Preface: I love the "Consensus MMA Rankings" that are periodically released. The concept is great (while kind of obviously a good idea) and informative. That said, I'd like to start an open discussion about some points I think could be improved or clearer. It is possible some of the items below may have been fixed already, but I am not currently aware of it.
1) "Consensus" implies that the panel meets, discusses, and comes to a single consensus view. The method used is not consensus, so the name should in my view be "Composite" MMA Rankings.
2) The explanation of the rankings says "A formula is used to "normalize" the data so all fighters are awarded points from those lists that do not include a full 25 fighters. This formula ensures that each ranking site awards the same number of total points regardless of how many fighters they choose to rank." This sounds convoluted and obscures the formula, making it seem like some arcane method which perhaps would not be not robust under public scrutiny. If you look closer, you'll see that unranked fighters all get the same score for sub-25 lists. I don't agree with that method. I have a much simpler option that works, and will generate (I believe) the same or better results. Simply sum the rankings of each fighter and divide by the number of sites that ranked that fighter (straight average the ranking). Then order them from lowest number to highest number. The one with the lowest number (around 1) will be the #1 ranked fighter, and so on. There is no need to complicate it by awarding 25 points to the #1 ranked fighter, etc. It's not clear how the points are allocated to unranked fighters for sites that list 10 fighters - I am not confident the method currently used is "fair"/robust purely because I haven't got the detailed info to assess it.
3) Check out Bantamweight. Joe Benavidez is listed. The 5 "most detailed" ranking sources, i.e. those that bothered to submit 25 fighters, didn't rank him. Presumably because they are also the most informed, and realise that he should not be ranked in BW. Dallas and Tim for example. As a result, Joe B is ranked at BW with a spuriously low ranking. That needs to be fixed. I'd say remove him from BW if he's made a commitment to FlyW. Simply bump everyone else to fill the gap. The compiler should take his cues from the "Best" ranking sources - again, the ones that bother to give 25 fighters.
4) I'm not sure if the rankings currently exclude votes that failed to take into account the latest events in MMA. They should! The panel should all be producing their rankings as at a certain date, taking into account all events previous to that date. If they haven't done that, their votes skew the rankings.
5) If one voter has a "blind spot" i.e. makes a mistake by omitting a highly ranked fighter, that can skew the rankings. Example would be if one guy failed to include Munoz in his MW list. To ensure this doesn't happen, the compilers should run an automated script that checks if all the top 15 or so fighters as at last month's ranking are included in every submission. Where it detects one missing, the submitter is sent an automated email asking them to check their rankings, highlighting the missing fighter, and they then either confirm or revise/re-submit.
6) We can see last month's rank but it would be nice if we could click to see a chart tracing the fighter's ranking over the last 12 or 24 months.
7) Flyweight? Probably time to start doing it.
8) Retired? Should not be on the list. Fedor is still there, for some reason. It just looks silly when the label says Retired and he's still ranked. Brock, for example, is not listed.
9) Not fought for X months? Pick a reasonable number like 18 or 24. Anyone with no fights for X months should be temporary suspended from the listings. Let's say for example GSP is injured for more than 24 months. I'd temporarily remove him and put him at the bottom of the list as a note: *GSP - injured. 25 months since last fight. Last ranking, #1.
10) When you click the fighter's name, you should get a page showing the last 5 fights and the next fight. Right now, we get just "the last fight" and "next fight". But it seems inaccurate: Ben Henderson's last fight listed is Pettis at WEC! This needs fixing.
It would be interesting to hear other suggestions as I'm sure I've been non-exhaustive.