The Whys and Wherefores of the USAT-SBN Consensus Rankings

There's been some discussion of the USAT-SBN Consensus Rankings in the FanPosts so I thought I would do a round up of some of the comments that were made so they could reach a larger audience and we could explain a little bit about the reasoning behind how we compile the rankings.

Rob Joyner started it at MMA Payout with this charming bit of prose:

In the case of the USAToday/SBNation/BloodyElbow rankings, the rankings amalgam simply doesn’t pass the taste test. The chefs over at BloodyElbow, in their infinite wisdom, have concocted a recipe that they think is superior because they have tossed in the entire spice rack to make the entree taste better and they then say don’t blame me when folks blanch at the taste of the dish. Their consensus rankings are only as good as the sum of its parts and and the vast majority of those parts face a credibility gap.

I’m sure some from the site will say I’m making an appeal to authority and empty the rest of the rhetorical and philosophic tool box in an effort to argue the value of those rankings but such arguments don’t make the methodology or sources or rankings any more credible.

His alternative proposal is for ESPN and USA Today to unite and use a "a mix of the very best online and print reporters, media and high end bloggers as pollsters would be ideal.

Here's the response I made to Joyner in the comments to Geedub's post:

Joyner predicts that I’d go after him for using an appeal to authority fallacy and he is correct. Just because he admits to making a massive mistake in reasoning doesn’t mean that’s not a fault with his argument.

Here’s the thing, the meta-rankings idea came from my work in politics. In the last couple of political cycles its been shown that an average of credible polls is more accurate than any single poll regardless of methodology.

Therefore we thought it would be fun and informative to compile the rankings of as many credible sources as we could find and produce a consensus rankings system.

The biggest problem with Joyner’s proposal is that doing rankings well takes a lot of thought and work and celebrity rankers are the least likely to have the time to really do quality rankings.

The second problem with an “insider” rankings panel is that the rankers are inevitably going to be very aware of the business and politcial implications of their rankings. People looking to get in good with Zuffa will puff up UFC fighters, those on the outs with Zuffa will seek to damage the credibility of their fighters. This already happens quite a bit (see WAMMA and MMA Weekly).

Rankings compiled by fans who don’t have an axe to grind and who have the time to really focus on doing the rankings are just as accurate if not more accurate than those done by fight insiders who have neither the time to be thorough nor any interest in being objective.

And Brent Brookhouse had this to say:

...no one can really argue that the meta’s reflect the general feelings of the MMA fanbase but still insist that there must be something horribly wrong with them. Yes, sometimes someone thinks that the #8 guy is better than the #4 guy…but that’s the thing with rankings. A) they’re subjective and B) taking a large sample of a diverse group of them and compiling tat data will provide the most realistic picture.

It seems like people get hung up on the fact that they don’t regularly read a few of the sites included as somehow meaning that their rankings aren’t “credible” enough. But I’m sorry, rankings are far from a perfect science, and if I’m supposed to believe that because site a is read more than site b it instantly makes them more credible I can’t get on board with that.

MMAWeekly is one of the more widely read sites on the internet, and I haven’t agreed with their rankings for YEARS now. But I’m sure they’re more “credible” in most peoples eyes than some of the other sites we use. But by using a diverse group the major outliers are minimized while still being represented by appropriate weight as they do reflect the feelings of some portion of the MMA fanbase.

The last point I’d like to make is that no fighter is making any money off of our rankings. We’re not crowning champions, we’re not organizing fights. I seriously do wonder if there is something as simple as jealousy every time someone gets in a tizzy (as my grandma would say) over them. To act so upset makes so little sense to me. In the end rankings are for fun. The idea that someone may be over ranking Aoki isn’t taking money out of anyone’s pocket. It’s just such a strange thing to me that people have to continually go after the consensus rankings like we’re trying to corrupt the sport by putting together a list that represents the diverse views of the MMA community.

Actually, one more point…it’s not like we include any site that is willing to put out rankings. We make a decision if we find a site credible enough, but we’re not going to eliminate sites that we think make the grade because we don’t agree with their rankings. That would be manipulation of the data…which is what we’re not trying to do.

Pramit Mohapatra over at FightTicker decided to do something constructive and come up with his own rankings system. His system takes 5 big time sources: WAMMA, Sherdog, MMA Weekly, FIGHT Magazine and Inside MMA, gives them 60% of the total, then takes computer generated sources (FightMatrix) and fan votes (MMA Playground) and gives them each 20%.

That's cool but I just have to point out that is basically the original Meta-Rankings system we came up with in January 2008. We stopped doing it because we concluded that 1) the computer rankings require so much tweaking to make them work that they're basically as subjective as everyone else and 2) MMA Playground doesn't get enough votes to truly represent fan sentiment.

Pramit is also ultimately making an appeal to authority, he says it himself:

...we expect the experts to be most knowledgeable, the least partial, and the most flexible in their ability to absorb knowledge and come out with insightful rankings.

He's expecting the "experts" to be the least partial and then he's including WAMMA, Sherdog and MMA Weekly in that list? Those three may be known for many things, but impartiality is not one of them.