Statistical Experiment: are Fantasy Sports games of Skill or Chance?

The UIGEA 2006 law specifically forbids online gambling in the United States, but makes an exception for Fantasy Sports on the basis that these are primarily games of skill, not chance.

Great, but I still had my doubts. I was about to start a Fantasy Sports site, and I needed to be sure. Empirically sure.

So I decided to do a little experiment. I signed up on Fanduel, DraftKings, and Victiv, which are fantasy sports sites for traditional sports such as Football and Basketball (things I know very little about). I created 22 teams and entered into 22 different contests, all with different win probabilities. I satisfied all the games’ rules – pick a player in each position, spend less than your team budget, get as close to spending all your budget as you can etc. The games were various types (50/50, tournament) but all were in the salary-cap format.

This meant that it was possible to do a Skill vs Chance test.

How? In blackjack for example, there is an optimal way to play by following the game rules. Over a large number of games, if you play optimally, your expected win rate [ E(x) ] is 48%. This can be tested by anyone – newbie, idiot, or skilled pro. Follow the exact procedure over a large enough number of games, and your actual win rate will approximate towards E(x).

So with my experiment, the hypothesis was this: I’m an idiot player. I know nothing about which players to pick. If I follow the rules optimally (spend all my budget efficiently, pick different players in each roster, pick a player in each position etc), then my expected win-rate should approximate towards E(x) if this is a game of Chance. That’s how games that are primarily games of chance behave. If you follow the rules perfectly, your actual win-rate equals the expected win-rate over a large sample size.

If you think of a coin toss, no matter how skilled or unskilled you are, if you toss the coin randomly without cheating (following rules), then W(x) = E(x). In our case, if my actual win-rate falls within a confidence interval of E(x), then the hypothesis is true: Fantasy Sports are a game of chance.

But if my result falls outside the confidence interval, then the hypothesis is false: Fantasy Sports are primarily games of Skill. Follow? And to be doubly sure, we’ll set a very high confidence interval of 99%.

Here’s the original data (bit messy, I’m sorry). After 22 contests, here is the summary of the results:

n=22
E(x) = 0.3037588192
Std. Dev = 0.1589799795
Actual Win-Rate = 0.1363636364

Using a t-distribution for n=22, 99% confidence interval to test hypothesis, we get:

0.2082098135 < X < 0.3993078249

Our actual result (0.1363636364) falls way outside this range, and so our hypothesis is false: this is NOT a game of Chance. Hence, this is primarily a game of skill.

I believe this result makes sense – I lost badly over 22 contests b/c of my inferior knowledge. And like the stock market, there is no upper limit to the amount of knowledge you can have about players & teams in a sport. Since there is no upper limit, there is no skill cap to Fantasy Sports. There is no single optimal way to play. Knowledge is virtually limitless, and it changes with each game. Results depend on your analysis, preparation and knowledge. 

To get an average, expected result, you have to be atleast as knowledgeable as the group average. If you’re an idiot at Football like me, you’ll fall way outside the mean.

That is exactly how the stock market (stock info/analysis) and the job market (skill level) behaves. If you’re exceptional at the things that matter in those particular contests, you’ll generally get exceptional results. If you’re an idiot, it will be obvious. Nobody calls those games of chance.

There are ofcourse certain problems with this study: small sample size, did I really play optimally etc. But I’m pretty confident that if this experiment were to be repeated by any idiot player who knows nothing about a sport, they would get a result similar to mine. 

Fantasy Sports are a game of skill because they depend primarily on knowledge & analysis, not on chance. That’s what we at Vulcun believe. And we have some math that agrees.

– Ali