A few weeks ago, I asked the internet to guess how many coins were in ahuge jar (below). For more than 27 years, my parents had saved their spare change. My mother recently trucked the whole load to a bank to cash in, and in so doing finally learned the stockpiles actual value, or at least the value as calculated by that particular coin-counting machine. The update from Mom got me wondering: Might someone be able to guess that amount? What about our collective estimateis the crowd really as wise as some say it is?

The mathematical theory behind this kind of estimation game is apparently sound. That is, the mean of all the estimates will be uncannily close to the actual value, every time. James Surowieckis best-selling book, Wisdom of the Crowd, banks on this principle, and details several striking anecdotes of crowd accuracy. The most famous is a 1906 competition in Plymouth, England to guess the weight of an ox. As reported by Sir Francis Galton in a letter to Nature, no one guessed the actual weight of the ox, but the average of all 787 submitted guesses was exactly the beasts actual weight.

Galton, who also happens to be the inventor of eugenics, was shocked to find such value in democratic judgment.

The notion that the hive is more intelligent than the individuals comprising it is a seductive one, and a keystone of todays bottom-up Big Data revolution. Its democratic ideology, open-source goodness, the invisible hand, and New Age humility all wrapped into a big networked hug.

But is it true? The results of the coin jar experiment offer some clues. I wont bore you with the finer points of asymmetric non-parametric one-sample T tests, but lets put it this way: The crowd was waaaay off.

The actual value of the coins was $379.54. The mean value (x) of the 602 guesses submitted was $596.12, about 57 percent too high. Whats more, a massive (by crowd-wisdom standards) 40 percent of the individual guesses were closer to the actual value than that of the crowd.

After looking at the histogram above, keen observers will notice that the data are highly skewed. In these cases, how do you summarize what the crowd thinks?

The mean, or average, is still considered the gold standard for aggregating crowd wisdom, although Francis Galton himself recommended using the vox populihis term for median, or the middle-most guess where half of the other guesses are higher and half are lower. In our case, a weighted mean might be the most appropriate value, giving more influence to guessers who reported having actually done the math.

The problem is, people who claim to have done some math were far less accurate (x =$724.81) than those who made a snap judgment (x =$525.02). This may explain why estimates submitted from .edu or gmail addresses were less accurate than guesses submitted from hotmail and yahoo addresses. (Here are the data.)

Excerpt from:

Turns Out the Internet Is Bad at Guessing How Many Coins Are in a Jar