Harnessing the Wisdom of Crowds
PredictWhen was built as an experiment to attempt the harness the ‘wisdom of crowds’. James Surowieki opens his excellent book of the same name with an example from 1907 when Francis Galton found that the crowd at a county fair accurately guessed the weight of an ox when their individual guesses were averaged.
The wisdom of crowds theorises that information aggregated in groups can result in better decisions than that made by any single member of the group. Crucially, it is most effective when each member of the group makes their own decision independent of other members of the group, based on information known to them.
ScienceNow has produced this short amusing film explaining the concept:
The Project & the Plugin
The web provides an ideal platform to aggregate the opinion of the wise, connected crowd (that’s you!). PredictWhen is a free wordpress plugin that allows wordpress site owners to invite predictions from their users of when an event of their choosing will happen. You can download the Predict When plugin here. We hope the plugin will serve as a useful tool to allow us to test the theory on a wide scale thanks to the popularity of the WordPress CMS.
In the county fair example summarised above, it is significant that most of the people attending the fair were well informed when it came to livestock. We hope by making the plugin available bloggers and site owners can invite predictions on subjects of interest to themselves and their readers, this will likely improve the accuracy of the averaged prediction.
Time here is the common unit of measurement. Preserving the independence of individual predictions is vital to arriving at an accurate averaged prediction from the group. To that end users must make a prediction before they see the predicted date based on the aggregated data from the ‘crowd’. The plugin also produces a chart showing the distribution of individual predictions over time, but again users don’t see the values on the x axis to allow them to interpret the data until they have made their prediction.
Score! An incentive to give considered predictions
Predictions are shown to be more accurate when the individual making them has given them careful consideration, and still more so when they have a vested interest in the prediction being accurate. In prediction markets for example, users are literally ‘investing’ a currency of some sort, whether real or virtual, in the likelihood of an event occurring. In Fantasy Sports, users make player selections to win points to beat rivals and or win prizes. With that in mind PredictWhen also includes a scoring option so that when an event has occurred site owners can identify and reward those who were most accurate and scored highest. Crucially, the scoring calculation is not solely based on accuracy but also on foresight. So a prediction that is a week out but made a year prior to the event occurring will score higher than a prediction that is a day out but made a month prior. Scoring utilises the standard wordpress registration system.
Open or Closed
Questions are either Open i.e. the event being asked to predict hasn’t happened yet, or Closed, where it has. When using the plugin it is up to the site owner to close a question when an event has occurred. If you’re using scoring it is wise to specify a third party source when giving further information about the question. For example ‘event will be deemed to have occurred when reported by the BBC on their website’. For questions created on PredictWhen.com we’ll do our best to keep on top of what has and hasn’t happened but will rely on, you guessed it, the crowd, to assist with this so tell us if you spot a question is open when the event has occurred.
About the PredictWhen.com website
This website, predictwhen.com serves as a directory of predictions being powered by the free plugin. When a site owner or blogger using the plugin publishes a new prediction on their site they have the option to list the prediction here on predictwhen.com We hope in time this will grow to be an interesting data resource. While predictions powered by the plugin are listed here if users of predictwhen.com want to make a prediction they have to visit the site where the prediction is published to do so. It’s your prediction so we send the traffic your way.
For people who want to invite predictions but who don’t have a wordpress site, you can build one and publish it right here on predictwhen.com with the same functionality that is available to people using the plugin.
Calculating the average explained
As you can imagine some questions prompt extremely diverse date ranges. We use a t distribution instead of a normal distribution to estimate population parameters and a 95% confidence interval for the sample mean. We highlight the upper and lower ranges on the charts as in many instances a date range will sometimes be more illuminating than a specific date. We have some alternative formulae to arrive at the average but require some more data for events that have happened to compare the accuracy of the methods.
