Do you know what the price of gas will be in six months?  How about the stock price of Google at closing bell on September 8, or the extent of the expected U.S. troop strength drawdown in Afghanistan by year’s end?  As an individual, predicting the outcome of these events with any degree of confidence or accuracy may seem the stuff of science fiction, but UCI cognitive scientists say it’s possible.   

“As a group, people are collectively more intelligent than you might think,” says Mark Steyvers, cognitive sciences professor.  “We all possess knowledge about different things; some more than others.  If we pool these pieces of knowledge together, we can get a pretty accurate look at the big picture.”

It’s called the wisdom of crowds effect, and the scientists are hoping to get your help putting it to the test. 

Steyvers and professors Michael Lee and Bill Batchelder are part of a national research team that has received a grant from the Intelligence Advanced Research Projects Activity (IARPA) to develop a statistical model based on the crowds concept that can, with an increasingly higher degree of accuracy, predict the outcome of future events. 

Working with the UCI scientists are researchers from the University of Maryland, University of Michigan, The Ohio State University, Fordham University, Wake Forest University, Wichita State University and ARA, a private research company.

They are one of five national teams to receive money under this program and must meet yearly goals in order to receive continued funding from the government agency. 

In July, the UCI/multi-university team launched their model via a software program called Forecasting Ace.  Through volunteer participation, the program will collect individual opinions on the likelihood of outcomes in a variety of future events.  

Before providing perspectives, participants must complete a short questionnaire in which they self-rate their subject matter expertise in areas including science and technology, business and the economy, politics and policy, military and security, and sports and health.    After completing the self-assessment, participants can choose which questions they’d like to answer.  The software then aggregates and analyzes the collected data using models of human decision-making developed by the UCI scientists.

“Finding wisdom in crowds is fundamentally a cognitive science problem because it’s about how people acquire knowledge and make judgments,” says Lee.  This concept, while relatively new, is related to cultural consensus theory work that Batchelder has been developing for the past 30 years.  The newer computer and cognitive modeling aspect draws on the expertise of Lee and Steyvers who have been perfecting these methods for the past four years together at UCI through numerous lab experiments and analyses of real-world behavioral data. The professors, along with two student researchers, recently won a best paper prize for this research at the 2011 annual meeting of the Cognitive Science Society. 

The qualitative and quantitative results from these various studies have helped them develop new methods for identifying experts by combining self-reported expertise and behavioral responses in their models. 

As an event that is part of the forecasting challenge draws near, such as predicting the likelihood of party leaders reaching a decision on the U.S. debt ceiling, the team runs the data through the model. During this process, “ace analysts” – true experts in their field – are identified and their opinions are appropriately weighted, helping the researchers to arrive at a single, relatively accurate forecast of the outcome of the up-coming event. 

While they currently don’t know the degree of accuracy until the event does or doesn’t occur, continued data aggregation is helping them hone their models to account for more complex and accurate analyses. 

The resulting applied project has implications for government agencies interested in improving the accuracy of intelligence analysis.  It may also have cross application in the business and policy communities as well as the medical field by helping to improve forecasts and diagnoses.

Funding for the project began in May and will continue through the end of April 2012, at which point funding may be extended. The research program aims to achieve a 20 percent increase in forecast accuracy when compared with state-of-the-art alternatives. Years two, three and four will aim for 35-, 50-, and 50+ percentage point increases, respectively.

-Heather Wuebker, Social Sciences Communications
-photo © istockphoto.com/BartCo