Intell site tests crowdsourcing's ability to predict future

The intelligence community, which uses everything from spies to satellites to gather information, is now looking to the general public, rolling out a crowdsourcing initiative to let people weigh in on the likelihood of potential future events and their probability of occurring.

The beta Aggregative Contingent Estimation system (ACES) website, called Forecasting ACE, is funded by the Intelligence Advanced Research Projects Activity. It launched July 15.

It will aggregate respondents’ answers to essentially an online survey. IARPA will then use the results to see if the group can make accurate predictions, reported Adam Rawnsley in Wired.

Respondents will be polled about potential social, political, scientific, military and economic events. Potential questions include “What is the probability that the [World Health Organization] will declare a flu pandemic in 2011?” and “How many members will Facebook have by the end of 2011?” Contributors are asked to join the project but will remain anonymous, according to the site.

Unlike other crowdsourcing initiatives that equally weight all answers, ACES will more heavily weigh answers from the most accurate predictors. "Our system will not rely on simple averaging, but will combine forecasts from many people in a way that substantially outperforms simple averaging of judgments,” according to information on the site. “We will be testing many ideas and do not know yet which will work best.”

“We’re trying to make good use of everybody’s individual opinions and trying to determine what aspects of them might be important and would lead to a good forecast,” Dr. Dirk Warnaar, the principal investigator for the ACES project at Applied Research Associates (ARA), developer of the tool, told Wired.

A work in progress, ARA also plans to add a tool to give users feedback on their forecasting accuracy to help them improve their success rate and “start thinking about this problem a little deeper than they otherwise would,” he added.

ARA also plans to add collaboration capabilities to the site, although the specifics of the social aspect and have yet to be determined. One possibility is allowing users to elaborate on how they arrived at their predictions and let other users vote on those responses.

“We’re thinking that people will be interested in competing with others and maybe learning how to become better forecasters,” Warnaar said.

ACES is currently being tested; IARPA expects the project to take four years to complete. How the forecasts would be used has not yet been decided. One possibility is National Intelligence Estimates, assessments of particular topics used by senior analysts.

Other federal, state and local agencies also have begun embracing crowdsourcing. One of the most recent is a logo contest from the Interior Department. The Defense Advanced Research Projects Agency is crowdsourcing the design of a new military vehicle, and municipal government such as New York City, Boston and Oakland County, Mich., also are pursuing crowdsourcing programs.

About the Author

Kathleen Hickey is a freelance writer for GCN.

Reader Comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above