Agencies look to sharpen their data mining skills
Connecting state and local government leaders
Ever since the demise of Total Information Awareness'John Poindexter's ill-fated attempt to use massive amounts of commercially available data, combined with intelligence from federal agencies, to identify possible terrorist activities'the federal government has been circumspect in talking about its data mining programs.
Ever since the demise of Total Information Awareness'John Poindexter's ill-fated attempt to use massive amounts of commercially available data, combined with intelligence from federal agencies, to identify possible terrorist activities'the federal government has been circumspect in talking about its data mining programs.
TIA fell prey to citizens' fears of the government invading their privacy. Since the public outcry against the program in 2003 and 2004, the government has moved to allay those fears, taking such steps as creating chief privacy officers at many agencies assigned to handle vast amounts of personal information.
But efforts continue to sift data and identify subtle patterns of suspicious behavior, as analysts look for ways to anticipate possible terrorist activity.
According to the Government Accountability Office, data mining is 'the application of database technology and techniques'such as statistical analysis and modeling'to uncover hidden patterns and subtle relationships in data and to infer rules that allow for the prediction of future results.'
In a May 2004 report (www.GCN.com/479), GAO found that 52 of 128 federal departments and agencies were using or planning to implement data-mining programs.
Russ Travers, deputy director of information sharing and knowledge development at the National Counterterrorism Center, said that improved data mining is necessary to help analysts deal with one of their biggest problems: how to find the nuggets of actionable intelligence hidden inside huge amounts of data from multiple sources.
'Analysts are completely overwhelmed by the terabytes of information out there,' he said. 'In that massive volume there is a lot of information that's completely bogus.'
The intelligence community has restrictions on what kinds of information it can have access to, Travers said, while law enforcement agencies and the Homeland Security Department have different rules.
The drive to share information 'is blurring the distinction between foreign and domestic [operations],' he said. 'You could draw a bright red line before, but you can't anymore.'
As a result, NCTC is experimenting with data-mining technologies using the data that is accessible to intelligence agencies to find 'non-obvious' relationships, he said.
NEXT STORY: Where the data meets the road