Mining big data, no data scientist required
- By Rutrell Yasin
- Oct 11, 2012
Data is everywhere. So how do you find meaning among vast collections of information to make timely and informed decisions? That’s the challenge Chiliad is helping government, education and health care organizations solve.
The company just unveiled the latest version of its software analytics suite, Discovery/Alert 7.0, which lets government analysts and clinicians discover information and knowledge across multiple data sources without specialized training in analytics and data sciences, according to Chiliad officials.
Often analysts know there is valuable content hidden in the vast amounts of information collected in their systems, but they need a data scientist to extract it and help make sense of it, said Rick Brown, senior director of product marketing at Chiliad. However, there is a shortage of data scientists who can help derive value from the data, so governments need tools that can help their analysts sort through structured and unstructured data from any source without having to replicate it or transform it.
Chiliad’s Discovery/Alert is being used by analysts at the FBI and Homeland Security Department to monitor all of the flights, ships, trucks, automobiles and passengers entering the United States. Now tasks that once took 32,000 hours to complete take 30 minutes, Chiliad executives said.
Discovery/Alert consists of software nodes that are installed at each original data location, which could be any type of system collecting or storing information, such as sensors or even mainframe computers. These nodes collaborate to create a virtual consolidated data center allowing analysts to query structured and unstructured data simultaneously, as if all data resources were in a single repository. Each Discovery/Alert node compares findings across all sources to produce a single, relevant and accurate global ranking without performance bottlenecks.
“The reality is that at a big data scale you will never be able to put all critical information into a single location,” said Rob Enderle, principal analyst at the Enderle Group.
Chiliad lets agencies work with their existing ecosystems through virtual consolidation, which reduces the time and costs associated with deploying, maintaining and scaling big data projects, Enderle said.
To support government and healthcare-related security requirement’s, Discovery/Alert 7.0 supports role-based security permissions at the document and collection level while integrating with secure authentication systems, such as Kerberos as well as others.
The suite is one of a number of big data products being developed for public-sector use. Other examples include a federal version of the Cleversafe Dispersed Compute Storage and the YottaDrive cloud storage systems for intelligence and surveillance sensor data.