Mining big data, no data scientist required

Data is everywhere. So how do you find meaning among vast collections of information to make timely and informed decisions? That’s the challenge Chiliad is helping government, education and health care organizations solve.

The company just unveiled the latest version of its software analytics suite, Discovery/Alert 7.0, which lets government analysts and clinicians discover information and knowledge across multiple data sources without specialized training in analytics and data sciences, according to Chiliad officials.

Often analysts know there is valuable content hidden in the vast amounts of information collected in their systems, but they need a data scientist to extract it and help make sense of it, said Rick Brown, senior director of product marketing at Chiliad. However, there is a shortage of data scientists who can help derive value from the data, so governments need tools that can help their analysts sort through structured and unstructured data from any source without having to replicate it or transform it.

Chiliad’s Discovery/Alert is being used by analysts at the FBI and Homeland Security Department to monitor all of the flights, ships, trucks, automobiles and passengers entering the United States. Now tasks that once took 32,000 hours to complete take 30 minutes, Chiliad executives said.

Discovery/Alert consists of software nodes that are installed at each original data location, which could be any type of system collecting or storing information, such as sensors or even mainframe computers. These nodes collaborate to create a virtual consolidated data center allowing analysts to query structured and unstructured data simultaneously, as if all data resources were in a single repository. Each Discovery/Alert node compares findings across all sources to produce a single, relevant and accurate global ranking without performance bottlenecks.

“The reality is that at a big data scale you will never be able to put all critical information into a single location,” said Rob Enderle, principal analyst at the Enderle Group.

Chiliad lets agencies work with their existing ecosystems through virtual consolidation, which reduces the time and costs associated with deploying, maintaining and scaling big data projects, Enderle said.

Discovery/Alert has an intuitive Web-based user interface built on HTML5, Cascading Style Sheets and JavaScript, so it can be used on any compatible Internet-connected device. Users can query data using conversational language. A host of built-in visualization tools help users find more relevant information and uncover new relationships between concepts. As new data is added or existing data modified, Discovery/Alert evaluates any changes based on the original query and can alert users, Chiliad officials said.

To support government and healthcare-related security requirement’s, Discovery/Alert 7.0 supports role-based security permissions at the document and collection level while integrating with secure authentication systems, such as Kerberos as well as others.

The suite is one of a number of big data products being developed for public-sector use. Other examples include a federal version of the Cleversafe Dispersed Compute Storage and the YottaDrive cloud storage systems for intelligence and surveillance sensor data.

About the Author

Rutrell Yasin is is a freelance technology writer for GCN.


  • business meeting (Monkey Business Images/

    Civic tech volunteers help states with legacy systems

    As COVID-19 exposed vulnerabilities in state and local government IT systems, the newly formed U.S. Digital Response stepped in to help. Its successes offer insight into existing barriers and the future of the civic tech movement.

  • data analytics (

    More visible data helps drive DOD decision-making

    CDOs in the Defense Department are opening up their data to take advantage of artificial intelligence and machine learning tools that help surface insights and improve decision-making.

Stay Connected