4 stages of the big data process

The phrase "garbage in, garbage out" is appearing with increasing frequency in discussion of big data – and with good reason. Thomas Redman, the self-styled Data Doc, has been posting some good articles on All Analytics about what he calls the "D4 process" that characterizes four increasingly difficult stages of the big data development for governments and corporations.

1. Data acquisition.  You can acquire potentially interesting data with little regard for quality. On the flip side, however, if you don’t ensure quality at this step, you make later steps increasingly difficult. Even machine created and harvested data is not immune from errors.

2. Discovery. It is more difficult to discover something truly interesting when the data is bad. Additionally, it's possible to draw invalid conclusions from relatively scrubbed data if error patterns aren't understood.

3. Delivery.  It is even more difficult to get someone to use the results when they don’t trust the data. The fruit of big data applications is in the "small data" or the "connected dots. One small data quality issue or missed connection between data sets will destroy the credibility of the project.

4. Dollars. While it is nearly impossible to make money from poor-quality data, the government can't deliver poor-quality data, period.

About the Author

Connect with the GCN staff on Twitter @GCNtech.


  • business meeting (Monkey Business Images/Shutterstock.com)

    Civic tech volunteers help states with legacy systems

    As COVID-19 exposed vulnerabilities in state and local government IT systems, the newly formed U.S. Digital Response stepped in to help. Its successes offer insight into existing barriers and the future of the civic tech movement.

  • data analytics (Shutterstock.com)

    More visible data helps drive DOD decision-making

    CDOs in the Defense Department are opening up their data to take advantage of artificial intelligence and machine learning tools that help surface insights and improve decision-making.

Stay Connected