10 steps to speed the journey from data prep to insight

10 steps to speed the journey from data prep to insight

Preparing data for analysis is a slow, difficult, tedious process that can cause major delays and inefficiencies when it comes time for analysis.  So improving the methods and technology to process data can greatly improve and speed up the time to insight.

Half of the respondents to a recent TDWI Research survey spent more time cleaning the data than using it. Most (73 percent) spent at least 41 percent of their total time preparing the data for analysis. Little more than a third spent between 61 and 80 percent of their time preparing data, and 8 percent spent nearly all of their time on data preparation.

Nearly all respondents thought the process of preparing data for business intelligence and analytics activities could be improved, especially when it comes to the accuracy, quality and validity of their data and its consistency across datasets.

Below are 10 recommendations from TWDI on ways organizations can streamline the data analytics process, improve productivity and help IT staff better serve users: 

1. Cut the time needed for data preparation by evaluating current data preparation procedures to eliminate unnecessary routines. Increase automation and standardized processes for incorporating and integrating new data.

2. Use data catalogs, glossaries and metadata repositories as part of the data cleansing process and enable reuse of data by others.

3. Automation makes it easier to find relevant data, while a centralized data catalog facilitates data discovery and collaboration.

4. Use technologies and methods to increase repeatability so that scripts, workflows and other elements can be reused. Develop a collaborative framework to encourage sharing of repeatable elements.

5. Consider self-service data preparation solutions for existing business intelligence and data analytics as well as data integration, transformation, data quality and metadata management. Test these self-service capabilities on smaller datasets before trying them on bigger, more complex sources.

6. Integrate self-service data preparation with self-service business intelligence and visual analytics. Evaluate new technologies enabling nontechnical users to prepare data themselves. Encourage experienced users to share their expertise with nontechnical users.

7. Create a flexible structure for analytics. Data transformation rules and associated technology should allow for tailoring data preparation processes according to the data source.

8. Ensure self-service data preparation, follows guidelines and regulations to avoid data chaos.

9. Include data preparation in data governance objectives and educate staff that following the guidelines leads to better data and data analytics.

10. Create centers of excellence to improve IT staff responsiveness to users’ needs. A team comprised of both business and IT leadership will ensure data projects align with the organization’s overall strategy.

About the Author

Kathleen Hickey is a freelance writer for GCN.

inside gcn

  • When cybersecurity capabilities are paid for, but untapped

Reader Comments

Mon, Aug 22, 2016 Sonal Goyal

Very valid point on self service data preparation and integration of tools into data analytics. The more data we collect, the greater the problem of data preparation and data quality. I would also like to suggest that our self service preparation tool Reifier(www.nubetech.co) eases the pain of data preparation and data quality and helps organizations focus on their business problems.

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group