10 steps to speed the journey from data prep to insight

10 steps to speed the journey from data prep to insight

Preparing data for analysis is a slow, difficult, tedious process that can cause major delays and inefficiencies when it comes time for analysis.  So improving the methods and technology to process data can greatly improve and speed up the time to insight.

Half of the respondents to a recent TDWI Research survey spent more time cleaning the data than using it. Most (73 percent) spent at least 41 percent of their total time preparing the data for analysis. Little more than a third spent between 61 and 80 percent of their time preparing data, and 8 percent spent nearly all of their time on data preparation.

Nearly all respondents thought the process of preparing data for business intelligence and analytics activities could be improved, especially when it comes to the accuracy, quality and validity of their data and its consistency across datasets.

Below are 10 recommendations from TWDI on ways organizations can streamline the data analytics process, improve productivity and help IT staff better serve users: 

1. Cut the time needed for data preparation by evaluating current data preparation procedures to eliminate unnecessary routines. Increase automation and standardized processes for incorporating and integrating new data.

2. Use data catalogs, glossaries and metadata repositories as part of the data cleansing process and enable reuse of data by others.

3. Automation makes it easier to find relevant data, while a centralized data catalog facilitates data discovery and collaboration.

4. Use technologies and methods to increase repeatability so that scripts, workflows and other elements can be reused. Develop a collaborative framework to encourage sharing of repeatable elements.

5. Consider self-service data preparation solutions for existing business intelligence and data analytics as well as data integration, transformation, data quality and metadata management. Test these self-service capabilities on smaller datasets before trying them on bigger, more complex sources.

6. Integrate self-service data preparation with self-service business intelligence and visual analytics. Evaluate new technologies enabling nontechnical users to prepare data themselves. Encourage experienced users to share their expertise with nontechnical users.

7. Create a flexible structure for analytics. Data transformation rules and associated technology should allow for tailoring data preparation processes according to the data source.

8. Ensure self-service data preparation, follows guidelines and regulations to avoid data chaos.

9. Include data preparation in data governance objectives and educate staff that following the guidelines leads to better data and data analytics.

10. Create centers of excellence to improve IT staff responsiveness to users’ needs. A team comprised of both business and IT leadership will ensure data projects align with the organization’s overall strategy.

About the Author

Kathleen Hickey is a freelance writer for GCN.


  • Records management: Look beyond the NARA mandates

    Pandemic tests electronic records management

    Between the rush enable more virtual collaboration, stalled digitization of archived records and managing records that reside in datasets, records management executives are sorting through new challenges.

  • boy learning at home (Travelpixs/Shutterstock.com)

    Tucson’s community wireless bridges the digital divide

    The city built cell sites at government-owned facilities such as fire departments and libraries that were already connected to Tucson’s existing fiber backbone.

Stay Connected