Big data spawns new breed of 'data scientist'

The rise of big data has sparked demand in government, universities, and industry for a new breed of data scientists, those with a rare blend of expertise in statistics, technology and business analysis who can extract useful knowledge from big data flows.

“Those are people who are in very short supply in the private sector and the public sector,” said Anne Lapkin, a research vice president with Gartner. “It's an entirely new skill set.”

Related coverage:

What you need to know about big data

Apache Hadoop: Big data's big player

Have you delved into your data lately?

Interestingly, Lapkin said, data scientists don’t generally come from academic computer science programs. Instead, she added, many members of the first generation of data scientists come from scientific research labs, such as the Jet Propulsion Laboratory at the California Institute of Technology. 

“They have a very, very big data problem because they have all of these masses of data that are coming back from the Mars exploration project and so on,” Lapkin said. “They have people who they have trained to do nothing but look at the data, analyze the data, figure out different datasets, build new types of queries, and so forth.”

With the boom in big data analysis in financial markets in the 1980s, however, many of the best data scientists were lured to the financial sector.  “You had a bunch of people who should have been astrophysicists and gone to work in the space program going to work on Wall Street to make a ton of money,” Lapkin said.

Fortunately, for the public sector — which traditionally has difficulty competing with salaries in the private sector — universities are beginning to offer data science programs that teach the skill sets needed to manage big data solutions. Still, most analysts expect it will be a number of years before the supply of data scientists comes close to meeting the demand.

About the Author

Patrick Marshall is a freelance technology writer for GCN.


  • business meeting (Monkey Business Images/

    Civic tech volunteers help states with legacy systems

    As COVID-19 exposed vulnerabilities in state and local government IT systems, the newly formed U.S. Digital Response stepped in to help. Its successes offer insight into existing barriers and the future of the civic tech movement.

  • data analytics (

    More visible data helps drive DOD decision-making

    CDOs in the Defense Department are opening up their data to take advantage of artificial intelligence and machine learning tools that help surface insights and improve decision-making.

Stay Connected