Big data spawns new breed of 'data scientist'

The rise of big data has sparked demand in government, universities, and industry for a new breed of data scientists, those with a rare blend of expertise in statistics, technology and business analysis who can extract useful knowledge from big data flows.

“Those are people who are in very short supply in the private sector and the public sector,” said Anne Lapkin, a research vice president with Gartner. “It's an entirely new skill set.”

Related coverage:

What you need to know about big data

Apache Hadoop: Big data's big player

Have you delved into your data lately?

Interestingly, Lapkin said, data scientists don’t generally come from academic computer science programs. Instead, she added, many members of the first generation of data scientists come from scientific research labs, such as the Jet Propulsion Laboratory at the California Institute of Technology. 

“They have a very, very big data problem because they have all of these masses of data that are coming back from the Mars exploration project and so on,” Lapkin said. “They have people who they have trained to do nothing but look at the data, analyze the data, figure out different datasets, build new types of queries, and so forth.”

With the boom in big data analysis in financial markets in the 1980s, however, many of the best data scientists were lured to the financial sector.  “You had a bunch of people who should have been astrophysicists and gone to work in the space program going to work on Wall Street to make a ton of money,” Lapkin said.

Fortunately, for the public sector — which traditionally has difficulty competing with salaries in the private sector — universities are beginning to offer data science programs that teach the skill sets needed to manage big data solutions. Still, most analysts expect it will be a number of years before the supply of data scientists comes close to meeting the demand.

About the Author

Patrick Marshall is a freelance technology writer for GCN.


  • senior center (vuqarali/

    Bmore Responsive: Home-grown emergency response coordination

    Working with the local Code for America brigade, Baltimore’s Health Department built a new contact management system that saves hundreds of hours when checking in on senior care centers during emergencies.

  • man checking phone in the dark (Maridav/

    AI-based ‘listening’ helps VA monitor vets’ mental health

    To better monitor veterans’ mental health, especially during the pandemic, the Department of Veterans Affairs is relying on data and artificial intelligence-based analytics.

Stay Connected