Big data spawns new breed of 'data scientist'

The rise of big data has sparked demand in government, universities, and industry for a new breed of data scientists, those with a rare blend of expertise in statistics, technology and business analysis who can extract useful knowledge from big data flows.

“Those are people who are in very short supply in the private sector and the public sector,” said Anne Lapkin, a research vice president with Gartner. “It's an entirely new skill set.”

Related coverage:

What you need to know about big data

Apache Hadoop: Big data's big player

Have you delved into your data lately?

Interestingly, Lapkin said, data scientists don’t generally come from academic computer science programs. Instead, she added, many members of the first generation of data scientists come from scientific research labs, such as the Jet Propulsion Laboratory at the California Institute of Technology. 

“They have a very, very big data problem because they have all of these masses of data that are coming back from the Mars exploration project and so on,” Lapkin said. “They have people who they have trained to do nothing but look at the data, analyze the data, figure out different datasets, build new types of queries, and so forth.”

With the boom in big data analysis in financial markets in the 1980s, however, many of the best data scientists were lured to the financial sector.  “You had a bunch of people who should have been astrophysicists and gone to work in the space program going to work on Wall Street to make a ton of money,” Lapkin said.

Fortunately, for the public sector — which traditionally has difficulty competing with salaries in the private sector — universities are beginning to offer data science programs that teach the skill sets needed to manage big data solutions. Still, most analysts expect it will be a number of years before the supply of data scientists comes close to meeting the demand.

About the Author

Patrick Marshall is a freelance technology writer for GCN.


  • Records management: Look beyond the NARA mandates

    Pandemic tests electronic records management

    Between the rush enable more virtual collaboration, stalled digitization of archived records and managing records that reside in datasets, records management executives are sorting through new challenges.

  • boy learning at home (Travelpixs/

    Tucson’s community wireless bridges the digital divide

    The city built cell sites at government-owned facilities such as fire departments and libraries that were already connected to Tucson’s existing fiber backbone.

Stay Connected