Big data spawns new breed of 'data scientist'
- By Patrick Marshall
- Feb 07, 2012
The rise of big data has sparked demand in government, universities, and industry for a new breed of data scientists, those with a rare blend of expertise in statistics, technology and business analysis who can extract useful knowledge from big data flows.
“Those are people who are in very short supply in the private sector and the public sector,” said Anne Lapkin, a research vice president with Gartner. “It's an entirely new skill set.”
What you need to know about big data
Apache Hadoop: Big data's big player
Have you delved into your data lately?
Interestingly, Lapkin said, data scientists don’t generally come from academic computer science programs. Instead, she added, many members of the first generation of data scientists come from scientific research labs, such as the Jet Propulsion Laboratory at the California Institute of Technology.
“They have a very, very big data problem because they have all of these masses of data that are coming back from the Mars exploration project and so on,” Lapkin said. “They have people who they have trained to do nothing but look at the data, analyze the data, figure out different datasets, build new types of queries, and so forth.”
With the boom in big data analysis in financial markets in the 1980s, however, many of the best data scientists were lured to the financial sector. “You had a bunch of people who should have been astrophysicists and gone to work in the space program going to work on Wall Street to make a ton of money,” Lapkin said.
Fortunately, for the public sector — which traditionally has difficulty competing with salaries in the private sector — universities are beginning to offer data science programs that teach the skill sets needed to manage big data solutions. Still, most analysts expect it will be a number of years before the supply of data scientists comes close to meeting the demand.