Overcoming the Big Data challenges

Although government agencies are excited about the prospect of using Big Data to solve pressing problems, they realize they must first address a wide range of concerns, from business to technology issues.

The biggest concern, cited by 57 percent of those in the survey, was overall costs (see figure 1). Eric Sweden, program director for enterprise architecture and governance for the National Association of State Chief Information Officers (NASCIO), points out that Big Data initiatives are competing for dollars against other high-priority issues -- such as legacy modernization, consolidation and shared services, cloud computing, mobile services and cyber security.

Figure 1


The business case for Big Data must start with “a purpose, the rationale and the need for investing now,” says NASCIO's Sweden. However, this bumps up against another key challenge noted by almost half of the people in the survey – the inability to demonstrate a return on investment for Big Data initiatives.

The survey indicated a wide ranging disagreement about the ability to demonstrate an ROI for Big Data analytics projects: 42 percent of agencies agreed that it is difficult, 34 percent say it isn’t; and 24 percent neither agreed nor disagreed (see fig. 2).

Figure 2


“The government always wrestles with ROI because it’s so hard to measure,” says Bob Gourley, former CTO of the Defense Intelligence Agency (DIA) and founder of Crucial Point LLC, a technology research and advisory firm. “I wish they would talk about return on mission – how the return to the agency from the Big Data protect helps it accomplish its mission. For government, ROI is so ambiguous, while industry can articulate it so well.”

Infrastructure concerns

The survey indicates that many agencies are also concerned about whether their current IT infrastructures (including network bandwidth, storage, and processing capabilities) are agile and flexible enough to handle Big Data projects effectively. In addition, the perennial business intelligence concerns are doubly relevant in a Big Data initiative.

“The biggest challenge is applying enterprise data management principles, such as data quality and data governance, to Big Data,” says Michael Daconta, former metadata program manager for the Homeland Security Department and author of "Information as Product: How to Deliver the Right Information to the Right Person at the Right Time." “If you can’t manage your current data and have data silos, you will also get silos with Big Data, but on a massive scale.”

In the survey, 57 percent of respondents mentioned strains on existing IT infrastructure, and 47 percent noted the difficulty of integration as critical challenges to Big Data.

“With the strain on the existing IT infrastructure, not everyone will move as quickly into Big Data,” Gourley says. “There is a mix of systems and priorities. If you modernized in the last three years and are ready for a Big Data solution it shouldn’t be a problem.”

Security is another key issue, mentioned by 65 percent of respondents. “We need to make sure the technology is secure so there are no unauthorized intrusions into your data set and no unauthorized queries,” Gourley says. “Nobody gets access to data unless they are authenticated and authorized.”

He says it’s significant that while there is a National Institute of Standards and Technology (NIST) guide for almost every type of security, none has yet been produced for Big Data clusters. He says this speaks to the embryonic state of the technology.

A narrow focus

In these early days of Big Data, Gourley contends the best answer to most of the challenges is to narrowly focus Big Data initiatives on specific problems. An extremely targeted approach can help address another concern expressed by 56 percent of respondents -- difficulty in uncovering actionable insights.

“Except in a few obvious niche cases, people are dipping their toes in this Big Data issue and that’s the right approach,” Daconta says. “The government is rightfully a little more cautious than the commercial side, because they have little or no competition.”

At the same time, some observers believe the issues identified in the survey are largely due to the fear of the unknown. “I don’t think IT infrastructure is a big challenge anywhere,” says Thomas Redman, a consultant and author of "Data Driven: Profiting from Your Most Important Business Asset." “Big data is fundamentally about innovation. It is more a cultural and people problem than a technology one.”

To Redman, it’s telling that all these concerns in the survey are significantly lower for agencies that are fully implemented or just starting out on Big Data initiatives versus those that are just beginning to investigate the technology. He draws an analogy to parents of a two-year-child.

“Usually the experience is nothing like what you expect it will be,” he says. “That’s the way it is with anything involving Big Data. It’s no different than the transition from the agrarian to the industrial age: nobody knew what it would be like. Many times organizations try something and after three or four swings, they begin to get good at it, or they try a different path.”

Methodology and survey demographics

n August 2012, 193 subscribers of Federal Computer Week, Government Computer News and other 1105 Government Information Group publications responded to an email survey about the status of their Big Data initiatives. Only subscribers whose agency had a Big Data project either deployed, in development or under consideration were included in the survey. In addition, only respondents who were either a buyer or user of Big Data tools and services, or the requisite IT infrastructure, were included as well.

Seven out of 10 respondents were technology decision makers (CIOs or other IT managers or professionals), while 30 percent were senior managers, consultants, directors or other non-technologists. More than 40 percent were from federal civilian agencies, 32 percent were from military agencies and 25 percent were from state or local government agencies.

About this report

This report was commissioned by the Content Solutions unit, an independent editorial arm of 1105 Government Information Group. Specific topics are chosen in response to interest from the vendor community; however, sponsors are not guaranteed content contribution or review of content before publication. For more information about 1105 Government Information Group Content Solutions, please e-mail us at [email protected].