Why you should care about Big Data

Why you should care about Big Data

What do these diverse projects have in common?

  •  Fewer combat troops in Afghanistan are being injured or dying from “improvised explosive devices” (IEDs) because lethal mines are being discovered before they detonate.
  •  A pattern-matching tool is being developed to find schools that provide fake student visas to potential terrorists.
  •  Citizens can go online and, in a few clicks, see how their tax dollars are spent, according to award size, jobs created, status, location, and other variables.

These projects represent the different ways that local, state and federal government agencies are accomplishing their missions by leveraging "Big Data."

Different people define Big Data in different ways, but it’s typically described in terms of the three Vs:

  1. The volume of information;
  2. The variety of information; and
  3. The velocity of information (which is the speed at which data becomes available and can be analyzed).

The three Vs usually drive organizations to deploy new techniques and technologies, since the new data sets often are incompatible with their existing business intelligence and analytics infrastructures. "Some people mistakenly think Big Data simply means they can’t afford to back up all the data they currently have," says Bob Gourley, former CTO of the Defense Intelligence Agency (DIA) and founder of Crucial Point LLC, a technology research and advisory firm. "But when you say Big Data, that usually implies a new way of doing analysis to make sense out of the data."

In a survey conducted in August 2012 by the 1105 Government Information Group, 63 percent of the almost 200 respondents agreed that unless they implement and use Big Data it will be more difficult to meet their agency's mission (see Figure 1). As a result, even in economically tough times, 49 percent of the respondents expect to increase their Big Data budgets while another 46 percent plan to maintain their budgets.

Figure 1


Eric Sweden, program director for enterprise architecture and governance for the National Association of State Chief Information Officers (NASCIO), says such survey findings reflect agencies' growing need to use data to identify and resolve the full portfolio of issues facing them.

“As we go forward, the concept of issues management will be more and more complex,” he says. “The 63 percent who say they need Big Data to accomplish their mission are recognizing that complexity, and the need to gather information from a more diverse set of sources – video, audio, newspaper articles, as well as traditional transactional data -- to develop the knowledge to make critical, strategic decisions.”

Overall, the survey pinpointed the chief benefits of “Big Data” activities as greater accuracy and speed, more efficiency, and better decision-making (see figure 2).

Figure 2


Financial fraud detection is among the first wave of government Big Data “sweet spots.” Other leading uses today include bioinformatics (like the human genome project); cyber security; rapid return of geospatial data on queries; and better search capabilities, such as USA.gov. It is important to note that federal civilian agencies are more apt than other agencies to cite “customer service” and “reduced expenditures” as benefits from Big Data.

Michael Daconta, an author and former metadata program manager for the Homeland Security Department, says these projects reflect one or more of the three main “motivators” of Big Data use in government today: The projects obviously address the agency’s mission (like intelligence); they promote the growing demand for transparency (usa.gov); or they produce good media coverage and avoid bad media coverage.

Sweden of NASCIO points out that it’s difficult, if not impossible, to determine when data will become relevant to future issues and decisions. For example, NASCIO documented cases where data more than a century old was still relevant to property line decisions. This highlights a key difference of using Big Data versus traditional digital information such as internal memos: The way to apply the data sometimes isn’t as readily apparent. Agencies need creativity to find effective and innovative ways to leverage these huge stores of data.

No one questions that the opportunities are there, though. From the National Institutes of Health (NIH) and NASA  to the U.S. Geological Survey, government agencies have enormous stockpiles of valuable information they can put to creative use.

“I was at a conference and some people from the NIH said they have access to five million pictures of tumors,” says Thomas Redman, a consultant and author of "Data Driven: Profiting from Your Most Important Business Asset." “Nobody in the private sector has access to that. And so even if government is not providing the analysis around the data, they can provide the data for researchers. To me, that’s good government. If you want to stimulate the economy, you can get that data out and enable the private sector to create businesses out of this data.”

According to the 1105 Government Information Group survey, Defense Department agencies are somewhat more advanced in their Big Data initiatives than most other branches of government. “We know the Department of Defense has a lot of sensors over the battlefield,” Gourley says. “They have turned into a pioneer of rapidly making sense of structured and unstructured data.”

Indeed, according to a recent wire service article, the Pentagon noted that increases in real-time information have translated into a 10 to 12 percent decrease in the number of deaths among Western coalition troops in Afghanistan this year.

It is important to note that most government Big Data projects are nascent. About one in five survey respondents indicated they had fully implemented a Big Data project (see figure 3) and most of the implemented Big Data initiatives are by federal agencies. In other words, state and local governments lag the federal government.

Figure 3


Gourley points to the General Services Administration (GSA) as a Big-Data pioneer on the civilian side. “They are in the press so much with heat, and they don’t get enough love for their Big Data solution that is being used by over 1,000 government web sites,” he says. “One of the applications is search for U.S. government web pages, including the White House. It provides the things you expect from search, like suggested search.”

He attributes the GSA’s pioneering success to personal leadership at the agency. “They reached out to industry and asked about best practices and implemented fast,” he says. “We are in the early stages of Big Data, and there are many challenges to work out. But if you come up with a smart idea it goes like wildfire.”

Methodology and survey demographics

In August 2012, 193 subscribers of Federal Computer Week, Government Computer News and other 1105 Government Information Group publications responded to an email survey about the status of their Big Data initiatives. Only subscribers whose agency had a Big Data project either deployed, in development or under consideration were included in the survey. In addition, only respondents who were either a buyer or user of Big Data tools and services, or the requisite IT infrastructure, were included as well.

Seven out of 10 respondents were technology decision makers (CIOs or other IT managers or professionals), while 30 percent were senior managers, consultants, directors or other non-technologists. More than 40 percent were from federal civilian agencies, 32 percent were from military agencies and 25 percent were from state or local government agencies.

About this Report

This report was commissioned by the Content Solutions unit, an independent editorial arm of 1105 Government Information Group. Specific topics are chosen in response to interest from the vendor community; however, sponsors are not guaranteed content contribution or review of content before publication. For more information about 1105 Government Information Group Content Solutions, please email us at [email protected]