NSA releases open source tool for high-volume data flows

NSA releases open source tool for high-volume data flows

The National Security Agency released an open source software product that automates data flows among multiple computer networks, even when data formats and protocols differ.

As the volume and rate of data grows and as the number of systems, protocols, and formats increase, so too does the complexity and need for greater data management insight and agility.

Niagarafiles (Nifi) is a dataflow system based on the concepts of flow-based programming and was designed to manage dataflow in massive distributed computing systems operated by numerous teams and organizations.

Joseph L. Witt, the lead developer of Nifi, said it "provides a way to prioritize data flows more effectively and get rid of artificial delays in identifying and transmitting critical information."

Over the past several years, Nifi has developed a strong community of both developers and operators within the U.S. government, according to the Nifi proposal on the Apache Incubator Wiki. In open sourcing Nifi, the NSA lets private sector programmers examine the code, and potentially improve it through additional enhancements and applications. At the same time, the government can gain from their related research advances.

The tool could benefit the U.S. private sector in various ways. For example, commercial enterprises could use it to quickly control, manage, and analyze the flow of information from geographically dispersed sites – creating comprehensive situational awareness, the NSA said in its announcement.

The Nifi code is available to the public through the Apache Software Foundation. It is the first in a series of releases of in-house software products by NSA's Technology Transfer Program.

"NSA's innovators work on some of the most challenging national security problems imaginable," said Linda L. Burger, director of NSA’s tech transfer program. "Their research breakthroughs often have broad, commercial applications, too. We use open source releases to move technology from the lab to the marketplace, making state-of-the-art technology more widely available and aiming to accelerate U.S. economic growth."

In 2011 the NSA’s code for the agency's Accumulo project was also released through the Apache Software Foundation. 

About the Author

Connect with the GCN staff on Twitter @GCNtech.


  • Russia prying into state, local networks

    A Russian state-sponsored advanced persistent threat actor targeting state, local, territorial and tribal government networks exfiltrated data from at least two victims.

  • Marines on patrol (US Marines)

    Using AVs to tell friend from foe

    The Defense Advanced Research Projects Agency is looking for ways autonomous vehicles can make it easier for commanders to detect and track threats among civilians in complex urban environments without escalating tensions.

Stay Connected