NSA releases open source tool for high-volume data flows

NSA releases open source tool for high-volume data flows

The National Security Agency released an open source software product that automates data flows among multiple computer networks, even when data formats and protocols differ.

As the volume and rate of data grows and as the number of systems, protocols, and formats increase, so too does the complexity and need for greater data management insight and agility.

Niagarafiles (Nifi) is a dataflow system based on the concepts of flow-based programming and was designed to manage dataflow in massive distributed computing systems operated by numerous teams and organizations.

Joseph L. Witt, the lead developer of Nifi, said it "provides a way to prioritize data flows more effectively and get rid of artificial delays in identifying and transmitting critical information."

Over the past several years, Nifi has developed a strong community of both developers and operators within the U.S. government, according to the Nifi proposal on the Apache Incubator Wiki. In open sourcing Nifi, the NSA lets private sector programmers examine the code, and potentially improve it through additional enhancements and applications. At the same time, the government can gain from their related research advances.

The tool could benefit the U.S. private sector in various ways. For example, commercial enterprises could use it to quickly control, manage, and analyze the flow of information from geographically dispersed sites – creating comprehensive situational awareness, the NSA said in its announcement.

The Nifi code is available to the public through the Apache Software Foundation. It is the first in a series of releases of in-house software products by NSA's Technology Transfer Program.

"NSA's innovators work on some of the most challenging national security problems imaginable," said Linda L. Burger, director of NSA’s tech transfer program. "Their research breakthroughs often have broad, commercial applications, too. We use open source releases to move technology from the lab to the marketplace, making state-of-the-art technology more widely available and aiming to accelerate U.S. economic growth."

In 2011 the NSA’s code for the agency's Accumulo project was also released through the Apache Software Foundation. 

About the Author

Connect with the GCN staff on Twitter @GCNtech.


  • Records management: Look beyond the NARA mandates

    Pandemic tests electronic records management

    Between the rush enable more virtual collaboration, stalled digitization of archived records and managing records that reside in datasets, records management executives are sorting through new challenges.

  • boy learning at home (Travelpixs/Shutterstock.com)

    Tucson’s community wireless bridges the digital divide

    The city built cell sites at government-owned facilities such as fire departments and libraries that were already connected to Tucson’s existing fiber backbone.

Stay Connected