NSA releases open source tool for high-volume data flows

NSA releases open source tool for high-volume data flows

The National Security Agency released an open source software product that automates data flows among multiple computer networks, even when data formats and protocols differ.

As the volume and rate of data grows and as the number of systems, protocols, and formats increase, so too does the complexity and need for greater data management insight and agility.

Niagarafiles (Nifi) is a dataflow system based on the concepts of flow-based programming and was designed to manage dataflow in massive distributed computing systems operated by numerous teams and organizations.

Joseph L. Witt, the lead developer of Nifi, said it "provides a way to prioritize data flows more effectively and get rid of artificial delays in identifying and transmitting critical information."

Over the past several years, Nifi has developed a strong community of both developers and operators within the U.S. government, according to the Nifi proposal on the Apache Incubator Wiki. In open sourcing Nifi, the NSA lets private sector programmers examine the code, and potentially improve it through additional enhancements and applications. At the same time, the government can gain from their related research advances.

The tool could benefit the U.S. private sector in various ways. For example, commercial enterprises could use it to quickly control, manage, and analyze the flow of information from geographically dispersed sites – creating comprehensive situational awareness, the NSA said in its announcement.

The Nifi code is available to the public through the Apache Software Foundation. It is the first in a series of releases of in-house software products by NSA's Technology Transfer Program.

"NSA's innovators work on some of the most challenging national security problems imaginable," said Linda L. Burger, director of NSA’s tech transfer program. "Their research breakthroughs often have broad, commercial applications, too. We use open source releases to move technology from the lab to the marketplace, making state-of-the-art technology more widely available and aiming to accelerate U.S. economic growth."

In 2011 the NSA’s code for the agency's Accumulo project was also released through the Apache Software Foundation. 

About the Author

Connect with the GCN staff on Twitter @GCNtech.


  • business meeting (Monkey Business Images/Shutterstock.com)

    Civic tech volunteers help states with legacy systems

    As COVID-19 exposed vulnerabilities in state and local government IT systems, the newly formed U.S. Digital Response stepped in to help. Its successes offer insight into existing barriers and the future of the civic tech movement.

  • data analytics (Shutterstock.com)

    More visible data helps drive DOD decision-making

    CDOs in the Defense Department are opening up their data to take advantage of artificial intelligence and machine learning tools that help surface insights and improve decision-making.

Stay Connected