NASA’s upgraded Sherlock data warehouse makes it easier for researchers to access nationwide air traffic data almost immediately.
As a plane takes off from Charlotte, N.C., the local air traffic control system monitors it until handing off to the air traffic controllers along the flight’s next leg. But now, thanks to upgrades in its Sherlock Air Traffic Management Data Warehouse, NASA will be able to monitor the entire flight in near-real time so it can more quickly analyze the entire U.S. air traffic system.
Until recently, NASA researchers and its partners could only see data from 77 individual FAA air traffic facilities, which made tracking a flight across the county difficult and slow. For example, the data from the air traffic facility at the Charlotte airport would show the surrounding 60-70 miles, but data for the remainder of the flight would be in a different file, generated by the next control center to monitor the flight.
The fragmented data also made it difficult for air traffic management researchers to quickly perform large-scale analyses of the U.S. air traffic system, to find where improvements will have the most benefit in terms of environment, safety and efficiency.
With algorithms provided by the ATAC Corporation of Santa Clara, Calif., researchers now get all the info in one place. Michelle Eshow, NASA’s lead for real-time air traffic management automation software development for the Aviation Systems Division, said ATAC’s algorithms are able to combine the data from the 77 FAA facilities into one nationwide, near-real-time file on air traffic data.
“Believe it or not, that’s a really hard thing to do and something NASA had not done before,” Eshow said.
NASA receives the data from FAA via a secure connection that the two agencies have had for years, Eshow said. They are working on getting the data in XML format, but for now it is being sent in different forms of binary, which is decoded by software when it reaches NASA. The software then sends it on to a Linux file system.
The front end of the Sherlock system is built on a variety of open source and commercial products. GeoServer is an open source tool that is used to visualize geospatial data. Oracle’s Apex handles basic search and download functions within the database. The heavy lifting of big data management is handled by Hadoop.
Sherlock contains over 36 terabytes of raw data, with more collected daily. The merged flight data will soon be available for all of the archived files going back to 2011, which will allow researchers to study many billions of flight tracks to better identify trends in the national traffic picture -- which in turn will help predict future traffic.
Editor's note: This article was changed Aug. 18 to correct the source of the algorithms. It's the ATAC Corporation of Santa Clara, Calif.