A cautionary tale of IT and OT dependencies
- By Mark Rockwell
- Feb 13, 2017
In a modern-day cautionary tale, NASA discovered that even small changes to security software can cause big problems for operational controls technology and even endanger critical equipment and agency missions.
According to a Feb. 8 report from the space agency's inspector general, a security patch that shut down monitoring equipment in a large NASA engineering oven resulted in a fire that destroyed spacecraft hardware inside it. Because the computer reboot to accommodate the software upgrade also crippled fire alarm activation, the fire in the oven wasn't discovered for three and a half hours.
When vulnerability scanning was used to identify software flaws, physical equipment failed, resulting in a communications blackout with an Earth science spacecraft during an orbital pass. As a result, the pass was rendered unusable and data could not be collected until the next pass.
Yet another incident involved a computer problem in a climate system in which a safety feature was disabled, causing the heat in a data center to jump 50 degrees in minutes, resulting in a shutdown of the center.
Like many large enterprises, NASA has been automating many of its isolated, manually controlled technologies in favor of more sophisticated and interconnected IT equipment. But the agency's approach to integrating cyber, IT and physical systems is still a work in progress, and gaps in standards, training and security best practices must be remediated, according to the OIG.
Knowing which IT systems incorporate OT components is especially critical for NASA, it said, because applying traditional IT security practices to OT systems can cause underlying systems to malfunction.
According to the OIG, 65 percent of the agency's critical infrastructure -- including environmental monitoring and control systems for heating, cooling, ventilation, power, rocket propulsion testing systems, spacecraft and aircraft command and control systems -- are managed and supported by OT, or hybrid OT/IT systems, yet NASA has not developed a centralized inventory of OT systems or established a standard protocol to protect systems that contain OT components.
The report's list of NASA's OT shortcomings is probably familiar to IT managers at large organizations working to get a handle on the increasing number of internet-connected, or interconnected, devices and systems that were previously manually operated.
In reply comments, NASA outlined plans to inventory areas of technological interdependency and to come up with practices to guard against problems like those described in the IG report. It also said it will define and segment operational technology and industrial control systems from agency IT.
NASA expects improvements to be in place by Oct. 1, 2018.
A longer version of this story was first posted to FCW, a sister site to GCN.
Mark Rockwell is a senior staff writer at FCW, whose beat focuses on acquisition, the Department of Homeland Security and the Department of Energy.
Before joining FCW, Rockwell was Washington correspondent for Government Security News, where he covered all aspects of homeland security from IT to detection dogs and border security. Over the last 25 years in Washington as a reporter, editor and correspondent, he has covered an increasingly wide array of high-tech issues for publications like Communications Week, Internet Week, Fiber Optics News, tele.com magazine and Wireless Week.
Rockwell received a Jesse H. Neal Award for his work covering telecommunications issues, and is a graduate of James Madison University.
Click here for previous articles by Rockwell.
Contact him at [email protected] or follow him on Twitter at @MRockwell4.