Bandwidth issues haunt National Weather Service
A recent systemwide failure at the NWS College Park data center that impacted networks nationwide illustrates the agency’s struggles to supply web access for increasingly complex weather models.
Internet bandwidth issues at the National Weather Service are hampering the agency’s ability to get its forecasts and warnings out to the public, according to a recent report by the Washington Post. Most recently, NWS experienced a systemwide failure at its College Park data center that impacted networks nationwide.
According to March 30 status reports from the National Oceanic and Atmospheric Administration’s National Centers for Environmental Prediction, several weather offices and forecasters at air traffic control centers nationwide lost communications with the NWS Network Control Facility (NCF), impacting the dissemination and reception of weather data, including observations. Several websites, including weather.gov, went down. NSWChat was inaccessible outside of the NCEP network and the Weather and Climate Operational Supercomputing System was not ingesting external partner data.
By mid-morning, NCEP Central Operations restarted a control unit on the College Park firewall that was experiencing a memory problem, and internet connections and data dissemination improved.
The problems, which lasted for seven hours, seem to be the latest in a long series of IT infrastructure problems.
The chat issues, for example, have become so intractable that employees at the Birmingham, Ala., Weather Service office decided to switch to Slack two days before tornadoes hit the state.
“In the interest of public safety and due to factors beyond our control, NWS [Birmingham] will be SWITCHING to Slack Chat as our PRIMARY means of realtime communication until such a time that NWS Chat is proven stable, reliable and has a reliable backup service in place,” according to an email sent by warning coordination meteorologist John De Block and obtained by the Washington Post.
In reaction, NWS headquarters informed the Birmingham office to switch back to NWS Chat and not use alternative platforms. With a second tornado outbreak in the forecast, NWS forecasters pleaded with partners outside affected areas to stay off NWS Chat to conserve bandwidth. In spite of appeals to limit use, “the chat service still went dark for a time as a deadly tornadic storm was swirling across the South,” the article said.
NWS Public Affairs Director Susan Buchanan told the Post the problems resulted from a “combination of increased web traffic associated with the severe weather in the Southeast and the loss of one data center on March 9 due to a water pipe burst at NWS HQ in Silver Spring.” She added that NWS plans to test an off-the-shelf product this spring as a replacement to the existing chat platform.
Neil Jacobs, former acting head of NOAA, told the Post that many of the agency’s infrastructure problems are tied to the fact NWS runs on internal hardware rather than through cloud platforms.
Over the last 10 years, NWS networks have suffered failures, the Post reported in December 2020. Demand for weather data, combined with the complexity and size of data models, prompted the agency to limit the amount of data major users can access from NWS websites.
NOAA has acknowledged the importance of robust IT infrastructure. In its July 2020 cloud and data strategies, the science agency noted that its data resources are growing so vast that they place “a premium on our capacity and wherewithal to scale the IT infrastructure and services to support this growth.” The document stated that “modernizing our infrastructure requires leveraging cloud services as a solution to meet future demand.”