NASA on the trail of metal shavings
Experiencing a mysterious surge in equipment failure in your data center? The culprit may be microscopic metallic shavings, warns NASA Goddard Space Flight Center researchers.
'For data centers, it is a serious problem,' said NASA chief parts engineer Henning Leidecker, who is part of a research team tracking the problem. 'More serious than many know about.'
The researchers are looking at how aging or inexpensively produced hardware can shed tiny conductive filaments of zinc typically less than a few millimeters long, known as zinc whiskers.
Because they are conductive, such whiskers can short-circuit electronic equipment if they gather in sensitive regions of electronic equipment. Usually it can take decades for a data center to collect a dangerous number of whiskers, though as many data centers continue to age, this will be a problem.
Zinc whiskers may come from a variety of sources. Mostly, they come from raised-floor tiles, though computer equipment ' and even screws, nuts, washers and bus rails ' can be sources too. The whiskers may be created through the movement of equipment or floor tiles, nicks and scrapes to the material, unequal thermal expansion, bending of the material, or defects in the manufacturing process.
Compounding the problem is that zinc whiskers are impossible to see: They are smaller than the threshold of the size of any object that can be perceived by the naked eye. They might be as small as 1/100th of a micron thick. They are also plentiful. Leidecker has seen cases of as many 14,000 whiskers per centimeter of surface. An observer must look parallel to the infested surface, light the surface and keep the background dark. Only then can the observer see the light that bounces off the filings.
Through a series of experiments, the NASA research team has been characterizing how the whiskers grow under various environmental conditions and the voltage breakdown that occurs through the whiskers. In addition to the basic research, the NASA team has built a rich multimedia library of literature documenting the phenomenon (See GCN.com/846).
Despite their diminutive stature, the filings can cause big problems. Colorado reported that, in 2005, a data center had been off-line for 30 days because zinc whiskers from floor tiles were blown by the air conditioning into computers, shorting many of them out.
NASA has also reported zinc whiskers causing mischief in satellites, flight control equipment and telecommunications systems.
Often, the whiskers remain unnoticed until some external event shakes them from their locations. Leidecker recounted a case in which a state data center saw a sudden rise of server and router failure immediately after a fire. The firefighters charging through the center must have dislodged the filaments, which then floated into the air-circulation system, he concluded.
Leidecker said shedding from zinc and other metals such as cadmium can be mitigated by adding small amounts of lead to the mixture. U.S. laws, however, limit the amount of lead that can be added to metals.
Data center operators should be able to keep equipment safe by adding filters to air circulation systems, though few data center operators seem to take this precaution.
Metal shavings 'are not a unique event,' Leidecker said. 'They may not even be rare, but the problem is underappreciated.'