AF grounded when software shuts down without warning
Connecting state and local government leaders
Halfway through a huge training exercise in Florida last March, Air Force computer operators got a chilling preview of what the year 2000 date-code problem could do to military readiness. As part of Internal Look '96, a joint service exercise based on a hypothetical conflict in the Persian Gulf, the operators were rushing to complete an air tasking order early in the morning of March 21. Equivalent to the master plan for the air component
Halfway through a huge training exercise in Florida last March, Air Force computer
operators got a chilling preview of what the year 2000 date-code problem could do to
military readiness.
As part of Internal Look '96, a joint service exercise based on a hypothetical conflict
in the Persian Gulf, the operators were rushing to complete an air tasking order early in
the morning of March 21. Equivalent to the master plan for the air component of a mission,
the ATO, in this case, contained detailed flight plans for 2,700 simulated aircraft
sorties.
But as they booted up their workstations and launched the Contingency Theater Automated
Planning System (CTAPS) used to generate ATOs, operators found they could not open a
crucial application. Frantic phone calls to CTAPS operators at Air Force bases across the
country quickly proved that the lock-out was systemwide.
"It was the year 2000 problem in spades," Lt. Gen. John Fairfield, Air Force
deputy chief of staff for communications and information, said in a recent interview. In
fact, it was an automated license manager problem, but Fairfield and other Air Force
officials have been citing the incident as an example of the havoc faulty date-codes could
wreak on Jan. 1, 2000.
The license manager in question was embedded in Unix office automation software from
Applix Inc., Westborough, Mass., that was bundled with CTAPS. According to Col. Carl
Steiling, program manager for Theater Battle Management Core Systems at the Electronic
Systems Center, Hanscom Air Force Base, Mass., CTAPS operators use an early Applix
application known as Asterix "to cut and paste text into CTAPS as they build the
ATO."
Automatic license managers are used by many commercial software vendors to control the
number of users in client-server networks, to prompt users for upgrades and, in some
cases, to make software unusable after a license expiration date. Unknown to the Air
Force, the version of Asterix on CTAPS was set to shut down March 21, the day the Air
Force's enterprise license expired.
"It was a two-year license, for some damned reason, instead of a 20-year
license," Fairfield said. "The system stopped, shut down and we were supposed to
have 2,700 sorties going."
Although CTAPS does not depend on the Asterix software to run, operators had no other
immediate means of manipulating text. "It was necessary for them to do the job,"
Steiling said. "Fortunately, this was a command post exercise, so there were no
actual aircraft flying. But what if it had happened during a contingency?"
As it turned out, that particular version of Asterix never should have been bundled
with CTAPS. Rick Smith, who heads Applix's government marketing in Washington, said his
company had agreed to extend the Air Force's license, but somewhere in the system
integration process, the wrong copy of the code got sent to the CTAPS integrator, Science
Applications International Corp of San Diego.
"It was human error," Smith said. "Someone grabbed the wrong tape during
the development process and an earlier version [with an active license manager] was put
in. We have always recognized that for many government users a license manager can present
difficulties, and we had no objection to extending the license period for the Air
Force."
Steiling provided a slightly different explanation. "It was a communications
breakdown between the people who did the contract [for an extended license] and those who
were building the system," he said. Either way, Steiling agreed, the wrong version of
Applix ended up in CTAPS.
Hours after the lock-out was discovered, the Air Force contacted Applix and began
working on a fix. By late afternoon, Applix posted segments of replacement code on Air
Force World Wide Web sites, and CTAPS users around the world began downloading and
installing it.
"Within about 14 hours, we were able to get the system running properly
again," Steiling said. "It was a great example of how the Web can be used to
disseminate software quickly."
Steiling said the incident also prompted his organization to conduct a careful survey
of the date-related license features in all its mission-critical software. "The
lesson is that we have to make sure that the configuration control of all the commercial
software in our systems is immediately visible to the purchasing agent as well as the
developing agent," he said.
"We even went so far as to set the time clocks on our systems all the way to
December 1999, to make sure we wouldn't have any license manager issues 'til then,"
Steiling added. Asked why they didn't set the clocks beyond that date, Steiling laughed
and said, "Well, that's another issue."