Systems managers recount tales from the code trenches
Systems managers recount tales from the code trenches
By Claire E. House and Trudy Walsh
GCN StaffExactly what was it like to be on the front line of the date code fix-it effort?
Eight state and local government systems professionals talked about their work on the year 2000 problem.
Sharing their experiences were Tim Blevins, information systems chief of the Kansas Revenue Department; Mario Diaz, deputy general manager of the Atlanta Aviation Department's Hartsfield Atlanta International Airport; Charlie Hartman, year 2000 project administrator of the Maine Human Services Department; Julio Natera, comput-computer operations chief of the Miami Information Technology Department; Roy Robinson, manager of Albuquerque, N.M.'s, Water Utility Division; Kevin Sifferlen, deputy director of Indiana's Criminal Justice Institute; and Jim Steinbruegge, year 2000 director of the Missouri Transportation Department's Information Systems Division.GCN/State & Local:
'How is your job different because of year 2000 work?
BLEVINS: It's been a 30 percent drag on resources.
We spent a lot of time and resources on analyzing information, aging the data, getting databases copied.
ROBINSON: We had to spend a lot of time on research, testing to see if the components of the water utility system were noncompliant. It turns out that our software wasn't compliant.
It was custom software, installed in the mid-1980s. And our computer maintenance management system ran on a 1980s vintage Wang mainframe, which wasn't year 2000-ready either.
That's one thing we found out, that everything we did to get ready for the year 2000 had to be done anyway.
STEINBRUEGGE: I was the assistant director here before I retired. Then the Information Systems Division staff got hit by a ton of turnover.
A year ago I came out of retirement to work on the year 2000 full time. But as long as they give me time to play golf every week, I'm fine. I'm handling just the information systems, and my cohort, Kirk Juranas, an engineer in Kansas City, is handling the operations side. That's probably the best move I ever made.GCN/State & Local:
'How did you plan for year 2000 work?
BLEVINS: We take what we call snapshot production transactions on our IBM AS/400, Unix and mainframe environments. Then we move them over and run these same transaction sections through mitigated code in our current environment. It's all year 2000-compliant software, but the data has all the old dates in it. We take four passes on it.
Then we age the data, push the date forward a year, then sprinkle the data with key dates: leap day, Jan. 1, about 20 other significant dates.
Then we take the entire transaction set and run it again to see if we get the same results. Then we change the system operating dates and rerun the whole thing again, when the system thinks it's ahead of year 2000. Taking four passes on the system has been an efficient way to shake out some bugs.
We're spending a lot of time on disaster recovery contingency planning. We test all this stuff in a sterile environment. Then we have a whole certification process with third-party vendors who certify our system from independent auditors. We finished that last month.
HARTMAN: We have three levels of plans. We have one for the actual program that provides benefits. Then a recovery plan for how we get the system up and running if it's not working. Then what's called the zero day plan, for the day on which this is going to be a problem.
For example, we give Medicaid cards to people every month that they use to get prescription drugs or physician care. The cards are printed by computers the fourth week of every month. So the zero day of Medicaid cards is the fourth week of January. We have two-year authorization for durable medical equipment for Medicaid. So in January 1998, we had things authorized to January of 2000.
GCN/State & Local:
Tim Blevins of the Kansas Revenue Department advocates close scrutiny of date code. Blevins says he found system assessments from five years ago that turned out to be untrue.
'What have you encountered in your year 2000 work that you didn't expect, and how did you handle it?
BLEVINS: Many of these systems had been mitigated four or five years ago. But detailed test data from several years ago wasn't kept because it was so long ago. Unless you have somebody who's going to be there through the whole process, you're going to be vulnerable. We had some system assessments from several years ago that proved untrue. So you really need scrutiny.
DIAZ: I think one of the things we didn't expect was finding as many systems as we did in different areas of the airport. Secondly, I think prior to my arrival in January, was a failure to recognize that to solve the year 2000 problem at an airport, you need not only technical expertise but also expertise in airport operations and management'specifically, knowledge of what systems the airport is required to operate and maintain, and what systems are a direct responsibility of the airlines, as outlined in Federal Aviation Administration regulations.
The scope ended up being much wider than the scope put together by the original team.
We are responsible for systems that control airfield lighting, employee badge security, restricted access area security, heating, ventilation, air conditioning, and electrical power for both terminal processes and initial airplane power-up. We also have to make sure vendors, such as airlines and fuel suppliers, are ready.
Through her year 2000 work with Maine's Human Services Department, Charlie Hartman says she has learned not to blink an eye when status checkers ask her group to test systems and gather data in different ways.
HARTMAN: The way it sort of takes over everything around you. Back in January, my group said, 'All right, we'll fix the lines of code and change the programs.' Then suddenly, it took on a life of its own and went up the ranks through management to the commissioner's level and began to take on a much broader meaning. Now it's, 'Do you have a contingency plan for your contingency plan?' and 'What about pacemakers? They have computer chips.'
We have different agencies that come check our progress. They'll ask how we tested our Cobol code'did we write down testing procedure beforehand, did we print reports, are they on single- or double-sided pages? If we did single-sided, they may need double-sided. It's agonizing, but we're trying to be as cooperative as possible, so we write everything over again.
A normal person would think this is odd, but now we're inured to it.
If we've tested something three ways and we're asked to test it a fourth way, we'll do it'this is how much it's taken over everybody's hearts, minds and souls.
NATERA: We haven't encountered any problems. We have a Unisys NX5800 mainframe that holds all of the citywide applications for police, fire, finance and many other things. What we've done so far is convert all the data fields from six to eight to carry the century. So we really haven't found any problems. We are currently undergoing testing to see how the computer will behave.
ROBINSON: The only curve that was thrown was that there really weren't any curves thrown. I did find that younger people were fairly accepting of the year 2000 issue. Older folks are a little more scared, the ones who went through the Depression.
SIFFERLEN: We really haven't had any. We're a small agency of about 35 people and several divisions that administers state and federal public safety grants. About two years ago, we upgraded to Microsoft Windows NT Server 4.0 and Windows 95. Everyone's connected to the same server, and we're all running Microsoft Office 97. There are no special or older databases involved that we had to fix. We have all corporate industry downloads in place in time, so unless Bill Gates is lying to all of us, we plan on being fine.
STEINBRUEGGE: One problem was getting the department to understand that year 2000 was about more than information systems. Once we got all that rolling, it made life a lot easier. Getting over that hump was the hard part.
The other hurdle was that everybody else just looked at the applications. I've been working on the infrastructure and our frame relay network. We found out that part of that wasn't ready. We had to look at the operating systems, the database managers and the utilities associated with all this. We spent a lot of time in the client-server world, checking the NetBIOS chips and the continuing service packs from Microsoft.
My contention has been that if the infrastructure doesn't work, nothing works. I took a holistic view, which I guess came from 35 years in the business.
WESTBROOK: The embedded systems, embedded chips'I hadn't really given it any thought until later on. We hired a company to crawl around in all the county offices and inventory all the equipment anybody thought had a possibility of a problem and run it through their database and come back with a certification or a failure. We inventoried about 500 pieces of equipment, and we ended up submitting 175 types of equipment to this company's database. We've already gotten back 110 that are fully compliant and 10 that aren't. We're also worried about things like generators, security systems. We have a wastewater treatment plant, and it has so much equipment in there, we've hired a consultant to go in and inventory it.
GCN/State & Local:
Roy Robinson, manager of the Albuquerque, N.M., water utility division, is excited about working this New Year's Eve. 'I'll be awake at least, not asleep by 11 p.m.'
'How are you planning for contingencies, and how do you think it will go?
STEINBRUEGGE: Our Governor's Council on Year 2000 is working with state agencies. We're working with our public service commission, the people who regulate water, utilities and banks. We don't see any big showstoppers. We don't see the world coming to an end.GCN/State & Local:
'What is an example of one of your contingency plans?
WESTBROOK: We issue building permits, and we're simply going to have available in reports the permits that have been issued and are still active, and we'll have the forms available for people to issue a permit manually. That will back up an application from Hansen Information Technologies Inc. of Sacramento, Calif. The app holds data in an Oracle Corp. database running on a Unix server that users access through Windows.GCN/State & Local:
'What have you learned from your year 2000 experiences?
BLEVINS: I found out that when we need to, we can go into break-fix mode very quickly. That's the mode people will be in right after year 2000 hits. When something breaks, you have to immediately go in there and fix it.
ROBINSON: Concentrate on the big picture. Don't sweat the small stuff. We're not going to go check every light switch or thermostat. We can operate a lot of things manually.
We'll have almost three days of full water storage in reservoirs all through town. And if the electricity goes out, it actually will be helpful for water conservation. Without electricity, water use is cut to less than half the normal amount. You can't run a washing machine or dishwasher without electricity, and most people don't like to take showers in the dark.GCN/State & Local:
'Once year 2000 is out of the way, what do you want to focus on most?
DIAZ: One of the systems we plan to upgrade is our financial accounting system. We have purchased Sun Microsystems 3500 and 5500 servers and are working with Oracle to upgrade our financial accounting system here at the airport. Our revenue control system, which tracks transactions in the parking system, will be replaced and integrated into this system. One of the benefits of the Y2K effort has really been to focus our attention on the need in some cases to upgrade systems and in others to fully replace them.
WESTBROOK: We've got some specific business issues that are of great concern to us, things such as collecting data at the point of origin by getting handheld field devices to county workers.
Cops have them now, but inspectors and other people who do work in the field should have them, too.
We're also looking to make further use of the Internet to make information available to people and let people do county business through the Internet.GCN/State & Local:
'Is there something about year 2000 you haven't seen covered by the press that you would like to know more about?
WESTBROOK: I'm listening constantly because I'm always learning. I think this ground is getting covered fairly well. My main concern at this point is public perception. I suspect that there will be some small issues, some inconvenience issues that will occur. I want people to understand that these things are going to happen and that they'll be corrected quickly and not to assume the worst.GCN/State & Local:
'Where will you be on New Year's Eve?
BLEVINS: We'll have a site at the capitol in Topeka to bring up systems statewide. We're working with the Federal Emergency Management Agency and Kansas' chief information officer, Don Heiman. The state lifted restrictions on having champagne in state office buildings for that night only'but I won't be partaking, that's for sure.
DIAZ: Probably here. Celebrating.
HARTMAN: I'm going to be in Times Square. But the state is setting up a bunker in Augusta where there will be representatives from agencies at midnight.
NATERA: I'm planning to be in the computer room with a few programmers waiting for the change. That will be critical because we deal with public safety and we don't want to have any problems.
STEINBRUEGGE: We'll be here in our support center in Jefferson City. We'll have people on standby, ready with snow plows and radio systems with backup powers. We'll have people ready to take stop signs anyplace. We haven't run into anything yet that isn't fixable.
ROBINSON: I'll be in the emergency preparedness center in the city, ready for whatever. This will probably be the most exciting New Year's Eve I've had in a few years. I'll be awake at least, not asleep by 11 p.m.
WESTBROOK: I'm going to be right here, probably. I plan to have my network infrastructure people here that night, so we'll hopefully see what's going on with the network fairly quickly. If it's functioning, I'll have my application people come in Saturday morning and start running through the applications. Hopefully by Monday morning, everyone will be able to come in and do their jobs.