Virginia's data centers pass a test

Newly consolidated centers return quickly in simulated disaster

IN APRIL, the Virginia Information Technology Administration (VITA) conducted the first large-scale disaster recovery tests of the consolidated data centers that will house the state's executive agencies' IT infrastructure.

'That disaster recovery test proved to be much more successful than past tests,' said Fred Duball, director of VITA's Service Management Organization. 'We were able to do things quicker'and go further than we had in the past.'

Within the first 24 hours of the 72-hour test, all hardware in the recovery data center was up and running, and within 36 hours, 90 percent of the applications and infrastructure were running.

'We can't bring it all up in 36 hours, but we don't need to,' said Doug McVicar, a Northrop Grumman vice president and VITA program manager.

The goal is to identify critical systems and ensure that they can be restored with a minimum of lost time and data.

'The way we have the replication set up, we had a 30- minute data loss,' said Mike Elkins, Northrop Grumman's director of data center transformation. In previous years, days of data would have been lost during recovery.

VITA is a program to consolidate the state's IT assets and infrastructure into a centralized, standardized system to improve performance, security, reliability and efficiency. It was created in 2002, and the agency entered into a partnership with Northrop Grumman in 2006 to provide IT services.

'It's more than just a contract,' McVicar said. The 10-year agreement calls for the company to build two new data centers for $60 million. The primary center is in Chesterfield County near Richmond, and the backup facility is in economically depressed Russell County. As part of the agreement, the company located the facilities in areas in need of economic development. In return, VITA opened the IT services contract to local agencies and school districts in addition to state agencies to maximize returns for the company.

During the first three years of the contract, the data centers and infrastructure are being modernized and agencies transitioned to the new facilities. The data centers are up and running with mainframe computers installed. New servers are being put in place, and a new network from Verizon links the state's 2,000 agency offices. The centers have a workforce of about 1,000 employees. All were formerly state employees, and Northrop Grumman has hired 600 of them. All 85 executive branch agencies will be moved into the data centers during the next year.

'We've got a solid year to go' to get everything up and running, Duball said.

In the final seven years of the contract, Northrop Grumman will operate the managed service, which includes 195 service-level agreements, and provide technical refresh for all network and desktop elements.

The recovery test simulated a disaster in which the primary data center would be off-line for more than 24 hours. The company spent two months working with agencies to prepare for the test, establishing priorities for recovery and creating pseudo networks to replicate data without interrupting routine operations.

'We start from power off and go to full power on; load, test and validate all of the equipment'; and test data to make sure it is up-to-date, Elkins said.

'The biggest lesson was sequencing,' McVicar said. 'There is a logical order in which different operations should be brought up. It could only be guessed at before. Now we have the recipe.'

Several weeks of retesting and examination followed the initial 72-hour test to establish the proper priority and order for restoring services.

This flexibility is one of the advantages of the VITA partnership, Duball said. In the old environment, the state contracted separately for disaster recovery testing and was limited by the contract to a single test each year.

Plans still call for annual tests with Northrop Grumman, but VITA is rethinking the time frame and might do additional tests focusing on specific areas.

Reader Comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above