The stumbling blocks of IT consolidation

It seems like a no-brainer: Budget-stretched governments operating a multitude of servers, networks and data centers stand to save millions of dollars through IT consolidation.

But there’s a catch. Local, state and federal agencies have to spend money before they can save it, and projects to rationalize IT infrastructure often require rigorous justification and fast payback times just to make it out of the gate.

Cost isn’t the only stumbling block, say government IT planners. Despite its obvious benefits, individual agencies may object to the loss of oversight implicit in consolidation and fret over the potential impact of greater centralization on customer service.

Alex Pettit, CIO for the state of Oklahoma, has observed this back-and-forth first hand. Oklahoma, where IT is handled agency by agency, has an opportunity to shake out duplication, he said. For example, state agencies operate more than 30 wide-area networks. Separate T1 circuits snake across the state, linking agencies headquartered in Oklahoma City to outlying municipalities such as Woodward in the northwest.

“If you look at these [networks] agency-by-agency, they make sense: you optimize for the agency,” Pettit said. “But it’s very suboptimal for the state -- particularly when covering all 77 counties.”

Related coverage:

Census consolidation by the numbers

7 secrets of IT consolidation success 

The IT consolidation challenge: Meeting agency expectations 

Wanted: Enterprise architects for data center consolidation

On the other hand, some agencies see a downside in centralization. Agencies sending their networks out to Woodward perceive that control as having value. In addition, agencies worry about a decline in responsiveness, flexibility and customer service.

“We are definitely interested in cutting the cost,” Pettit said. “But, at the same time, producing a better service is part of the focus as well. There has to be some sort of way we can provide for a higher quality of service at a lower price point, because we do have so much redundancy in service today.”

The balancing act Pettit described will play out across the country as all levels of government grapple with IT consolidation. Steve Carter, vice president for Digital Infrastructure Services at the Uptime Institute, an education and consulting organization that focuses on data centers, acknowledged the contending forces.

Consolidation and IT optimization can potentially generate a return of $2 for every $1 invested over a 5-year span, Carter said. Technologies such as virtualization and practices gleaned from previous projects can help guide consolidation and move agencies toward those kinds of returns, he noted.

“But it does take upfront seed money,” Carter said. “In a tight budget time, that is the real crux of the issue. It’s just a matter of obtaining the funding and the political will to make that happen.”

A federal case

The political will has manifested itself most noticeably in the federal sector. Vivek Kundra, the federal government’s CIO, reported in April that 39 data centers have been shuttered this year and another 98 facilities will close by the end of 2011. This effort is part of a broader program to close at least 800 data centers by 2015. Kundra pegged the savings from consolidation at more than $3 billion.

But even in the federal sector, data center consolidation isn’t a slam-dunk. When Input, a market researcher now part of Deltek, interviewed government executives involved in consolidation last summer, one of their key concerns was the lack of seed money.

“Nearly two thirds of government interviewees cited lack of upfront funding as their major obstacle,” said Angie Petty, senior principal research analyst at Deltek Information Solutions.

Some agencies have made progress despite the fiscal limitations, Petty noted, citing the 39 data center closures thus far this year. She pointed to the Census Bureau, General Services Administration and the State and Agriculture departments as making headway in consolidation. GSA and USDA have moved thousands of users to cloud computing, while State and Census have reduced costs through virtualization.

Server virtualization involves running multiple applications as virtual machines on a server, rather than operating a separate server for every application. Storage virtualization creates a single pool of storage, which can be provisioned as needed among users.

Census’s virtualization campaign is helping it close a secondary data center ahead of schedule, a move that the bureau expects will save several million dollars. That center was put in place to handle spillover from the 2010 census and provide additional space for infrastructure expansion. But the virtualization of both server and storage resources means Census will be able to host additional IT requirements in its primary data center.

As a result, the secondary facility will close by the end of the fiscal year, according to Brian McGrath, Census’ CIO. Originally, the data center was to remain open until November 2013. Another six data centers, contained within multipurpose facilities, will also shut down this fiscal year.

“We’re seeing some nice cost avoidance,” McGrath said, citing reduced hardware needs and savings on power, cooling and floor space. Census now offers its freed-up data center space to other agencies for their IT gear. The first such customer, a Commerce Department organization, is set to go online in June.

The Federal Energy Regulatory Commission, meanwhile, engaged in a server virtualization and data center consolidation program in fiscal 2010. “To date, we have been able to consolidate where we are running over 80 percent of our environment using virtual machines and have reduced the physical inventory significantly,” a FERC spokesman said.

The spokesman said it has not been difficult to fund consolidation in the current fiscal environment. The agency has simply redeployed IT budget dollars to pursue virtualization.

“Consolidation should be addressed when it comes time to refresh hardware,” he said. “Budget is usually set aside for maintenance and upgrades, and reusing this money addresses both the technology refresh and consolidation.”

These early wins aside, Petty suggested the next wave of projects -- getting to a total of 800 closures in the next four-and-a-half-years -- won’t prove as easy.

“I believe the longer-term closures will be more difficult, complex, and costly,” she said. “The jury is still out on whether the savings realized from the initial consolidations from [fiscal 2011] will be enough to fund the more complex task of consolidating an additional 663 data centers by 2015.”

Petty added that energy savings, which stem from running fewer machines and facilities, will be reflected in budgets other than IT. That situation may place those dollars out of reach for subsequent consolidation projects.  “So, the savings rewards from reduced energy consumption will not necessarily help fund additional IT consolidation projects,” Petty said.

States: step by step

State governments lack a unified consolidation directive to drive them, but a number of states pursue the objective nonetheless. At least nine states are actively planning consolidation, while others are already in implementation, said Chris Dixon, manager of state and local market analysis at Deltek Information Solutions, who is working on a study of state consolidation projects.

“There will be a good, solid five to 10 years of data center activity nationwide,” Dixon said.

Some states have been consolidating for some time. New York completed mainframe consolidation in 2004, shrinking 16 data centers to four. That project reduced software and hardware maintenance costs as well as technology refreshment and power/cooling expenses, said Angela Liotta, director of marketing, communications and Web services at the NYS Office For Technology.

Another consolidation phase may be in the offing.

New York Gov. Andrew Cuomo’s Spending and Government Efficiency Commission plans to review government operations with an eye toward cutting costs and improving service delivery. Consolidation and shared services are among the opportunities the SAGE commission will review, Liotta said. The commission will submit recommendations to the governor on an ongoing basis and provide a final report by June 1, 2012.

This phased approach to consolidation -- taking a number of whacks at the efficiency piñata -- will likely surface in other state government projects. Financially struggling agencies may find the stepwise approach easier to manage, according to industry executives.

“States are certainly strapped in terms of [capital expenditure] money, but they can’t ignore the overall [return on investment] and total cost of ownership benefits of doing a consolidated infrastructure,”  said Jim Smid, data center practice manager at Iron Bow Technologies. “They’re finding out how to work their way there in a phased approach.”

Smid recommends that agencies adopt a blueprint as a step-by-step guide toward their consolidation goals.

Local government: shared services

Local governments, meanwhile, are pursuing shared services and IT consolidation. In Charlotte, N.C., the city’s Business Support Services department acts as a shared services organization for the city and Mecklenburg County as well as other counties and municipalities in the region. Business Support Services provides procurement, fleet asset management, and public safety radios, among other services.

IT infrastructure is among the most recent function to move under Business Support Services’ roof. In March 2010 the department was directed to complete a city-wide infrastructure consolidation by January 2011. No special funding would be allocated to complete the task.

“With that set of constraints it was a pretty difficult thing to try to get done,” noted Chuck Robinson, director of Business Support Services.

The department repurposed funds from its budget and obtained help from the tech refresh portion of the enterprise technology budget, Robinson said. The consolidation effort, which tapped Exervio (now Perficient) and Gartner for consulting assistance, covers infrastructure such as routers, switches, storage-area networks, and servers. As for software, city lines of business with specific, critical applications continue to manage them.

Now that the city has consolidated its IT infrastructure, the next step is to optimize it. “We inherited a lot of bad infrastructure,” Robinson said. “Were now pulling together a capital plan that moves us into a much more sustainable, secure and reliable infrastructure.”

In one example, Business Support Services aims to eliminate aging servers and move city customers onto blade servers and virtualized services.

Alan Shark, executive director of Public Technology Institute, which focuses on local government technology, said virtualization has been embraced by most agencies. “Six or seven years ago it was looked upon as science fiction,” he said, noting the past practice of one application per server.

Business Support Services’ capital plan may also break with tradition. Robinson said Charlotte has relied on a pay-as-you-go approach and operating budgets when it comes to funding technology. He said that situation will have to change if the city is to upgrade its infrastructure.

Money isn’t the only issue with Charlotte’s IT consolidation. Robinson cited “interesting and sometimes contentious” discussions with city departments. “People know the shortcomings of their infrastructure and how to manage it,” he said. “Now, if they turn it over to somebody else, there’s the whole question of ‘are you going to manage it in a way that is good for me and serves my purpose?’”

Another sticking point: establishing service-level agreements with city customers. A lack of performance metrics among the city’s former IT enclaves makes it difficult to determine requirements. As a consequence, Robinson said his department monitors a given piece of infrastructure and builds service-level agreements “as we go along” in an iterative process.

But while Charlotte nails down service levels, Oklahoma debates whether to consolidate at all. The state legislature will ultimately decide on whether the state consolidates all of its IT, rationalizes a portion of its holdings, or stands pat.

“We’re waiting for them to decide what they want to do about it,” Pettit said. “It’s not certain at this point what will happen.”