Crunch time
States find IT consolidation is key to staying creative amid budget shortfalls<@VM>Sidebar | Disaster recovery on the cheap<@VM>Sidebar | This year's budget, last year's economy
Talking to Teri Takai, Michigan's chief
information officer, one could be
reminded of ancestral tales from the
Great Depression ' of grandmothers making
clothes out of flour sacks and scraping their
lips getting the last bit of lipstick from the tube.
Takai and her team are using the same spirit
of innovation to squeeze more computing
power from Michigan's limited computer dollars.
As an example, by consolidating the state
government's 25 data centers into three,
Michigan has saved $9.5 million with an expected
return on investment of $19.1 million.
When times get tough, states have to make
do with what they have. And probably no state
has had to do more with less in the past few
years than Michigan, battered
by a downturn in the auto industry.
Michigan's budget woes
culminated in a five-hour shutdown
of the state government in
September.
Waxing and waningA year ago, headlines were
trumpeting state and local
budgets as relatively bright spots in the economy,
having recovered from the blows of the
recession following the 2001 terrorist attacks.
Many states were seeing an increase in revenues
because of the housing boom of the first
half of this decade.
But a number of factors have recently constrained
state and municipal budgets. The
housing market's slump has accounted for
decreased revenues, particularly in states
such as California and Florida. Louisiana
is still struggling two years after Hurricane
Katrina devastated coastal areas.
Although recent media reports have described
the downturn in state economies as
sudden, Doug Robinson, executive director at
the National Association of State Chief Information
Officers, said there were signs of a
downturn about a year ago. Robinson talks to
state CIOs regularly and said many of them
were talking about travel restrictions and hiring
freezes in January.
When the federal government runs out of
money to fund information technology projects,
it can always borrow more. But states
can't. Every state except Vermont has a constitutional
or statutory requirement to balance its
budget. And even without a legal requirement,
Vermont always balances its budget.
States' fiscal situation isn't following any particular
pattern, said Scott Pattison, executive
director at the National Association of State
Budget Officers. In the late 1990s, almost every
state's revenues went up 'and they all came
down after Sept. 11, 2001. In the 1980s, there
was a geographical pattern to the downturn,
with states in the manufacturing-heavy Rust
Belt taking a financial hit.
Now it's a mixed bag, Pattison said. For example,
North Carolina is doing fine fiscally, yet
South Carolina is having trouble making ends
meet.
When Michigan Gov. Jennifer Granholm
took office in 2003, one of her first goals was to
reduce the cost of IT.
The second objective was to improve security.
'When you have a distributed configuration,
it's difficult to maintain security,' Takai said.
Doing more with lessMichigan government had two e-mail systems
that were configured in 70 different ways.
Many of its 25 data centers had been converted
from old conference rooms. One of the data
centers had computers and power switches
located next to the water intake.
When Takai started on the path of consolidation,
participation was voluntary. 'But we kept
running up against the 'you can't touch my
stuff' thing,' she said. Undaunted, Takai drew a
line in the sand and said that
no new computer equipment
would go into anything other
than the three data centers.
Michigan's IT Department
also was able to obtain funding
from the federal Homeland Security
Department to support
its consolidation project. States
allocate DHS funding by giving
80 percent to local governments and retaining
20 percent for state use.
All of Michigan's agencies applied to receive
that 20 percent. The IT Department had just
undergone a continuity-of-operations planning
study, and state officials determined that
it was essential for Michigan's IT services to
stay up and running. Officials reasoned that if
the DHS money went to the IT Department, it
would help all of the state's agencies. The consolidation produced some side benefits, too.
'Moving all these servers into three locations from 25, we had to touch all of them,' said Pat Hale, deputy director of infrastructure
services at the IT Department.
The IT staff discovered
that the state had 180 old Microsoft
NT servers and 4,600 NT PCs. NT servers and
PCs can make the state network susceptible to
viruses because antivirus software is not readily
available for NT environments.
The IT staff also found older equipment that
still had some life. Some agencies replaced their
PCs every three years; others replaced them
every 12 years. By redeploying PCs that had
good use left in them, Michigan migrated its
old NT environment, said Ken Theis, the IT
Department's chief deputy director. The department
replaced 100 of the NT servers by redistributing
them.
'The perfect solution would be to get new
ones, but we didn't have the money,' Hale said.
The lack of funding wasn't a hindrance to
moving forward, Takai said. 'It would have
been easy to say, 'We don't have any money, so
there will be no new innovations.' But it's been
quite the contrary. The budget crunch has
helped us become more innovative.'
Consolidation feverThe state has caught consolidation fever.
Michigan had 27 separate grant systems that it
is consolidating into one, and it also consolidated
its geographic information systems.
Michigan is using Microsoft Active Directory
and NDS Migration Suite from Quest Software
to move to a unified directory service. This will
allow everyone on the network ' every agency
' to have a single source for authentication on
the network, said Paul Christman, director at
Quest Software's state and local division.
Once the state is using one directory for its
network, 'then you can really do something
with it,' Christman said. Plus, there will be
fewer passwords to remember.
Enabling integrated services and consolidating
data centers 'is like building the foundation
for a house,' Christman said. 'Later on you can
make it solar-powered or wind-powered. But
right now we're building a foundation.'
Takai and her team are building this foundation
one migration at a time. And if it takes a
little longer, that's OK.
'We know we have to change the infrastructure,'
she said. 'Without investment it might
take longer. But it doesn't mean we can't make
strides in changing it.'
The Alabama Criminal Justice
Information Center (ACJIC) was
stuck between a rock and a hard
place. On one hand, the information it
kept track of was being requested
around the clock by police officers,
correction officials and others. On the
other, the center had little money to
spend on extras, such as measures to
ensure disaster recovery.
But managers found that, with some
creative thinking, they could not only
get a disaster recovery plan with limited
funds but also improve the
uptime of their service.
Started in 2004, ACJIC serves as
the information hub for all the criminal
justice data collected by the state.
The system performs 18 million
transactions per month for as many
as 15,000 people a day. Not bad, considering
only 10 people run the show.
'We do a ton under budget constraints,'
said Maury Mitchell, director at
the agency.
As the operation got ramped up, one
missing piece that became increasingly
evident was disaster recovery.
'Disaster recovery has not always been
on the forefront of state government
because of budgetary restraints,'
Mitchell said. 'Everyone is just struggling
to get operations done.'
The organization did have tape backups
of the data itself. But even small disruptions
could stop operations.
'Five years ago ... we had one rack with
three servers sitting in the hallways at
the administrative offices, under a sprinkler
system,' said Jeff Matthews, IT
director at the center.
Also, the system was run from a state
mainframe in Montgomery.
What to do? One option would be to
consider companies that offered commercial
backup services.
That would be a costly approach, however
' as much as $90,000 per year.
Instead, Mitchell and his crew cobbled
together their own recovery plan.
They found another state agency with
some extra rack space it was willing to
rent out for around $12,000 a year,
where ACJIC could house its backup
server, an ES7000 from Unisys, and a
20 T storage area network.
The center also deployed backup software
to duplicate the data in
Montgomery. 'Once we got the solution
in place, it didn't cost a lot,' Mitchell said.
How did the center get the additional
money for the disaster recovery setup?
By appealing to the other agencies that
used the service, Mitchell said. The
agencies saw how such continual service
would benefit their own operations, so
they all agreed to chip in to maintain the
backup services.
State budgets tend to be a lagging
indicator of the shape of the economy.
'If you think about states collecting
income taxes, that trails the
economy by 12 months,' said Paul
Christman, director at Quest
Software's state and local division.
If the incomes in a state go
down, the lower tax revenues won't
show up for a year, when taxes are
collected.
So despite all the hoopla about
the housing market tailspin in
Florida, it hasn't trickled down yet
into budgets in Florida's Orange
County, said Joe Giovanelli, management
information systems manager
at the county tax collector's
office.
The county's budget is approved
by the Florida Department of
Revenue, and Giovanelli said he
hasn't yet seen a slowdown.
Nonetheless, the county is
streamlining operations by consolidation.
Communication between the
main office and the agencies has
improved substantially, Giovanelli
said. Consolidating servers onto
eight Hewlett-Packard blade
servers also cut tape backup time in
half.
The county is even consolidating
eight county agencies' copiers, fax
machines and scanners, which
were growing old and required
expensive maintenance. It is replacing
them with HP multifunction
printers that can fax, scan and copy.
'But I'll be honest with you,'
Giovanelli said. 'We haven't really
saved any paper.'