Patch tactics

Alabama National Guard finds a cure for patch management headaches

Key to successful patching: Don't be pushy

The point of a reliable, automated patch management system is to get those security patches rolled out, installed, and the endpoints rebooted as quickly as possible.

Not necessarily. When a mission is critical, sometimes it's best to wait.

'With the restrictions that the military has, you have to have a product that is user friendly, so that if you are in the middle of something you can pause the patch,' said Jim Crawford, information security specialist for the Alabama National Guard. 'Get a product that does that if you're concerned about productivity.'

The Alabama National Guard has standardized on the PatchLink Update system to automate its security patch management on more than 2,000 workstations and servers.

'They are a pretty typical user,' said PatchLink vice president of security technologies Chris Andrew. Some agencies, such as the Veterans Affairs Department with 300,000 nodes or NASA with 80,000 nodes, are larger. But 'the most typical installation is like this one.'

PatchLink tests and sends critical patches out to customer servers within 24 hours of receiving them from the vendors. It is up to the customer to decide when and how to deploy them.

Downloading and installing patches can interfere with some work, and rebooting the client to make the patch effective can be a problem if the user is in the middle of something critical. The user has the option of halting the process so it can be finished up at a more convenient time.

But an unpatched computer is at risk and eventually the job will have to be done, so administrators can set limits to how long the patching is delayed. At some point, a curfew can force the installation and the reboot.

The Alabama National Guard also has patching policies for each workgroup, taking office hours into account so that the installation and rebooting can be done routinely with as little disruption as possible.

UP TO DATE: The Alabama National Guard's Jim Crawford tried a lot of security patch management systems but has had the most success with PatchLink's Update.

WPN photo by Cary Jobe

Keeping security patches up to date at 120 National Guard sites throughout Alabama was becoming something of a headache for the Guard's IT administrators.
'We support about 2,200 computers and servers,' said information security specialist Jim Crawford.

That headache was only going to get bigger, because guardsmen issued Common Access Cards by the Defense Department must be given access to the Army's Internet domain.

'We are going to have to support our 9,000 traditional National Guardsmen on a full-time basis,' Crawford said. That support will come by adding an additional six to eight computers at each National Guard site.

The Alabama National Guard has turned to the Update patch management system from PatchLink Corp. of Scottsdale, Ariz., to handle the growing job. The Army requires the Guard to be fully updated on its patches within 30 days of release, a deadline that Alabama now is able to beat by pushing out patches received through the Update subscription.

'In a week I can get it 90 percent patched,' Crawford said. 'In two weeks I'm 100 percent.'

Software patches are a critical component in securing IT systems. This puts a premium on being able to trust a patch and automate its installation.
PatchLink Update was not the first tool the Alabama Guard turned to for getting this job done. It started with Version 1 of Microsoft Corp.'s System Management Server, Crawford said.

'It didn't work at all,' he said. 'Then we migrated to v. 2.0 and it worked pretty good,' except that it had to be heavily tailored for the Guard's environment. 'That's when I got disgusted with Microsoft.'

He also tried the Hercules vulnerability management tool from Citadel Security Software Inc. of Dallas. But Hercules could make unexpected changes.

There also were some problems with the 12-Kbps circuits that connected most of the Alabama sites. Both SMS and Hercules could eat up that bandwidth with batch distributions of patches. The state recently finished building out a Bell South network with T1 links to each National Guard site, and bandwidth no longer is a critical issue. But even before the upgrade, PatchLink was able to push out patches on an as-can basis without using up all the bandwidth.

PatchLink Update is a server-agent system with an online service for delivering new vendor patches to the customer's server. Agents residing on client devices scan the system to determine what patches are needed and if they can be safely installed. They also can confirm successful installation and alert administrators when installation fails or creates a problem.

Update confirms installation by searching for signatures created by the patch in each software platform, said Chris Andrew, PatchLink vice president of security technologies. If the signature is not present, something went wrong with the installation.

New patches are delivered to Update servers daily. PatchLink also distributes third-party workarounds for vulnerabilities that do not have vendor patches. The patches and fixes are sent out after being tested by PatchLink against a library of up to 150 image configurations to help ensure against conflicts and unexpected results. The company's service level agreement calls for patches deemed critical to be delivered to customers within 24 hours, and longer for moderate and less important patches.

Andrew said PatchLink has a 'high degree of confidence' in the testing, but that given the wide variety of end-user platforms, it is not foolproof.
'We provide the next level of assurance on top of what the vendor provides,' he said. 'At this point, best practice for our customers is not to take that patch and roll it out, but to do their own tests' and install it in phases to limit any negative impact.

The Alabama Guard keeps a pretty tight rein on its system profiles, with 'maybe three different hardware configurations,' Crawford said. This helps speed his local testing and installation. 'When I get it, it takes me an hour to push it out in my lab to see if it blows up anything,' he said. 'Then I'm patching.'

The Update client agent has a small footprint, about 5MB, and runs in the background as a system service with negligible impact on performance, Andrew said. Because it runs as a system service, it can patch a locked-down system that the user is not supposed to modify.

The agent also helps with ensuring the patching on the nearly 400 laptops that go home each night and connect to the network remotely. When remote users access the network over a virtual private network, the PatchLink server on the intranet is the first thing they hit. If the machine is not up to date, patches can be pushed out. In some rural areas of Alabama, VPN connections are not possible. Those connections over the Internet are pushed to the same PatchLink server for discovery and patching. Performance is about the same with each type of connection, Crawford said.

'I don't see a lot of difference,' he said. 'I can't tell which connection was used when the patching was done.'

Reader Comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above