A late entry in firewall assessment gets kudos for performance, speed.
Every network manager worries about security, not just to keep out hackers but to keep
erroneous information from flowing through the network.
Until recently, the only strong firewall protection ran under Unix. There now are
firewalls for Microsoft Windows NT, but some security experts say NT firewalls are not as
secure, manageable or robust.
After evaluating several firewalls for NT, the National Software Testing Laboratories
Inc. has two words for those experts: Guess again.
NSTL of Conshohocken, Pa., bombarded seven top-selling NT firewalls with nearly 300
forms of attack and found no significant security loopholes.
Thats a big improvement over the labs past tests of Unix as well as NT
products, which had exposed dozens of flaws. And the newest products do an excellent job
of locking down NTs own potential vulnerabilities.
No firewall product has reached security nirvana. Some just arent as strong as
others in terms of manageability or performance. The lab found at least two areas in which
products were weaker than it had hoped: logging and address translation.
Logging is a concern both in the amount of information that can be logged as well as in
firewall behavior as the log fills up. Furthermore, NSTLs performance stress test
showed that network address translation significantly slowed things down.
Two products stood out: Firewall-1 from Check Point Software Technologies Inc. and
Guardian from NetGuard Inc. Both earned the Reviewers Choice designation.
Check Point Firewall-1 had strong security, good performance and a great management
interface. It was especially easy to manage remotely.
NetGuard Guardian had an admirably clear management interface that laid out key tasks
in a clear, straightforward way. It also was one of the fastest of the group.
An honorable mention goes to Eagle NT from Raptor Systems Inc., which has a
much-improved management interface and trailed the two leaders by a slim margin in the
area of reporting tools.
When NSTL took its first look at NT firewalls last year, there were only four NT
products [GCN, Aug. 4, 1997, Page 35]. This year, seven vendors sent products to be
tested, and several others declined to take part.
The only thing new about the products is the NT platform. The firewalls have the same
architectures as their Unix predecessors. Whatever the operating system, a firewall acts
as a packet filter or an application proxy.
A packet-filtering firewall inspects each packet it receives and decides whether to
forward or to drop the packet after checking a table of access control rules.
Stateful inspection, a variation on packet filtering, goes beyond simple filtering. It
tracks the state of each connection the firewall handles. This keeps attackers from
hijacking a connection while it is opening or closing.
A proxy firewall acts as an intermediary for users. Instead of simply passing along
user requests, the proxy firewall terminates an applications connection at the
firewall and then sets up a separate connection to the desired server.
In circuit relay, the user logs in to the firewall via a session layer mechanism such
as the Socks security protocol or Microsoft Corp.s Winsock driver. That in turn sets
up another session on the users behalf.
After years of debate over which approach gives better security, vendors are beginning
to blend the two. Some, such as NetGuard and Secure Computing Corp., offer both proxies
and packet filtering. Even Check Point, the first vendor with a stateful-inspection
product, will provide proxies if users request them.
Nearly all the firewalls in this test support network address translation, which is a
way of hiding all the IP addresses on the protected side of a firewall from the external
interface. Hiding the addresses keeps would-be attackers from discovering the internal
In this evaluation, all the products but HeatSeeker Pro from Fortress Technologies Inc.
could do address translation. The client-based HeatSeeker Pro works differently from the
other tested products. It resides on all protected machines instead of acting as a
gatekeeper. That doesnt preclude the use of network address translation, which is
available at extra cost.
Two products, Raptors Eagle NT and Secure Computings BorderWare, worked
only with address translation enabled. Many firewalls also support virtual private network
features such as authentication and encryption, but the lab did not evaluate them in this
Security is a firewalls reason for being, so it was the most important test
criterion. The lab looked not only for common firewall vulnerabilities to the ping of
death and finger bomb, but also for potential loopholes in Windows NT configurations. That
step is especially important, because NT Servers out-of-the-box configuration ease
could make it vulnerable.
The installation routines included steps to secure NT, such as replacing the default
adapter driver with a packet driver hardened by removal of unnecessary services.
As in previous security tests, the lab used Safesuite, an intrusion detection tool from
Internet Security Systems Inc. of Atlanta. Safesuite 5.0 pokes and prods each firewall
with 288 different attacks.
Because all the products ran under NT, the lab could test on a single hardware
platform. NSTL installed each package under preloaded Windows NT Server 4.0 and the latest
Service Pack the product could support.
That was a major change from past firewall tests, in which vendors brought in and
installed their own software and hardware. Standardizing on one platform ensured an even
comparison because the only variables in the test bed were the firewalls.
All tests had two interfaces that linked protected and unprotected network segments,
and they had a simple rule set that allowed only a few services.
The security tests allowed no access from the unprotected to the protected segments. In
the other direction, access to just six common Internet applications was allowed.
To minimize the risk of configuration errors, the lab required vendors to set up their
own products for the security tests.
Safesuite looks for hundreds of potential vulnerabilities, and its output can be
voluminous. To help net managers perform triage, Safesuite puts vulnerabilities into
high-, medium- and low-risk categories.
Internet Security Systems defines high-risk vulnerabilities as those that let an
attacker get into the system running the firewall software. A few high-risk
vulnerabilities, such as compromisable proxies, are specific to firewalls.
Many other high-risk vulnerabilities are specific to Windows NT. Examples include
sharing of resources using NetBIOS names, using outdated system files that expose password
lists and failing to apply security patches after attacks are discovered.
Microsoft usually supplies patches for security flaws within 48 hours after they have
been announced. The patches are downloadable from the Web at
Safesuite classifies vulnerabilities as medium-risk if they might compromise services
but not the system itself. A notable example is Syn flooding, or bombarding a firewall
with so many requests for TCP sessions that it cannot handle legitimate requests. Syn
flooding attacks do not compromise the target machine, but they effectively take it out of
Safesuite also identifies several denial-of-service attacks that can be mounted against
NT, such as locking up a system by sending it improperly formed ping packets.
Results were good. Safesuite turned up almost no vulnerabilities, either in the
firewall code or in NT configurations. In last years test of fewer than 100 attacks,
NSTL found dozens of problems including some high- and medium-risk vulnerabilities.
This year, running nearly 300 attacks, the lab turned up a scant seven vulnerabilities,
all in one firewall, and none serious. The seven were false alarms triggered by daemons
running on an intercept course.
The clean bill of health may come largely from the popularity of intrusion detection
products such as Safesuite, now resold by several firewall vendors as a firewall
configuration-checker. Knowing that Safesuite checks for the best-known vulnerabilities,
firewall vendors have added safeguards against them.
Digital Equipment Corp.s AltaVista Firewall 97 was the only product to reveal any
vulnerabilities, all of which were false positives.
Digital configured its firewall to allow inverse Domain Name System lookups and DNS
zone transfers, which can be exploited to find details about protected networks.
Digital confirmed that the firewall supports the services, but only for its unprotected
interface; it does not advertise information about the protected network. Even so,
firewalls should not respond to zone transfer requests, and Digital has disabled the
feature in the Unix version.
The labs test also showed the Digital firewall would open a File Transfer
Protocol connection back to itself, a medium-risk event that could allow a
denial-of-service attack. This too was a false positive; the FTP version used in the
firewall is not vulnerable to this form of attack.
Safesuite turned up a few low-risk vulnerabilities, such as support for the Finger
command on the firewalls external interface. This is innocuous, because Finger
doesnt respond with information about the protected network.
Digital officials said the firewall can be configured to shut down these potential
Besides scanning for firewall and NT vulnerabilities, the lab also evaluated each
firewalls ability to screen executable content such as ActiveX and Java applets.
Applets can hold malicious code that gets in through services a firewall is supposed to
For example, an attacker could write a Web page containing a malicious ActiveX applet
that reformats the users hard drive. A firewall configured to allow Web access
would let this through unless it had some means of screening for executable content.
To test whether the firewalls could block executable code, NSTL set up a Web server
with pages containing ActiveX and Java applets, and the lab asked each vendor to configure
its firewall to block the applets. The lab worked with a Web browser on the other side of
the firewall to evaluate performance.
Five of the seven products blocked executable code: those of Check Point, Fortress,
Microsoft, Raptor and Secure Computing.
The packages worked as advertised, blocking both ActiveX and Java objects. The lab
checked only whether firewalls would block the executable content outright. Firewalls can
collaborate with third-party products such as SurfinGate from Finjan Inc. of Santa Clara,
Calif., which lets executable content pass through the firewall after inspecting it for
NSTLs security results do not in any way certify these firewalls as safe. The
tests involved the worlds foremost experts on the productsthe vendors who
built themconfiguring their own software to withstand well-known attacks in a
carefully controlled lab environment.
Because security depends on correct configuration, the lab took a close look at
manageability. NTs graphical environment should make this task a snap. Yet the lab
found major differences, especially in terms of audit logsa critical capability when
it comes to tracking down miscreants.
Why the differences? For one thing, theres a lot of variation in firewall
interface designs. Some interfaces are easier to navigate than others. Also, there are
substantial differences in what each interface permits the net manager to do. That makes
direct comparisons difficult or even impossible.
Rather than try to compare apples with oranges, NSTL drew up three common scenarios
faced by any manager responsible for a firewall and then evaluated the ease of
accomplishing the tasks within the scenarios.
The scenarios tested remote management, access control and logging. Vendors did not
configure firewalls for this part of the review.
In the first scenario, the labs hypothetical network manager was awakened in the
middle of the night about an attack in progress.
All the products could notify the manager via e-mail or pager about specific attacks.
But it was easier to set up notification in some products than in others. Raptors
Eagle took top honors for the most notification options, including e-mail, pager and
paging through an e-mail server.
The manager then decided to shut down the firewall for the night. All products allow
remote reconfiguration, including remote shutdown. NetGuards Guardian excelled with
the most straightforward menus and fewest steps to disable the firewall remotely.
NSTL expected to find an encrypted link between the firewall and the net manager at
home. Surprisingly, products from Fortress and Secure Computing did not encrypt
communications between the firewall and a remote management station.
That might not be a problem on a point-to-point dial-up link, but on any shared-media
network, a lack of encryption poses a security threat. An attacker could grab packets from
the network and follow the net managers commands or, worse, gain access to the
Because networks are vulnerable to attack around-the-clock, the lab asked vendors about
technical support hours. All except Fortress and Secure Computing have round-the-clock
tech support; Check Point, Digital and Raptor charge for it. Some of Secure
Computings resellers give optional round-the-clock support.
In the second scenario, the lab looked at how easy it is to set up access control
policies. Here, the network manager set two policies. The first allowed inbound access
from only a few source IP addresses all the time, outbound access for any address between
6 a.m. and 11 p.m., and outbound access for only a few addresses from 11 p.m. to 6 a.m.
All the products could restrict access based on IP address and time of day. In rating
ease of use, the lab did find some minor differences. Setting time-of-day restrictions was
slightly more difficult in the Microsoft and Raptor firewalls.
The second hypothetical policy had the same time-of-day restrictions but allowed access
based on user identification rather than source IP address. Whats the difference?
Outside the network, road warriors who dial in are unlikely to be assigned the same IP
addresses as when theyre in the office. And inside the network, more and more net
managers are using the Dynamic Host Configuration Protocol to assign different IP
addresses for different user sessions.
Because static IP addresses say little about whos using them, it has become
increasingly important that the firewall identify who requests a resource, not just which
address is making the request.
Fortunately, all the products allow some form of log-on based on user ID.
Digitals AltaVista establishes user credentials based on FTP or Telnet user names
and passwords. Thats risky in itself because both applications transmit names and
passwords in the clear.
But the Digital product also supports SecurID token authentication from Security
Dynamics Inc. of Bedford, Mass., which uses an external database and strong cryptography
to establish user credentials. Firewalls from Check Point and Raptor also support SecurID.
The Check Point, Microsoft, Raptor and Secure Computing products support Radius, a
remote-access dial-in user service that is a standards-based means of authentication.
Check Point and Raptor also support TACACS+, or the Terminal Access Controller Access
Control System, a means of authenticating user IDs on network equipment from Cisco Systems
Inc. of San Jose, Calif.
Fortress HeatSeeker Pro made a chore of configuring policies by user ID. The
firewall relies on Windows user profiles for defining access. That eliminates duplicate
access lists but relies on the crypto strength of the Windows password-protection scheme,
which some security experts say isnt strong enough.
The third management scenario examined each firewalls logging ability. A
firewalls audit log is its flight recorder. Without adequate and accurate logging,
theres no way of saying how, when and where an intruder tried to enter.
You would expect firewalls to give the manager a clear picture of whos using the
firewall at all times. Yet several products didnt meet that expectation.
In this scenario, the labs hypothetical manager wanted to get control of the
logging process. The manager had to pick out unauthorized access attempts, track
historical usage, and determine where and how logging took place.
The manager wanted to produce two reports: one showing all unauthorized access attempts
and one showing all Web requests in a 24-hour period. That sounds easy, but the volume of
log data makes it more difficult than you might think.
Firewalls generally log every connection they handle. Lets look at a typical Web
connection. Any given Web page may embed a dozen or more objects, each representing a
separate connection. The typical user will jump from page to page.
If the firewall is logging each connection, the log grows exponentially to gargantuan
proportions, even in small organizations. Searching gigabytes of log entries to find a
specific event simply isnt practical. Instead, the firewall product must provide
some means of sorting log entries by criteria identified by the manager.
All the products reviewed, except Microsofts Proxy Server, have reporting tools.
Of these, Fortress was the easiest to use. The log entries from Microsoft Proxy
Server can be sorted using external applications such as Microsoft SQL Server or Excel,
but they are extra-cost add-ons. If you dont have them, you cannot access the
critical information in the database.
Because firewall logs grow so rapidly, the lab also examined the products
behavior when their logs and disks had filled up. The lab was specifically interested to
see whether the firewalls were susceptible to denial-of-service attacks caused by filling
up the logs with garbage entries.
NSTL asked vendors to describe how their firewalls react to full-log situations, and
they presented five ways of handling the situation.
First, a firewall might wrap its log when full, rewriting the oldest entries first.
Second, it might delete the old log file, then create a new one using the same filename.
Third, it might continue operating but stop logging. Fourth, it might shut down
altogether. Finally, it might let the manager choose.
There is no one right approach. In shops where security is paramount, keeping a
complete log is the most important action, even if it means shutting down the firewall and
thus access to the network.
In other organizations, network uptime is so critical that it may be best to keep the
firewall running even if logging is temporarily disabled.
The first three approaches maintain the firewall operation but could let attackers
cover their tracks or operate undetected. The fourth approach ensures a complete audit
trail but restricts access to the network.
NetGuard and Secure Computing wrap their logs when full. Microsoft and Raptor delete
the old logs. Check Point and Fortress disable logging, and Digital shuts down access to
No product gives the manager the option of choosing among these approaches.
Attackers might also lock up a firewall by filling its hard disk. In theory, this
shouldnt be possible, because a machine acting as a firewall should allow no access
to services that involve writing to disk.
Nevertheless, if an attacker does somehow gain access to the firewall and fills the
disk, only the Microsoft and NetGuard products shut down. The others treat a full disk the
same as a full log file.
Because log files are a potential vulnerability, managers would be wise to define a
large amount of log space, perhaps in a separate disk partition.
The products from Microsoft, NetGuard, Raptor and Secure Computing let net managers
define log size.
For even more effective security, the manager could set up a separate machine for
logging, possibly one that creates read-only media on a CD-recordable or optical drive.
All the products except Digitals permit logging on a separate system. But only
the Check Point, NetGuard and Raptor firewalls could transmit log data in encrypted form.
The others do not encrypt logs, potentially exposing the data to capture as it crosses the
As in past tests, the lab also measured firewall performance. Performance is a concern
for NT products, for a couple of reasons. First, NT is an increasingly popular choice on
intranets, which typically use 10- or 100-Mbps interfaces rather than the 1.5-Mbps T1 or
slower interfaces that connect to the Internet.
Second, security expertsmany with deep roots in the Unix worldoften charge
that NT isnt as speedy a performer as Unix.
To see how theyd stand up on fast networks, the lab pounded the firewalls with
extremely heavy loads. The lab placed Fast Ethernet switches on both sides of the firewall
and used full-duplex links, ensuring full 100-Mbps loads simultaneously in both directions
with no collisions.
The lab placed traffic generators behind the switches, each capable of slinging data at
100-Mbps, and placed Hypertext Transfer Protocol and FTP servers on both sides of the
firewall, each fielding requests from the traffic generators on the other side.
The lab also made other key changes from past firewall tests. The most important
change, as noted earlier, is that the lab supplied the firewall hardware. NSTL chose a
powerful dual-CPU, 200-MHz Pentium Pro system with 256M RAM. Thats more horsepower
than firewalls typically have, but it ensured that the only variable in the tests was the
firewall software itself.
Another key difference was the decision to send Web and FTP traffic separately. Over
the last year, the lab has seen substantial difference in the performance of proxies for
In this test, the lab chose two of the most common Internet and intranet
applicationsWeb and FTP trafficand measured them separately.
NSTL decided to require network address translation from all products that support it.
This reflects a real-world reality: Translation is an increasingly attractive option for
organizations that face a shortage of IP addresses.
The downside is that translation degrades performance. It requires rewriting IP
addresses for each packet that traverses the firewall. As noted, all products except
Fortress HeatSeeker Pro support address translation.
To ensure correct and consistent setups, the lab required vendors to configure their
firewalls for this part of the test. The lab asked them to enable logging, run six popular
Internet apps and deny all other traffic.
Once the labs test bed was set up with a pair of clients and servers on both
sides of the firewalleach with two 100Base-T interfacesthe theoretical maximum
aggregate throughput was 100 Mbps in each direction, or an aggregate 200 Mbps.
The lab used a protocol analyzer to verify that forwarding rates on the wire were 100
Mbps in each direction.
NSTL equipped all the machines with two 100Base-T interfaces, each having its own IP
Traffic was both nonmeshed and bidirectional. In the nonmeshed pattern, Client
Interface 1 requested uniform resource locators from Server Interface 1, and Client
Interface 2 requested URLs from Server Interface 2.
In this bidirectional pattern, the client machines on either side of the network
simultaneously requested URLs from servers across the firewall. The pattern ensured that
the lab gave both firewall interfaces a full load.
The lab began by baselining the test, generating traffic between two switches with no
firewall. Measurements with a protocol analyzer showed the link was 100 percent utilized
in both directions.
No existing firewall can handle 100-Mbps bidirectional loads, and the number of
firewalls on 100-Mbps segments is still low.
But that number is growing, and one of the major concerns for administrators is the
maximum forwarding rate a device can sustain. Although firewall performance is important,
it is far less important than security or manageability.
NSTLs tests ran traffic at full bore in each direction, measuring the forwarding
rate in each direction and then adding the two numbers. Because the lab used switched
full-duplex links throughout the test bed, it was theoretically possible to achieve
aggregate throughput of 200 Mbps.
But the fastest result for network address translation was just 62.6 Mbps, recorded for
Web traffic by Microsofts Proxy Server. Proxy Server also handled FTP traffic
fastest but far below the theoretical maximum.
The next-highest performers were the products from NetGuard, Secure Computing and Check
Point. Clearly, address translation degrades firewall performance.
The effect was even plainer in the results for HeatSeeker Pro, which doesnt do
address translation. Its FTP rate was nearly twice that of most other products, and it
moved Web traffic more than 10 percent faster than the next-fastest translation-enabled
Proxy-based products from Digital and Raptor posted substantially lower FTP forwarding
rates than HTTP rates. That points up the need to evaluate proxy performance based on the
applications in use at a given site.
Some products moved traffic faster in one direction than the other. NetGuards
firewall, for example, moved both FTP and HTTP traffic about 10 Mbps faster from the
protected network than to it. Raptors Eagle showed the same behavior with FTP
Microsofts Proxy Server moved FTP traffic nearly 20 Mbps faster from the outside
in than from the inside out.
All vendors were asked to send a firewall that supported Windows NT Server 4.0, Service
Pack 3, 100Base-T interfaces, remote management and network address translation.
All tests ran on 100Base-T segments except for the remote management scenario, which
incorporated a management console on a 10Base-T segment. The lab tested all products on
the same hardware and swapped in a new hard drive for each entry.
In the Intermark custom application developed by NSTL for testing Internet
applications, the lab set up two FTP and Web serversone on both sides of the
firewall. NSTL equipped both servers with two 100Base-T adapter cards, each with a unique
To ensure maximum throughput to each firewall, the lab placed servers and clients
behind Fast Ethernet switches, which in turn connected to each firewall interface.
Intermark reported the number of successfully completed transactions per minute. NSTL
derived average throughput by dividing the amount of data transferred by the time required
to complete each test.
Helen Holzbaur is a network project manager at National Software Testing
Laboratories Inc. of Conshohocken, Pa.