MTR MITRE TECHNICAL REPORT Firewall Test Tools Evaluation August 1997 Glenn C. Everhart G021 U G021 This document was prepared for authorized distribution only. It has not been approved for public release. MITRE MITRE Department Approval: ApprovalNameHere MITRE Project Approval: ApprovalNameHere Table of Contents Section Page 1 Introduction 3 1.1 Background 3 1.2 Purpose 3 1.3 Document Organization 3 2 MITRE Firewall Test Tool 5 2.1 Tool Description 5 2.2 Rule Sets/Firewall Security Policy 6 2.3 Packet Generation 6 2.3.1 MFTT Version 1 6 2.3.2 MFTT Version 2 7 2.4 Packet Observation 7 2.4.1 Version 1 7 2.4.2 MFTT Version 2 8 2.5 Packet Correlation With Firewall Security Policy 8 2.6 Other Features 8 3 ISS Firewall Scanner 11 3.1 Tool Description 11 3.2 Rule Sets/Firewall Security Policy 11 3.3 Packet Generation 12 3.4 Packet Observation 12 3.5 Packet Correlation With Firewall Security Policy 12 3.6 Other Features 13 4 Chisel\ 15 4.1 Tool Description 15 4.2 Rule Sets/Firewall Security Policy 15 4.3 Packet Generation 15 4.4 Packet Observation 16 4.5 Packet Correlation With Firewall Security Policy 16 4.6 Other Features 16 5 Ballista 17 5.1 Tool Description 17 5.2 Rule Sets/Firewall Security Policy 17 5.3 Packet Generation 17 5.4 Packet Observation 18 5.5 Packet Correlation With Firewall Security Policy 18 5.6 Other Features 18 6 Free Utilities 19 6.1 Tool Description 19 6.2 Rule Sets/Firewall Security Policy 20 6.3 Packet Generation 20 6.4 Packet Observation 20 6.5 Packet Correlation with Firewall Security Policy 21 6.6 Other Features 21 7 Summary 23 8 Recommendations and Futures 27 Abstract Many organizations use firewall systems to prevent computer network based attacks on their systems from the outside. These firewall systems have an inside and an outside network connection and are given a set of rules concerning what network traffic they should allow to pass through, and what they should block. These rules are expressions of the organization’s security policy. Determining whether these rules are set up to do what the firewall owner wanted them to do is a task which is often overlooked. A number of tools exist for testing firewalls to determine whether they are functioning correctly. Some are commercial, one is an in-house MITRE tool, and some are free utilities. This document examines all three tool types. Of all the tools examined, only the MITRE tool has the capability of comparing test results to an expected security policy. Keywords: Firewalls, Firewall Test Tools, Security Section 1 Introduction 1.1 Background As computer networks are more universally deployed, concerns over the results of inadequate security have grown. One very common technique in securing a previously isolated system or network is to install a firewall between this previously-isolated system or network and the wider networks to which it is being connected. This firewall is a specialized filtering device which selectively allows only some traffic to pass through it, thus providing a measure of resistance to external attacks on the security of what is inside the firewall. Firewall systems can be either programs on general purpose computers, or specialized custom boxes. In any case, a security policy describing what network traffic may pass through the firewall and what must be blocked must exist. This policy is fed into the firewall to direct its operation. Some test tools are now available which can be used to check that a firewall is operating according to a specified security policy. 1.2 Purpose The purpose of this document is to present an evaluation of firewall test tools according to tool type: MITRE test tool, commercial test tools, and free utilities. 1.3 Document Organization This document contains 8 sections. Section 1 is this introduction. Section 2 is an evaluation of the MITRE Firewall Test Tool. Sections 3 through 5 present evaluations of commercial tools. Section 6 reviews free utilities. Section 7 is a summary, and Section 8 is a set of recommendations. Section 2 MITRE Firewall Test Tool 2.1 Tool Description The MITRE Firewall Test Tool has two versions. Version 1 is the currently released version, which integrates several other tools and adds a rule routine. This version generates TCP and UDP packets and runs in several phases. Version 2 is being worked on currently. This version has a more integrated packet generation and observation facility, generates a wider variety of packets, and is designed to perform correlations in real time rather than in a correlation phase. Both versions are evaluated herein. The MITRE Firewall Test Tool (MFTT) operates on a system with probes on either side of the firewall under test (see Figure 1) and monitors traffic at both points. It also injects traffic into the system under test at the “Outside net” port and receive responses at that location. Traffic is directed through the firewall by this tool. The MITRE Firewall Test Tool runs on Solaris. The MITRE Firewall Test Tool (MFTT) is designed to probe a firewall and determine whether its actions agree with its stated rules. It is not designed to probe a firewall as a computer system, but to determine whether it does what it claims as a filter. The MFTT contains a rule-checking routine which decides whether a particular packet should have been passed through the firewall based on rules fed into it. When a packet is not passed which the tool’s rule routine claims should have been passed, or when a packet is passed through the firewall under test which the firewall test tool’s rule routine claims should have been blocked, the MFTT reports a discrepancy. This report allows one to determine what the firewall is actually enforcing. Since firewalls of filtering type can base decisions on protocols, source addresses, or destination addresses (or more complex information), it is infeasible to generate all possible IP packets and see what is passed. Rather, the tool contains a program able to generate tests for addresses mentioned in the rules, and is being expanded to generate tests for boundaries of rules. For example, if packets from addresses A through B using protocol P are to be blocked, the Version 2 MFTT will generate test packets from A-1 to A+1, and B-1 to B+1, and possibly a small number inside the range, using protocol P to test the firewall. The rule set must be input as well, so that the tool’s rule-checking routine can distinguish correct from incorrect operation. The rule set should reflect a site’s security policy, though no formal comparison to a site policy is implemented in the MITRE tool. The MITRE tester is however unique in having the ability to select tests automatically to check a set of firewall rules. 2.2 Rule Sets/Firewall Security Policy The MITRE Firewall Test Tool accepts a set of rules which correspond generally to rules which would exist as inputs to a commercial firewall. The purpose of a firewall is to enforce a specific firewall security policy. The firewall security policy is represented as a set of rules to the firewall. The MFTT also has the capacity to allow a user to represent a firewall security policy as a set of rules which are input to it. The MFTT does not however attempt to directly read any firewall vendor rule sets. Rather, they must be input to the MFTT in a MFTT- specific format. 2.3 Packet Generation 2.3.1 MFTT Version 1 MFTT V1 generates TCP and UDP packets to scan IP services. The packet generator records packets generated and responses to them. (Because passive observation is available, generated packets, normal network traffic packets, or both can be used. Thus MFTT V1 can be operated without internally generating any packets.) Packets are generated with source addresses in a set specified by the user, destination addresses in a set specified by the user, and IP port numbers (corresponding to net services) generated in a range. The range generated is obtained from the analysis of rules input to the program or from user specifications, so that scans of TCP and UDP port ranges are generally done. Packets can also be generated for certain predefined protocols (FTP (file transfer), HTTP (worldwide web), NNTP (network news), telnet (remote terminal), SMTP (electronic mail or email) and a few more) to generate connections with certain protocols and observe a more extended set of traffic. These predefined packets test for certain known weaknesses in their protocols, and give the tool a chance to see the effects of the traffic on stateful packet-inspecting firewalls. V1 does not generate or check for packets with source routine or source port numbers set nonstandardly. 2.3.2 MFTT Version 2 The Firewall Test Tool (V2) contains (in addition to the abilities in V1) its own packet generator which can generate packets with any desired contents to be used for probes. The packets generated are chosen by analysis of the set of firewall rules that the firewall under test claims to be using and generates test cases to see whether the boundary conditions are satisfied. (The chosen traffic can be added to by the user as well.) Also, a number of predefined traffic scripts which use UDP-based services are being added to the tool. The packet generator, however, is generally used in its mode of providing traffic for a large number of known protocols. Packets to and from selectable addresses, with selectable port numbers (which correspond to protocol numbers) can be generated. Addresses produced by the rule-based generator test addresses mentioned in the rules, as well as cover boundary cases of protocol numbers. The test case generation cannot handle wildcard addresses currently. Because the wildcarding does not carry information about what addresses exist either within the firewall, or on the tester machine outside, the intent is to avoid conducting any unauthorized attacks on machines at sites where the firewall was tested. Therefore, one must manually specify addresses which are within the wildcard ranges and acceptable to use for testing to complete the setup of the packet generation function. (Nevertheless, plans are to add the ability to generate boundary tests for “wildcard” ranges of addresses or protocols.) The V2 test tool will be modified to generate and use source-routed packets (which must go through particular IP addresses, and are thus useful in various breakin techniques) and to generate packets with unusual source port numbers (used in some cases to spoof protocols). 2.4 Packet Observation 2.4.1 Version 1 MFTT V1 observes traffic actively, passively, or both. Its passive observation can be done at one side of the firewall only (relying on reply packets from the other side of the firewall to signal what traffic passed through the firewall) or at both sides. If desired, V1 can observe existing traffic, yet still flag apparent violation of firewall filtering rules. V1 also can use active observation in which it receives packets it has generated. The more extensive tests built into the V1 generation routines for NNTP, SMTP, etc., receive their replies directly, rather than via passive probes. Typically, MFTT V1 is configured to look at a firewall from the outside in. The ability of V1 to observe packets on both sides of the firewall means that it can conduct more precise tests than could any single observation point tester tool. If for example a packet for port 302 is sent to the firewall and nothing inside is listening for that packet, it may be that no reply will be seen at the outside of the firewall. This is indistinguishable from the situation where the firewall blocks packets to port 302 until a probe at the inside of the firewall is added. Then it can be seen that the packet is getting through the firewall, but it is not being responded to from inside. This capability is present in only a few other tools. V1 records traffic into files and allows comparison at any time afterwards. 2.4.2 MFTT Version 2 V2 will retain all the capabilities of Version 1, but it will add the ability to read packets from both sides of a firewall as they are passed and to determine during the traffic flow whether anything undesirable is happening. It will also be able to keep better track of what packets are and are not being replied to, to facilitate more extensive probes of firewalls with both TCP and UDP packets. (Considering that firewalls are evolving toward more stateful packet inspection and filtering this added capacity is necessary. Some protocols –notably HTTP - are becoming more complex, and firewalls are having to track them, which is why a test tool must also be able to generate some of this complexity.) 2.5 Packet Correlation With Firewall Security Policy MFTT reports where anomalies are found in the traffic passed through the firewall. These anomalies can mean either that the firewall is not functioning as specified, or that the specifications of the firewall rules have not been correctly entered. If no anomalies are found, then the firewall’s actual rules agree with those found via the tests. The anomalies are not currently summarized by the MFTT; instead, the information indicates what traffic did or did not pass through the firewall by revealing the packet information. Thus, for example, if DNS (Domain Name Service) traffic is able to reach within the firewall, the display will show many DNS packets passing through the firewall as anomalies, indicating that they appear to violate this or that rule. Users are told which rule or rules were violated by each anomaly. MFTT V2 will attempt to generalize these cases. The object is to give a summary of what is not occurring as expected, but details have yet to be finalized. 2.6 Other Features The MMFTT offers a graphical user interface which makes it easier to use than command line-driven tools. It does not attempt generic tests of a firewall save for those tests mentioned above, such as checking that the security policy (as embodied in the rules given to the tool) is being enforced accurately. MFTT can check previously collected data against a rule set also, so that discrepancies between rules input to it and rules under which the firewall was running can be investigated without rerunning data collection. Also, the MITRE tool is able to perform its tests based on purely passive sniffing (observation of traffic using “receive- everything” operation of network hardware)., Therefore, checks that a firewall is operating correctly can be made without introducing network traffic which can cause alarms or legal problems. Section 3 ISS Firewall Scanner 3.1 Tool Description The ISS Internet Scanner system (of which the ISS Firewall Scanner is a part) started out as a system scanner, and is still available in a free form able to test systems for a set of known vulnerabilities. A commercial version (discussed here) is now available from Internet Security Systems, Inc. (at www.iss.net on the Internet) which adds more tests, including some which apply to firewalls. The ISS Internet Scanner system runs on Unix systems. It deals with vulnerabilities visible from the outside of the systems such as problems with weak authentication protocols, but would not, for example, test for improper group permissions within a host. The ISS Firewall Scanner is licensed to be able to communicate with a finite number of IP addresses from some single machine. ISS Internet Scanner is heavily focussed on finding and pointing out system vulnerabilities, which it can do for Unix, VMS, or Windows NT. The ISS Firewall Scanner is connected typically outside of firewall and tests the firewall and systems behind it for their security properties. It observes responses at its attachment point (and only there). Figure 2 shows the ISS Firewall Scanner architecture. Figure 2. ISS Firewall Scanner Architecture 3.2 Rule Sets/Firewall Security Policy The ISS Firewall Scanner contains nothing to read or use any sets of firewall rules. No provision is documented for input of any such rules nor for output of any rule based exceptions. 3.3 Packet Generation The ISS Firewall Scanner generates packets that contain the source address that of the machine on which it operates and the destination address of one or more of the licensed IP addresses. It generates probes for known problems in a variety of applications, and can send packets with illegal source routing or source port numbers through a firewall. When the ISS Firewall Scanner is testing a firewall-protected configuration, it expects to be on a machine outside the firewall sending traffic in through the firewall. It relies on IP replies or rejects (and timeouts) sent from within for the necessary replies to this traffic. Packet generation depends on the user list of ports to be probed, plus the built-in vulnerability tests in the product The ISS Firewall Scanner generates probes of network services, with or without making full TCP connections, and it sets source routing and source ports (IP options) to attempt to bypass firewall restrictions. (Setting source port numbers is a way to attempt to fool a firewall about what protocol is being passed. Source routing tells the network to route a packet through a particular node, but it may also be used to allow a spurious node to capture replies, or even fool a firewall into determining that a packet was sent from an internal source.) In addition to scanning services (by scanning IP ports), ISS is able to perform more extensive tests of FTP, SMTP (email) , and HTTP protocols to test for problems, and it can perform some brute force tests against systems checking login names and backwards names, for example. The ISS Firewall Scanner also has facilities to exercise protocols through firewall types using applications proxies, including those running the SOCKS suite of programs. 3.4 Packet Observation The ISS Internet Scanner generally (and thus its ISS Firewall Scanner component) relies on IP replies, rejects, and timeouts, since it has no provision for passive packet observation (sniffing) inside a firewall. It uses various tricks for probing a system (including partial TCP setups) to find active ports, within the limits of what one machine can do. Thus it uses low level IP information for discovering packets. It can indicate when different services can be seen, and it can list vulnerabilities of given hosts. The ISS Internet Scanner can use the trick of partially setting up a TCP connection to see whether an internal system will respond. Such partial connections may not be logged in some firewall audit trails, permitting probes for vulnerable services to be done as a “stealth scan”. Even this “stealth scan” requires some kind of reply, or a timeout, from the inside of a firewall. 3.5 Packet Correlation With Firewall Security Policy To test ISS Firewall Scanner packet correlation with the rules of firewall security policy, the user must arrange for the ISS Firewall Scanner to generate traffic which might violate a firewall’s rules. The user must then examine the ISS Firewall Scanner reports to deduce manually whether such rules are being enforced. A few tests (the stealth scan and IP spoofing tests, for example) which would be reported as vulnerabilities may violate firewall rules directly, but these are reported as host vulnerabilities. 3.6 Other Features ISS has a very large suite of known vulnerabilities and probes for them…probably the largest of the tools evaluated. It can probe a firewall as another system with great thoroughness, and also probe systems behind a firewall for a rather large set of possible security holes. A firewall built on a general purposewould benefit from this type of probing, since someone breaking into a firewall might be able to disable its filtering capabilities completely, even though the filtering engine were working perfectly. This is however a somewhat different problem from verifying that a firewall is following its rules accurately. ISS does not claim to test this. It does generate port scans, proxy tests, tests for loose source routes, and many other tests of other areas of a firewall’s function. Section 4 Chisel 4.1 Tool Description Chisel is a tool from Network Tools, Inc. (networktools.com) whose primary use is to generate heavy network loads in order to validate that network servers can handle them properly. It can measure response times. Nevertheless, Chisel’s load generator can generate traffic through a firewall that appears to be sent from a variety of addresses and to generate stress tests, particularly of a few known and common protocols (SMTP (email), FTP, or HTTP (web) traffic). It must be added that as of this writing, Chisel is not yet on the market, and its evaluation has been done using literature and a demonstration movie furnished by the vendor. Chisel will run on Windows NT. Chisel is attached to a network at one point, typically just outside a firewall, and generates traffic to test various aspects of the system from its one attachment point. Figure 3 shows the Chisel architecture. Figure 3. Chisel Architecture 4.2 Rule Sets/Firewall Security Policy Chisel has no provision for reading or using any firewall rule sets or security policies. 4.3 Packet Generation Chisel can generate large amounts of traffic from the machine on which it runs, making packets appear to be sent from a variety of places and permitting tests of several application ports at a time. By generating simulated traffic as though it were coming from actual (multiple) users, Chisel can exercise many of the high level features of HTTP, email, and FTP. In this way, Chisel can test traffic at all OSI (Open System Interconnect) layers, though only for these protocols. Chisel can be used to test multiple machines (apparently one at a time), but test generation to test firewall rules must be manual. No automated way to select traffic patterns based on firewall rules is included. 4.4 Packet Observation Chisel runs on a single machine, with only one network connection, relying only on replies, rejects, or timeouts for its return information. Chisel has apparently no provision for trying any spoofing attacks such as using IP source routing, different source and destination port numbers, or the like, and can only create packet formats needed for its traffic tests. A variety of reports on what traffic was received exist, so that systems or services not responding or responding incorrectly can be identified. 4.5 Packet Correlation With Firewall Security Policy Users must manually correlate packet information with firewall policy. Chisel offers no capability to perform this function. Rather, Chisel’s focus is to ensure that a firewall does not disrupt the services it knows how to test (email, HTTP, and FTP) under heavy work load, allowing the user to ensure that services are blocked where they should be blocked. 4.6 Other Features Chisel is unique in that it alone, of the tools examined, generates heavy mixed-service loads to test network functions. This type of testing might find problems other tools would miss, such as when a firewall stops filtering all of some type of traffic when it is overloaded and just lets that traffic pass. It is limited in what services it tests, though, and it does not scan for host security holes, nor does it validate firewall rules in any formal way. Section 5 Ballista 5.1 Tool Description Ballista is a combined system and network scanner from Secure Networks, Inc. (secnet.com) which can generate almost any packet contents to be sent through a firewall for testing purposes. It can monitor traffic which is allowed to pass through the firewall, as well as run tests for a large number of system security vulnerabilities. Ballista is a very complete tool for generating and monitoring many types of packets, with sufficient options to allow almost any desired tests to be conducted. It is the closest in features to the MITRE Firewall Test Tool of any tool examined here. Ballista is attached to a network in one or two places. For firewall testing it generally is connected both outside and inside the firewall and injects and observes traffic to test the firewall system and systems inside the firewall. Traffic of almost any type can be generated via its programmable generator. Ballista runs on Unix or Linux systems; its architecture is shown in Figure 4.. Figure 4. Ballista Architecture 5.2 Rule Sets/Firewall Security Policy Ballista has no provision for reading or using any sets of rules for firewall control. 5.3 Packet Generation Ballista’s packet generator interprets a packet-generating language called Cape, which provides control over basically all IP packet fields. Thus, it can generate packets with any desired addresses (source or destination), protocol types, or other contents, and it comes with a number of template scripts for this purpose. The scripts perform functions such as “test whether IP encapsulation can traverse the firewall”. One must select these scripts to test for specific packet filtering capabilities. The claimed intent of Cape is to “perform complex protocol level spoofing and attack simulations with relative ease” and with little programming knowledge. Ballista also can generate application type tests (FTP, mail, RPC (Remote Procedure Call), NFS (Network File System), telnet, rlogin (remote login), rexec (remote execute), popd (post office protocol daemon), and HTTP protocols), in which longer sequences of packets are generated and results monitored from the outside of a firewall. These tests are useful in that they closely approximate real traffic and thus provide a view of more realistic behavior of these protocols as they pass through a firewall. Ballista also can attempt password guessing attacks on these protocols. 5.4 Packet Observation Ballista can observe the packets generated by Cape and its other built-in tests via a snooping interface which promiscuously reads (sniffs) the network on the inside of the firewall, so it can obtain clear indications of what has or has not passed. Also, application tests have normal replies which Ballista can parse to see whether the firewall is functioning correctly. Ballista can produce a variety of reports documenting test results, but again Ballista has no automatic correlation with what a firewall rule set might predict. It is able to correlate packets it observes with what it generated. 5.5 Packet Correlation With Firewall Security Policy Ballista has no features to relate packets to the firewall security policy. This correlation must be done manually. Ballista has no provision for adding firewall rules, so the presumption is that the user can run tests knowing what a firewall should allow to pass. While the tool can exercise some common protocols well, it must be driven by someone sufficiently expert to understand the functions of a firewall. Such a person must design what packets will be generated via Cape scripts and analyze the results manually, comparing traffic allowed to pass with the firewall rules. The effort for a skilled person might be minimal, as claimed, but the understanding of firewall operation and protocols must be assured. 5.6 Other Features Ballista is primarily a network and system test tool, not a firewall test tool. While it can be used to test firewalls, this use requires some expertise. As a network tester, Ballista most closely approximates the capabilities of the MITRE Firewall Test Tool does, and it has considerable ability to test system and network holes. Were it able to relate its test cases to firewall rules, it would be a complete superset of the MFTT. Section 6 Free Utilities 6.1 Tool Description A number of free utilities overlap in the firewall testing area when used in combination. These tools are netcat (a program which generates almost arbitrary IP packets), SATAN (which generates many variants of IP packets and protocols in scanning a system and which tests for many system security vulnerabilities), Argus (a packet logger which captures summary information of traffic and allows queries about network use based on these logs), and tcpdump (a general purpose promiscuous mode packet dumper which can dump packets from an Ethernet port and display them). There are many others. The tools discussed here all operate in Unix or Linux environments. Others able to run on other operating systems exist also. Diferent free utilities are written to be connected to firewall systems both inside and outside firewalls. Many have rather sophisticated suites of system tests. Figure 5 shows a general layout for the use of free utilities. These tests must be correlated by hand between tools to achieve wide coverage of possible system issues. Figure 5. Free Tools Analysis / Testing Layout 6.2 Rule Sets/Firewall Security Policy None of the tools discussed here has provision for checking rules which express security policy, nor for flagging traffic which may violate any such policy. 6.3 Packet Generation The netcat program can generate most IP packets needed for test purposes, including packets that spoof addresses or ports. It is command-line driven and usable by shell scripts, but requires some skill to use. However, it significantly reduces the skill level needed to mount spoofing or fragmentation attacks. Netcat is able to listen for TCP responses and a large class of UDP responses as well. SATAN uses a number of protocols in its operation. SATAN was designed as a network scanning program able to find system vulnerabilities for various kinds of network attacks, and as such it can exercise some of the higher level protocols like mail and FTP and look for replies to what it generates. Using the two tools together, a skilled individual can generate a rather comprehensive set of attacks, though again none of these tools offer anything to assist in deciding what attacks to generate to test a firewall. 6.4 Packet Observation SATAN can observe what it generates (the SATAN engine runs from outside the firewall only, so it must rely on the machines inside to reply to or reject traffic). For other packets, the tcpdump program can observe traffic on an ethernet and write it to a file so that it can be compared later. This sniffing ability is therefore not suited for real-time comparisons (as the MFTT is being enhanced to do) but can be made to display what is seen. Filtering tcpdump’s output into a more manageable amount of data requires some post filtering (perhaps via a Unix pipe filter). Other tools (e.g. Argus) can capture a smaller, filtered subset of data, but they lose parts of packets which can be important for some protocols. Thus, filtering tools exist, but require considerable skill to use. Netcat can also observe (and log to a journal file if desired) many responses to TCP or UDP packets it sends. (The UDP case presumes the same port is used in the response.) Argus also observes raw packets on a network interface and records a small amount of information about them. Query programs which are part of the tool can then access the modest amount of data collected and yet answer questions later about what protocols and connections were made to/from where. This is useful where it is necessary to see whether an attack was made sometime in the past and where it came from. Such capability coupled with the ability to generate attacks makes it simpler to measure which attacks were passed by a firewall. 6.5 Packet Correlation with Firewall Security Policy Correlating packets with a firewall security policy is entirely manual using free tools. Apart from the alerts SATAN can produce and the ability of netcat to look at TCP or UDP packets that it generates, these tools will not indicate anything about the security policy or implications of the traffic they can produce. 6.6 Other Features The major advantage of these free tools is that they are free. A person with sufficient skills and with sufficient time can select tools and can test firewall actions or system security, but can expect to spend significant time doing so. Also, these tools use mostly command-line interfaces, requiring greater skill in operation than commercial tools. Section 7 Summary When a firewall is being used to provide basic system isolation, an enterprise must be able to ensure that the firewall works correctly. Where such testing is carried out periodically (as good practice would suggest), it should be easy to do and largely automatic. For testing firewalls against their claimed rules, the MFTT is the only one found which relates its tests to the rules of the firewall. It will show that the rules are being enforced within the limits of what a test tool can do. None of the other tools examined relate tests to rules, and thus they require significantly more skill and time to discover improper settings. Test scope will always have limitations. If a firewall contains a single back door source address, for example, this would be difficult to identify in any test suite, by any tool, save by chance. (In sufficiently sensitive sites this could be a good argument for using two firewalls in series, perhaps one of them a public domain firewall whose source code is available.) Still, a test suite automatically designed to test all the rules of a firewall will generally catch more holes than a manual procedure. The commercial tools examined offer additional tests of systems and firewalls for system security and for functions operating under heavy loads which are also valuable, but do not overlap those of the MITRE tool. System security tests of firewalls as separate computing systems are worth conducting, to ensure that the firewall function cannot be bypassed, but they are not doing the same function as testing the firewall’s rules themselves. Both tests are needed to show that a firewall can operate properly. TextHere Section 8 Recommendations and Futures Where a firewall is present, it needs to be as trustworthy as the most trustworthy system on a network. This implies it should be tested periodically to ensure it is doing what it should. The kind of testing to be done depends on the skill, time, and money mix available to the testing organization. An organization with little money but someone available who can run some of the freely available tools to check the firewall as a system and to run some tests of traffic passing should certainly use those tools. If it is possible to use commercial tools, however, such tools as ISS or Ballista should certainly be run to check the firewall as a system and to check for firewall reaction to source routed packets and the like. These two tools cover similar sets of holes, so that the most sensible action is to use one or the other. Firewall rule sets are however not always easy to set up properly, so tests with the MITRE firewall test tool should be done as well, to ensure that the rules given to a firewall are valid. This should be done even where the rules given to the vendor firewall appear clear and unambiguous; experience has shown that the MITRE tool often finds surprises even with very simple rule sets. Finally, if it is possible to test your firewall under heavy load, this should be done also. There are cases where firewall code may behave differently under load, perhaps by skipping tests which might take too long. Chisel could be useful here, since it is designed to generate heavy loads on demand, and can report anomalous behavior for several protocols. Future firewall test tools should combine the aspects of testing the firewall rule conformance, firewall system security tests, and load tests to make their use simpler and less costly. It is however becoming clear that firewalls can cover only a small part of what is needed to secure a network of systems. The introduction of ever more complex protocols and of wider variety in responses to packets is making it clear that the notion of blocking undesired access at the network border is not feasible without controls within. Some particularly dangerous attacks and protocols can be efficiently blocked, so firewalls will continue to be needed components in providing security. Nevertheless, systems inside the firewalls will increasingly need to be involved with protection of internal assets also. Significant opportunity exists in the area of finding ways for systems to cooperate with firewalls in identifying and tracking activities counter to site policies iii CONFIDENTIAL Section Page iv iii 1 ii 9 13 16 18 21 23 25 27