Texas A&M Network Security Dave Safford Doug Schales Dave Hess ABSTRACT: Texas A&M University UNIX computers recently came under extensive attack from a coordinated group of internet crackers. This paper presents an overview of the problem and our responses, which included the development of policies, procedures, and tools to protect university computers. The tools developed include "drawbridge", an advanced internet filter bridge, "tiger scripts", extremely powerful but easy to use programs for securing individual hosts, and "xvefc", (XView Etherfind Client), a powerful distributed network monitor. In this paper we will NOT discuss the technical aspects of the intruders methods of attack, as examples tend to promote additional attacks. Instead, we have developed a machine checking program which helps managers detect and correct all the weaknesses we have seen exploited. A BRIEF HISTORY OF THE INCIDENTS: On Tuesday 25 August, we were notified by Ohio State that a specific TAMU machine was being used to attack one of their computers over internet. The local machine turned out to be a Sun workstation in a faculty member's office. Unfortunately, this faculty member was out of town for a week, so rather than trying to convince the department head to let us in without the faculty present, we decided to monitor network connections to the workstation, and if necessary, disconnect the machine from the net electronically. This decision to monitor the machine's sessions rather than immediately securing it turned out to be very fortunate, as this monitoring gave us a wealth of information about the intruders and their methods. Our initial monitoring tools were very simple, but as the significance of what we were seeing became apparent, we rapidly improved the tools to the point that we were able to watch the intruder's entire session in real time, keystroke by keystroke. This monitoring led to the discovery that several outside intruders were involved, and that many other local machines had been compromised. One local machine had even been set up as a cracker bulletin board machine, that the crackers would use to contact each other and discuss techniques and progress! By Thursday 27 August we felt we had enough information about which machines had been compromised, and how they had been broken into, that we could effectively clean them up. In addition, the severity of the modifications the intruders were making, particularly on the bulletin board machine, made it imperative to stop the intrusions, so we contacted the respective system managers, and arranged to shut down all machines, and scheduled the system cleanup for the next day. On Friday 28 August, we worked on the known affected machines, closing the security holes that had been used to break in, and brought them all back up on the network. On Saturday 29 August, we received an emergency call from one of the system managers, saying that the intruders had broken back into the cracker bulletin board machine. Concerned about the integrity of their research data, they asked for their machines to be physically disconnected from the rest of the network. On Monday 31 August, we analyzed the logs of the new break-in, and determined that 1) the crackers were much more sophisticated than we had been led to believe, and 2) many more local machines and user accounts had been compromised than we initially knew. We even found a file containing hundreds of captured passwords. It appeared that there were actually two levels of crackers: the high level was the more sophisticated, and had a thorough knowledge of the technology; the low level were the "foot soldiers" that merely used the supplied cracking programs with little understanding of how they worked. Our initial response had been based on watching the later, less capable crackers, and was insufficient to handle the more sophisticated ones. After much deliberation, we decided that the only way to protect the computers on campus was to block certain key incoming network protocols, reenabling them to local machines on a case by case basis, as each machine had been cleaned up and secured. The problem was that if the crackers had access to even one unsecure local machine, it could be used as a base for further attacks, so we had to assume all machines had been compromised, unless proven otherwise. The recommendation to filter incoming traffic was presented to the Associate Provost for Computing on Monday afternoon, and approved. We bought or borrowed the necessary equipment for the filter and monitor machines late that afternoon, and worked all night on the design and coding of the filter. Particular effort was made in the design to achieve the necessary security with the minimum of impact to local users. The filter was completed and installed by 5PM Tuesday 1 September. On Thursday 3 September, our monitor logs showed an obviously automated attack by ftp which was sequentially probing every machine on campus. Here again we decided to monitor this attack, as we were unsure what it could accomplish. This decision to observe, rather than immediately block, turned out to be very fortunate. Shortly after midnight on friday we were notified by CERT that another site was monitoring their machines, and had noticed that the crackers had broken back into our machines. We immediately went in and analyzed our logs, and determined that the crackers had used ftp to install a program that allowed them to tunnel past our filter's blocks. In addition, the crackers were now installing some very sophisticated trap doors and trojan programs throughout a large number of machines, apparently to ensure that they were never locked out again. These system modifications were particularly nasty in that they went to great pains to conceal them, including patching the modified binaries to match the original timestamps and checksums. At this point we completely redesigned the filter to keep the crackers out, and installed the new version by 5AM Saturday. The new version, while providing much greater security, also was unfortunately more visible to valid users. The good news was that the new filter interrupted the new sophisticated crackers while they were working (apparently they did not anticipate so rapid a response), and we discovered one machine still had over 40 megabytes of the cracker's tools. This captured information provided us the most detailed information yet as to the cracker's methods. Since the new filter was installed, we have observed no successful intrusion attacks against our machines, despite continued logging of probes and continued attempts. (We did log one incident of a local student trying to attack an off campus machine, but that is another topic.) Our efforts since then have centered in three areas: improving the filter (largely for ease of use, and throughput), improving the monitoring tools (to reduce manpower requirements), and developing a program to help local system managers check their machines for proper security configuration. RESPONSE OVERVIEW: Our response to the intrusion incidents has three major thrusts: monitoring, filtering, and cleaning. The purpose of monitoring is to record network events so that intrusion attempts can be recognized and tracked. Our intitial monitor tools have evolved into a distributed client/server design called "efd" (etherfind daemon). Filtering is used both to protect unsecure machines from attack, and to restrict the modes of attack to the designated "bastion" hosts. Our current filter package is called "drawbridge". The goals of cleaning are both to check workstations for signs of intrusion, as well as to prepare candidate bastion machines for secure operation. This is done with our package "tiger scripts". These three packages are described more fully later, but first we need to describe the implementation architecture. The following diagram shows an overview of the filter and monitor implementation. In traditional secure gateways, a filter and secure bastion host are used, and all traffic to or from internet is forced through them. This typically means that users need proxy clients for external access, such as for telnet and ftp, so that they all do not have to log on to the bastion host for external access. In our case, the filter allows arbitrary protocol filtering on a host by host basis, so that each department can set up its own bastion host, with its own service configuration, (subject to the campus wide minimum standards.) This provides a reasonable level of both security and flexibility for educational and research requirements. For a host to be enabled at all beyond the default incoming permissions for mail, it must pass the very thorough "tiger scripts", as described later. The monitor node is placed outside the filter so that it can record connection attempts which are blocked by the filter. This placement has been crucial to recognizing intrusion attacks, but does place the monitor itself at risk. To minimize this risk, we placed both the filter and monitor in a controlled access machine room, and configured the monitor to accept no other network requests other than validated efc requests. The filter is similarly programmed only to respond to secure filter update requests. \ \/\ T1 to internet \ --------- |WAN | |Router | |(cisco)| --------- | ----------- | |Secure | ethernet |-----|Monitor | "efd" | |(Sun4/60)| | ----------- | --------- |Filter | |Bridge | "drawbridge" |(486PC)| --------- | | ethernet to campus \/ ---------------------------------------------- | | --------- --------- | | secured | | | | ... ... .... | | (checked with "tiger scripts") | | hosts | | --------- --------- MONITOR: (efd) The goal of monitoring is to record security related network events, by which intrusion attempts can be detected and tracked. This is a very difficult problem in general. The communication data rates make this problem somewhat like trying to take a sip of water from a fire hose; our campus has some 30 terabytes of internal data transfer per day, and our internet connection is on the order of ten gigabytes per day, with an average of 50,000 individual connections during that period. Clearly we have to be both very selective and flexible in what we monitor, and we need automated tools for reviewing even these resultant logs. Another problem is that of monitor placement. It is important that monitors be placed so that critical segments can be observed, and so that the monitors themselves are secure. Our solution has been the development of a securely distributed version of etherfind. Etherfind is very powerful and flexible in its filtering. By distributing this in a secure manner, we are able to monitor a large network from a convenient, secure central location. efd has two front end clients: a command line version suitable for routine monitoring (efc - etherfind client), and an xview based version, xvefc, for rapid, flexible ad hoc queries. In use, we normally monitor only connection establishment messages to or from campus. This results in roughly 50,000 lines per day, which are both highly informative for detecting unusual events, and relatively easy to analyze. Since starting this logging, we have detected one successful obvious intrusion, one misconfigured mailer, one misconfigured router, and numerous probable blocked intrusion attempts. If a suspicious pattern of connection attempts is seen, efc can easily be used to record all packets of interest for more detailed analysis. FILTER: (drawbridge) Our first line of defense is the filter bridge running our "drawbridge" program. We initially looked at using the filtering built into our WAN routers (cisco), but determined that our requirements, particularly in the need for supporting potentially different filtering to each of our roughly 4,500 machines in our class B network, were too complex for the router. In addition we needed something that could handle full ethernet bandwidth, was itself very secure, and which could be implemented VERY rapidly. D. Brent Chapman presented an interesting analysis of the limitations of current filter implementations in his "Network (In)Security THrough IP Packet Filtering", Proceedings of the Third UNIX Security Symposium, September 1992. Our "drawbridge" program, along with its support filter specification language and compiler, address his critical recommendations with respect to both functionality and ease of specification. We decided on a PC with two SMC 8013 (AKA Western Digital) ethernet cards, and based our first software implementation on pcbridge, by Vance Morrison. This initial version was soon rewritten from scratch in turbo C++, to make the addition of needed features somewhat easier. The current filter design provides "allow" based filtering per host with separate incoming and outgoing permissions. (Allow based filtering prohibits all connections except those that are explicitly allowed.) For both performance and configuration management, the filter tables are created on a support workstation, based on a powerful filter configuration language, and then securely transferred to the filter machine, either at boot time, or dynamically during operation. The support machine does all the hard work of parsing the configuration file, looking up addresses, and building the tables, so that the filter itself need only perform simple O(1) table lookups at run time. Updating the tables dynamically is made secure with a DES challenge/response and authentication. Our current default configuration allows any outgoing connection, but allows in only smtp (mail). Several campus and departmental servers have been set up as bastion hosts which are able to receive incoming telnet, ftp, nntp, and gopher requests. MACHINE CLEANUP: (tiger scripts) Our third major thrust has been the development of an automated tool for checking a given machine for signs of intrusion, and for network security related configuration items. The resultant tool is actually a set of bourne shell scripts (for high portability). The major goal of the tiger scripts is to provide a simple way to check a machine prior to allowing it bastion host status, and greater internet connectivity. For ease of use, the tiger scripts label all outputs with an error classification: --FAIL-- The problem that was found was extremely serious. --WARN-- The problem that was found maybe serious, but will require human inspection. --INFO-- A possible problem was found, or a change in configuration is being suggested. --ERROR-- A test was not able to be performed for some reason. The checking performed covers a wide range of items, including items identified in CERT announcements, and items observed in the recent intrusions. The scripts use Xerox's cryptographic checksum programs to check for both modified system binaries (possible trap doors/ trojans), as well as for the presence of required security related patches. Currently the tiger scripts have been configured only for SunOS 4.1.x releases. The programs are largely table driven for ease of porting, and we are working on ports to Solaris 2.0, SVR4, Ultrix, AIX, Next, etc. POLICIES: The policies and procedures need to provide both security and flexibility. Our resultant decision was to filter incoming traffic other than mail to all machines, and then allow case by case requests for bastion hosts status, based on successful demonstration of basic security configuration with the tiger scripts. In special cases, we have had requests for allowing incoming requests to special servers which are not easily checked, such as for embedded robot controllers. In these cases, we have allowed the connections, but implemented special monitors on these services. Long term policy questions which remain unanswered include how to handle updates in response to critical CERT announcements, and how to handle OS updates? Obviously we need some way to coordinate both periodic and quick response bastion host reviews. Our filter configuration language does support machine classes, so we could do something such as "disable ftp to all SunOS 4.1.1 machines" in response to a CERT announcement of a respective problem, but it would be nice to have a mechanism to communicate such announcements to the respective managers BEFORE cutting off access. The problem on a large campus is maintaining a contact list for a large number of machines, given the high rate of turnover in student managers. In addition, the information in our filter configuration file may rapidly become outdated, as managers update their machines hardware and software. Our current plan is to require annual security checks with the current tiger scripts, enforced with the warning of loss of bastion host status. In the case of aperiodic security events or announcements, we will attempt to evaluate the time criticality of responding, and require appropriate event specific checking. As the tiger scripts are so easy to run, we anticipate that this requirement will not be a significant burden to system managers. CONCLUSIONS: In response to a significant series of intrusions from internet, we have developed a set of policies and tools for monitoring, filtering and checking. Each of these three areas has proved critical; the monitoring because it has yielded significant information about the intruders and their methods, the filtering for its ability to protect machines from attack, and the checking tools for their ability to automate the task of checking and cleaning a large number of machines.