Honeypot Essentials By Anton Chuvakin

Honeypot Essentials Anton Chuvakin, Ph.D., GCIA, GCIH WRITTEN: 2003 DISCLAIMER: Security is a rapidly changing field of human endeavor. Threats we face literally change every day; moreover, many security professionals consider the rate of change to be accelerating. On top of that, to be able to stay in touch with such ever-changing reality, one has to evolve with the space as well. Thus, even though I hope that this document will be useful for to my readers, please keep in mind that is was possibly written years ago. Also, keep in mind that some of the URL might have gone 404, please Google around.

Summary: The paper covers honeypot (and honeynet) basics and definitions and then outlines important implementation and setup guidelines. It also describes some of the security lessons a company can derive from running a honeypot, based on the author experience running a research honeypot. The article also provides insights on techniques of the attackers and concludes with considerations useful for answering the question “Should your organization deploy a honeynet?” I. While known to security processionals for a long time, honeypots recently became a hot topic in information security. However, the amount of technical information available on their setup, configuration, and maintenance is still sparse as are qualified people able to run them. In addition, higher-level guidelines (such as need and business case determination) are similarly absent. This paper will cover some of the honeypot (and honeynet) basics and definitions and then will outline important implementation issues. It will also cover security lessons a company can derive from running a research honeypot. What is a honeypot? Lance Spitzner, a founder of Honeynet Project (http://www.honeynet.org) defines a honeypot as "a security resource who's value lies in being probed, attacked or compromised". The Project differentiates between research and production honeypots. The former are focused on gaining intelligence information about attackers and their technologies and methods, while the latter are aimed at decreasing the risk to a company IT resources and providing advance warning about the incoming attacks on the network infrastructure. Honeypots of any kind are hard to classify using the

"prevention - detection - response" metaphor, but it is hoped that after reading this paper their value will become clearer to readers. This paper will focus on operating a research honeypot or a "honeynet". The term "honeynet", used in this article, originated in the Honeynet Project and means a network of systems with fairly standard configurations connected to the Internet. The only difference of such network from a regular production network is that all communication is recorded and analyzed, and no attacks targeted at third parties can escape the network. Sometimes, the system software is slightly modified to help deal with encrypted communication, often used by attackers. The systems are never "weakened" for easier hacking, but are often deployed in default configurations with minimum of security patches. They might or might not have known security holes. Honeynet Project defines such honeypots as "high-interaction" honeypots, meaning that attackers interact with a deception system exactly as they would with a real victim machine. On the other hand, various honeypot and deception daemons are "low-interaction", since they only provide an illusion to an attacker, and the one that can hold their attention for a short time only. Such honeypots has value as an early attack indicator, but do not yield in-depth information about the attackers. Research honeypots are set up with no extra effort to lure attackers - blackhats locate and exploit systems on their own. It happens due to the widespread use of automatic hacking tools, such as fast multiple vulnerability scanners and automatic penetration scripts. For example, an attacker from our honeynet has attempted to scan 200,000 systems for a single FTP vulnerability in one night using such tools. Research honeypots are also unlikely to be used for prosecuting intruders, however researchers are known to track hacker activities using various covert techniques for a long time after the intruder broke into their honeypot. In addition, prosecution based on honeypot evidence has never been tested in the court of law. It is still wise to involve a company legal team before setting up such hacker study project. Overall, the honeypot is the best tool for looking into the malicious hacker activity. The reason for that is simple: all communication to and from the honeynet is malicious by definition. No data filtering, no false positives and no false negatives (the latter only if the data analysis is adequate) are obscuring the picture. Watching the honeypot provides insight on intruders' personalities and can be used to profile attackers. For example, in recent past the majority of penetrated Linux honeypots are hacked by Romanian attackers. What are some of the common sense prerequisites for running a honeynet? First, honeypot is a sophisticated security project, and it makes sense to take care of security basics first. If your firewall crashes or your intrusion detection system misses attacks, you are clearly not yet ready for a honeypot deployment. Running a honeypot also requires advanced knowledge in computer security. After running honeynet for my employer, a member of Honeynet Research Alliance, I can state that operating a honeynet presents an ultimate challenge a security professional can face. The reason is simple: no "lock it down

and maintain secure state" model is possible for such a deception network. It requires an in-depth expertise in many security technologies and beyond. Some of the technical requirements follow. Apparently, the honeypot systems should not be allowed to attack other systems or, at least, such ability should be minimized. This requirement often conflicts with a desire to create a more realistic environment for malicious hackers to "feel at home" so that they manifest a full spectrum of their behavior. Related to the above is a need for a proper separation of research honey network from a company production machines. In addition to protecting innocent third parties, similar measures should be utilized to prevent attacks against your own systems from your honeypot. Honeypot systems should also have reliable out-of-band management. The main reason for having this capability is to be able to quickly cut off the network access to and from the honeypot in case of emergency (and they do happen!) even if the main network connection is saturated by an attack. That sounds contradictory with the above statement about preventing outgoing attacks, but Murphy Law might play a trick or two and “human errors” can never be totally excluded. Honeynet Research Alliance (http://www.honeynet.org/alliance/) has guidelines on Data Control and Data Capture for the deployed honeynet. They distill the above ideas and guidelines into a well-written document "Honeynet Definitions, Requirements, and Standards" (http://www.honeynet.org/alliance/requirements.html) . The document establishes some "rules of the game", which has a direct influence on honeynet firewall rule sets and IDS policies. Data Control is a capability required to control the network traffic flow in and out of the honeynet in order to contain the blackhat actions within the defined policy. For example, rules such as 'no outgoing connections', 'limited number of outgoing connection per time unit', ‘only specific protocols and/or locations for outgoing connections’, 'limited bandwidth of outgoing connections', 'attack string filtering in outgoing connections' or their combination can be used on a honeynet. Data Control functionality should be multilayered, allow for manual and automatic intervention (such as remote disabling of the honeypot) and should make every effort to protect innocent third parties from becoming victims of attacks launched from the honeynet (and launched they will be!). Data Capture defines the information that should be captured on the honeypot system for future analysis, data retention policies and standardized data formats which facilitate information sharing between the honeynets and cross-honeynet data processing. Crosshoneypot correlation is an extremely promising area of future research, since it allows for a creation of an early warning system about new exploits and attacks. Data Capture also covers the proper separation of honeypots from production networks to protect the attack data from being contaminated by the regular network traffic. Another important aspect of Data Capture is timely documentation of attacks and other incidents occurring in the honeypot. It is crucial for research to have a well-written log of malicious activities and configuration changes performed on the honeypot system.

II. Lets turn to practical aspects of running a honeynet. Our example setup consists of three hosts (see diagram): a victim host, a firewall and an IDS. This is the simplest configuration to maintain, however, a workable honeynet can even be set up on a single machine if virtual environment (such as VMware or UML-Linux) is used. Combining IDS and firewall functionality by using a gateway IDS (such as “snort-inline”) allows one to reduce the requirement to just two machines. A gateway IDS is a host with two network cards that analyzes the traffic passing through it and can make packet-forwarding decisions (like a firewall) and send alerts based on network packet contents (like an IDS). Currently, the honeynet uses Linux on all systems, but various other UNIX flavors will be deployed as "victim" servers by the time this paper is published. Linux machines in default configurations are hacked often enough to provide a steady stream of data on blackhat activity. "Root"-level system penetration within hours of being deployed is not unheard of. UNIX also provides a safe choice for a victim system OS due to its higher transparency and ease of reproducing a given configuration. The honeypot is run on a separate network connection – always a good idea since the deception systems should not be seen as owned by your organization. Firewall (hardened Linux "iptables" stateful firewall) allows and logs all the inbound connections to the honeypot machines and limits the outgoing traffic depending upon the protocol (with full logging as well). It also blocks all IP spoofing attempts and fragmented packets, often used to conceal the source of a connection or launch a denial-of-service attack. Firewall also protects the analysis network from attacks originating from the honeypot. In fact, in the above setup, an attacker has to pierce two firewalls to get to the analysis network. IDS machine is also firewalled, hardened and runs no services accessible from the untrusted network. The part of the rule set relevant to protecting the analysis network is very simple: no connections are allowed from the untrusted LAN to an analysis network. IDS (Snort from www.snort.org) records all network traffic to a database and a binary traffic file via a stealth IP-less interface and also sends alerts on all known attacks detected by its wide signature base (approximately 1650 signatures as of July 2002). In addition, specially designed software is used to monitor the intruder's keystrokes and covertly send them to a monitoring station. All data capture and data control functionality is duplicated as per Honeynet Project requirements. 'Tcpdump' tool is used as the secondary data capture facility, bandwidthlimiting device serves as the second layer of data control and the stealth kernel-level key logger backs up the keystroke recording. Numerous automated monitoring tools, some custom-designed for the environment, are watching the honeypot network for alerts and suspicious traffic patterns. Data analysis is crucial for the honeypot environment. The evidence, in the form of system, firewall and IDS log files, IDS alerts, keystroke captures and full traffic captures, is generated in overwhelming amount. Events are correlated and suspicious ones are analyzed using the full packet dumps. It is highly recommended to synchronize the time via Network Time Protocol on all the honeypot servers for more reliable data correlation.

SIM software can be used to enable advanced data correlation and analysis, as well logging the compromises using the Incident Resolution Management system. Unlike in the production environment, having traffic data available in the honeypot is extremely helpful. It also allows for reliable recognition of new attacks. For example, a Solaris attack on "dtspcd" daemon (TCP port 6112) was first captured in one of the Project's honeypots and then reported to CERT. Several new attacks against Linux samba1 servers were also detected recently The above setup has gone through many system compromises, several massive outbound denial-of-service attacks (all blocked by the firewall!), major system vulnerability scanning, serving as an Internet Relay Chat server for Romanian hackers and other exciting stuff. It passed with flying colors through all the above "adventures" and can be recommended for deployment. III. What insights have we gained about the attacking side from running the honeynet? It is true that most of the attackers "caught" in such honeynets are "script kiddies" i.e. the less enlightened part of hacker community. While famous early honeypot stories (such as those described in Bill Cheswick's "An Evening with Berferd" and Cliff Stolls' "Cuckoo's Nest") dealt with advanced attackers, most of your honeypot experience will probably be related to "scipt kiddies". Opposite to common wisdom, companies do have something to fear from the script kiddies. The number of scans and attacks aimed by the attackers at Internet-facing networks ensures that any minor mistake in network security configuration will be discovered fairly soon. Every unsecured server, running a popular operating system (such as Solaris, Linux or Windows) will be taken over fairly soon. Default configurations and bugs in services (UNIX/Linux ssh, bind, ftpd and now even Apache web server and Windows IIS are primary examples) are the reason. We have captured and analyzed multiple attack tools using the above flaws. For example, fully automated scanner that looks for 25 common UNIX vulnerabilities, runs hundreds of attack threads simultaneously and deploys a rootkit upon the system compromise is one such tool. The software can be set to choose a random A class (16 million hosts) and first scan it for a particular network service. Then, on second pass the program collects FTP banners (such as "ftp.example.com FTP server (Version wu-2.6.1-16) ready.") for target selection. On third pass, the servers, that had misfortune of running a particular vulnerable version of the FTP daemon, are attacked, exploited and backdoored for convenience. The owner of such tool can return in the morning to pick up a list of IP addresses that he now "owns" (meaning, has privileged access to). In addition, malicious attackers are known to compile Internet-wide databases of available network services complete with their versions so that the hosts can be compromised quickly after the new software flaw is discovered. In fact, there is always a race between various groups to take over more systems. This advantage can come handy in case of a local denial-of-service war. While "our" attackers have not tried to draft the honeypot in their army of "zombie" bots, they did use it to launch old-fashioned point-to-point denial of 1

Samba is a Linux/UNIX implementation of a Microsoft Server Message Block (SMB) protocol

service attacks (such as UDP and ping floods and even the ancient modem hang-up ATH DoS). The attacker's behavior seemed to indicate that they are used to operate with no resistance. One attacker's first action was changing the 'root' password on the system clearly, an action that will be noticed the next time system admin tries to log in. Not a single attacker bothered to check for the presence of Tripwire integrity checking system, which is included by default in many Linux distributions. On the next Tripwire run, all the "hidden" files are easily discovered. One more attacker has created a directory for himself as "/his-hacker-handle", something that every system admin worth his/her salt will see at once. The rootkits (i.e. hacker toolkits to maintain access to a system that include backdoors, Trojans and common attack tools) now reach megabyte sizes, and feature graphical installation interfaces, suitable for novice blackhats. Research indicates that some of the script kiddies "own" networks consisting of hundreds of machines that can be used for DoS or other malicious purposes. The exposed UNIX system is most often scanned for ports 111 (RPC services), 139 (SMB), 443 (OpenSSL) and 21 (FTP). Recent (2001-2003) remote "root" bugs in those services account for this phenomenon. The system with vulnerable Apache with SSL is compromised within several days. Another benefit of running a honeypot is a better handle on the Internet noise. Clearly, security professionals who run Internet-exposed networks are well aware of the common Internet noise (such as CodeRed, SQL, MSRPC worms, warez site FTP scans, etc). Honeypot allows one to observe the minor oscillations of such noise. Sometimes, such changes are meaningful. In the recent case of MS SQL worm, we detected a sharp increase of TCP port 1433 access attempts, just before the news of the worm became public. The same spike was seen when the RPC worms were released. The number of hits was similar to a well-researched CodeRed growth pattern. Thus, we concluded that a new worm was out. Additional value of the honeypot is in its use as a security training platform. Using the honeypot the company can bring up the level of incident response skills of the security team. Honeypots incidents can be investigated and then the answers verified by honeypot's enhanced data collection capabilities. 'What tool was used to attack?' - here it is, on the captured hard drive or extracted from network traffic. 'What did they want?' look at their shell command history and know. One can quickly and effectively develop network and disk forensics skills, attacker tracking, log analysis, IDS tuning and many other critical security skills in the controlled but realistic environment of the honeypot. More advanced research uses of the honeypot include hacker profiling and tracking, statistical and anomaly analysis of incoming probes, capture of worms and analysis of malicious code development. By adding some valuable resources (such as e-commerce systems and billing databases) and using the covert intelligence techniques to lure attackers in, more sophisticated attackers can be attracted and studied. That will increase the operating risks.

IV.

Abuse of the compromised systems

The more recent OpenSSL incidents are more interesting since the attacker does not have root upon breaking into the system (such as, user "apache"). One might think that owning a system with no "root" access is useless, but we usually see active system use in these cases. Here are some of the things that such non-root attackers do on such compromised systems: 1."IRC till you drop" Installing an IRC bot or bouncer is a popular choice of such attackers. Several IRC channels dedicated entirely for communication of the servers compromised by a particular group were observed on several occasions. Running an IRC bot does not require additional privileges. 2."Local exploit bonanza" Throwing everything they have at the Holy Grail of root access seems common as well. Often, the attacker will try half a dozen different exploits trying to elevate his privileges from mere "apache" to "root". 3. "Evil daemon" A secure shell daemon can be launched by a non-root user on a high numbered port. This was observed in several cases. In some of these cases, the intruder accepted the fact that he will not have root. He then started to make his new home on the net more comfortable by adding a backdoor and some other tools in "hidden" (".. " and other non printable names are common) directories in /tmp or /var/tmp. 4. "Flood, flood, flood" While spoofed DoS is more stealthy and harder to trace, many of the classic DoS attacks do not require root access. For example, ping floods and UDP floods can be initiated by non-root users. This capability is sometimes abused by the intruders, using the fact that even when the attack is traced the only found source would be a compromised machine with no logs present. 5. "More boxes!" Similar to a root-owning intruder, those with non-root shells may use the compromised system for vulnerability scanning and widespread exploitation. Many of the scanners, such as openssl autorooter, recently discovered by us, do not need root to operate, but is still

capable of discovering and exploiting a massive (thousands and more) system within a short time period. Such large networks can be used for devastating denial of service attacks (for example, such as recently warned by CERT). V. Conclusion As a conclusion, we will try to answer the question: "Should you do it?" The precise answer depends upon your organization mission and available security expertise. Again, the emphasis here is on research honeypots and not on "shield" or protection honeypots. If your organization took care of most routine security concerns, has a developed in-house security program (since calling an outside consultant to investigate your honeypot incident does not qualify as a wise investment) and requires a first hand knowledge of attacker techniques and last minute Internet threats - the answer tends towards a tentative "yes". Major security vendors and consultancies or Universities with advanced computer security programs might fall into the category. If you are not happy with your existing security infrastructure and want to replace or supplement it with the new cutting edge "honeypot technology" - the answer is a resounding "no". Research honeypots will not *directly* impact the safety of your organization. Moreover, honeypots have their inherent dangers. They are analyzed in papers posted on the Honeynet Project site. The dangers include uncertain liability status, possible hacker retaliation and others. ABOUT THE AUTHOR: This is an updated author bio, added to the paper at the time of reposting in 2009. Dr. Anton Chuvakin (http://www.chuvakin.org) is a recognized security expert in the field of log management and PCI DSS compliance. He is an author of books "Security Warrior" and "PCI Compliance" and a contributor to "Know Your Enemy II", "Information Security Management Handbook" and others. Anton has published dozens of papers on log management, correlation, data analysis, PCI DSS, security management (see list www.info-secure.org) . His blog http://www.securitywarrior.org is one of the most popular in the industry. In addition, Anton teaches classes and presents at many security conferences across the world; he recently addressed audiences in United States, UK, Singapore, Spain, Russia and other countries. He works on emerging security standards and serves on the advisory boards of several security start-ups. Currently, Anton is developing his security consulting practice, focusing on logging and PCI DSS compliance for security vendors and Fortune 500 organizations. Dr. Anton Chuvakin was formerly a Director of PCI Compliance Solutions at Qualys. Previously, Anton worked at LogLogic as a Chief Logging Evangelist, tasked with educating the world about the importance of logging for security, compliance and operations. Before LogLogic, Anton was employed by a security vendor in a strategic product management role. Anton earned his Ph.D. degree from Stony Brook University.