Assessing The Performance Impact Of The Linux Kernel Audit Trail

  • December 2019
  • PDF

This document was uploaded by user and they confirmed that they have the permission to share it. If you are author or own the copyright of this book, please report to us by using this DMCA report form. Report DMCA


Overview

Download & View Assessing The Performance Impact Of The Linux Kernel Audit Trail as PDF for free.

More details

  • Words: 8,758
  • Pages: 43
Honours Year Project Report

Assessing the Performance Impact of the Linux Kernel Audit Trail

By Qiu Jiang

Department of Computer Science School of Computing National University of Singapore 2004/05

Honours Year Project Report

Assessing the Performance Impact of the Linux Kernel Audit Trail

By Qiu Jiang

Department of Computer Science School of Computing National University of Singapore 2004/05

Project No: H089020 Advisor: Prof. Sandeep Kumar Deliverables: Report: 1 Volume

Abstract Security breach into information systems is almost always caused by unpatched or unknown vulnerabilities. To counter this threat, system administrators often rely on effective auditing to discover such security violations. The benefits of auditing include accountability, reconstruction of higher-level events, intrusion detection, and problem analysis. Furthermore, the mere presence of auditing is a powerful deterrence to potential attackers. Despite these well-understood benefits, system administrators are reluctant to turn on auditing on their systems because of the fear of performance overhead. In this thesis, we design, implement and benchmark a low overhead auditing subsystem for the Linux 2.4 kernel.

Subject Descriptors: C5 Computer System Implementation D.4.6 Security and Protection

Keywords: audit, kernel, operating system, linux

Implementation Software and Hardware: Linux Kernel 2.4

Acknowledgement I would like to express my utmost gratitude to my project supervisor, Assistant Professor Sandeep Kumar, who provided me with inspiration and guidance that set me on the right course in this study. Also, my hat goes off to Jeremy Banford and Nenad Ocelic for their attempt to contribute a high quality kernel-level auditing tool to the open source community.

List of Figures 3.1 3.2 3.3

Audit records are generated at system calls . . . . . . . . . . . . . . . . . . . . . 12 Security sensitive system calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Auditor device with double buffering . . . . . . . . . . . . . . . . . . . . . . . . . 17

4.1 4.2 4.3 4.4

Integer, float and double opreations . Memory and file system results . . . Process related results . . . . . . . . Communication results . . . . . . . .

. . . .

4

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

21 23 24 26

List of Tables 3.1 3.2

System call group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Threat level classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4.1

Linux kernel compilation macrobenchmarks, time in seconds . . . . . . . . . . . . 27

5

Table of Contents Title

1

Abstract

2

Acknowledgement

3

List of Figures

4

List of Tables

5

1 Introduction 1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Report Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 1 2 3

2 Related Work 2.1 Introduction . . . . . . . . . . 2.2 Audit Tools . . . . . . . . . . 2.2.1 Built-in Logging Tool 2.2.2 Custom Audit Trails . 2.3 Auditing Benchmarks . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

4 4 5 5 5 7

3 System Design and Implementation 3.1 Naive Approach . . . . . . . . . . . . 3.2 Improved Approach . . . . . . . . . 3.2.1 Kernel Patches . . . . . . . . 3.2.2 Auditor Device . . . . . . . . 3.2.3 Audit Daemon . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

9 9 10 10 15 18

4 Evaluation 4.1 Test Environment . . . . . . . . . . . . . . . 4.2 Microbenmarks . . . . . . . . . . . . . . . . 4.2.1 Arithmetic Operations . . . . . . . . 4.2.2 Memory and File System Operations 4.2.3 Process Related Benchmarks . . . . 4.2.4 Communication Benchmarks . . . . 4.3 Macrobenchmarks . . . . . . . . . . . . . . 4.4 Analysis . . . . . . . . . . . . . . . . . . . . 4.4.1 Ideal Scenario . . . . . . . . . . . . . 4.4.2 Worse Case Scenario . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

19 19 20 20 22 24 25 27 27 28 28

. . . . .

. . . . .

. . . . .

6

4.4.3

Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5 Conclusions and Future Work 5.1 Future Work . . . . . . . . . . . 5.1.1 Tamper Proof Audit Trail 5.1.2 LSM Integration . . . . . 5.2 Conclusions . . . . . . . . . . . .

. . . .

. . . .

. . . .

References

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

31 31 31 31 32 33

7

Chapter 1

Introduction Information security is becoming an ever critial aspect of computing today. Widely publicized security breaches, for example (Leyden, 2004) and (CERT, 2003), highlight the need for a forensic tool that will greatly assist system administrators in understanding the modus operandi of an attack and assessing the extent of damage to their systems. This information can also help fix the vulnerabilities that lead to successful exploitation. In this thesis, we describe the design and implementation of a kernel-level subsystem that can record process activity at a very fine granualarity. We also argue that, contrary to popular belief, there is little performance impact resulting from a well-implemented kernel-level auditing system.

1.1

Background

An audit trail, as defined in (NIST, 1995), is a sequential record of computer events about an operating system, an application, or user activity. The process of constructing an audit trail and its subsequent examination is called auditing. The benefits of auditing include individual accountability, reconstruction of higher-level events, intrusion detection, and problem analysis. Additionally, the knowledge of the existence of an auditing mechanism can be a powerful deterrence to potential attackers. The ultimate goal of auditing is the reconstruction of high-level and human-understandable events that attackers commit to breach security. Examples of these high-level events are user-

1

logon, buffer-overflow, privilege-escalation and password-stealing. However, currently there is no mechanism that can be used to describe such high-level events. Instead, what is often used is a record of system events that are at a very fine granularity. It is hoped that with such a record, system administrators can understand the chain of high-level events that lead to successful attacks and, more importantly, patch the very security vulnerability which made the attacks possible in the first place. To this end, operating system vendors have implemented a kernel-level auditing solution that records the arguments and return values of every system call executed on the system. With such a fine granularity of record generation and storage comes the fear of performance overhead as a result of auditing. Many users and system administrators are loathe to install a full featured auditing system, or to turn it on during normal daily activity because of the fear of performance degradation on the host platform, in spite of the aforementioned benefits, To better understand the cause of this performance degradation, and to investigate solutions to mitigate the overhead, we proposed and implemented a system that significantly extends the original LinuxBSM (Banford, 2001) (Ocelic, 2001) project. The improved LinuxBSM consists of three components: (1) a set of kernel patches that augment the audit record generation of the security-relevant system calls, (2) a kernel-space auditing device that provides a means for user-space applications to interact with the kernel audit subsystem and, (3) a user-space audit daemon that stores audit records into the local filesystem. These are described in detail in Chapter 3.

1.2

Contributions

The contribution of our work is two fold. We have designed and implemented a kernel-level security auditing system for the Linux operating system that significantly improves over the most recent system (Banford, 2001) (Ocelic, 2001). Secondly, we have performed in-depth benchmarking to quantify the actual performance overhead in practice. Based on these results, we can say that a well designed and implemented auditing system incurs little overhead, so system administrators can reap the full benefits of auditing without the fear of its performance 2

impact.

1.3

Report Organization

The report is organized as follows. We review the state-of-the-art in kernel-level auditing in Chapter 2. Chapter 3 describes the design and implementation of our improved LinuxBSM. Chapter 4 examines the performance impact of the auditing system, and we conclude and discuss future work in Chapter 5.

3

Chapter 2

Related Work 2.1

Introduction

Given the benefits of auditing, many application-level software such as relational database management systems and web servers provide comprehensive audit capabilities with extensive information that can even be used in performance tuning. Even operating systems, such as the recent Microsoft Windows Server 2003 (Microsoft, 2003) and Sun’s Solaris 10 (Sun, 2005) allow comprehensive and highly configurable auditing policies. Application-level auditing is commonly known as logging. They provide many benefits such as system-health monitoring, application debugging and performance tuning. However, with the specific aim of security auditing, we restrict our investigation to kernel-level auditing in the following sections. In this chapter, we survey the auditing tools available either as part of the operating system, in section 2.2.1, or as a custom built solution in section 2.2.2. Then, in section 2.3, we look at some audit benchmarks that were performed in the past and understand how it has become a major obstacle to audit adoption.

4

2.2 2.2.1

Audit Tools Built-in Logging Tool

At a basic level, the various Unix-like operating systems provide a logging utility called klogd (Schwarz, 2000), the kernel logging daemon that intercepts and logs Linux kernel messages in much the same way as the system logging daemon. The kernel can add an entry to the kernel log by calling printk() that behaves almost identically to the C library printf() function. Invoking the printk() function stores the new entry in a circular buffer and subsequently the user-space klog daemon retrieves these messages from the buffer and feeds them into the log file via the syslogd daemon. Klogd is remarkable in that it aims to provide a unified logging framework to the kernel. This enables critical kernel messages to be retrieved and reviewed easily by developers as well as administrators and acted upon. However, klog entries are entirely customary and there is no common standard on how, when and what to audit. Furthermore, the requirement for security auditing mandates a custom auditing system that allows the effecient recording of data-intensive audit trails. In recognizing this need, and in enabling rich and flexible auditing, various operating system vendors, as well as the open source community, have implemented their auditing solution.

2.2.2

Custom Audit Trails

The SunShield Basic Security Module (Sun, 2000) provides an auditing system for both SunOS (Sunshield) and Solaris (Sunshield BSM). The BSM permits the actions of specific users to be recorded and written to an audit file. Sun’s BSM is the originial inspiration to develop a high quality audit tool for the open source Linux operating system, which is known as LinuxBSM. LinuxBSM (Banford, 2001) is an initiative at University of California, Davis to provide a comprehesive auditing package for Linux that is fully compliant with the U.S. Government’s C2 standards for security, defined in the standards document ’Trusted Computer System Evaluation Criteria’ (DoD, 1985) or more commonly called the Orange Book. Specifically, the LinuxBSM

5

is an auditing tool that aims to bring the capabilities of Sun’s Solaris Basic Security Module to Linux. LinuxBSM monitors and records system events on the kernel level, equipping system adminstrators a mechanism to trace the activities in the system. The project’s development stopped in November 2000 and its capabilities at that time included auditing a total of six system calls for the Linux 2.2 kernel. LinuxBSM2 (Ocelic, 2001) is the next evolution from LinuxBSM and it consists of several architectural improvements over the original LinuxBSM, including a flexible system call audit record generator. This tool allows an administrator to specify desired audit record fields for new system calls in a plain text file. The file is used in inserting the relevant source codes to enable auditing of these system calls automatically. In this way, kernel developers can avoid a certain amount of manual duplication of identical code section in audit record construction. The project is also targeted at the same Linux 2.2 kernel and the numbers of system calls audited remains at six. Meanwhile, major kernel changes took place in 2001 and the first release of new version 2.4 kernel was released at the same time. LinuxBSM2 cannot be used in the new kernel due to these breaking changes. Similarly, development stopped towards the end of 2001. Linux Auditing Subsystem (SuSE, 2003), LAuS, is a very similar auditing system implemented by SuSE Inc. The system is composed of kernel-resident and user-space components that facilitate highly-configurable and robust logging of system call invocations. It is currently distributed as part of Red Hat Enterprise Linux 3 and SuSE Linux Enterprise Server 8 that are based on the Linux 2.4 kernel. The design principle of LAuS is significantly different from the improved LinuxBSM and we will describe its key features as follows. • LAuS adds a new field to the task structure to store audit parameters to enable selective auditing of processes. If a processes is marked to be audited, then relevant audit fields will be recorded at the system call dispatch point. This has the disadvantage of the need to modify individual system call handlers for each supported hardware architecture. • LAuS stores the audit records in a memory buffer in the kernel-space. This buffer is like a queue in that the audit records are read, one at a time, by invoking the read() system call on the auditor device. Such a design necessitates frequent disk write operations and hence 6

places a I/O load on host systems. Also, the rate at which audit records are generated may outpace that of dequeuing from the memory buffer and this results in audit data loss. The improved LinuxBSM solves these problems by avoiding an architecture-dependent design and employing a double-buffer strategy, as detailed in Chapter 3. The abovementioned audit solutions, including the improved LinuxBSM, share a common need for kernel patches. This is because the Linux kernel, by default, does not have any provision for system call interception or monitoring. Hence, a kernel patch is the precursor for any system call level auditing systems. However, the recent development in the Linux Security Modules (Wright, Cowan, Smalley, Morris, & Kroah-Hartman, 2002) project introduces a lightweight, general purpose access control framework as loadable kernel modules. The LSM framework, itself a set of kernel patches, is on the verge of being integrated into the mainstream kernel. It provides access to various kernel objects such as processes, IPCs and file system. Any access control policy can be implemented as a LSM security kernel module. Currently there are some problems with LSM, such as the lack of support for persistent security attribute binding without resorting to external metadata storage. However, once LSM is stablized in the mainstream kernel, the improved LinuxBSM can leverage the access to auditable kernel objects without having to place system call hooks manually.

2.3

Auditing Benchmarks

Despite the well understood benefits of auditing, many system administrators are reluctant to install an auditing system primarily due to two reasons. 1. Auditing tools have not been standardized and often the format of audit records are customary to individual applications. As a result, audit records from different auditing systems exist as information islands and there is no coherent view across these potentially useful data. This is a classical data warehousing problem and solutions to it include standardizing audit format for the various auditing sources, as proposed by (Bishop, 1995). Alternatively, we can perform extraction, transform and load (ETL) by custom-built data

7

adapters and store the result into a central repository. This method is favored by hybrid intrusion detection systems such as Prelude (Mathieu Blanc, 2003). 2. The second major hurdle to widespread adoption of auditing is due to the fear of its performance impact on the host system. To this end, we find surprisingly little literature on documented performance benchmarks on auditing. (Nitzberg, 1994) performed an early analysis of the performance impact of the audit component available in the HP-UX version 8.0, a hybrid between AT&T System V and BSD Unix, and found that the worse case impact is as high as 85% degradation. This might explain the fear of implementing an auditing solution. More recently, (Veronique Abily, 2000), used the SunShield BSM (Sun, 2000) to measure the performance impact of an intrusion detection system and discovered that the auditing component contributed to about 4% performance hit. This result is much more encouraging than the earlier study. However, the result is buried as part of benchmarking process of an intrusion detection system. In chapter 4, we describe the benchmark results of our auditing system that provide conclusive evidence about the fact that auditing does not impact system performance in most systems.

8

Chapter 3

System Design and Implementation This chapter describes the design and implementation issues in the improved LinuxBSM. It starts with a discussion on the obvious (and inefficient) approach to kernel auditing in section 3.1. Section 3.2 explains the approach employed by the improved LinuxBSM to what and how to audit in the improved LinuxBSM.

3.1

Naive Approach

As argued in (Sekar, Bowen, & Segal, 1999), regardless of how an attack is delivered, any damage to a host is effected via the system calls made by a process running on the host computer. Therefore, auditing system calls is an efficient way of analyzing (potential) security breaches. Note that this is different from auditing policies in the Microsoft Windows Server family (Microsoft, 2003) where a level of abstraction above system calls is provided. A naive approach to system call auditing would involve ’capturing’ system calls in the architecture dependent system call handler1 . As user-space programs invoke system calls, threads of execution trap into the kernel from where the system call is dispatched. At this point, an audit record containing the system call number and associated arguments can be generated and written to the local file system. However, there are at least two problems with this approach. 1. Invoking system calls can have different post-conditions and return values. For example, 1

For example, in the Linux kernel, the system call handler resides in entry.s

9

the read() function call can succeed or fail and when it succeeds, the file offset can have different values based on how many bytes are read. As a result, if only a set of generic parameters are recorded in the audit trail, then it will be remarkably more difficult to distinguish one system call from another. 2. More seriously, if every system call results in a disk write operation to the file system, it is clear that auditing will become a bottleneck to system performance.

3.2

Improved Approach

To overcome the abovementioned problems, the improved LinuxBSM consists of three components, a set of kernel patches to the security sensitive system calls, a kernel device driver that records audit events in memory buffers (shared with the user-space daemon), and a user-space daemon that periodically writes to the local file system. This design enhances the availability of the host on which it runs while providing the benefits of auditing. We describe the three components in the following sections.

3.2.1

Kernel Patches

The main design policy of the system is to maximize its utility and minimize its load in a production environment. To maximize its utility, we must recognize the ingenuity of attackers and their pattern of intrusions. In an ideal situation, we will audit all system calls. A system administrator will have a complete record of events to reconstruct the steps of an attack. However, such an approach will not only overwhelm the system administrator with mostly normal audit records, the system on which the audit trail runs will incur a significant performance hit. Performance degradation due to auditing on Unix is reported (Nitzberg, 1994) to range from negligible impact for the arithmetic component of the benchmark to 85% degradation for system call overhead tests. Therefore, we need to strike a balance between security and performance. (Bernaschi, Gabrielli, & Mancini, 2002) divided the Linux system call set into nine groups, as shown in Table 3.1 according to their intended functionalities. They assert that it is safe to disregard 10

Table 3.1: System call group Group

Functionality

1

File system and device

2

Process

3

Loadable kernel module

4

Memory

5

Time and Timer

6

Interprocess Communication

7

System Information

8

Reserved

9

Not Implemented

system calls from group 4 to 9 as they cannot be used to subvert a system. For example, unimplemented system calls (group 9) obviously cannot be used to subvert a system. So are reserved (group 8) and system information related (group 7) calls. More interestingly, system calls that deal with memory management (group 4), timers (group 5) and interprocess communication (group 6) are useless to an attacker who seeks to control either the running system or the running process. We are left with system calls from group 1 to group 3 where either the file system, processes or kernel modules are involved. They also classifed the system calls, as in Table 3.2 according to four levels of security threats, from allowing full control of the system to harmless. The result is that a total of 102 system calls are identified as security sensitive. In our implementation, we apply a set of kernel patches to a stock Linux kernel version 2.4 as shown in Figure 3.1. These patches are placed at critical system call points to capture security related information to a buffer. The set of system calls chosen are based on the study done by (Bernaschi et al., 2002), with the exception of signals and real-time signals.

11

Table 3.2: Threat level classification Threat Level

Description

1

Allows full control of the system

2

Used for a denial of service attack

3

Used for subverting the invoking process

4

Harmless

Figure 3.1: Audit records are generated at system calls

12

As can be seen from Figure 3.2, the system calls are categorized into three threat levels. • Level 1 refers to system calls that allow full control of the system. • Level 2 identifies those calls that can be used for a denial of service attack on the host machine • Level 3 includes system calls that can be used for subverting the invoking process. Note that by convention, the smaller the threat level number, the more dangerous are the system calls. Also, there exists a total order across the threat levels; for if a system call allows the attacker to take full control of a system, he can subsequently launch a denial-of-service attack on the local machine trivially. Together, these system calls deal with file systems, devices, processes and module managements that are the core functionality of the Linux kernel. By auditing these specific system calls, we can discover actions that lead to successful or attempts of security breaches. For example, various attempts have been made in the past to effectively examine audit trails in order to discover important events that lead to a security breach. Such a forensic discovery process can be greatly assisted by two types of tools. The first type of log analyzers, such as Xlogmaster(Greve, 1998) and Swatch(Hansen & Atkins, 1993), focus mainly on the effective filtering of audit events according to user-defined rules. The problems of these tools soon become obvious to practioners that they require highly experienced professionals who know what they are looking for from audit trails. Security breaches are often the result of successful exploitation of unprecedented vulnerabilities in operating systems or application programs. This calls for a tool that can automatically trace from compromised objects to the initial break-ins. The answer to that is the second type of audit trail analyzers, such as Tudumi(Takada & Koike, 2002) amd BackTracker(King & Chen, 2003). In short, they employ techniques from infomation visualization to present the chain of events in a graphical user interface (GUI). In this thesis, we merely state that auditing security related system calls can lead to a successful identification of security vulnerabilities. 13



                                                                                     

                                                                            

        

          

  

                    

    

       

Figure 3.2: Security sensitive system calls

14

3.2.2

Auditor Device

Since we have previously argued that write operations to the file system is expensive and should be minimized, we create a character device in the kernel-space that writes audit records as they are created by the hooks in the modified system calls to memory buffers. The reason that the auditor is a character device and not a block device is that we do not require the nonsequential access to the memory buffer. Nonsequential access to block devices such as floppy drives and CD-ROM drives are not relevant to the auditor device. At run time, the auditor device will fill up a memory buffer with audit records sequentially and a user-space audit daemon will clear the audit records once the memory stream is full, also sequentially. Implementing it as a block device would incur considerable effort with no apparent benefits. The other important task of this auditor is to signal to the user-space audit daemon that the memory buffer is full and the audit daemon should copy the contents in the buffer out. In this way, disk write is only performed periodically and the periodicity can be adjusted via the size of the buffer based on empirical results of performance impact on the system. Storing audit records in a memory buffer solves the problem of frequent write but it creates another problem. When the content of a full buffer is being copied to the user-space, some audit records will be lost as the the buffer is in the read mode2 . At first thought, it is tempting to solve this problem by simply implementing the memory buffer as a queue data structure. In this way, as soon as the buffer is full, we can start dequeuing from the other end of the buffer to accommodate the incoming audit records. However the rate of clearing the content into user-space might not equal or exceed the rate at which new audit records are generated. Hence, audit loss becomes inevitable in some circumstances. In the security sensitive environment, we must assume that an attacker has access to the mechanism with which auditing is done. He will exploit this vunerability so that the record of the key events are lost during the so-called buffer jams. In our improved LinuxBSM, we apply the double-buffer strategy, shown in Figure 3.3, to avoid audit record loss. Specifically, the kernel auditor device associate two memory buffers 2

From the perspective of the user-space daemon

15

and label them as the read and write buffer respectively. From the user-space daemon’s point of view, read buffer refers to the daemon’s reading from this buffer. Write buffer is the memory region that audit records are appended as they are generated. Once the write buffer is full, the device driver sends a signal to the user-space daemon and immediately swaps the two buffers. In this way, while the audit daemon writes the content of read buffer to the local file system, fresh audit records are written to the newly appointed write buffer. The double-buffer strategy is reminiscent to graphics rendering devices in that while the front buffer is rasterized onto screen display the back buffer can be used to store the content of the next frame to render. This design is due to the original LinuxBSM. It might be argued that circumstance might occur where audit records are generated at a rate that is faster than that of being cleared by the user-space audit daemon and hence propose a triple-buffering strategy. We believe that while triple-buffering certainly adds a higher level of assurance against audit record loss and in fact, a variable number of buffers can be used in an adaptive manner depending on the rate of audit growth, empirical results indicate that double-buffering is sufficient even at high-load situations, as we can see from Chapter 4.

16

17 Figure 3.3: Auditor device with double buffering

3.2.3

Audit Daemon

The communication between the kernel-space and user-space in the Linux operating system is facilitated through the use of signals. Signals, in this case, are used as software interrupts that allow asynchronous notifications from kernel-space to user-space about events being raised. Specifically in our design, as soon as we reach the point where storing one more audit record will overflow the write buffer, the kernel notifies a user-space audit daemon with a SIGIO signal to swap the contents in the buffer out of the kernel-space. A daemon is similar to a service in the Windows operating system that is an autonumous background process. The audit daemon copies the content from the read buffer in the kernel and writes to the local file system partition or sends the audit data via network to another computer. Before the audit daemon is running, the kernel auditor device will not generate any audit records. The audit daemon is started with a set of command line parameters including the range of process numbers, user ids and group ids to audit. These allow the selective auditing of events that are deemed useful by the system administrator. Besides setting the audit parameters, the audit daemon also registers the signals and attaches signal handlers to them. In this way, it can perform the necessary actions such as writing to filesystem and closing the audit device based on the received signals. Once the initialization of opening the device, setting up the memory buffers and registering signal handlers are done, the audit daemon informs the kernel with a specific IOCTL command. As soon as the kernel is aware that the audit daemon is running, it can start auditing predefined system calls.

18

Chapter 4

Evaluation In this thesis, we benchmark the performance overhead of our improved LinuxBSM that audits a total of 82 security related system calls. Following the standard practice of benchmarking the kernel, for example as done for Linux Security Modules (LSM) (Wright et al., 2002), we use lmbench (Larry W. McVoy, 1996) for microbenchmarking. lmbench is developed specifically to measure the performance of core kernel system calls and facilities such as file system access, context switching and memory access. For macrobenchmarking, we follow the oft used convention of measuring kernel compile times.

4.1

Test Environment

We compare the standard Linux 2.4.22 kernel against the 2.4.22 kernel incorporating the improved LinuxBSM with the audit daemon running. The tests are run on a single-processor 1.6 GHz Pentium 4 computer with 256MB of SDRAM and one 20GB Ultra ATA/133 hard disk drive. In most tests, the performance overhead of the improved LinuxBSM is within the boundary of the inevitable experimental errors. In certain cases we even have better results with auditing and that is almost certainly due to random errors that are part of any experiment. In addition, the size of each buffer, which is used by the auditor as described in Chapter 3, is configured to be 250 kilobytes. The results are shown in section 4.2 and 4.3.

19

4.2 4.2.1

Microbenmarks Arithmetic Operations

Figure 4.1 shows that, for basic arithmetic opreations, besides empirical disrepancies, there is no difference between the unpatched kernel and improved LinuxBSM. This is to be expected as these CPU-bound tasks do not invoke any system calls that we audit.

20

 !! !                          

 

 

"!





 

  



"!







 











  #



 !! !           

        

  

"!

 

 

 

 

"!

 

 

 

 







  #



 "! !        

        

  

"!

 

 

 

 

"!

 

 

 

 







  #

 Figure 4.1: Integer, float and double opreations

21

4.2.2

Memory and File System Operations

Just as auditing has no performance impact on arithmetic operations, so is its lack of overhead on memory latencies, as shown in Figure 4.2. In our test machine we have both level 1 and level 2 cache and access to these memory is effected through architecture specific assembly code where no auditing is performed. Furthermore, main and random memory access are not audited too because so far attackers are unable to manipulate kernel memory directly without crashing the system. Since no auditing is done to memory operation, no consistent performance impact could possibly be observed. On the other hand, file system operations are fully audited due to their sensitive nature. In the Linux operating system, most objects, such are directories and devices, can be considered as part of the file system. As a result, we expect and observe a large proportion of filesystem related audit records. Therefore, this is the most likely bottleneck of our auditing mechanism. However, Figure 4.2 shows encouraging results as we fail to observe significant performance differences.

22

!&#"""  









$ #

 

 

  

 

$#

  



 

 







!  %!



"&"##"!""  

             

 

 



$ #   

  



  

  

$#  

 

 

 

   

  ! 









%! 







Figure 4.2: Memory and file system results

23



!%! $!!       







  







"!   

 

 

 







"!

 

 

 



















 

    #  



  !   " " !!  !    #         !    

"!  

 





 











"!  







 











   





 













#  

Figure 4.3: Process related results

4.2.3

Process Related Benchmarks

The performance overhead of auditing context switch, which occurs when the scheduler calls context switch() to run the next process in the run queue, is shown in Figure 4.3. Benchmarks are performed on 2, 8 and 16 processes with 0, 16 and 64KB of data associated with the switching processes. Again, except for random errors, we fail to observe audit-related performance impact. Also shown in Figure 4.3 is process related benchmarks such as process creation fork() and overlay exec() system calls. These system calls, among others, are audited but we do not see any discernible performance difference.

24

4.2.4

Communication Benchmarks

Finally, we measure the bandwitdth and latency of memory related operations and interprocess communications. Again, we fail to observe, as shown in Figure 4.4, any significant difference between the unpatched kernel and the modified kernel with the auditor running.

25

"" '!&"!&!%! $"%"!%   

    ! 

  

   

'!#&    

 

 









'&  



 

 











 $ !



 











($ 



"" '!&"!!)&%!%

 

         " "         '!#&  



 

      



'& 



 

       





 



 



  $ !



 

($    Figure 4.4: Communication results

26



 

Table 4.1: Linux kernel compilation macrobenchmarks, time in seconds

4.3

unpatched

auditd running

% Overhead with auditd

313

316

1.0

Macrobenchmarks

We perform the widely used kernel compilation macrobenmark, measuring the time to build the Linux kernel. As before, we run the compilations on a single-processor 1.6 GHz Pentium 4 computer with 256MB of SDRAM and 20GB Ultra ATA/100 disk.The command make bzImage is executed with the results shown in Table 4.1.

4.4

Analysis

Our observation of the negligible performance overhead resulting from kernel auditing requires some explanation. We present an approximate calculation of the performance impact here which is in line with the observed results and therefore provides credence to it. The memory size, B, used by the auditor is configured to be 250 kilobytes for each buffer. The buffers are described in detail in Chapter 3. Empirical observation shows that the auditorinduced disk write occurs at 5 second frequency during intensive I/O operations. The hard disk drive running on the test PC has the following relevant parameters. • Number of sectors per track, denoted by P: 63 • Number of bytes per sector, denoted by Q: 512 • Track-Track seek time, denoted by s: 0.8 ms • Average seek time, denoted by S: 8.5 ms • Rotational Delay, denoted by R: 4.2 ms • Data Transfer time, denoted by D: 0.595 ms per track

27

4.4.1

Ideal Scenario

In the ideal scenario where the partition to which audit data is written is unfragmented, the audit daemon approximately writes to consecutive free blocks and little time is lost because of seek delay. Under such an assumption, we have T, the total time, required for the write operation as T =

B ∗ (S + R + D) PQ

where B is the buffer size, P is the number of sectors per track, Q is the number of bytes per sector, S is the average seek time, R is the rotational delay and D is data transfer time. The first multiplier

B PQ

calculates the number of tracks written. The second multiplier, (S + R + D)

calculates the time required to write one full track. The result of this multiplication is the total time, T, required for the write operation. Hence, at every 5 second intervals we spend 250000 ∗ (8.5 + 4.2 + 0.595) ≈ 106ms 63 ∗ 512 in write operations. This is about 2% (106 ∗ 10−3 /5 ≈ 0.0212) and that is close to the performance overhead observed in our benchmarks.

4.4.2

Worse Case Scenario

In the worse case scenario, the partition is heavily fragmented due to frequent read-write operations. In this case, the disk access incurs a seek on every sector that is written. In this scenario,

T =

B ∗ (S + R + D + s) PQ

Hence, at every 5 second intervals we spend 250000 ∗ (8.5 + 4.2 + 0.595 + 0.8) ≈ 109ms 63 ∗ 512 in write operations. This is again about 2% (109 ∗ 10−3 /5 ≈ 0.0218) performance overhead. 28

4.4.3

Remarks

With these hardware parameters, we understand that the negligible audit performance overhead is due to the rapid advances in hardware technology. It is also worth noting that lmbench benchmarking suite emulates both I/O intensive and typical interactive sessions in its testing. The result of it therefore covers both scenarios in which auditing is highly desirable. While lmbench provides us with a set of performance measurements of typical system operations on the Linux kernel, we aim to measure the worst case scenario with the audit subsystem turned on. In order to do that, we create a program that repeatedly calls open() and close() on a file in the local file system 10,000,000 times. The process that executes this program does virtually nothing except invoking audited system calls and it provides us with a satisfactory upper-bound of the performance overhead of the audit subsystem. int main() { int fd; int counter = 0; while (counter < 10000000) { fd = open("abc"); close(fd); count++; } return 0; } Executing the above program on the same test machine takes about 74.2 and 45.7 seconds with auditing turned on and off repsectively. This indicates an upper bound of 62.4% performance overhead. In reality, processes invoke system calls in a manner that is more akin to lmbench’s simulation than to our explicitly constructed upper bound. Across this spectrum of system call

29

intensities, system administrators will have to understand the opreating environment and make a judgment call on whether to turn on kernel-level auditing.

30

Chapter 5

Conclusions and Future Work 5.1

Future Work

5.1.1

Tamper Proof Audit Trail

One requirement for forensic analysis on compromised computers is that the audit trails be tamper proof. Such a framework is described in (Schneier & Kelsey, 1999), and we plan to study and implement the framework to our improved LinuxBSM. This will remarkably increase the reliability to LinuxBSM as a forensic tool.

5.1.2

LSM Integration

As described in earlier, we intend to port our auditing mechanism to the LSM framework. This has two advantages. 1. As part of a standardized effort to provide enhanced security to the Linux operating system, our auditing system can be more easily accepted by system administrators. 2. The current improved LinuxBSM is based on the Linux 2.4 kernel. By porting it to the LSM framework, it automatically works with the latest Linux 2.6 kernel.

31

5.2

Conclusions

In this thesis, we provide a much more fine-grained auditing system for the Linux operating system to the Linux 2.4 kernel. Also, we discover that given a well implementd auditing system, performance overhead is negligible as compared to an unpatched kernel. With these, we believe the scale of auditing adoption by system administrators will improve considerably and security vulnerabilitiy exploits can be discovered and rectified in a much more timely fashion.

32

References Banford, J. (2001). Linuxbsm. http://linuxbsm.sourceforge.net/, , July, 2001. Bernaschi, M., Gabrielli, E., & Mancini, L. V. (2002). Remus: A security-enhanced operating system. ACM Transactions on Information and System Security, 5 , February, 2002, 36– 61. Bishop, M. (1995). A standard audit trail format. Proceedings of the 18th NIST-NCSC National Information Systems Security Conference (pp. 136–145), Baltimore, Maryland, October, 1995. CERT (2003). Cert advisory ca-2003-23 rpcss vulnerabilities in microsoft windows. http://www.cert.org/advisories/CA-2003-23.html, , September, 2003. DoD (1985). Trusted computer system evaluation criteria. Washington, DC 20301-1000: Department of Defense. Greve, G. C. F. (1998). The xlogmaster. http://www.gnu.org/software/xlogmaster/xlogmaster.html, , June, 1998. Hansen, S. E., & Atkins, E. T. (1993). Automated system monitoring and notification with swatch. Proceedings of the Seventh Systems Administration Conference (LISA VII) (USENIX Association: Berkeley, CA), , November, 1993, 145. King, S. T., & Chen, P. M. (2003). Backtracking intrusions. Proceedings of International Conference on Graph Theory, Ann Arbor, MI 48109-2122, October, 2003. Larry W. McVoy, C. S. (1996). lmbench: Portable tools for performance analysis. USENIX Annual Technical Conference (pp. 279–294), San Diego, CA, January, 1996: USENIX Association. Leyden, J. (2004). Students find 44 unix flaws as homework. http://www.theregister.co.uk/2004/12/16/unix flaw homework/, , December, 2004. Mathieu Blanc, Laurent Oudot, V. G. (2003). Global intrusion detection: Prelude hybrid ids (Technical report). Villeurbanne, France: ExaProbe. Microsoft (2003). Windows server 2003 product documentation. Redmond,WA, 98052: Microsoft Corporation. NIST (1995). Introduction to computer security: The nist handbook. Gaithersburg, MD: NIST. Nitzberg, S. (1994). Performance benchmarking of unix system auditing. Master’s thesis, Monmouth college, Monmouth, IL. 33

Ocelic, D. (2001). Linuxbsm2. http://linuxbsm2.sourceforge.net/, , December, 2001. Schneier, B., & Kelsey, J. (1999). Secure audit logs to support computer forensics. ACM Transactions on Information and System Security, 2 (2), May, 1999, 159–176. Schwarz, M. A. (2000). Take command: klogd: The kernel logging daemon. Linux Journal, , August, 2000, 20. Sekar, R., Bowen, T., & Segal, M. (1999). On preventing intrusions by process behavior monitoring. Proceedings of the Workshop on Intrusion Detection and Network Monitoring (pp. 29–40), Santa Clara, CA, April, 1999. Sun (2000). Sunshield basic security module guide. Palo Alto, CA: Sun Microsystems. Sun (2005). Solaris 10 product documentation. Santa Clara, CA 95054: Sun Microsystems. SuSE (2003). Linux audit-subsystem design documentation. Waltham, MA 02451: Suse Inc. Takada, T., & Koike, H. (2002). Tudumi: Information visualization system for monitoring and auditing computer logs. Proceedings of Sixth International Conference on Information Visualisation, London, England, July, 2002. Veronique Abily, M. D. (2000). Benchmarking a distributed intrusion detection system based on asax: Preliminary results. Lecture Notes in Computer Science, 1907 , April, 2000. Wright, C., Cowan, C., Smalley, S., Morris, J., & Kroah-Hartman, G. (2002). Linux security modules: General security support for the linux kernel. Proceedings of the 11th USENIX Security Symposium (pp. 17–31), San Francisco, CA, August, 2002: USENIX Association.

34

Project Information Page Project Type

:

Honours Year Project

Project Area

:

Software System

Project Title

:

Assessing the Performance Impact of the Linux Kernel Audit Trail

Project No

:

H089020

Students Name

:

Qiu Jiang

Project Advisor

:

Prof. Sandeep Kumar

Date of Completion

:

April 2005

Deliverables

:

Report 1 Volume

:

Linux Kernel 2.4

Implementation Software and Hardware

Abstract Security breach into information systems is almost always caused by unpatched or unknown vulnerabilities. To counter this threat, system administrators often rely on effective auditing to discover such security violations. The benefits of auditing include accountability, reconstruction of higher-level events, intrusion detection, and problem analysis. Furthermore, the mere presence of auditing is a powerful deterrence to potential attackers. Despite these well-understood benefits, system administrators are reluctant to turn on auditing on their systems because of the fear of performance overhead. In this thesis, we design, implement and benchmark a low overhead auditing sub-system for the Linux 2.4 kernel.

Subject Descriptors: C5 Computer System Implementation D.4.6 Security and Protection

Keywords: audit, kernel

35

Related Documents