Aix Monitor Users

Monitoring AIX Users Audience: AIX Administrators and End Users Date: July 16, 1999 Monitoring user activity is an important system administration task. Here are a few useful AIX commands for monitoring user activity. Command

Description Display information about previous logins, including login last date/time, logout time, originating remote host. who List users who are currently logged on. ps -fu "userid" List processes being run by "userid". Text file located in the home directory of each Korn shell user. .sh_history The file contains the last 50 commands issued by the user. /var/adm/sulog Text file containing all "su" activity. Binary file containing a listing of all failed login attempts. The /etc/security/failedlogin file can be read by only root, using "who failedlogin" AIX has two other facilities for monitoring user activity on a more granular level. The first is "Accounting" which monitors user's connect time and system usage (cpu by command, diskstorage and printer usage). The second is "Auditing", a security related function, which provides a detailed audit trail of each user's activity, including priviledge failures, commands run, files they view/create/delete, and more. Both Accounting and Auditing are part of the base AIX operating system. However, both must be configured to run. The attachment HTML files discuss how to set up accounting and auditing. See your AIX documentation for more information.

AIX System Security Audit [audit.all.cmd] AIX System Security Audit -----------------------------------------------------------------------------Contents About This Document Related Documentation Overview Auditing Events and Objects Auditing Mode: BIN and STREAM Starting and Stopping Audit Auditing Configuration Auditing a User Auditing an Object Disk Space Consideration Understanding the Output Common Problems with Auditing

Data Overload Fixes for Auditing Subsystem -----------------------------------------------------------------------------About This Document This document is intended to simplify the use of the auditing system provided in AIX and applies to all versions of AIX. It includes information on what auditing offers, what its requirements are, and what common problems may be encountered. The intention is not to answer every question about auditing, but to provide a starting point for understanding and setting up auditing. Related Documentation Documentation for System Auditing can be found in Chapter 5 of the "System Management Guide" for AIX version 3.2, Chapter 3 of the "System Management Guide: Operating System and Devices" for AIX version 4, and Chapter 8 in the IBM Redbook "Elements of Security: AIX 4.1" (GG24-4433-00). The AIX and RS/6000 product documentation library is also available: http://www.rs6000.ibm.com/resource/ -----------------------------------------------------------------------------Overview The auditing subsystem provides the means to record security-related information and to alert system administrators of potential and actual violations of the system security policy. The information collected by auditing includes: the name of the auditable event, the status (success or failure) of the event, and any additional event-specific information related to security auditing. -----------------------------------------------------------------------------Auditing Events and Objects A list of audit events built into AIX, along with a list of predefined audit objects, can be found in the file /etc/security/audit/events. In general, auditing events are defined at the system call level. A single operation at the command line would result in records of several events in the

audit trail. For example, when viewing a file using the cat or more command, you would see the following records logged into the audit trail: FILE_Open (file is opened) FILE_Read (file is read) FILE_Write (file is written to standard output) PROC_Create (process creation for more OR cat) PROC_Execute (command execution) PROC_Delete (process completion) Auditing all possible events can produce a huge amount of data. Through audit controls (that is, modifying the configuration files), you can select events to be recorded. Audit events are grouped into classes. The events can be defined by which events are in a class. While the class names are arbitrary, they, rather than individual event names, are associated with user IDs when the audit subsystem is active. Auditing objects are just individual files that will be monitored. Three operations can be audited: read, write, and execute. Objects are not associated with user IDs. Audit records are generated whenever an audited object is referenced by any user (including root). To add further audit objects, extend the /etc/security/audit/objects file. -----------------------------------------------------------------------------Auditing Mode: BIN and STREAM There are two modes of operation for auditing: BIN and STREAM. BIN mode writes the audit trail to alternating temporary files (bins), then to a single trail file. STREAM mode writes to a circular buffer that is read synchronously through an audit pseudo-device (/dev/audit). An audit can be started in one OR both of these modes. Using the audit configuration setup shipped with AIX, /etc/security/audit/config, the BIN mode alternates between /audit/bin1 and /audit/bin2. When one BIN is full (the binsize parameter determines the size of the bin), the audit switches to the other BIN file while adding the accumulated data in the first file to the audit trail (defined in /etc/security/audit/bincmds), /audit/trail. Use "audit shutdown" to be certain

that all audit records have been added to /audit/trail. The BIN mode audit record is in binary format. You can read it with audit commands such as auditpr. In STREAM mode, the default AIX configuration provides a program to read the STREAM buffer and processes each record with the commands found in /etc/security/audit/streamcmds. These commands format the output into human-readable form and write it in /audit/stream.out. This file is NOT cumulative; it is restarted every time the audit is restarted. The STREAM audit trail can be read in real time by reading /audit/stream.out, or by sending output directly to a terminal or printer. -----------------------------------------------------------------------------Starting and Stopping Audit There are five audit subcommands for invoking auditing. They are as follows: audit start

- to activate the audit subsystem (This is the only correct way to start audit.) audit shutdown - to stop auditing subsystem, processing final BIN records and removing the /audit/auditb file that is used as an "active" indicator by the audit modules audit off - to suspend auditing temporarily audit on - to resume audit after audit off audit query - to display the status of auditing NOTE: Using audit commands in the wrong order can confuse the auditing subsystem. If the auditing subsystem gets confused, reset everything by deleting all files in the /audit directory (except trail, stream.out and bin files). Auditing can be run at the discretion of the system administrator. Depending on the environment, it is usually not necessary to have auditing running at all times. If it is configured to monitor a large number of events or objects at all times, the amount of data generated would be so substantial that its overhead would outweigh its benefit. It is worthwhile to take time to configure auditing to collect selected information. To start auditing at system startup, add the following line to the /etc/rc file, just prior to the line reading dspmsg rc.cat 5 'Multi-user initialization completed': /usr/sbin/audit start

If auditing is running at all times, make sure to run or to add the following in the /usr/sbin/shutdown script to properly shut down: /usr/sbin/audit shutdown -----------------------------------------------------------------------------Auditing Configuration All auditing related configuration files reside in /etc/security/audit. The /etc/security/audit/config file contains the key audit controls. It has the following stanzas: start

-

bin and stream

-

classes

-

specifies whether BIN or STREAM (or both) should be used for auditing contain controls for each mode; the names of the BIN files are specified here defines several groups (classes) of auditing events

The predefined classes are: general, objects, SRC, kernel, files, SVIPC, mail, cron, and TCPIP. New classes can be defined using the auditing events in the /etc/security/audit/events file. All audit classes except the objects class are associated with user IDs. For example, audit the events defined as "general" and "TCPIP" for user root. user stanza

-

lists specified users and the audit classes assigned to them; each username must be the login name of a system user or the string 'default'

An example of this stanza is as follows: users:

root = general joe = general, files, TCPIP default = general

When auditing starts, it ALWAYS audits the events specified for every user ID defined in the config file and ALL the objects defined in /etc/security/audit/objects. If the objects' audit records are not wanted, remove or comment out (using an *) the objects defined in the objects file. If there are specific classes of events that are not wanted as audit records,

specify No_Events for that specific class in the config file. For example: files = No_Events or tcpip = No_Events The objects file contains all objects to be audited when auditing is active. A user defined object is displayed as: /home/joe/my.stuff:

r = "JOE_READ" w = "JOE_WRITE"

The names JOE_READ and JOE_WRITE are referenced in the /etc/security/events file to define the format of the auditpr output: JOE_READ = printf "%s" JOE_WRITE = printf "%s" NOTE: There is no need to add the newly added objects to the objects stanza in the /etc/security/audit/config file, since the objects line is not referenced. Only the objects file is referenced. The streamcmds file has commands that are entered for STREAM audit records. The default file contains one command. Enter: /usr/sbin/auditstream | auditpr > /audit/stream.out & Adding the -v flag for the auditpr command improves this command at the expense of having more information. Without -v, full path names for files are not shown in the audit output; only file descriptors are recorded. To limit the amount of data collected during the auditing operation, use the -c option on the auditstream command to select a specific class of events as defined in the config file, or use the auditselect command to select specific events. For example: NOTE: This command must be all on one line in the streamcmds file. This command will collect only FILE_Open event records.

/usr/sbin/auditstream | /usr/sbin/auditselect -e "event == FILE_Open" | auditpr -v > /audit/stream.out & NOTE: The following command will limit data collection to only the TCPIP class of events as defined in the config file. /usr/sbin/auditstream -c tcpip | auditpr -v > /audit/stream.out & The bincmds file contains commands that are entered whenever a BIN file fills or when auditing is shut down. The file distributed reads like the following: /usr/sbin/auditcat -p -o $trail $bin The environment variables in the preceding command are defined while auditing is active. The auditselect command can be added to select specific events, reducing the amount of audit records. The bincmds file will only collect audit records that match USER_SU or USER_Login audit events. Enter: /usr/sbin/auditselect -e "event== USER_SU || event== \ USER_Login" $bin > /audit/trail.login -----------------------------------------------------------------------------Auditing a User For example: 1. To audit classes, use the fastpath command smit chuser. *User NAME AUDIT classes

[joe] [general, files]

A user stanza should be displayed for joe in /etc/security/audit/config file. 2. At the command line, edit the /etc/security/config file. In the classes stanza, add the following new class: procmon = PROC_Create, PROC_Delete, PROC_Execute In the users stanza, the following could exist: joe = procmon The newly assigned audit classes will take effect at the next login for user joe. 3. The BIN mode audit trail can be read with the following:

auditpr -v < /audit/trail | more The STREAM mode audit file /audit/stream.out can be viewed directly. Remember that the /audit/stream.out file is rewritten each time the auditing subsystem is started. Save the old stream.out before starting auditing. If you do not want the objects audit records when auditing a user ID, comment out the objects defined in the /etc/security/audit/objects file or rename this file. -----------------------------------------------------------------------------Auditing an Object In the following example, all processes writing to the /etc/utmp file will be audited. 1. Edit the /etc/security/audit/objects file to add the following: /etc/utmp: w = "UTMP_WRITE" 2. Edit the /etc/security/audit/events file to include the following: * /etc/utmp UTMP_WRITE = printf " %s " The audit record is displayed as follows: UTMP_WRITE

root

OK

Wed Jul 12 12:12:25 1995

init

In this case, the init process owned by root wrote to the file. NOTE: The length of an audit event or object name cannot exceed 15 characters. This limit is defined in the header file /usr/include/sys/audit.h, ah_event [16]. The following error message usually indicates an invalid event or object name. "auditevents (): Invalid argument" -----------------------------------------------------------------------------Disk Space Consideration Each record in the audit trail takes about 50 to 150 bytes depending on what mode is used and whether the verbose mode flag is specified. This means

that 1MB of data could contain about 6800 entries. -----------------------------------------------------------------------------Understanding the Output It is important to specify what information should be reviewed while auditing. Although you can configure auditing to record events of interest, there may still be too much data to be useful when viewed all at once. The auditselect command can be used with auditpr to sort through volumes of information and pull out only that which is needed for a specific report. It can be used to pull all data from a specific time period, for a specific user, or for a specific event, or any combination of these three. For example: /usr/sbin/auditselect -f /audit/pick \ /audit/trail | /usr/sbin/auditpr -v The /audit/pick file reads as follows: command == rlogin && \ time >= 08:00:00 && time <= 17:00:00 && \ data >= 04/01/96 && date <= 04/12/96 This command reports the use of the rlogin command within the specific time interval (8AM-5PM between April 1 and April 12). The compressed trail data from the binmode auditing is not in the same format at AIX version 3.2 as it is in 4.1 or later. There is a utility to convert the data from a pre-AIX Version 4 format to the Version 4 format. It is a command called auditconv. -----------------------------------------------------------------------------Common Problems with Auditing Errors When Starting Audit o There are certain errors that appear when running audit start. Error Message: ** failed setting kernel audit objects This occurs when there is a syntax error in the /etc/security/audit/objects

file. o Error Message: auditbin: ** failed backend command /etc/auditcat -p -o /audit/trail -r /audit/bin1 This error can be corrected by removing or renaming the BIN files. It is sometimes helpful to run audit shutdown again and then to retry audit start. o Error in config file: It is necessary to have the "user" stanza in the /etc/security/audit/ configuration file or the following error will display when you start auditing: Unable to find the user stanza in /etc/security/audit/config If it is not obvious that the user stanza is missing, verify that each of the classes are defined on a single continuous line. -----------------------------------------------------------------------------Data Overload Given the way that cron and the TCPIP code is written, each sets up its own set of audit events. These events will get written into the audit trail regardless of how the config files are set up. The workaround is to use auditselect to exclude these events when generating the audit report. TCPIP sessions, ftpd, rexecd, and telnetd all call auditproc() to set up process auditing using the class tcpip in /etc/security/audit/config. The same thing is done in the cron code (at, cron, and cronadm) for the cron class in /etc/security/audit/config. These events will be written into the audit trail. The best thing to do is to filter them using auditselect. For example: auditselect -e"event!=AT_JobAdd && event!=AT_JobRemove && ..." This will exclude events AT_JobAdd and At_JobRemove and so on. Or select on the command name: auditselect -e"command!=cron && command!=at && ..."

This excludes recording related to commands, at, cron, and so on. -----------------------------------------------------------------------------Fixes for Auditing Subsystem As defects arise, updates to the AIX audit code are made. Techdocs Ref:90605195614808

4FAX Ref:4824

Setup of System Accounting in AIX 3.2 [acct.setup.32.cmd] Setup of System Accounting in AIX 3.2 -----------------------------------------------------------------------------Contents About This Document Related Documentation What to Install to Get Accounting Working on a System Steps for Modifying the Root User Profile Steps for Setting Up System Accounting Fields of the Daily Usage Report Fixes and Problems Diagnosing Problems -----------------------------------------------------------------------------About This Document This document describes the accounting setup procedures for AIX Version 3.2. The setup of InfoExplorer some errors. and provides the 3.2.

system accounting as documented in the 3.2 AIX has This document describes the Version 3.2 setup procedures corrections to enable system accounting under AIX version

Access to the InfoExplorer and system documentation for AIX Version 3.2 require the setup person to be root. In this document, root user will be used for setting up system accounting and the user adm for running the reports. Modify the root user .profile to provide access to the required executables. Modification of the root profile is optional, but will probably help with administration.

Related Documentation InfoExplorer -----------------------------------------------------------------------------What to Install to Get Accounting Working on a System Make sure the following product is installed: bosext2.acct.obj - 'lslpp -l bosext2.acct.obj' This command should report the state as COMMITTED or APPLIED if the fileset is installed. To get the latest level accounting software, refer to the section "Fixes and Problems" later in this document. -----------------------------------------------------------------------------Steps for Modifying the Root User Profile 1. Log in as root. 2. Edit the .profile file with your favorite text editor. 3. Locate the PATH statement in the .profile file. It must include the /usr/sbin/acct and /var/adm/acct directories as shown in the following example: PATH= /usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin: /usr/sbin/acct:/var/adm/acct ; export PATH 4. Locate the export statement in the .profile. If the PATH statement given above was not used, the PATH variable must be included. -----------------------------------------------------------------------------Steps for Setting Up System Accounting 1. Log in to the system as root user. As described previously, root is the accounting administrator. 2. Enter: su - adm -c /usr/lib/acct/nulladm

/var/adm/wtmp

/var/adm/pacct

This command ensures correct permissions and provides access to the wtmp and pacct files. 3. Update the /etc/acct/holidays file for the current year using a favorite text editor. The following notes will help clarify the format of this

file: o An asterisk (*) in column 1 denotes a comment. o Define prime time on the first data line (the first line that is not a comment) using a 24-hour clock. The line will consist of three 4-digit fields, in the following order: - current year - when (hhmm) prime time begins - when (hhmm) prime time ends Leading blanks are ignored. Midnight can be entered as either 0000 or

2400.

For example, to specify the year 1992 with prime time beginning at 8:00 a.m. and ending at 5:00 p.m., specify: 1992

0800

1700

o Define the company holidays for the year on the next data line. Each line contains four fields in the following order: - day of the year - month - day of the month - description of the holiday The day-of-the-year field contains the number of the day of the year in which the holiday falls. It must be a number from 1 through 365 (366 on a leap year like 1992). 4. The file systems that will be included for accounting information need to be configured in the /etc/filesystems file. For each file system that will be included, add the following information to its stanza: account = true 5. Indicate the data file that will be used for printer data by adding the following line to the /etc/qconfig stanza for that printer: acctfile = /var/adm/qacct This entry must be added to the queue stanza and not the device stanza. If the entry is added by editing the /etc/qconfig file, enter the following command to re-digest the qconfig file: enq -d Another method of adding this entry to the qconfig file would be to enter the command smitty chque. Then follow the next steps:

1. Select queue name. 2. Enter /var/adm/qacct for the ACCOUNTING FILE pathname. These steps will update the appropriate stanza in /etc/qconfig file and will re-digest the qconfig file. NOTE: The printer accounting will not record usage for network printers, transparent printers, or PostScript printers. In addition, all print queues must use the same accounting file: /var/adm/qacct The pac command (printer accounting command) is more versatile because it allows separate accounting files for each printer. For more information, see InfoExplorer. 6. Create directories /var/adm/acct/nite, /var/adm/acct/fiscal, and /var/adm/acct/sum if they do not already exist. These directories should be owned by the user adm with a group ID of "adm". Use the following series of commands to create these directories: su - adm cd /var/adm/acct mkdir nite fiscal sum chown adm:adm nite fiscal sum chmod 755 nite fiscal sum exit 7. Modify the crontabs file for the adm user for automated accumulation of accounting data. The crontabs file for the adm user is in /var/spool/cron/crontabs/adm. To edit the adm crontab, log on as root and enter: su - adm -c crontab -e This will bring up the adm crontabfile in a vi session. You may need to uncomment or add some entries, and times may need to be modified to suit the installation's time requirements. An example of the entries needed is as follows: 10 23 * * 0-6 /usr/lib/acct/runacct 2> \ /usr/adm/acct/nite/accterr > /dev/null 0 23 * * 0-6 /usr/lib/acct/dodisk > /dev/null 2>&1 0 * * * * /usr/lib/acct/ckpacct > /dev/null 2>&1 15 4 1 * * /usr/lib/acct/monacct > /dev/null 2>&1 o The first line starts runacct at 11:10 pm (10 23), each Sunday

through

Saturday (0-6). runacct runs the daily reporting. o The second line starts disk accounting at 11:00 pm (0 23), each Sunday through Saturday (0-6), before starting runacct. o The third line checks the /var/adm/pacct file to ensure that it does not get too large at 0 minutes past each hour (0 *) every day (*). If the free space in /var falls below 500 blocks, then ckpacct

turns

off accounting until space is made available. A loss of accounting data will result during the period that accounting is turned off. ckpacct will turn accounting on again when more space is available. THERE IS NO NOTIFICATION unless the MAILCOM variable is set as follows: MAILCOM="mail root adm" The variable can be set in the ckpacct and runacct scripts or in the

/etc/environment file. If MAILCOM is set in both places the setting in ckpacct and runacct will be used. o The fourth line runs the monthly accounting reports at 4:15 am on the first of every month. 8. Turn on process accounting by adding the following line to the /etc/rc file. /usr/bin/su - root -c /usr/sbin/acct/startup The startup command records the time that accounting was turned on and

cleans up the previous day's accounting files.

If the system is not going to be rebooted at this time, run the preceding startup command from the root command line to start process accounting. -----------------------------------------------------------------------------Fields of the Daily Usage Report The meaning of the fields in the accounting Daily Usage Report are as follows: UID User ID number LOGIN NAME

Login name of user CPU PRIME Cumulative CPU minutes during prime hours CPU NPRIME During non-prime hours KCORE PRIME Cumulative minutes spent in the kernel during prime hours KCORE NPRIME During non-prime hours BLKIO PRIME Cumulative blocks transferred during prime hours BLKIO NPRIME During non-prime hours RW/WR PRIME Cumulative blocks read/written during prime hours RW/WR NPRIME During non-prime hours CONNECT PRIME Cumulative connect time (minutes) during prime hours CONNECT NPRIME During non-prime hours DISK BLOCKS Cumulative disk usage PRINT Queuing system charges (pages) FEES Fee for special services # OF PROCS Count of processes # OF SESS Count of login sessions # OF SAMPLES Count of count of disk samples -----------------------------------------------------------------------------Fixes and Problems Fixes for AIX Version 3.2.5 and later can be downloaded via the Internet with the FixDist service. On very large systems, if there are over 1000 users in the /etc/passwd file, the /usr/sbin/acct/dodisk script must be changed. At line 136 in dodisk, the diskusg command must be changed to read as follows: diskusg -Unnnnn $args > dtmp nnnnn is the new maximum number of users for which disk accounting should be done. (When the maximum number is too small, the dodisk script will return an error message that tells the user to use -uxxxx. It should say -Uxxxx.)

On AIX 3.2.5 and Greater o IX38748 corrects the problem of a user missing the output from individual users for the CPU and DASD output. o IX39408 corrects the problem of process accounting failing on a diskless workstation. o IX42292 corrects the problem of the runacct account command truncating the wtmp file during its nightly run, causing connection accounting to be lost for users who do not log off. o IX40232 corrects the problem of daily and monthly per-user memory accounting being off by a factor of exactly 200. o IX42322 corrects the problem of a 64MB limit on system process accounting. With this problem, any process that used more than 64MB of memory caused an overflow, and the process size reported by the ac_mem field was inaccurate. o IX43161 corrects the problem of acctprc2 core dumping with more than 500 users. o IX41228 corrects the problem of acctdisk entering into an infinite loop if the input file is corrupted. o IX40305 corrects the problem of the /var/adm/acct/sum/login log file not being updated in 3.2.5. -----------------------------------------------------------------------------Diagnosing Problems General Information Needed o Which accounting command is being used? acctcms acctcom acctcon acctdisk

acctmerg acctprc acctprcl acctprc2

All of these accounting commands accept input from standard input (as in acctcms < /usr/adm/pacct) and redirect to standard output (as in acctcmd < /usr/adm/pacct > /tmp/report). Find out which accounting file is being used as standard input and where the output is being directed by entering: acct_cmd < In_file > Out_file o Defaults are standard in and standard out. Exact syntax is very

important. o How is accounting started, via cron or via command line?

the

If accounting is started via cron, note the crontab file entries. If accounting is run from root, then the entry is in /usr/spool/cron/crontabs/root. If accounting is run from adm, then entry is in /usr/spool/cron/crontabs/adm.

When Accounting Fails 1. Check to see what state accounting is in. Look at the /usr/adm/acct/nite/active file. States are as follows: setup wtmpfix connect1 connect2 process merge fees disk queueacct mergetacct cms userexit cleanup 2. Check accterr for additional messages. 3. Refer to section "How To Fix General Accounting Problems" in InfoExplorer. 4. Restart runaccount. Other Items to Check o Is the /usr directory out of space? o Does wtmp have records with inconsistent data stamps? o Is cron failing? (Check mail for root and adm.) Techdocs Ref:90605194214786

4FAX Ref:2486