Useful Unix Commands For Websphere

Useful UNIX commands for WebSphere & Portal Administrator These are some useful commands that i came across when administrating WebSphere Application Server or WebSphere Portal Server. Note these commands are specifically written for LINUX but most of them would for other UNIX platforms like Solaris, AIX and other flavours of LINUX like SUSE or REDHAT. Performance related Commands to monitor system performance ------------------------------------------------------------------------1) Find the process that uses most CPU ps -eo pcpu,pid,user,args | sort -k 1 -r | head -2 (eg) output of the above command when websphere server process is utilizing the most of the CPU

%CPU PID USER COMMAND 71.7 31237 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath /p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere /AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=54859 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv02/properties:/cust/IBM/WebSphere/AppSer

2) Find the last 10 process that use the most CPU ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10

%CPU PID USER COMMAND 2.1 6375 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath /p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServ status.socket=2132 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties 0.3 13886 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath /p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServ status.socket=25042 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/propertie 0.2 17271 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath /p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServ status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties 0.2 17871 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath /p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServ status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties 0.2 17713 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath /p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServ

status.socket=9140 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties 0.1 18499 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath /p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServ status.socket=11309 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/propertie 0.1 17504 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath /p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServ status.socket=8969 -classpath /cust/IBM/WebSphere/AppServer/profiles/Dmgr01/properties:/

3)Find cpu usage mpstat -P ALL or mpstat

Linux 2.4.21-27.ELsmp (machine1.www.com) 11:55:00 intr/s 11:55:00 184.92 11:55:00 123.76 11:55:00 61.16

AM

CPU

%user

AM

all

1.51

0.00

0.41

AM

0

1.50

0.00

AM

1

1.51

0.00

02/02/2008

%nice %system %iowait

%irq

%soft

%idle

0.04

0.01

0.09

97.94

0.41

0.04

0.01

0.11

97.93

0.42

0.04

0.00

0.07

97.95

4)Find CPU usage with the process listing in runtime and much more statistics in terms of memory, virtual memory, etc. top

12:00:20 up 118 days, 15:16, 2 users, load average: 0.07, 0.02, 0.00 572 processes: 570 sleeping, 2 running, 0 zombie, 0 stopped CPU states: cpu user nice system irq softirq iowait idle total 2.7% 0.0% 1.1% 0.0% 0.0% 0.0% 95.9% cpu00 1.9% 0.0% 0.3% 0.0% 0.0% 0.0% 97.6% cpu01 3.5% 0.0% 1.9% 0.1% 0.0% 0.0% 94.2% Mem: 11819592k av, 5088092k used, 6731500k free, 0k shrd, 295304k buff 3998132k active, 682312k inactive Swap: 2048276k av, 0k used, 2048276k free 1258592k cached PID USER COMMAND 17719 root 19358 root 17771 root

PRI

NI

16 16 15

0 0 0

SIZE

RSS SHARE STAT %CPU %MEM

306M 306M 2532 2532 306M 306M

3992 S 896 R 3992 S

0.7 0.5 0.4

2.6 0.0 2.6

TIME CPU 22:14 0:00 3:48

1 java 1 top 0 java

18058 root 1 root 2 root migration/0

16 15 RT

0 0 0

458M 458M 512 512 0 0

4032 S 452 S 0 SW

0.1 0.0 0.0

3.9 0.0 0.0

12:55 1:19 0:00

0 java 0 init 0

5)Continuous CPU usage monitoring and find the average CPU used. a) sar -u 2 5 b) sar -o output.file 12 8 >/dev/null 2>&1 & c) nohup sar -o output.file 12 8 >/dev/null 2>&1 &

Linux 2.4.21-27.ELsmp (machine1.web.com) 12:04:34 12:04:36 12:04:38 12:04:40 12:04:42 12:04:44 Average:

PM PM PM PM PM PM

CPU all all all all all all

%user 0.25 0.00 7.00 21.05 3.00 6.26

%nice 0.00 0.00 0.00 0.00 0.00 0.00

02/02/2008 %system 0.00 0.50 7.25 1.50 0.50 1.95

%iowait 0.00 0.25 0.00 0.00 0.00 0.05

%idle 99.75 99.25 85.75 77.44 96.50 91.74

Disk Usage related Commands to monitor Disk space -------------------------------------------------------1)Disk Usage on all mounts df -h -T

Filesystem /dev/sda1 /dev/sda7 /dev/sda6 none /dev/sda5 /dev/sda3

Type ext3 ext3 ext3 tmpfs ext3 ext3

Size 2.9G 57G 2.0G 5.7G 2.0G 2.0G

Used Avail Use% Mounted on 785M 2.0G 29% / 9.3G 45G 18% /cust 82M 1.8G 5% /lc 0 5.7G 0% /dev/shm 33M 1.8G 2% /tmp 64M 1.8G 4% /var

2)prints just one line with the total size of the directory, the below example show the size of the /usr/IBM/Websphere WebSphere Portal Server directory. du -ch | grep total

bash-2.05b# du -ch | grep total 6.3G total

Network related Commands ------------------------------------1) a) dig (domain information groper) - is a flexible tool for interrogating DNS name servers. b) nslookup - Queries a name server for a host or domain lookup. c) host - DNS lookup uility. The below example shows the information like CNAME , Aliases, ipaddres , etc about www.google.com a) dig

bash-2.05b# dig www.google.com ; <<>> DiG 9.2.4 <<>> www.google.com ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1088 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 7, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.google.com.

IN

A

;; ANSWER SECTION: www.google.com. www.l.google.com. www.l.google.com.

585119 69 69

IN IN IN

CNAME A A

www.l.google.com. 216.239.51.99 216.239.51.104

;; AUTHORITY SECTION: l.google.com. l.google.com. l.google.com. l.google.com. l.google.com. l.google.com. l.google.com.

60734 60734 60734 60734 60734 60734 60734

IN IN IN IN IN IN IN

NS NS NS NS NS NS NS

g.l.google.com. b.l.google.com. f.l.google.com. c.l.google.com. a.l.google.com. e.l.google.com. d.l.google.com.

;; ;; ;; ;;

Query time: 2 msec SERVER: 10.5.50.184#53(10.5.50.184) WHEN: Sun Feb 3 09:51:46 2008 MSG SIZE rcvd: 196

b) host

bash-2.05b# host www.google.com www.google.com is an alias for www.l.google.com. www.l.google.com has address 216.239.51.104 www.l.google.com has address 216.239.51.99

c) nslookup

bash-2.05b# nslookup www.google.com Server: 10.5.50.184 Address: 10.5.50.184#53 Non-authoritative answer: www.google.com canonical name = www.l.google.com. Name: www.l.google.com Address: 216.239.51.99 Name: www.l.google.com Address: 216.239.51.104

2) List all network interfaces, the below example list two interfaces eth0 and eth1 and a loopback. infconfig -a

bash-2.05b# ifconfig -a eth0 Link encap:Ethernet HWaddr 00:11:43:32:49:99 inet addr:10.2.150.74 Bcast:10.5.50.127 Mask:255.255.255.192 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:357854361 errors:0 dropped:0 overruns:0 frame:0 TX packets:383621806 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000

Mb)

RX bytes:3168219417 (3021.4 Mb)

TX bytes:3724305382 (3551.7

Interrupt:17 eth1

Link encap:Ethernet HWaddr 00:11:43:32:49:9A inet addr:10.5.154.138 Bcast:10.5.54.127 Mask:255.255.255.128 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:18 lo

Mb)

Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:92809439 errors:0 dropped:0 overruns:0 frame:0 TX packets:92809439 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1454141744 (1386.7 Mb) TX bytes:1454141744 (1386.7

3) To find more information about a network interface, the below example shows detail information about interface eth0. bash-2.05b# ethtool eth0

Settings for eth0: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: Not reported Advertised auto-negotiation: No Speed: 100Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: off Supports Wake-on: d Wake-on: d Link detected: yes

File related commands ---------------------1) find text in the matching files and show the filename with the pattern. The below example shows the list of filenames and the location of the hostname in those files, which will be helpful when changing hostnames for your websphere installation. find . -name '*.xml' -print | xargs grep 'hostName' /dev/null

bash-2.05b# find /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/ -name 'server*.x | more /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/por ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http .0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="portalserver.example.com" endPointR ndPoint_1130375480668 NamedEndPoint_1130375480669 NamedEndPoint_1130375480670 NamedEndPo 2 NamedEndPoint_1130375480673"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n20 serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serve er/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1130386859873" hostName="n2034indchn /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n20 serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serve er/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1130361145826" hostName="n2033indchn /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/por verindex:ServerIndex xmi:id="ServerIndex_1" endPointRefs="NamedEndPoint_1 NamedEndPoint_ /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n20 ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http .0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="n2011indchn2.example.com" endPointR ndPoint_1130375717655 NamedEndPoint_1130375717656 NamedEndPoint_1130375717657 NamedEndPo 9 NamedEndPoint_1130375717660"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n20 x:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="ht /5.0/serverindex.xmi" xmi:id="ServerIndex_1173388344070"hostName="n2033indchn2.example.c /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n20 x:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="ht /5.0/serverindex.xmi" xmi:id="ServerIndex_1173388421900" hostName="n2034indchn2.example. /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/templates/servertypes/APPLICATION_ ndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www. erindex.xmi" xmi:id="ServerIndex_1" hostName="$(node.host.name)"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/backup/base/cells/portalserverNode ml:<serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:s server/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="portalserver.exampl /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/backup/base/templates/servertypes/ ndex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex= mas/5.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="$(node.host.name)">

2) Soemtimes you might run out of diskspace and want to clean up some files, so this command will help you find all large files. In this example it will list all the files greater than 10 mb in

size. find / -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5

bash-2.05b# find /usr/IBM/WebSphere -xdev -size +1024 -exec ls -al {} \; | sort -r -k 51 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node -rw-r--r-1 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node -rw-r--r-1 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node

3) When sending SystemOut.log or SystemErr.log or trace.log to your developers or to IBM support Please use Tar and gzip on the fly to reduce the size of the file and the transfer time to FTP,email or http upload. To pack on the fly... tar -cvf - FILE-LIST | gzip -c > FILE.tar.gz To unpack on the fly.. gunzip < FILE.tar.gz | tar -xvf To pack on the fly and email the file to IBM Support tar -cvf - SystemOut.log | gzip -c | uuencode SystemOut.tar.gz | mail -s "PMR 09921,227,000" -b "SystemOut.log for server crash " [email protected] To find selective files and tar it up on the fly... find /opt/IBM/WebSphere/PortalServer/log -name 'System*.log'| xargs tar -rvf System_logs.tar 4) Sometimes you might encounter a situation for example when running xmlaccess or wsadmin where you might want to write stdout to the file as well as to the console so that you have a copy even when the standard output disappears because of the screen buffer size. ls -l | tee > filename xmlaccess.sh -in ExportRelease.xml -username wpsadmin -password wpsadmin -url http://localhost:9080/wps/config | tee filename ./wsadmin.sh | tee wsadmin.log 5) find files that is recently updated which might be useful to see what changes are made to the system recently (e.g) below command shows the files that are modified within the last 20 minutes(-mmin) and 20 days (-mtime) find /usr/IBM -mmin -20 find /usr/IBM -mtime -20

/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/dmgr/SystemOut.log /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/ffdc/dmgr_exception.log /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/activity.log /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer0.ser /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer1.ser /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer2.ser /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs

6) Find a memory usage of the system. free cat /proc/meminfo

bash-2.05b# free total cached Mem: 11819592 1440200 -/+ buffers/cache: Swap: 2048276

used

free

shared

buffers

4938416

6881176

0

298148

3200068 0

8619524 2048276

bash-2.05b# cat /proc/meminfo total: used: free: shared: buffers: cached: Mem: 12103262208 5056954368 7046307840 0 305303552 1474797568 Swap: 2097434624 0 2097434624 MemTotal: 11819592 kB MemFree: 6881160 kB MemShared: 0 kB Buffers: 298148 kB Cached: 1440232 kB SwapCached: 0 kB Active: 3718616 kB ActiveAnon: 2968776 kB ActiveCache: 749840 kB Inact_dirty: 789328 kB

Inact_laundry: 237596 Inact_clean: 0 Inact_target: 949108 HighTotal: 11140992 HighFree: 6681628 LowTotal: 678600 LowFree: 199532 SwapTotal: 2048276 SwapFree: 2048276 Committed_AS: 6039100 HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 2048

kB kB kB kB kB kB kB kB kB kB kB

4) To repeat a command contimuously. watch -d=10 -n 1 "netstat -a"

Every 1s: netstat -a| grep ESTABLISHED Sun Feb 3 21:48:04 2008 tcp 0 ESTABLISHED tcp 0 ESTABLISHED tcp 0 ESTABLISHED tcp 0 ESTABLISHED tcp 0 ESTABLISHED

0 n1212indchn2.www.com:9900 n1312indchn2.www.com:11530 0 n1212indchn2.www.com:9353 n1312indchn2.www.com:11549 0 n1212indchn2.www.com:9354 n1312indchn2.www.com:11550 0 n1212indchn2.www.com:9201 n1312indchn2.www.com:11544 0 n1212indchn2.www.com:9352 n1312indchn2.www.com:11545