Tools For Debugging_03

ES-eEnabling Utilities for 2005-2006 Strategy

Debugging in Unix

Srinivas Adyapak

Wipro confidential

1

Common Utilities for Developers in Unix •Common Utilities required by Developers to improve their code and enhance their productivity • These utilities will be available in most of the Linux installations. Debugging

 cscope, ctags – Code browsing  strace (truss in Solaris) – Debugging a running process  pstack - Stack trace of a running process  pmap - Memory used by a process  nm - Symbol names in an object or exe  ddd – A GUI for Command line debugging

Process & Task Management

 ps - Snapshot of process status  top - Display task details (usage of memory, cpu..)  time - Code performance  kill – terminate a process

File Management

 lsof - Open files and Processes opening the files  df, du – available disk space, disk usage

Networking

 netstat - displays the contents of various network related data structures  ifconfig – configuring a network device  ping - Sends ICMP ECHO_REQUEST packets to network hosts

• Refer to man pages for more details on each of these utilities Wipro confidential

2

Common Utilities for Developers in Unix

Other utilities

 sar – Displays the CPU activity  vmstat - reports virtual memory statistics of process, virtual memory, disk, trap, and CPU activity  iostat - reports terminal and disk I/O activity and CPU utilization

• Refer to man pages for more details on each of these utilities Wipro confidential

3

cscope • Instead of doing a “grep” on a source code of few million lines, it is easier to examine source code thro’ a code browsing tool such as cscope. •Useful in Code browsing – locating variables, function calls, functions, macros in files • Human-readable cross-reference can be got by for ctags eg. ctags -x *.c • Not limited to C –> C++, Python, Perl, hardware designed languages such as Verilog. • Builds the cross-references on all the source files to examine – includes .c, .h file extensions • cscope vsftpd-2.0.5/* - builds the cross-references for all files under the directory vsftpd-2.0.5 • cscope rebuilds the cross-reference only if a source file has changed or the list of source files is different. • Can directly navigate to the occurrence of the source. Wipro confidential

4

cscope 

Cscope version 15.5

          

Find this C symbol: Find this global definition: Find functions called by this function: Find functions calling this function: Find this text string: Change this text string: Find this egrep pattern: Find this file: Find files #including this file: Find all function definitions: Find all symbol assignments:

Press the ? key for help

Wipro confidential

5

cscope 

Functions called by this function: str_netfd_alloc

               

File Function 0 netstr.c bug 1 netstr.c str_empty 2 netstr.c vsf_sysutil_recv_peek 3 netstr.c vsf_sysutil_retval_is_error 4 netstr.c die 5 netstr.c die 6 netstr.c vsf_sysutil_read_loop 7 netstr.c vsf_sysutil_retval_is_error 8 netstr.c die 9 netstr.c die a netstr.c str_alloc_alt_term b netstr.c bug c netstr.c vsf_sysutil_read_loop d netstr.c vsf_sysutil_retval_is_error e netstr.c die

          

Find this C symbol: Find this global definition: Find functions called by this function: str_netfd_alloc Find functions calling this function: Find this text string: Change this text string: Find this egrep pattern: Find this file: Find files #including this file: Find all function definitions: Find all symbol assignments:

Line 31 bug("poor buffer accounting in str_netfd_alloc"); 36 str_empty(p_str); 39 retval = vsf_sysutil_recv_peek(fd, p_readpos, left); 40 if (vsf_sysutil_retval_is_error(retval)) 42 die("vsf_sysutil_recv_peek"); 46 die("vsf_sysutil_recv_peek: no data"); 55 retval = vsf_sysutil_read_loop(fd, p_readpos, i + 1); 56 if (vsf_sysutil_retval_is_error(retval) || 59 die("vsf_sysutil_read_loop"); 63 die("missing terminator in str_netfd_alloc"); 65 str_alloc_alt_term(p_str, p_readbuf, term); 72 bug("bytes_read > left in str_netfd_alloc"); 75 retval = vsf_sysutil_read_loop(fd, p_readpos, bytes_read); 76 if (vsf_sysutil_retval_is_error(retval) || 79 die("vsf_sysutil_read_loop");

Wipro confidential

6

cscope 

Functions calling this function: str_netfd_alloc

 

File Function Line 0 readwrite.c ftp_getline 94 str_netfd_alloc(

          

Find this C symbol: Find this global definition: Find functions called by this function: Find functions calling this function: str_netfd_alloc Find this text string: Change this text string: Find this egrep pattern: Find this file: Find files #including this file: Find all function definitions: Find all symbol assignments: Wipro confidential

7

strace  Traces the system calls used by program

• reports whether they pass or fail • It even can follow forks. 

Attach to processes and begins tracing

• strace also lets you attach to processes for just-in-time debugging. 

Lets you know what’s happening even if there is no debugger or source code



Useful in a live environment

Wipro confidential

8

strace  Each line in the trace contains the system call name, followed by its arguments in parentheses and its return value.

•

open("/dev/null", O_RDONLY) = 3

 Errors (typically a return value of -1) have the errno symbol and error string appended.

•

open("/foo/bar", O_RDONLY) = -1 ENOENT (No such file or directory)

 Usage

• • •

strace –c –p 2155 gives the count time, calls and errors for each system call and report summary for the process with pid 2155 strace –o strace.out –p 2155 strace <program>

Wipro confidential

9

strace 

For eg. If a normal user tries to delete the services file (owned by root)

•

strace rm /etc/services lstat64("/etc/services", {st_mode=S_IFREG|0644, st_size=362031, ...}) = 0 access("/etc/services", W_OK) = -1 EACCES (Permission denied) unlink("/etc/services") = -1 EACCES (Permission denied) write(2, "cannot remove `/etc/services\'", 29) = 29

• 

strace –o strace.out rm /etc/services will copy the output to strace.out

strace –c –p 4313 (gcalctool is a calculator started in Linux with pid 4313)

• After few seconds do a Ctrl-C -> with the following output % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------------------------------------------38.97 0.005311 19 285 poll 29.35 0.004000 68 59 gettimeofday 15.14 0.002063 172 12 writev 11.04 0.001504 7 213 26 read 5.51 0.000751 3 262 1 ioctl 0.00 0.000000 0 150 write 0.00 0.000000 0 1 uname ------ ----------- ----------- --------- --------- ------------------------------------------------------100.00 0.013629 982 27 total

Wipro confidential

10

pmap  pmap reports the memory map of a process  Amount of memory used by the process  Details of the map are also available in the /proc

• pmap –x 2352

2352: ./server Address Kbytes RSS Anon Locked Mode Mapping 00111000 1200 r-x-- libc-2.4.so 0023d000 12 r-x-- libc-2.4.so 00240000 4 rwx-- libc-2.4.so 00241000 12 rwx-- [ anon ] 006dc000 100 r-x-- ld-2.4.so 006f5000 4 r-x-- ld-2.4.so 006f6000 4 rwx-- ld-2.4.so 00769000 4 r-x-- [ anon ] 08048000 4 r-x-- server 08049000 4 rw--- server b7f5d000 4 rw--- [ anon ] b7f6e000 4 rw--- [ anon ] bfe59000 88 rw--- [ stack ] -------- ------- ------- ------- ---------------------------------------------------------------total kB 1444 -

Wipro confidential

11

pstack  pstack

• • •

prints the stack trace of the running process – quite useful along with strace. Attaches to the active process given on the command line The output is read bottom up.

Wipro confidential

12

nm  Used to retrieve information on symbol names inside an object file or executable file.  Helps in resolving problems due to name conflicts

•

If there are poorly defined headers tracking down the offending module becomes easier using nm.

Wipro confidential

13

ps  ps

• • •

powerful debugging tool Report a snapshot of the current processes use the –o option with ps and get many details such as virtual memory, cpu usage, current state and lot more • Most of the information that ps produces is available from the /proc filesystem ps –ef will list the following

UID root guest guest guest guest

PID PPID C STIME TTY 2184 2187 2188 2281 2352

1635 2184 2187 1 2188

0 09:00 ? 0 09:00 ? 0 09:01 pts/2 0 09:11 ? 0 09:19 pts/2

TIME 00:00:00 00:00:01 00:00:00 00:00:00 00:00:00

CMD sshd: guest [priv] sshd: guest@pts/2 -bash /usr/libexec/gam_server ./server

Wipro confidential

14

ps  ps ps -eo start,etime,state,pid,cmd STARTED 08:54:15 09:00:46 09:00:53 09:00:59 09:11:03 09:19:13

ELAPSED 27:26 S 20:55 S 20:48 S 20:42 S 10:38 S 02:28 S

S PID 1 2184 2187 2188 2281 2352

CMD init [5] sshd: guest [priv] sshd: guest@pts/2 -bash /usr/libexec/gam_server ./server

Wipro confidential

15

top 

top is a simple text based system monitoring tool



Displays system summary and details, status of individual tasks (memory, cpu, pid and more)



The top line prints the current time, uptime since the last reboot, users logged in, and the load average.



The memory line displays the

• •



Total physical RAM available on the system, Amount of usage, free, shared, along with the amount of ram in buffers.

Most importantly it displays per process the

• • •

Amount of physical memory consumed percentage of the available processor time a process is taking total amount of processor time the process has had Wipro confidential

16

top top - 10:57:09 up 2:02, 3 users, load average: 0.07, 0.12, 0.60 Tasks: 96 total, 1 running, 95 sleeping, 0 stopped, 0 zombie Cpu(s): 0.3% us, 2.4% sy, 0.0% ni, 97.2% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 126252k total, 118640k used, 7612k free, 4148k buffers Swap: 1048568k total, 16084k used, 1032484k free, 52500k cached PID

USER

22513 1 2 3 4 5 6 8

guest root root root root root root root

PR NI VIRT RES SHR S %CPU %MEM

16 16 34 RT 10 10 11 10

0 2124 0 1988 19 0 0 0 -5 0 -5 0 -5 0 -5 0

1008 796 R 652 564 S 0 0 S 0 0 S 0 0 S 0 0 S 0 0 S 0 0 S

1.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0

0.8 0.5 0.0 0.0 0.0 0.0 0.0 0.0

TIME+ COMMAND

0:05.98 0:03.00 0:00.00 0:00.02 0:00.09 0:00.04 0:00.00 0:00.29

top init ksoftirqd/0 watchdog/0 events/0 khelper kthread kblockd/0

Wipro confidential

17

lsof  Reports a list of all

• •

open files and processes that opened them.

 Open files in the system include

• • • •

Disk files Pipes Network sockets and Devices opened by all processes.

 glsof is a GUI for lsof

Wipro confidential

18

lsof  To list the internet ports/files opened by guest user • • COMMAND sshd sshd server server-test telnet

lsof –i | grep guest lsof –u guest –a -i

PID 2168 2605 3191 3198 3529

USER guest guest guest guest guest

FD 3u 3u 3u 0u 3u

TYPE DEVICE SIZE NODE NAME IPv4 7562 TCP 10.116.3.80:ssh->m2-14956.wipro.com:writesrv (ESTABLISHED) IPv4 9888 TCP 10.116.3.80:ssh->m2-14956.wipro.com:direcpc-si (ESTABLISHED) IPv4 12473 TCP *:8171 (LISTEN) IPv4 12508 TCP *:8181 (LISTEN) IPv4 16885 TCP 10.116.3.80:53112->efcg.wipro.com:telnet (ESTABLISHED)

 Examining a process (pid 3198)

• lsof –p 3198 COMMAND server-test server-test server-test server-test server-test server-test server-test

PID USER 3198 guest 3198 guest 3198 guest 3198 guest 3198 guest 3198 guest 3198 guest

FD cwd rtd txt mem mem mem 0u

TYPE DEVICE DIR 253,0 DIR 253,0 REG 253,0 REG 253,0 REG 253,0 REG 0,0 IPv4 12508

SIZE NODE NAME 4096 948823 /home/guest/testserver/webpages 4096 2 / 10971 948808 /home/guest/testserver/server-test 121396 1275458 /lib/ld-2.4.so 1528292 1275459 /lib/libc-2.4.so 0 [vdso] (stat: No such file or directory) TCP *:8181 (LISTEN)

Wipro confidential

19

lsof 

Sometimes you need to track down the user or process that's blocking you from unmounting a disk. • Umount /opt will give -> umount: /opt: device is busy • kill `lsof -t /opt` will kill all the users on the /opt



List all processes that have files open in /tmp •

lsof +D /tmp

Wipro confidential

20

netstat  Netstat gets the information about the status of network connections  Displays the kernel routing tables  Retrieve information on Network interfaces

•

netstat -nr

Kernel IP routing table Destination Gateway 10.116.3.0 0.0.0.0 169.254.0.0 0.0.0.0 0.0.0.0 10.116.3.1

Genmask Flags MSS Window irtt Iface 255.255.255.0 U 0 0 0 eth0 255.255.0.0 U 0 0 0 eth0 0.0.0.0 UG 0 0 0 eth0

Wipro confidential

21

netstat  Processes accepting connections for TCP

•

netstat -tpln

Proto Recv-Q Send-Q Local Address

tcp tcp tcp tcp tcp tcp

0 0 0 0 0 0

0 0.0.0.0:1353 0 0.0.0.0:111 0 127.0.0.1:50000 0 127.0.0.1:50002 0 0.0.0.0:52340 0 0.0.0.0:21

Foreign Address

0.0.0.0:* 0.0.0.0:* 0.0.0.0:* 0.0.0.0:* 0.0.0.0:* 0.0.0.0:*

State

LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN

PID/Program name

2352/server -

Wipro confidential

22

ifconfig  Allows to configure network interfaces (bring interfaces up or down)  Displays the currently active and inactive network interfaces.  Supports a variety of address families and hardware types

• Address famililes – inet (ipv4), inet6(ipv6), unix .. • Hardware types – ethernet, x.25, frame-relay .. • ifconfig eth0 - View the network settings on the first Ethernet adapter installed in the computer. eth0

Link encap:Ethernet HWaddr 00:03:FF:E3:08:1C inet addr:10.116.3.80 Bcast:10.116.3.255 Mask:255.255.255.0 inet6 addr: fe80::203:ffff:fee3:81c/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1640 errors:0 dropped:0 overruns:0 frame:0 TX packets:434 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:163039 (159.2 KiB) TX bytes:108893 (106.3 KiB) Interrupt:11 Base address:0x6000

Wipro confidential

23

sar     

Collect, report, or save system activity information. Displays the CPU activity Memory usage can be monitored Disk I/O activity Number of context switches - a good indication of how much time a process is wasting

•

sar –u -> cpu utilisation

Linux 2.6.15-1.2054_FC5 (localhost.localdomain) 10:10:02 AM 10:20:01 AM 10:30:02 AM Average:

CPU all all all

%user 14.20 16.73 15.47

12/05/2006

%nice %system %iowait %idle 22.05 57.01 6.74 0.00 21.21 59.62 2.44 0.00 21.63 58.32 4.59 0.00

Wipro confidential

24

sar 

sar –r -> prints the memory usage

10:10:02 AM kbmemfree kbmemused 10:20:01 AM 3500 122752 10:30:02 AM 2904 123348 Average: 3202 123050



sar –b -> prints the disk i/o

10:10:02 AM 10:20:01 AM 10:30:02 AM 10:40:01 AM Average:



%memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 97.23 4108 45604 1032052 16516 1.58 1952 97.70 2572 49116 1032052 16516 1.58 980 97.46 3340 47360 1032052 16516 1.58 1466

tps 52.00 24.33 22.05 33.22

rtps 34.31 10.30 11.81 19.08

wtps bread/s bwrtn/s 17.69 911.39 229.92 14.03 91.34 188.15 10.24 137.21 136.69 14.14 389.49 186.85

sar –w -> number of context switches/sec 10:10:02 AM cswch/s 10:20:01 AM 1749.10 10:30:02 AM 1997.61 10:40:01 AM 1269.78 Average: 1688.47

Wipro confidential

25

iostat  Reports the CPU statistics and Input/Output statistics for devices and partitions  Can be used with sar and vmstat  CPU utilization gives the percentage of idle time with/with out any outstanding I/Os requests (iowait, idle).  Blocks read and written is given for device utilization avg-cpu: %user %nice %system %iowait %idle 7.56 1.43 11.43 2.79 76.80 Device: hda dm-0 dm-1

tps Blk_read/s Blk_wrtn/s Blk_read 5.79 188.11 38.92 284468 11.79 186.26 38.90 281674 0.04 0.33 0.00 504

Blk_wrtn 58862 58832 0

Wipro confidential

26

ddd  DDD – Data Display Debugger is a popular GUI for command line debugger  Debugs executable binaries using GDB, DBX command line debuggers  Debugs programs in

• • •

Java (DDD with JDB Java debugger) Python (DDD with PYDB, Python debugger) Perl (Perl debugger)

 Advantages • User need not know much about the underlying command line debugger used with DDD • Available freely

Wipro confidential

27