Build Test Debug

airNgine Bug Fixing Document version: 1 Document date: November 30, 2006 Reference: http://aireka/C15/ Commercial in confidence

http://www.airwidesolutions.com/

airwide solutions Montreal

airwide solutions UK

1111 St-Charles Street West 8th floor, East Tower Longueuil, Quebec J4K 5G4 Canada

New Century Place 20 – 22 Queens Road Reading, Berkshire RG1 4AU United Kingdom

phone: +1 450 646 7700 toll free: +1 800 524 8792 fax: +1 450 646 7004

phone: +44 118 963 7272 fax: +44 118 964 2369

airwide solutions Ottawa

airwide solutions Australia

515 Legget Drive Suite 700 Kanata, Ontario K2K 3G4 Canada

Level 2, 173 Pacific Highway North Sydney, NSW 2060 Australia

phone: +1 613 270 0364 fax: +1 613 270 0497

phone: +61 2 99 29 41 11 fax: +61 2 99 29 41 99

Every reasonable effort has been made to ensure the information and procedures detailed in this guide are complete and accurate at the time of printing. However, information contained in this guide is subject to change without notice. © Copyright of airwide solutions 2005. All rights reserved. The copyright in this work is vested in airwide solutions and the information contained herein is confidential. This work (either in whole or in part) must not be modified, reproduced, disclosed or disseminated to others or used for purposes other than that for which it is supplied, without the prior written permission of airwide solutions. If this work (or any part of it) is provided to a party ("Other Party") under a contract between airwide solutions and the Other Party, then the use of the work by the Other Party shall be governed by the provisions of the contract.

airNgine

Bug FixingBu

Contents Contents....................................................................................................................................................3 1 Build Procedure.......................................................................................................................................4 1.1 Complete [Full] build........................................................................................................................4 1.1.1 Build Procedure for AirNgine 2.1.........................................................................................4 1.1.2 Build Procedure for airNgine media 3.0...............................................................................5 1.1.3 Build Procedure for airNgine push 3.0 ................................................................................6 1.2 Single module build.........................................................................................................................7 1.3 Purify build.......................................................................................................................................7 1.4 KB references..................................................................................................................................8 2 Debugging Procedure – DBX & Workshop ............................................................................................9 2.1 Appropriate UNIX Box for core debug:............................................................................................9 2.2 Debugging steps:.............................................................................................................................9 2.2.1 Mount the Ottawa Solaris 9 loadbuild directory via NFS: ....................................................9 2.2.2 Open Sun Workshop: .........................................................................................................9 2.2.3 Path Map Adjustment:.......................................................................................................10 2.2.4 Open the core for analysis ................................................................................................11 3 Testing Procedure..................................................................................................................................12 3.1 Air'n'gine Media 3.0 nTelos MMSC replica: 12 4 CR [ClearQuest] Procedure...................................................................................................................15 4.1 Stage description...........................................................................................................................15 4.2 Flow diagram.................................................................................................................................16 5 Heat Ticket Procedure...........................................................................................................................17 5.1 Heat ticket guidelines....................................................................................................................17 5.2 Flow diagram.................................................................................................................................18 6 Whom To Contact..................................................................................................................................19 6.1 Air’n’gine........................................................................................................................................19 6.2 Airgate...........................................................................................................................................19 6.3 MGA..............................................................................................................................................19 20

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

3

Bug FixingBu

1

Build Procedure This section describes the build procedure for various versions of airNgine. Build is made either in debug mode or in no-debug mode.

1.1

Complete [Full] build Complete build for airNgine is made when changes are made in common components. Source files of common components are used across the different modules in airNgine. CVSROOT is set to your local or remote repository as normal. Note: Be sure to put your own username for setting the CVSROOT environment variable where the example below contains sample username.

1.1.1

Build Procedure for AirNgine 2.1 Customer: WIN UK Branch: rel21w_branch Code location: /loadbuild on Solaris version: Solaris 8 Official Build Server: netra3 Build setup replication Server: vulcan Official Test Server: ultra3 (aeuser/aeuser) Checkout commands: Run the following commands from the directory where you want airNgine/src directory cvs co –r rel21w_branch airNgine/src cvs co –r rel21w_branch airNgine/preinstall Build commands: Execute the following commands in sequence to make a debug build of airNgine2.1. This will create a directory called msg/ under preinstall/ Note: To make no-debug build, specify CC_NODEBUG=true flag. airNgine/src>make clean airNgine/src>make CC_NODEBUG=true clean airNgine/src>make idl airNgine/src>make library airNgine/src>make airNgine/src>cd ../preinstall airNgine/preinstall>make Tar Commands: Tar up the msg directory making sure to include complete files not just the symbolic links cd /cvs/airNgine/preinstall/msg tar -cvfh msg_ae_media30_CR_xxxx_fix.tar * gzip msg_ae_media30_CR_xxxx.tar

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

4

Bug FixingBu

The following show how to compile/build the TMA. Go in tma/com/airwide/tma >./CompileFiles

then , in tma/ >./CreateJar >./StopTomcat >./StartTomcat

That's it !

Profile in build server: Set the following environment variables in the profile file as per the shell being used. CVSROOT=:pserver:[email protected]:/cvsroot CVSUMASK=007 LD_LIBRARY_PATH=/usr/lib/sparcv9/omni:/usr/lib/omni: /loadbuild/rel21w_branch/airNgine/3rdPartyDvlp/lib PATH=$PATH:/usr/local/bin export CVSROOT CVSUMASK PATH PS1 1.1.2

Build Procedure for airNgine media 3.0 Customer: NTELOS Branch: ae_media30 Code location: /loadbuild on Solaris version: Solaris 9 Official Build Server: homer Official Test Server: rocco1 (aentelos/aentelos) Temporary build server: tiger1 Checkout commands: Run this command from the directory where you want airNgine/src directory cvs co –r ae_media30 airNgine/src cvs co –r ae_media30 airNgine/preinstall Build commands: Execute the following commands in sequence to make a debug build of ae_media30. This will create a directory called msg/ under preinstall/ Note: To make no-debug build, specify CC_NODEBUG=true flag. airNgine/src>make clean airNgine/src>make CC_NODEBUG=true clean airNgine/src>make idl airNgine/src>make library airNgine/src>make airNgine/src>cd ../preinstall airNgine/preinstall>make Tar commands: Tar up the msg directory making sure to include complete files not just the symbolic links cd /cvs/airNgine/preinstall/msg tar -cvfh msg_ae_media30_CR_xxxx_fix.tar * gzip msg_ae_media30_CR_xxxx.tar

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

5

Bug FixingBu

Profile in build server: Set the following environment variables in the profile file as per the shell being used. CVSROOT=:pserver:[email protected]:/cvsroot CVSUMASK=007 LD_LIBRARY_PATH=/usr/lib/sparcv9/omni:/usr/lib/omni:/loadbuild/ae_media30 /airNgine/3rdPartyDvlp-solaris-9/lib PATH=$PATH:/usr/local/bin export CVSROOT CVSUMASK PATH PS1 1.1.3

Build Procedure for airNgine push 3.0 Customer: Vodafone Australia Branch: ae_push30 Solaris version: Solaris 9 Code location: /loadbuild on Official Build Server: homer Official Test Server: seve2 (aeuser/aeuser123) Checkout commands: Run the following commands from the directory where you want airNgine/src directory cvs co –r ae_push30 airNgine/src cvs co –r ae_push30 airNgine/preinstall Build commands: Execute the following commands in sequence to make a debug build of ae_media30. This will create a directory called msg/ under preinstall/ Note: To make no-debug build, specify CC_NODEBUG=true flag. airNgine/src>make clean airNgine/src>make CC_NODEBUG=true clean airNgine/src>make idl airNgine/src>make library airNgine/src>make airNgine/src>cd ../preinstall airNgine/preinstall>make Tar Commands: Tar up the msg directory making sure to include complete files not just the symbolic links cd /cvs/airNgine/preinstall/msg tar -cvfh msg_ae_media30_CR_xxxx_fix.tar * gzip msg_ae_media30_CR_xxxx.tar Profile in build server: Set the following environment variables in the profile file as per the shell being used. CVSROOT=:pserver:[email protected]:/cvsroot CVSUMASK=007 LD_LIBRARY_PATH=/usr/lib/sparcv9/omni:/usr/lib/omni:/loadbuild/ae_push30 /airNgine/3rdPartyDvlp-solaris-9/lib PATH=$PATH:/usr/local/bin

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

6

Bug FixingBu

export CVSROOT CVSUMASK PATH PS1

1.2

Single module build Build for a single component is made only when the changed component is not common module. Keep the same profile as mentioned above for the respective version of airNgine. Checkout commands: Run this command from the directory where you want airNgine/src/<module-Name> directory cvs co –r ae_push30 airNgine/src/<module-Name> Copy the Makefile.defaults from airNgine/src directory under /loadbuild of the respective version of airNgine. Build commands: Execute the following commands in sequence to make a debug build of the single module. Note: To make no-debug build, specify CC_NODEBUG=true flag. Go to ….airNgine/src/<module-Name> directory where you checked out this module. airNgine/src>make

1.3

Purify build Checkout commands: Same as mentioned above for different versions of airNgine. Build Commands: Execute the following commands in sequence to make a debug build of different versions of airNgine. This will create a directory called msg/ under preinstall/ Note: To make no-debug build, specify CC_NODEBUG=true flag. airNgine/src>make clean airNgine/src>make CC_NODEBUG=true clean airNgine/src>make idl airNgine/src>make CC_PURIFY=true library airNgine/src>make CC_PURIFY=true airNgine/src>cd ../preinstall airNgine/preinstall>make Tar commands: Same as mentioned above for different versions of airNgine. Profile in build server: export AE=/opt/airNgine/latest_ntelos/msg PATH = /usr/local/bin:/usr/bin: /usr/sbin:/usr/openwin/bin:/usr/dt/bin: /usr/ccs/bin: /usr/ucb/bin: .: ${AE} scripts: /opt/SUNWspro/bin export PATH

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

7

Bug FixingBu

. /opt/airNgine/latest_ntelos/msg/scripts/pminit.env export CVSROOT=:pserver:[email protected]:/cvsroot export CVSUMASK=007 export LD_LIBRARY_PATH=/loadbuild/ae_media30/airNgine/3rdPartyDvlp/lib: /lib:/usr/lib:/usr/local/lib: /opt/rational/releases/purify.sol.2003a.06.15.FixPack.0188/lib32 export LD_RUN_PATH=$LD_LIBRARY_PATH . /opt/rational/purifyplus_setup.sh 1.4

KB references The following documentation and links can help the reader understand this document: [1]

http://airbb/KnowledgeBase/viewtopic.php?t=760

[2]

http://airbb/KnowledgeBase/viewtopic.php?t=653&highlight=test

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

8

Bug FixingBu

2

Debugging Procedure – DBX & Workshop This section is tells how to debug a core file on Montreal servers from IDC itself. This section describes what needs to be done to open a core file [nTelos (ae_media 3.0 is taken for example here).

2.1

Appropriate UNIX Box for core debug: Transfer the core and executable file to a server running a later version of Solaris, so that the system libraries are the same as the customer system. You may get warnings about /usr/lib library ELF checksums otherwise. [Analyze the core on box running with same version of UNIX and UNIX updates as UNIX box at customer site] For Ex for nTelos: Machine used: rocco1 Account used: aentelos Directory: /global/support/cores_ae_media30/

2.2

Debugging steps: Please follow the following steps to debug the core file.

2.2.1

Mount the Ottawa Solaris 9 loadbuild directory via NFS: (Only if not already mounted!) As root: • Create the mount point: mkdir /ae_sol9_loadbuild • Edit /etc/vfstab and add the following: # Ottawa Solaris 9 loadbuild homer:/loadbuild - /ae_sol9_loadbuild nfs - yes proto=udp,intr,hard,bg • Mount the NFS volume in read-only mode: mount -F nfs -o ro=homer:/loadbuild /ae_sol9_loadbuild

2.2.2

Open Sun Workshop: • • • • • • • • • •

Login to gurux100 IDC Unix box using its userId/passwd. Set display by using your windows IP. Run xhost + command on command prompt on gurux100 []. Login to rooc1 as aentelos user, Run “/usr/dt/bin/ttsession -p -c ./workshop &” From the Windows menu of Workshop window, select Show Debugging Window. From the Windows menu of the Debugging window, select Session Manager. From the Session Manager window, select New Session. In the Debugging window you should see the (dbx) prompt appear in the bottom panel. At the dbx prompt type the command: dbxenv core_lo_pathmap on This command instructs dbx to use its pathmap to look for libraries that the core file makes reference to.

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

9

Bug FixingBu

2.2.3

Path Map Adjustment: Use the pathmap command to tell dbx where the correct libraries for the core file are located. Sometimes a core file is created on one system (the core-host) and you want to load the core file on another machine (the dbx-host) to debug it. However, the following problems with libraries may arise when you do so: The shared libraries used by the program on the core-host won't be the same libraries as those on the dbx-host. To get proper stack traces involving the libraries, you'll want to make these original libraries available on the dbx-host. dbx uses system libraries in /usr/lib to help understand the implementation details of the run time linker and threads library on the system. It may also be necessary to provide these system libraries from the core-host so that dbx can understand the runtime linker data structures and the threads data structures. The user libraries and system libraries can change in patches as well as major Solaris upgrades, so this problem can even occur on the same host, if, for example, a patch was installed after the core file was collected, but before running dbx on the core file. dbx may display one or more of the following error messages when you load a "mismatched" core file: dbx: core file read error: address 0xff3dd1bc not available dbx: warning: could not initialize librtld_db.so.1 -- trying libDP_rtld_db.so dbx: cannot get thread info for 1 -- generic libthread_db.so error dbx: attempt to fetch registers failed - stack corrupted dbx: read of registers from (0xff363430) failed -- debugger service failed Important: Example in Airwide – nTelos Scenario: Release build for nTelos customer is made on homer for nTelos and so nTelos customer server runs the same libraries as on homer. Core is debugged on rocco1 and when core is generated on nTelos server, same libraries and object files are needed at rocco1 as well to debug it. So root partition of the core-host [though homer is not core host but Homer has same libraries]has been exported over NFS and can be accessed on rocco1 [dbx-host machine]. If libraries were at <Path1 > at core host [nTelos server] and if we debug this core on rocco1 and libraries are accessible at <path2> on rocco1. Then tell the dbx to look at <path2> on rocco1 for the libraries it is searching in <Path1> pathmap <Path1> <Path2> You may have to set many pathmap commands for different warnings. Inside the core file there is an absolute path to where the libraries are located on the system where the core occurred ELF checksum warnings about the application libraries, one reason may be that the executable was compiled in debug but the libraries used were the nodebug version (as was the case for this example). Make sure to point to the correct libraries. pathmap /opt/airngine/rel3.0/3rdParty/lib /global/support/cores_ae_media30/ae_media30_3rdParty_lib

This command indicates to dbx where to find the airngine third party libraries. This is a local copy of the equivalent directory on server homer in Ottawa. The following will also work but will be much slower:

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

10

Bug FixingBu

pathmap /opt/airngine/rel3.0/3rdParty/lib /ae_sol9_loadbuild/ae_media30/airNgine/ preinstall/ 3rdParty/lib pathmap /loadbuild/ae_media30/airNgine/ae_media30_20051001 /ae_sol9_loadbuild/ae_media30/airNgine/ae_media30_20051001

This command indicates to dbx where to look for the source and object files. Note: if the core is from a no-debug (aka 'optimized') executable, you cannot use the source files. 2.2.4

Open the core for analysis From the Debugging window, select Debug menu, Load Core File. Select the correct executable and core file. Note: • The executable may not appear in the selection window if it lost executable permissions during transfer via FTP. • If there are any ELF checksum warnings or files cannot be found go back to step (2.2.2) and fix the pathmaps. • To check the ELF checksum of a library in a directory (to compare with that recorded in the core file, that you see in the warning), use the dump command. For example: /usr/ccs/bin/dump -Lv libaw_db_interface.so | grep CHECKSUM [10] CHECKSUM 0x18e2 ----but always under the operator’s full control.

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

11

Bug FixingBu

3

Testing Procedure

3.1

Air'n'gine Media 3.0 nTelos MMSC replica: Servers: rocco1/rocco3 (Sun cluster) account: aentelos/aentelos

Locations Use $AE for /opt/airNgine/latest_ntelos/msg/

For scripts: cd /opt/airNgine/latest_ntelos/msg/scripts/

Operation: start_ae/stop_ae this starts/stops the Air'n'gine as well as the OAM agent

For executables ( installed airNgine) cd /opt/airNgine/latest_ntelos/msg/bin

For configuration cd /opt/airNgine/latest_ntelos/msg/config

For Simulator tools /export/home/aentelos/TEST_TOOLS

CDRs: Found in /opt/airNgine/latest_ntelos/CDR Can use 'reader' utility to convert to text format from binary.

Logs: Log file for AirNgine: cd /var/log/ae_ntelos.log To run this log file with colorize to get more readable form use cd /opt/airNgine/latest_ntelos/msg/config $ oplog Or Tail –f ae_ntelos.log | $AE/run/ae-log-colorize.pl There is an alias called 'oplog' which allows to view the logs in real-time in colorized version (by event severity).

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

12

Bug FixingBu

Testing steps This is to test mm4 to mm1 scenario and it is a two step process. a) Installing build AirNgine Version on Test machine • Take the zipped tar file created in build procedure and ftp that into ftp rocco1 aentelos/aentelos cd /opt/airNgine/latest_ntelos/msg/patches/idc_support put • Unzip,untar it on rocco1 using cd /opt/airNgine/latest_ntelos/msg/patches/idc_support gunzip tar –xvf • Rename the lib,bin directory and copy them to $AE mv lib lib_idc mv bin bin_idc cp –r lib_idc $AE cp –r bin_idc $AE • Stop airNgine cd $AE/scripts stop_ae • Link lib and bin to installed lib_idc and bin_idc cd $AE rm bin lib ln –s lib_idc lib ln –s bin_idc bin • Start airNgine cd $AE/scripts start_ae b) Run Simulator Tools • For MM4 Message Injetion source $AE/scripts/pminit.env • Run Handset simulator cd /export/home/aentelos/TEST_TOOLS/handset_simulator [/home1/airngine] cd /home1/airngine/TEST_TOOLS/handset_simulator ./handset-simulator handsetSimulator.cfg • Run smsc simulator cd /export/home/aentelos/TEST_TOOLS/smsc_simulator cd /home1/airngine/TEST_TOOLS/smsc_simulator ./smsc_server -smpp-server-port 9213 -http-server-port 9313 -forward-wap -wap-serverhost rocco1 -wap-server-port 8888 ./smsc_server -smpp-server-port 9213 -http-server-port 9313 -forward-wap -wap-serverhost ggnss02 -wap-server-port 8888 • •

Execute “ kill –HUP <smpp_am process>” Run smtp-sink cd /export/home/aentelos/TEST_TOOLS/smtp_sink cd /home1/airngine/TEST_TOOLS/smtp-sink smtp-sink -v rocco1:2523 100 – rocco1

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

13

Bug FixingBu

•

smtp-sink -v ggnss021:2523 100 - idc Run mm4 message injection tool cd export/home/aentelos/TEST_TOOLS/mm4_msg_inject_scripts mm4_to_mm1_from_live_mmsc.sh

General: 

delrep_one.pl ultra3 22567 1

 ./vasp-server-verbose.pl 3080

 

vasp-server-verbose.pl 8090

    

MM7AMMOTask.cc:627 Couldn't map messageId=1 to ctag and xid

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

14

Bug FixingBu

4 4.1

CR [ClearQuest] Procedure Stage description Use windows client for ClearQuest to work to create/update/access CRs . 1. Create the CR for using ClearQuest ID/Passwd. Make the entries in the following fields:  Select the owner [User to whom the problem is assigned] from list.  Select the site as ‘GURGAON’ from the list.  Select appropriate ‘Type’ from the list.  Select appropriate ‘Severity’ from the list.  Select appropriate ‘Priority’ from the list.  Enter the appropriate text in Headline Edit Box.  Write the problem description in Description text area. 2. Change the stage of the created CR to ‘ToAnalyst’ [In Analysis] when started analyzing problem. 3. Change the stage of the created CR to ‘Analyse’ [Analyzed] when done with the analysis. 4. Change the stage of the created CR to ‘In Dev’ when started working on problem fix. 5. Change the stage of the created CR to ‘Resolved’ when completed the problem fix make the entries in Resolution/Notes tab about file changed and code change. 6. Change the stage of the created CR to ‘Closed’ when Heat ticket is closed.

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

15

Bug FixingBu

4.2

Flow diagram

Create CR

ToAnalyst

Analyse

InDev

Resolved

Closed

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

16

Bug FixingBu

5

Heat Ticket Procedure

5.1

Heat ticket guidelines • •

• •

When we receive or open a Heat ticket assignment, the initial Reason for Assignment will usually be Problem Investigation or Advice and Guidance. If in the course of your investigation you determine that a bug fix or CD is necessary, you should immediately create a new assignment to yourself with the Reason set to Bug Fix Required or Enhancement Request respectively. Right-click to Acknowledge the assignment. If you have committed a delivery date for this patch/enhancement to Operations, enter it in the Bug Fix Committed Release Date box. When the patch/enhancement is released, set the Status in the assignment to Completed and right-click to Resolve it.

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

17

Bug FixingBu

5.2

Flow diagram

Create an assignment and assign to IDC 3rd [1st line]

Acknowledge Heat Ticket

Analyze the problem

C Re ode qu Fix ire d?

Close the first assignment

Close the 1st assignment

Create a new assignment

Do required code fixes

Close the 2nd assignment

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

18

Bug FixingBu

6

Whom To Contact

6.1

Air’n’gine Jill Vieregge – Manager – R&D • Email: "VIEREGGE, Jill" <[email protected]> • SkypeId: ?? Tracee Smith – Team Lead – R&D • Email: "SMITH, Tracee" • Skype-Id: ?? Ken Kim – Architect – R&D • Email: "KIM, Ken" • Skype Id: ken_kim George MUNTEANU – Moving to R &D • Email: "MUNTEANU, George" • Skype Id: gmunteanu

6.2

Airgate. Jill Vieregge – Manager – R&D • Email: "VIEREGGE, Jill" <[email protected]> • SkypeId: ?? Tracee Smith – Team Lead – R&D • Email: "SMITH, Tracee" • Skype-Id: ?? Ken Kim – Architect – R&D • Email: "KIM, Ken" • Skype Id: ken_kim George MUNTEANU – Moving to R &D • Email: "MUNTEANU, George" • Skype Id: gmunteanu

6.3

MGA

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

19

Bug FixingBu

airwide solutions Commercial in confidence

http://aireka/C15/Document version: 1

20