Garbage Collection, Tuning And Monitoring Jvm In Ebs 11i And R12

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 Siddharth Gandhi Nov 2008

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

1

Contents Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12..........................1 Introduction.......................................................................................... ...................3 First Steps.......................................................................................................... ......4 Memory management.............................................................................................4 Garbage Collection Concepts..................................................................................5 Definition....................................................................................................... .......5 Generations................................................................................................. .........5 Types of Collectors...............................................................................................6 Measurement.......................................................................................................6 Few notable JVM parameters...................................................................................8 Monitor the JVM............................................................................. ........................10 Future changes - JRockit........................................................................................13 References and Quotes.......................................................................................... 14 Appendix.................................................................................... ...........................15

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

2

Introduction In many of our support tickets we might have encountered performance and memory issues with the middle-tier and the ubiquitous java.lang.OutOfMemoryError(s) statement in our stack trace. We might be encountering these performance issues for many reasons impacted by hardware (specifications, networking etc) or software (OS, application versions/patches/parameters) or just by external parameters like user behavior, current number of active users, bandwidth etc. In the Oracle Application ‘development’ space, plugging in these performance and memory issues will majorly boil down to JVM tuning, stopping memory leaks, JDBC optimizations and few others. In this whitepaper, we will brief upon

•

Garbage Collection,

•

Tuning JVM for better performance,

•

Understanding few of the JVM tuning parameters provided in Jserv.properties and opmn.xml file

•

Monitoring JVM

The document focuses only on the OACoreGroup JVM and not considers the JVM settings for other Oracle services like Forms & Reports, Discoverer, Configurator etc. Also assumed is that the reader has some familiarity with Java and E-Business Suite 11i and Release 12

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

3

First Steps Quoting Sun’s documentation, Out of Memory (OOM) errors are usually thrown… “… when the Java Virtual Machine cannot allocate an object because it is out of memory, and no more memory could be made available by the garbage collector.” And so follows the usual fix of increasing the JVM heap size, java -Xms1024M -Xmx1024M ….

But in this quick-fix world, we comfortably forget the following two important questions,

1. Why Java Virtual Machine is out of memory? 2. Why garbage collector is not able to make more memory available? Let’s remember the fact that physical memory is limited and on the other hand there are easier, faster and cheaper approaches to consider then upgrading the hardware. So before we proceed to tune our JVMs, let’s brush our basics for memory management and role of Garbage Collection (GC)

Memory management “Memory management is the process of recognizing when allocated objects are no longer needed, deallocating (freeing) the memory used by such objects, and making it available for subsequent allocation“ Explicit Vs. Automatic Memory Management Java employs automatic memory management unlike C and few other programming languages where memory management is developer responsibility. Explicit management many a times results into unexpected or erroneous program behavior and crashes. Whereas automatic management is handled by a program called a garbage collector (GC). GC automatically frees all memory no longer referenced. There are basically two ways you can alter the JVM’s behavior to improve performance. Firstly, you can influence memory use by changing how memory is allocated and organized by the JVM. Secondly, you can influence how garbage collection is performed.

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

4

Garbage Collection Concepts

Definition Objects that are referenced are said to be live. Objects that are no longer referenced are considered dead and are termed garbage. The process of finding and reclaiming the space used by these objects is known as garbage collection. The timing of garbage collection is up to the garbage collector.

Generations Memory in the Java HotSpot virtual machine is organized into three generations:

1. Young generation, 2. Old generation(or Tenured), and 3. Permanent generation (or Perm) This generational garbage collection, takes advantage of the observation that, most programs create: • •

Many objects that have short lives (for example, iterators and local variables). Some objects that have very long lifetimes (for example, high level persistent objects)

The young generation consists of an "eden space" and two "survivor spaces." The VM initially assigns all objects to the Eden space, and most objects die there. When the young generation area fills up, it causes Minor Garbage Collection. Any objects survived after Minor GC is moved to Tenured Generation. And when Tenured Generation area fills up it causes Major Garbage Collection i.e. Full GC. Major GC is slow and expensive as compared to Minor GC since it involves evaluating all live objects. The permanent generation holds objects that the JVM finds convenient to have the garbage collector manage, such as objects describing classes and methods, as well as the classes and methods themselves.

Figure 1

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

5

Types of Collectors To reclaim memory, HotSpot has four different collectors as of J2SE 5.0 (and the option to enable them): • • • •

The default or serial collector Throughput collector (-XX:+UseParallelGC) Concurrent low pause collector (-XX:+UseConcMarkSweepGC) Incremental collector (-XX:+UseTrainGC)

The default collector is a serial collector that pauses the application during minor and major collections. If the host machine has a single CPU, the serial collector will likely be as fast or faster as the other collectors. The throughput collector uses the same collection mechanism as the serial collector for major collections, but implements a parallel minor collector for the young generation space. On the other hand, the concurrent low-pause collector attempts to perform most of the work of a major collection without interrupting your application (pauses are low/short). Finally the incremental collector, by careful book-keeping, collects just a portion of the tenured generation at each minor collection, trying to spread the large pause of a major collection over many minor collections. However, the incremental collector is deprecated and will eventually be removed. Which collector works best for your application depends on the load of the application and the host system. However, unless you have a system with multiple CPUs, tuning memory configuration rather than garbage collection is likely to yield performance gains.

Measurement Before we consider the metrics, let’s understand various parameters, on which it’s evaluated. Throughput is the percentage of total time not spent in garbage collection, considered over long periods of time. Pauses are the times when an application appears unresponsive because garbage collection is occurring. Usually tolerable, but in an interactive graphics program even short pauses may negatively affect the user experience. Promptness is the time between when an object becomes dead and when the memory becomes available, an important consideration for distributed systems, including remote method invocation (RMI). Footprint is the working set of a process, measured in pages and cache lines. On systems with limited physical memory or many processes, footprint may dictate scalability.

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

6

Throughput and footprint are best measured using metrics particular to the application. On the other hand, pauses due to garbage collection are easily estimated by inspecting the diagnostic output of the virtual machine itself. Now let’s understand that how and what garbage collection details are generated in the E-Business Suite. Table 1 below briefs how and where to set the parameters in the appropriate file to generate the GC data and view it for specific E-Business Suite version Table 1 Suite

Generate the GC data

View the output {file location}

11i

Provide the command line argument -verbose:gc in the file $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties. For instance,

$IAS_ORACLE_HOME/Apache/Jserv/l ogs/jvm/OACoreGroup..stdout

wrapper.bin.parameters=-verbose:gc -hotspot -Xmx1024M -Xms128M -XX:MaxPermSize=128M or value of the s_jvm_options context variable in the file $APPL_TOP/admin/.xml <jvm_options oa_var=”s_jvm_options” osd=”Solaris”>verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=128M R12

Set the argument -verbose:gc in the start-parameters, javaoptions element in the file $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml. For instance,

$LOG_HOME/ora/10.1.3/opmn/OC4J~ oacore~default_group_*

<process-type id="home" module-id="OC4J" status="enabled"> <module-data> ... ... ... ...

Following are the sample GC details that are generated after setting the value -verbose:gc [GC 325407K->83000K(776768K), 0.2300771 secs] [GC 325816K->83372K(776768K), 0.2454258 secs] [Full GC 267628K->83769K(776768K), 1.8479984 secs]

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

7

But what does these values means and how to measure them?

Figure 2 1) Here we see two minor collections, and a 2) Major collection 3) and 4) Indicate the combined size of live objects before and after garbage collection, respectively. After minor collections the count includes objects that aren't necessarily alive but can't be reclaimed, either because they are directly alive, or because they are within or referenced from the tenured generation. 5) Is the total available space, not counting the space in the permanent generation 6) The minor collection took about 24 secs to run Simply said, the first statement of this output tells that we had 325407Kb used, but that after about quarter of a second of work, GC has freed 242407KB so you now have 776768KB used. Note: Additional flags like, -XX:+PrintGCDetails or -XX:+PrintGCTimeStamps can be used to print more information about the collections, for instance, [GC 111.042: [DefNew: 8128K->8128K(8128K), 0.0000505 secs]111.042: [Tenured: 18154K->2311K(24576K), 0.1290354 secs]

Few notable JVM parameters Few key options that should be considered for useful GC logging are: Table 2 Option

Meaning

-verbose:gc -XX:+PrintGCDetails

Enable verbose GC logging. Provide detailed statistics about collections. Give details about young generation survivor spaces and how many objects are promoted to the tenured space.

-XX:+PrintTenuringDistribution

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

8

Following are few of the options for controlling how the JVM allocates and organizes memory (refer Sun’s documentation for details) Table 3 Option

Meaning

-Xms -Xmx

Configure the initial Java heap size. Configure the maximum Java heap size. Instructs the VM to analyze the current operating environment and attempt to optimize settings for memory-intensive applications. This option also enables implicit garbage collection and adaptive memory sizing. It is intended for machines with large amounts of memory and multiple CPUs. Configure the ratio of new/old generation sizes Configure the initial size of the young generation space. Configure the maximum size of the young generation space.

-XX:+AggressiveHeap -XX:NewRatio -XX:NewSize -XX:MaxNewSize

Now let’s look at some of the default JVM parameters provided in one of the configuration files. For instance, the sample context file $APPL_TOP/admin/$CONTEXT_NAME.xml contains the following options, <jvm_options oa_var=”s_jvm_options” osd=”Solaris”>-verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=128M -XX:NewRatio=2-XX:+PrintGCTimeStamps -XX:+UseTLAB

Table 4 Option

Meaning

-Xms128M -Xmx512M -XX:NewRatio=2

Initiate with 128MB heap size Grow JVM heap size to maximum size of 512 MB If the size of young generation is 10 MB then the size of old generation should be around 20MB Maximum grow able size of permanent generation to 128M Uses thread-local object allocation blocks. This improves concurrency by reducing contention on the shared heap lock.

-XX:MaxPermSize=128M -XX:+UseTLAB

But all this tuning might go for a toss, if the developer calls System.gc() since this forces a major collection. Application can be prohibited to make this operational by providing -XX:+DisableExplicitGC

Few tips for JVM tuning in E-Business Suite  Upgrade to highest possible certified JDK version (JRE 6 is latest certified with both 11i and R12). To know current version of Java, search for the parameter s_jdktop in the file $APPL_TOP/admin/$CONTEXT_NAME.xml <JDK_TOP oa_var=”s_jdktop”>/oracle/apps/11i/bin/java/1.4/

 For OACoreGroup consider no more than 100 active users per JVM (Refer Appendix for SQL Script)

 There should be no more than 1 active JVM/OC4J instance per CPU. Ideally, there should be 1 JVM per 2 CPUs  Size your maximum heap size as either 512 MB or 1 GB. If you start with 512 MB and find that more memory is required, increase to a maximum of 1 GB. If more memory than 1 GB is required, use multiple JVM’s of 512MB or 1024 MB. (For multiple JVM’s, edit
9

oa_var=”s_oacore_nprocs”>2) This is to reduce GC time since more heap size

means more time to garbage collect  Setting -Xms and -Xmx to the same value avoid the need to memory allocation during runtime  With multi-CPU servers using JDK 1.4 and higher, the -XX:+UseConcMarkSweepGC -XX: +UseParNewGC parameters have been seen to improve performance and GC throughput on some environments.  Minor GC should be occurring at interval long enough to allow many objects to die young (i.e. lot of objects should die between two minor GC).  Review the frequency of collections, especially major collections (i.e. Full GC)  If full GCs are too frequent, consider increasing Xms and Xmx  Throughput is inversely proportion to amount of memory. Bigger the heap, more the time for GC, meaning low throughput.  Unless you have problems with pauses, try granting as much memory as possible to VM (512 is good start and fine tune as per load testing results)  100 users per OACore JVM has generally proven to be a practical and performant number  If heap usage is steadily increasing, then there may be a memory leak which require tweaking the application code  Do not use -XX:+UseParallelGC with -XX:+UseConcMarkSweepGC  Last but not the least, try experimenting with various JVM parameters referred in Table 2 and Table 3 above and many more available at Sun’s documentation site (refer References section), to arrive at the optimal solution

Monitor the JVM The basic text based monitoring tool available to us are the log files which contains the data produced by the JVM (using –verbose:gc option). For the details of these log files and how to understand the heap data available in these files refer the Measurement section above. Apart for this text based approach, there are many other visual tools available to us like JConsole, VisualVM etc The following section will discuss one of them, JConsole. JConsole provide information on performance and resource consumption of applications running on the Java platform using Java Management Extension (JMX) technology. This tool comes in JDK 1.5 and provides graphical view of the JVM heap, usage, threads, classes and other statistics in real time. Refer Figure 3 below. You can run JConsole either locally (from the server that you are monitoring as localhost) or remotely via a network connection with Oracle Applications. The safest and easier option is to run locally since using JConsole remotely is more complex, insecure and it may introduce unacceptable risk for a production environment. The JConsole interface in JDK 5 is composed of the following six tabs: 1. Summary displays summary information on the JVM and monitored values. 2. Memory displays information about memory use. 3. Threads displays information about thread use. 4. Classes displays information about class loading. 5. MBeans displays information about MBeans. 6. VM displays information about the JVM. Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

10

For instance, to check for memory leaks in your application, you can start a load test to create a constant load on the application. If the functionality you're testing during the load test does not contain a memory leak, the memory usage should stay between a specific upper and lower limit. Connecting Locally Using JConsole with Oracle E-Business Suite 11i Add the following lines to your jserv.properties file present under $IAS_TOP/Apache/Jserv/etc/ ,save the changes and bounce Apache server Note: To make this change permanent, refer Metalink note 270519.1 about AutoConfig wrapper.bin.parameters=-Dcom.sun.management.jmxremote wrapper.bin.parameters=-Dcom.sun.management.jmxremote.ssl=false wrapper.bin.parameters=-Dcom.sun.management.jmxremote.authenticate=false

Figure 3

Connecting Remotely Firstly, authentication is enabled by default so you will either need to configure your username/passwords or could choose to defeat authentication entirely by using the jserv.properties value below. Note: The following change allows everyone to attach to your JVM processes wrapper.bin.parameters=-Dcom.sun.management.jmxremote.authenticate=false

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

11

The second issue to overcome is that each JVM needs its own TCP port to accept connections. One possible approach is to customize the $IAS_ORACLE_HOME/Apache/Apache/bin/java.sh JVM startup script and provide a port number range from which you allocate a TCP port number. The following sample script provided by Steven Chan but may give you some ideas as to how you could overcome this issue in your own environment. Add this modification just before the existing line exec $JSERVJAVA $JAVA_ADDITIONAL_ARGS $ARGV 1>> $STDOUTLOG 2>> $STDERRLOG #### author Steven Chan #### start of modification for JConsole remote monitoring #### Not supported or recommended by Oracle Corporation #### #### If there is a clash of port numbers calculated, then #### the JVM will not startup. Script therefore needs to be #### improved to check port not already in use #### #### Use port number range 9600 - 9650 as an illustration if [ "$JDK_VERSION" = "1.5" ] ; then mJMXPORTMIN=9600 mJXMPORTMAX=9650 mNUMPORTS=`expr ${mJXMPORTMAX} - ${mJMXPORTMIN}` mRANDOM=$(((($RANDOM*${mNUMPORTS})/32767)+1)) mJMXPORT=`expr ${mRANDOM} + ${mJMXPORTMIN}` JAVA_ADDITIONAL_ARGS=" -DCLIENT_PROCESSID=$$ -Dcom. sun.management.jmxremote.port=${mJMXPORT}" fi #### end of modification

After starting your JVMs we then need to identify the TCP port they are listening on. This can be determined by using the following command: netstat -a | grep tcp | grep 96

The JConsole executable is present in $JDK_HOME/bin, where JDK_HOME is the directory where the JDK is installed. Once JConsole is started a connection dialog box opens, with tab lists as 'Local', 'Remote' and ‘Advanced'. Check that under 'Remote' tab 'Host or IP' field value should be localhost (for local connection) or the IP address of the remote machine. For the remote connection, we additionally need to provide the port we wish to connect to and also the username/password if appropriate. Then click on 'Connect' button and it will show the details of any JVMs running on the local/remote system started with the same user ID as JConsole, along with their process ID and class/argument information and it can be monitored. Note: JConsole works with the JRockit JVM which will be part of Fusion stack in future. But it’s recommended to use the tools that are packaged as JRockit Mission Control for monitoring JRockit JVM

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

12

Future changes - JRockit Acquisition of BEA and introduction of BEA Weblogic Application Server and JRockit JVM All the discussion above has been restricted to Sun HotSpot JVM. But with the acquisition of BEA, JRockit JVM will also be used across the Oracle Fusion Middleware and Oracle Applications product families. This will bring additional changes to JVM tuning for E-Business Suite associated with JRocket For instance, the Table 4 below lists options that work the same or similarly on both Sun Hotspot and the JRockit JVM but have different names depending upon the JVM you are running. Table 4 Sun Hotspot Option Name

JRockit JVM Option Name

Function

-XX:+AggressiveHeap

-XXaggressive:memory

Configures the memory system for memory-intensive workloads and sets an expectation to enable large amounts of memory resources to ensure high throughput. The JRockit JVM will also use large pages, if available.

-verbose:gc

-Xverbose:memory

Prints out log information about the memory system

-Xmn, -XXNewSize, -XXMaxNewSize

-Xns

Sets the size of the young generation

-XX:+UseConcMarkSweepGC

-Xgc:singlecon

Sets the garbage collector to use a concurrent strategy

-XX:+UseParallelGC

-Xgc:parallel

Sets the garbage collector to use a parallel strategy

Note: The certified plugs-ins for Forms is Jinitiator and Sun only.

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

13

References and Quotes General http://java.sun.com/j2se/reference/whitepapers/memorymanagement_whitepaper.pdf http://java.sun.com/docs/hotspot/gc1.4.2/example.html http://blogs.oracle.com/stevenChan/ http://java.sun.com/docs/hotspot/gc1.4.2/ http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html http://www.oracle.com/technology/software/products/jrockit/FAQ.html http://java.sun.com/developer/technicalArticSE/jconsole.html http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp http://edocs.bea.com/jrockit/geninfo/diagnos/migrate.html On upgrading the JDK in E-Business Suite: Note 246105.1- Upgrading to J2SE 1.4.2 with Oracle Applications 11i Note 304099.1- Using J2SE Version 5.0 with Oracle E-Business Suite 11i Note 401561.1- Using J2SE Version 6.0 with Oracle E-Business Suite 11i Notes on JVM tuning with respect to E-Business Suite Note 244040.1- Oracle E-Business Suite Recommended Performance Patches Note 370583.1- Basic troubleshooting of JVM consuming CPU or too many JDBC connections in Apps Note 244040.1- Oracle E-Business Suite Recommended Performance Patches

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

14

Appendix Script to determine "active users" for OACoreGroup {run as APPS user} REM REM SQL to count number of Apps 11i users , refer Metalink Note: Note:427759.1 REM Run as APPS user REM select 'Number of user sessions : ' || count( distinct session_id) How_many_user_sessions from icx_sessions icx where disabled_flag != 'Y' and PSEUDO_FLAG = 'N' and (last_connect + decode(FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'), NULL,limit_time, 0,limit_time,FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate and counter < limit_connects; REM REM END OF SQL REM

Tags for this document: JVM Tuning Java.lang.OutOfMemoryError JRockit JConsole Garbage Collection

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

15