Apache Tomcat Tuning

tomcat native with user-specified open ssl libraries quiet

looks like that tomcat native can not work without the openssl support. so we are going to build ssl also as part of our apr package. as usual we will start with apr. download the apr 1.2.7 bundle (1) ./configure --prefix=/oad/x/apr make make install (2) then go a grab a copy of openssl sources from www.openssl.org. i get 0.9.7f tarball, the version that corresponds to my fedora core 4 install. ./config --prefix=/oad/x/apr make make test make install

--openssldir=/home/rajeevj01/x/tmp shared

the documents and tests would be installed in x/tmp location and libraries are installed in /oad/x/apr location. (3) next, we un-tar and build the tomcat-nativve1.1.2 source ball with the following options. [rajeevj01@bdc31035e native]$ ./configure --prefix=/oad/x/apr --withssl=/oad/x/apr --with-apr=/oad/x/apr --with-java-home=/home/rajeevj01/sw/jdk1.5 --with-java-platform=2 (4) now, set up the tomcat server to use this native library , add the following in $tomcat_home/bin/setenv.sh script (or equivalent) ----------------------------------------------------------------------------------------ld_library_path=/oad/x/apr/lib:$ld_library_path export ld_library_path -----------------------------------------------------------------------------------------rajeevj01 at 2:41:57 am edt link to this entry | blog about this entry | notify aol this entry has 0 comments: add your own friday, june 16, 2006 building apr library on feodra core 4 worried

the libtcnative (the native tomcat connector) requires the openssl and apr libraries headers to be built. so the first step is : download and install the apr. i head over to http://apr.apache.org and grab the latest tarball. untar it and just run configure with the required prefix 1051 ./configure --prefix=/oad/x/apr

1052 1053

make make install

apr is now installed. we are planning to use ajp connector and so we build the non-ssl version of tomcat ajp/13 connector. [rajeevj01@bdc31035e native]$ ./configure --prefix=/home/rajeevj01/x/tcnative126 --with-apr=/home/rajeevj01/x/apr126 --without-ssl --with-javahome=/home/rajeevj01/sw/jdk1.5 --with-java-platform=2 if we do not want to use ssl, we can go ahead and remove the -lssl flag from the makefile also. but if you do that, the tomcat can not find the apr installed. ssl must be enabled for apr (why?) there are some version issues with tomcat 1.1.1(bundled with tomcat 5.5.16) and apr 1.2.7 , so download the tomcat 1.1.2 sources from http://tomcat.heanet.ie/native/1.1.2/source/ 1100 1101

make make install

this builds the native tc library. once we have the library, the next big question is how to use it. rajeevj01 at 4:18:18 am edt link to this entry | blog about this entry | notify aol this entry has 0 comments: add your own wednesday, june 14, 2006 ethereal for http packet captures silly

we need some packet capture tool to analyze the server headers and what the browser is sending to the server. after using couple of useless freeware tool, now i have settled on ethereal. it can do a lot of stuff and options seem quite daunting but the good thing is i can make it work to my purpose with minimal fuss. download and install ethereal and read the user guide anyway. to get up to speed very quickly, 1) start ethereal , to start capture , go to menu | capture |options 2) on this screen, select the interface (like vpn adapter or intel ethernet card etc) 3) in filter box, type tcp port 5400 one more filter that you can try is tcp port 5400 and host 10.141.11.189 etc. this will capture all traffic to and from that host going through your designated interface. 4) i do not like to see lots of colors , so menu | view | colorize packet list this lets you switch on/off the packet colors in display. 5) you can save the packets captured during the session with menu | file | export 6) in the capture window, each packet can be probed further for frame details etc.

all in all a really useful tool and w/o any freeware shit!!

rajeevj01 at 7:29:51 am edt link to this entry | blog about this entry | notify aol this entry has 0 comments: add your own monday, june 12, 2006 webchat gc tuning loopy

okay, so we have done one round of our jvm garbage collection data capture and this is what i can infer: + the eden space size is small , so minor collections are happening every now and then. + survivor space is not big, so after minor collections overflow is happening to tenured space at a very young age so this means, i should start with a bigger young generation. plus i am running a server application, so i will go with same -xms and -xmx options. here are my gc parameters ============================================================== -xss256k -xms512m -xmx512m -xx:newratio=2 -xx:survivorratio=6 -xx:+useparallelgc ============================================================= i may want to try with -xx:newsize=256m -xx:maxnewsize=256m so that young generation size is half the heap size to start with. right now, i will settle for young:tenure ratio as 1:2. with young generation = half the heap size we get young = 256 mb, eden = 6 * (256/8) == 192 mb. survivor space is 32 mb each. ================================================================= to check the jvm stats, we use $jstat -gcutil <delay in millis> > jstat.log like: $jstat -gcutil 200 > jstat.log we can get the vmid using the $jps command.

rajeevj01 at 7:22:48 am edt link to this entry | blog about this entry | notify aol this entry has 0 comments: add your own tuesday, june 6, 2006 how to make the eclipse profiler work for you silly

first things first. we want to do remote profiling and we do not want lots of over heads. the option was there to run one of the standard tools like optimizeit or jboss profiler etc. however, i could not set up jboss profiler quickly enough and optimizeit would have required tomcat to start with its own audit system. so finally i settled for eclipse plug-in profiler. this plugin can attach to remote servers. it just needs one library file and few command line options on server. required server changes a) catalina.sh script or the script that starts you server will require the following chages this one is for cpu profiling, see the link for more details on memory profiling etc. http://eclipsecolorer.sourceforge.net/index_profiler.html java_opts="-xrunprofilerdll:1 -xbootclasspath/a:jakartaregexp.jar:profiler_trace.jar:commons-lang.jar \ -d__profiler_package_filter=__a__% mainclass%:__m__sun.:__m__com.sun.:__m__java.:__m__javax.:__m__org.apache. \ -d__profiler_timing_method=1" for our setup, we need to include these options in post.tcl file along with other java command line options . this is a jvmpi plugin so some of the gc options you require may not be supported. b) copy common-lang.jar, jakarta-regex.jar and profiler_trace.jar to $tomcat_home/bin folder. in our case, this may mean the folder from where we bootstrap the tomcat application. so copy the 3 jar files to the port folder like to 48800/ folder. c) compile the agent using the following script ( better to compile on target server ). the default agent is compiled with gcc 3.2 and it may not work on all systems. g++ -o0 -dlinux -shared -wall -i. -i/home/rajeevj01/sw/jdk1.5/include -i/home/rajeevj01/sw/jdk1.5/include/linux profilerdll.cpp -o libprofilerdll.so d)copy the agent file to jdk_home/jre/lib/i386 folder or you can put it in ld_library_path also. after making the changes, start the server. check logs for any errors. once server startup is successful, we are ready to attach to this server to get profiling data. eclipse changes : you need to do few changes to your eclipse before you can open the profiler window. a) we require a new plugin. download it from the web link http://eclipsecolorer.sourceforge.net/index_profiler.html and copy to your eclipse plug-ins folder. then from menu | windows | open perspective | profiler b) you need to create a launcher configuration to connect to remote server. to do that , open menu | run and choose run .. option. under remote profiler node,

create a new configuration by clicking new button. add the server name and click on run button. thats it!!!! c) you have to patch the plugin for eclipse 3.1 version. link is http://sourceforge.net/tracker/index.php?func=detail&aid=1202373&group_id=48823&at id=454283 d) you may require unrar on windoze machines. see unxutils zip on sourceforge. http://unxutils.sourceforge.net/

rajeevj01 at 8:15:30 am edt link to this entry | blog about this entry | notify aol this entry has 0 comments: add your own monday, june 5, 2006 roadmap : for doing server performance tuning silly

1) chat is working okay for so-called 400 simultaneous users. we do not know when we get the apache on chdl-m05 blue box. so in the mean-while this is the plan i have for performance testing a) we need to profile the application and see what is taking time. right now, there are too many synchronization blocks around!! for profiling , we can take a look at jprobe/optimizeit or jboss profiler, whatever works for us. b) shut down apache debugging and dns lookup etc. point is you run a very lean server. should we compile the mod_status module at this point ? linux server data should give enough indications at this point! c) we need to do tomcat and jvm monitoring. we can do the monitoring using mc4j. d) we capture the linux box stats for different combinations. we need snapshots of following data 0) remove all the debugging options and make a lean server. 1) active thread count 2) open sockets this number should be less than my $ulimit -n output. 3) cpu utilization 4) memory ( i should not be swapping ) 5) jvm heap status we can use the jstat tool. what we want to understand is how much of the heap we are actually using under projected loads. this will give us a better idea of our heap usage. also, i would like to know if i am doing lots of gc around. the link for jstat is http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstat.html see the effect of using -server option etc. 6) apache status

we can use mod_status for this purpose (?) the parameters to vary are 1) apache server threads + number of child processes 2) jvm heap size, -server option, -xincgc option 3) thread stack size for apache as well as jvm threads

this entry has 3 comments: hide recent | add your own kernel-imposed limits on the system ------------------------------------------1.fd-max can be got at: cat /proc/sys/fs/file-max 2. threads-max can be got at: cat /proc/sys/kernel/threads-max comment from shrutikbdc - 6/5/06 7:56 am top command ----------------top -b -d 2 > top_stats this redirects the output of top to a file in plain text format every 2 seconds comment from shrutikbdc - 6/5/06 7:45 am memory vmstat <seconds> ; say "vmstat 2"

gives stats every 2 seconds

the following can be captured from vmstat output: procs r: the number of processes waiting for run time. b: the number of processes in uninterruptable sleep. w: the number of processes swapped out but otherwise runnable. field is calculated, but linux never desperation swaps. memory swpd: the amount of virtual memory used (kb). free: the amount of idle memory (kb). buff: the amount of memory used as buffers (kb). swap

io

si: amount of memory swapped in from disk (kb/s). so: amount of memory swapped to disk (kb/s). bi: blocks sent to a block device (blocks/s). bo: blocks received from a block device (blocks/s).

system in: the number of interrupts per second, including the clock. cs: the number of context switches per second.

this

cpu

these are percentages of total cpu time. us: user time sy: system time id: idle time

comment from shrutikbdc - 6/5/06 7:26 am mod_jk connector to plug jetty/tomcat loopy

in apache

we are going to use mod_jk to connect apache and jetty. step1) download and copy mod_jk.so to /usr/lib/httpd/modules download link is: http://tomcat.apache.org/download-connectors.cgi beware : do not download jk2 module, download jk module. this is very confusing! jk is the one that we need not jk2. you may want to rename the jk .so file. step2) add loadmodule directive to the apache httpd.conf file [ add where other loadmodule directives are ] loadmodule jk_module modules/mod_jk.so beware : the way to add modules may differ in apache 1.3 and apache 2. i dont think you require anything more than above one line with apache 2 that has been enabled with dso support. question: how do i check if my apache has been compiled with dso support? [rajeevj01@bdc31035e installs]$ /usr/sbin/httpd -l should list mod_so.c step3) go to /etc/httpd/conf.d directory [ comment: this may be specific to fc4 , if such a folder is not there then you have to specify the path to mod_jk.conf in apache httpd.conf file using an include <path-to-conf-file> directive, like, include mod_jk.conf ] create a mod_jk.conf file there with the following content --------------------------x-----------------------------------------------# rjha: adding the mod_jk options jkworkersfile /etc/httpd/conf/workers.properties jklogfile /home/rajeevj01/logs/mod_jk.log jkloglevel debug jklogstampformat "[%a %b %d %h:%m:%s %y] " jkmount /servlet/* ajp13 ------------------------------x----------------------------------------------replace /servlet/* with your dymanic scripts context path. step 4) create the logs dir if that does not exist step5) create the workers.properties file with following info -------------------------x-----------------------------------------------------

ps=/ # rjha: sample worker.properties lifted from geronimo worker.list=ajp13 worker.ajp13.port=8009 worker.ajp13.host=localhost worker.ajp13.type=ajp13 step 6) now configure your servlet container to listen on ajp13

port.

a) for jetty do the changes in $jetty_home/etc/jetty.xml rjha : uncommenting the ajp13 listener --> <arg> <set name="port">8009 <set name="minthreads">5 <set name="maxthreads">20 <set name="maxidletimems">0 <set name="confidentialport">443 b) for tomcat , do the changes in server.xml file. comment out the http connector and uncomment the ajp13 connector portions. thats it. now next set of things to do : 1) know more about jetty logging options 2) know more about mod_jk logging options 3) how servlets are deployed on jetty, the context mappings then we can start changing and debugging the jetty code. deploying the dit dot servlet happy

the dit dot servlet just prints one dot every 1/2 seconds. point is to replace the chat pipe servlet with this dit dot servlet on my machine and do the stress testing. i am going to deploy the dit dot servlet on my development box, with tomcat (ditch jetty for the moment). so the next steps in my wave of experiments is 1) download and install the latest tomcat (preferably from sources ?) no ? should do at the moment. download sources as a side-kick. 2) deploy the dit dot servlet with tomcat container 3) deploy the rest of the chat apps 4) now do the jvm stats monitoring , things like, how much of memory i am munching, how many threads etc. need to find out more about this. ============= web.xml file ==============================

binary

<web-app> <servlet> <servlet-name>vanillapipeservlet <servlet-class>com.bobo.chat.vanillapipeservlet <servlet-mapping> <servlet-name>vanillapipeservlet /pipe

we want to monitor our tomcat application jvm. the development boxes i have are linux boxes. on these machines, we just have to add the following 4 lines to java_opts in catalina.sh script ==================================================== # rjha: supply this argument to jvm on start-up # we want to monitor tomcat jvm remotely # development system, no password! java_opts="-dcom.sun.management.jmxremote \ -dcom.sun.management.jmxremote.port=9421 \ -dcom.sun.management.jmxremote.ssl=false \ -dcom.sun.management.jmxremote.authenticate=false" =================================================== no authentication, no passwords, i dont give a damn at the moment. then you should have the jdk 1.5 installed on your machines. on the monitoring machine, just fire jconsole and connect to target vm. right now, remote monitoring not working, only local monitoring is working. i can monitor the number of threads now!!!!

areas of performance monitoring of tomcat silly

sometimes you would like to know how your tomcat server is doing. how many threads are open on container etc. there are two ways you can do this 1) use the server status link on tomcat manager application log into tomcat manager application. you will see a server status application's list.

link on top of

2) use one of the jmx monitoring tools. catalina container properties can be exposed as jmx beans and we can use one of the jmx monitoring tools. i cover jmx tools list in a separate document.

mc4j one of the tools that we are going to use (covered in separate blog) . we need to check out visualgc also. plus there is a tool called hat for heap analysis. i need to pay attention to jvm tunining next. but for the starters, here is the points to ponder when tuning tomcat 1) server configuration file. need more info on this from the tomcat user's guide. http://tomcat.apache.org/tomcat-5.5-doc/config/index.html the config parameters should tell me about maximum processes, acceptcount, max-threads etc. need to do rtfm here. 2) jvm parameters may play a big role here, so we need some info on jvm tuning also + thread stack size + head min/max size + gc (incremental/ how much allocated to generational space) + -x arguments to jvm @ startup -xincgc (for incremental gc) 3) threading support how do i find out if my vm is using nptl libs or not ? start with the linux kernel version $uname -r if you are running a 2.6.x kernel then you should be ok. use this small program to print java library paths ================================================ public class printlibrarypath { public static void main(string[] args) { system.out.println(system.getproperty("java.library.path" )); } } ==================================================== then we can use ldd to see if nptl is linked in our jvm or not. [rajeevj01@bdc31035e jdk1.5]$ ldd /home/rajeevj01/bin/jdk1.5/jre/lib/i386/client/libjvm.so linux-gate.so.1 => (0x00835000) libm.so.6 => /lib/libm.so.6 (0x00111000) libdl.so.2 => /lib/libdl.so.2 (0x003e4000) libpthread.so.0 => /lib/libpthread.so.0 (0x00219000) libc.so.6 => /lib/libc.so.6 (0x0022b000) /lib/ld-linux.so.2 (0x00864000) [rajeevj01@bdc31035e jdk1.5]$ nm /lib/libpthread.so.0 | grep nptl nptl is linked!! -------------------------------------------------------------------------------------------------+ switch off servlet reloading + compile jsps a-priori in production usage. use jspc compiler. + we are not using tomcat in stand-alone mode, so *comment* out that connector. we need something to monitor the tomcat! something to monitor the jvm. use jconsole to see how the jvm is performing. use servlet statistics page to see how the tomcat installation is doing.

tomcat configurations

loopy

+ you should not allow any tracing/debugging options + you should disable the servlet re-loading + the number of threads should be tuned on the server. is there a way to increase the maxthreads programatically ? maxkeepaliverequests the maximum number of http requests which can be pipelined until the connection is closed by the server. setting this attribute to 1 will disable http/1.0 keepalive, as well as http/1.1 keep-alive and pipelining. setting this to -1 will allow an unlimited amount of pipelined or keep-alive http requests. if not specified, this attribute is set to 100. maxsparethreads the maximum number of unused request processing threads that will be allowed to exist until the thread pool starts stopping the unnecessary threads. the default value is 50. maxthreads the maximum number of request processing threads to be created by this connector, which therefore determines the maximum number of simultaneous requests that can be handled. if not specified, this attribute is set to 200. minsparethreads the number of request processing threads that will be created when this connector is first started. the connector will also make sure it has the specified number of idle processing threads available. this attribute should be set to a value smaller than that set for maxthreads. the default value is 4. descriptionacceptcount the maximum queue length for incoming connection requests when all possible request processing threads are in use. any requests received when the queue is full will be refused. the default value is 10.

java.sun.com has lots of articles on how to tune jvm settings. crux of the matter is that you should be able to + size up a heap for your purposes + thread stack size and numbers for your purpose right now, we are running with -xms128m -xmx256m -xincgc -xss128k you can do the required arithmetic. try tuning with server mode also, the warm-up may take some time but after that it should be ok! use some jmx remote management tool like mc4j etc to monitor the health of the jvm as well as tomcat instance running on top of it. this is a good reference to see what all options are available with jvm

http://blogs.sun.com/roller/resources/watt/jvm-options-list.html this one discusses the jvm threads http://java.sun.com/docs/hotspot/threads/threads.html and this one tuning the gc http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/ and finally we have the jvm tuning whitepaper http://java.sun.com/performance/reference/whitepapers/tuning.html

threads tuning params on linux 2.6 the issues here is to find out the system limits and application limits imposed on a running program on linux 2.6 kernel series. later we see how we can tune the parameters to spawn a lot of threads and do fast context switching as well. * application limit tuning * ================== 1) ulimit -s and ulimit -u ulimit has options to set thread stack size. check by typing ulimit -a. looks like this is not system wide setting but just applies to applications started in that bash shell. you can not go beyond the hard-limits of a parameter. the hard limits can be checked by typing ulimit -ah. 2) you can set linux thread stack size using $ulimit -s then you make sure that you use the same value with java -xss option when starting your jvm. that way the thread runs in a limited stack space and more and more threads can be spawned. * system limit tuning * ================ 1) check the system wide settings in following places $ cat /proc/sys/kernel/threads-max $grep totalpages /var/log/dmesg (totalpages/16 is the max # threads possible) $ cat /proc/sys/kernel/pid_max $cat /proc/sys/vm/max_map_count (* need to figure out what exactly is this *) * test program * ============= 3) run the test program to see how many threads are supported on your machine idea is , final limit on # max threads should only be imposed by the amount of ram in system. 4) how to do system wide ulimit settings ? check the contents of file /etc/security/limits.conf

bumping the ulimit happy

the ulimit command has two variants. ulimit-ah and ulimit -as. h is for the hard options (what is set in the config files for the user) and soft is the current value. now if you are root its all pretty ok,you can just use ulimit command to increases whatever you want. for other logins, we can not bump up the resources more than the hard limits. for such cases, we have to first increase the hard-limits by changing the /etc/security/limits.conf on linux 2.6 systems. ================================================= edit /etc/security/limits.conf and add the lines:rajeevj01 soft nofile 1024 rajeevj01 hard nofile 65535 also check the values in /proc/sys/fs/file-max. [ on 2.6.x systems, they are pretty ok!] ================================================== after bumping up the hard limits, use ulimit command for user login $ ulimit -n unlimited a: c: d: f: m: n: r: s: t: u: k: l: p:

max address space (kb) max core file size (kb) max data size (kb) maximum filesize (kb) max locked-in-memory address space (kb) max number of open files max resident set size (kb) max stack size (kb) max cpu time (min) max number of processes file creation mask, set by umask(2) . max number of logins for this user process priority, set by setpriority(2) .

for example, l2d2048n5 is a valid limits_string. for reading convenience, the following entries are equivalent: username l2d2048n5 username l2 d2048 n5

so far not able to locate any remote monitoring tools for linux box. most of the scripts there can be put as part of cron and flushed to log files. i can use a user-specific crontab files!! + install the redhat sysstat package $yum install sysstat + need to do some research on sar and crond to achieve the desired results. other useful commands are ps,iostat, vmstat, top ,mpstat [ but the hitch is i have to be on the machine.] one good link is http://people.redhat.com/alikins/system_tuning.html apart from that try reading the redhat sysadmin guide also http://www.redhat.com/docs/manuals/linux/rhl-9-manual/admin-primer/index.html need to do some research on flushing cron jobs

so its agreed , to use netstat to monitor the number of open sockets and the above utilities to monitor the system health. since i am going to run all the experiments on lan, i am not going to monitor the bandwidth usage as such!

i still can not locate any tool that allows for remote monitoring. anyways, i am not going to break my head over it. mostly, this is what i need to do a) how to monitor + open processes + open threads + memory + i/o, open sockets and open file descriptors. i think the most important is memory monitoring. when i check my /proc i dont think i am going to run out of fds and processes in a hurry! so limits are all ok. so we got to watch memory and cpu after bumping the of fd and thread stack size using ulimit command in a shell. to see the open sockets on my machine , i use this command lifted from site: =====================x===================================

parameters, kerneluser limits red hat's

one of the most twisted, but useful invocations is: $netstat -a -n|grep -e "^(tcp)"| cut -c 68-|sort|uniq -c|sort -n this will show you a sorted list of how many sockets are in each connection state. =====================x==================================== to see the memory and cpu usage , i am relying on top. for a quick snapshot , you may consider using vmstat also. $vmstat 1 5 gives 5 readings with 1 sec interval. b) gnome-system-monitor is a graphic utility that can be installed using yum if it is not there on your system. i have fc4 and gnome-system-monitor is installed. this shows the process list and memory usage. c) top, watch, free , ps etc. are usual unix commands.

the following gives a recipe to bump the number of allowed file descriptors. 1) setting up the hard limit in /etc/limits.conf rajeevj01 nofile hard 2048 2) then executing ulimit -n 2048 in a shell should increase the number of file descriptors. on recent systems , /proc/sys/fs/file-max is good enough for all the purposes (limited by hardware only). on my machine it is ~ 101k. 3) apache or tomcat uses some start-up scripts like apachectl or catalina.sh etc (or if started from /etc/rc.d/httpd etc) there, in the beginning of such scripts we can set the soft ulimit. (make sure the hard-limits have been bumped for apachectl user) ========================================= ulimit -s -n `ulimit -h -n` ========================================= if you are using pam, then you also need to do the following to honor the

limits.conf edit /etc/security/limits.conf and add to any config file in /etc/pam.d/ that handles logins ("login" and "ssh" example): session required /lib/security/pam_limits.so

1) packet analyzer/tcp mon to see what the server is sending and what the browser is receiving. should show the files/size/transfer time plus all the headers. useful for debugging. 2) bandwidth monitor and throttler what is my download speed right now, how can i make my speed simulate a modem speed. 3) network load/stress monitor how is my network pipe doing. what kinds of stress it is subjected to. is my webserver saturating my network card? 4) javascript debugger how to use venkman. microsoft script debugger. get rid of alert debugging. 5) javascript compiler/size muncher 6) web server log analyzer, web server and servlet container tuning tomcat, apache, weblogic 7) os resources monitoring and tuning.

apache connection tuning silly

the apache 2 has 256 maxclients limit out-of-box. the compiled-in hard-limit of maxclients(serverlimit) is 20,000 (way beyond our purpose). first we will look at some of the apache directives used in httpd.conf to tune the server. 1) serverlimit : this goes hand-in-hand with the maxclients. for prefork mpm , this is number of maxclients. in case of worker mpm , this together with threadsperchild determines the maxclients for worker mpm maxclients = threadsperchild * serverlimit 2) threadperchild number of threads created for a apache child process. default is 25. can be 64 determined by the threadlimit.

hard limit

3) maxclients : this is the number of simultaneous requests that apache can handle. below are some of the real world setting examples (this is one of the busiest sourceforge mirrors) ============================================= serverlimit 900 startservers 10 maxclients 28800 minsparethreads 25 maxsparethreads 75

threadsperchild 32 maxrequestsperchild 100 ================================================= as you can see, maxclients = serverlimit * threadsperchild ================================================= startservers 100 minspareservers 10 maxspareservers 10 serverlimit 50000 maxclients 50000 maxrequestsperchild 2000 ================================================= i do not know if we can start so many apache processes on a machine, even theoretically. on my linux machine, [rajeevj01@bdc31035e ~]$ cat /proc/sys/kernel/pid_max 32768 so this number needs to be tuned first. plus of course there is + this issue of open fds + and linux thread stack size + and increasing the number of processes

limit in shell using the ulimit command.

the next example config is taken from aol search team ===================================================== ## worker mpm settings # aol: fixed 2 servers and 200 threads each for a total of 400 max # concurrent requests. can be expanded to 1000 * 2 = 2000 via a # graceful httpd controlled restart. serverlimit threadlimit startservers maxclients minsparethreads maxsparethreads threadsperchild

2 1000 2 400 200 400 200

==================================================== after going through all such examples lets find a configuration that will allow us to run 1k simultaneous connections. this is our first goal. and perspective gained from here will determine how we go ahead with 2k simultaneous connections. for our ec 2.0 case lets try this setup: serverlimit 8 threadlimit 256 startservers 2 maxclients 1024 minsparethreads 128 maxsparethreads 256 threadsperchild 128

to achieve these numbers, we would need to + tune the number of file descriptors + and linux thread stack size. number of threads should not be a problem. we can try tuning serverlimit to 4 and increasing the threadsperchild. but only more data can tells us how to proceed ahead with tuning. finally, we have installed apache 2.0 with worker mpm. all you need to do is -with-mpm=worker switch when using the configuration. using the worker mpm, we can start apache with 4k threads (maxclients). see the blog about apache tuning to know more. idea is to start 4 ajp workers with 1k threads each and interface it with apache. here we have a caveat. we have no way of distinguishing one apache request from another. so the standard tricks of 1 ajp worker for one url is not going to work for ec2. so initially i am going to set up 1 apache - 1 ajp connector for 1k connections. we need 4k + delta for apache instances and and 1k+delta /tomcat , so in total we are talking about 4 jvms of 1200 threads each !!! but before we dive into jvm arithmetic, we need to know how to setup ajp13 with 1k connections. here , the trick is to use maxthreads setting for ajp connector in server.xml file. but then what maxprocessors are doing ? there were some discussions on tomcat list about maxprocessor attrib. but the long and short of it is that i just need to set maxthreads and i should be okay. i dont see a need to use apr (apache portable runtime), because in my case i will be holding on to the connections. so threads are not wasted. now, arithmetic 1 jvm - 1200 threads thread stack size 256k => 8 threads /mb so 256 mb means 1k threads so we start jvm with minheap 256 mb, maxheap 512 mb 512 mb x 4 = 2 gb for running 4 jvms. thread stack size can be tuned to 128k also. ($ulimit -s 128) lets try with one apache + one jvm first! one apache that can handle 1k simultaneous is covered in apache tuning blog. chat application is a push type application. we open a socket to server and browser keeps getting data on that open connection. here, the browser and server should not cache at any point because you need to push data to users as soon as they are available. to enable this behavior we need to set some response headers you will need an equivalent of these apache server headers ======================================================== header set expires "sat,6 may 1995 12:00:00 gmt" header set cache-control "no-store, no-cache" header append cache-control "must-revalidate , no-transform" header append cache-control "pre-check=0, post-check=0"

header set pragma "no-cache" ========================================================= when we pluged mod_deflate module in our apache installation , we never realized that we would hit the caching issues because of compression ;o) mod_deflate is to compress the response, in our case it can not work till the response is complete. so to do compression, server will keep caching the data coming from chat backend and only flush it when the response is complete. so now we need to do compression based on some conditions like some directories only etc. to do list for server tuning step 1) get a development box that has rh 4.0 installed. we need rh 4.0 because we need linux 2.6.x kernel series. verify the kernel with $uname -r step2) we need to bump up the file descriptors limit. we need to raise a ticket for this. verify the hard limits and then again verify that you can actually bump up the file descriptors numbers. we can get 8k or maybe 16k descriptors /process/shell. 3) use this program to verify ================================================================================== ==== [rajeevj01@bdc31035e sandbox]$ cat fd.c #include <sys/types.h> #include #include <stdio.h> void main(void) { int k = 3; while (dup(0) != -1) k++; printf("opens = %d fdsize = %d\n", k, sizeof(fd_set) * 8); } ============================================================ 4) change the thread stack size with $ulimit -s change it to 128k or 256k. for the moment lets stick with 256k. again we can verify the number of threads we can spawn with a program. 5) now fds and threads are okay. we need to move to apache settings. the apache setting that will let us use 1k simultaneous connections have been covered in a different blog. make sure we run a lean apache. check apache 2.0 perf docs also. also make sure that our apache has been compiled with all the required compiler switches. we need to arrive at final configurations after doing the required tuning. turn off debugging options/lots of logging in production systems. + tune with various parameters with sendfile on/off + we need to turn on keepalive on finally test your apache for 1k simultaneous connections using the ab tool. you may use grind/siege etc\. whatever you like, main thing is it should be verified.

6) now we need to move to tomcat. we are using jwal team's ntomcat to run tomcat after starting the jwal layer. important things to remember are + turn off re-loading + turn off debugging + jvm settings + server.xml changes + shutdown http connector in production systems + tune with and w/o apr. the ajp connector settings should change. this has been covered in a separate blog. after doing the setup , verify the connections. 7) jvm settings tuning + min / max heap size + -xss switch other things like incremental gc, starting the jvm in server mode and other tweaks we can think of. 8) now run the tas tool against our deployment and get the statistics. ================= programs to check max. threads ======================= [rajeevj01@bdc31035e sandbox]$ cat threads.c /* compile with: gcc -lpthread -o thread-limit thread-limit.c */ /* originally from: http://www.volano.com/linuxnotes.html */ #include <stdio.h> #include #include #define max_threads 20000 int i; void run(void) { char c; if (i < 10) printf("address of c = %u kb\n", (unsigned int) &c / 1024); sleep(60 * 60); } int main(int argc, char *argv[]) { int rc = 0; pthread_t thread[max_threads]; printf("creating threads ...\n"); for (i = 0; i < max_threads && rc == 0; i++) { rc = pthread_create(&(thread[i]), null, (void *) &run, null); if (rc == 0) { pthread_detach(thread[i]); if ((i + 1) % 100 == 0) printf("%i threads so far ...\n", i + 1); } else printf("failed with return code %i creating thread %i.\n", rc, i + 1);

} exit(0);

}

ajp13 connector options in detail happy

lets go throught the various attributes of ajp 13 connector enablelookups : set this to false. we do not want the dns lookups to happen. protocol this attribute value must be ajp/1.3 to use the ajp handler. connectiontimeout the number of milliseconds this connector will wait, after accepting a connection, for the request uri line to be presented. the default value is infinite (i.e. no timeout). in our case, we can use the default. minprocessors and maxprocessors are now not read at all (from 5.5.x series onwards ) port : whatever port you want. sample number of threads control how many concurrent requests can be served by this connector and that is a very important parameter for us.