Bartow 302 Java Perf Tuning

About Dan

•

•

Responsible for Turbotax Online performance and New Business Initiatives

•

Previously responsible for turbotax.com, intuit.com, quickbooks.com, quicken.com, back-end order systems, content services, and many others

•

The Intuit eStore cluster is the largest stateful JBoss cluster in the world

Best Buy, American Airlines, Sony Online Entertainment, MTV, Rubbermaid, Oprah, Plantronics, JCrew, Neiman Marcus, FAA, US Army, Kodak, American Eagle Outfitters, Cabelas, Finish Line, Lexmark, Alcatel, Deutsche Post, B&Q, Abbott Labs, United Health Group, Sun Microsystems, GlaxoSmithKline, AT&T Wireless, Yellowpages, and many others

Agenda • • •

Intros and Expectations Architecture Components - Web Applications Observing and Tuning

• •

Targeting each layer of the architecture

Discussion

Intros • • •

Your Name Company/Title/Role What you hope to learn during our time together

Architecture Components Web Applications

Network Infrastructure

Java Your Application

Operating System Database

Overview: 3 Tier Architecture

Load Balancer

F5 BigIP/Cisco CSS

Web Server 1

Web Server 2

Apache

App Server 1

App Server 2

Tomcat/JBoss/Websphere

Database

Oracle/SQLServer/MySQL

Observing and Tuning: Performance Analysis

What are your performance targets? How do you define them?

Load (Real or Virtual)

Concurrent users Transactions/second

Your Application/System

Observing and Tuning: A Capacity Model

Observing and Tuning: Performance Analysis A number of things can happen on the way to your targets ...or in every day operation

Response time gets slower Response time is fine and then it stops responding

Load (Real or Virtual)

Response time is fine, but has intermittent bursts of slowness

Your Application/System

Observing and Tuning: Tools of the Trade

Some things you’ll need along the way

GCViewer - Free - Garbage Collection Analyzer Good sys admins :) Samurai - Free - Thread Dump Analyzer Good network engineers :) A Java Application Profiler

Observing and Tuning: Load Balancer

Features and Focus Areas Bound by:

Load Balancing Algorithm (round robin...) OneConnect

CPU/Memory

Compression

Connections per Virtual IP SSL tx/sec

SSL Acceleration Health Checking

Observing and Tuning: Web Server - Apache

Features and Focus Areas Bound by: Content gzip Compression CPU/Memory Concurrent Threads in Use OS Level Network Contention

SSL Compression Threading Model OS Level Network Contention

Observing and Tuning: Web Server - Apache

Threading Models PreFork Worker

PreFork forks off an apache process for every request. This is the original model and works great. Worker model is multithreaded model. It’s much easier on memory because there are less processes in use.

The really important question is: Do you have enough threads?

Observing and Tuning: Web Server - Apache

Apache mod_status

A simple way to find out how many threads are in use!

Observing and Tuning: Operating System

Bound by: CPU Memory Config (File Descriptors) TCP Stack

Observing and Tuning: Operating System

File Descriptors: In Unix based operating systems, a network socket is actually just a special type of a file on the filesystem There’s a HARD and SOFT limit on file descriptors To find out the HARD: ulimit -H

To find out the SOFT limit: ulimit -n

This is a system wide maximum limit that can never be exceeded by a user or process

This is the limit of file descriptors a particular shell can have available to it

Changing this is a root level config change requiring a system reboot

This can be changed without a reboot but you will need to put it in a shell script or profile to get set every time the shell is started

Observing and Tuning: Operating System

TCP stack?? - what the heck is that? Its the set of configs and behaviors that your OS uses for network communication

http://www.sean.de/Solaris/soltune.html tcp_time_wait_interval Default = 240,000ms/4 Minutes Recommended = 60,000ms/1 Minute (sometimes lower)

tcp_conn_req_max_q tcp_conn_req_max_q0 Default = 1024 and 128

Observing and Tuning: Application Server

ThreadHandler Pool - Each thread handles a page request (JSP) Defaults are often too low or too high - JBoss = 100, WebSphere = 20 Find out how many are in use with the web admin browsers (JMX-Console...)

Observing and Tuning: Java Virtual Machine Tuning

Observing and Tuning: Java Virtual Machine Tuning

java -Xms1024m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=256m -XX:MaxNewSize=256m XX:+UseConcMarkSweepGC -XX:UseParNewGC -XX:CMSInitiatingOccupancyFraction=50

Min and Max Heap Size: Perm Generation New Generation

Garbage Collection Algorithm

Observing and Tuning: Java Garbage Collection A few different ways to clean up your garbage Single threaded (default - yikes) Parallel Concurrent Mark Sweep

The NEW and OLD generations use different collectors - set independently

Observing and Tuning: Java First - how do I see what my heap is doing under load?

GCViewer! Add these JVM startup args: -verbose:gc -XX:+PrintGCDetails -xloggc:/tmp/myGCFile.log

Observing and Tuning: Java

Single or Parallel Threaded Collection Full GCs get closer and closer together as load increases until throughput starts to drop

Observing and Tuning: Java

ConcurrentMarkSweep Collector Heap usage stays in a tight band between 30-60%

Observing and Tuning: Java A Java Thread Dump - Powerful! kill -QUIT <JAVA PROCESS ID>

Discussion - Q&A

Real performance problems? Architecture challenges? Organizational issues?

Thank you! For more information: Dan Bartow Sr. Manager - Performance Engineering Consumer Group [email protected] [email protected] www.turbotax.com