Clustering

Professional Open Source™

JBoss Clustering High Availability and Load Balancing

© JBoss, Inc. 2003-2005.

8/2/2005

1

Clustering Topics Professional Open Source™

– – – –

Features How to use it Design overview Future

© JBoss, Inc. 2003-2005.

2

1

Load Balancing != Fault Tolerance Professional Open Source™

Load balancing is used for scalability, not for fault tolerance

3

© JBoss, Inc. 2003-2005.

Load Balancing != Fault Tolerance Professional Open Source™

Failover and state replication is used for fault tolerance

State Replication

Why do you want to use clustering?

© JBoss, Inc. 2003-2005.

4

2

Clustering Has a Cost Professional Open Source™

Clustering always has a cost – fail-over with state  state replication  serialization

What about the cache? – In a cluster, you never have an exclusive access to the DB

Raw Performance != Scalability – (1 + 1) != 2

5

© JBoss, Inc. 2003-2005.

Cluster Management: Installation Professional Open Source™

To install a cluster you can either: – Manually install & configure each node separately – Bootstrap all nodes from a central location  NetBoot

NetBoot – Microkernel installed on every machine – JBoss configuration & applications on a central web server – Supply URL to micro-kernel • JBoss boots up from micro-kernel

© JBoss, Inc. 2003-2005.

6

3

Cluster Manager: Net Boot Professional Open Source™

NetBoot Client

NetBoot Client

NetBoot Client NetBoot Client

NetBoot Server

Configuration Files

Web Server Administrator

Deployment Packages

7

© JBoss, Inc. 2003-2005.

Cluster Management: Deployment Professional Open Source™

Remember JBoss’ deployment easiness? – Simply copy a file in /deploy

The farm service brings this simplicity to clustering – We see the cluster as farm of nodes  Farm Service

Farm Service Features: – Hot-deploy across cluster – Undeploy across cluster – 24x7 availability: simple redeployment across cluster

© JBoss, Inc. 2003-2005.

8

4

Cluster Manager: Farming Professional Open Source™

Package Replication

Deploy Packages on Single Cluster Node

9

© JBoss, Inc. 2003-2005.

Cluster Management: Auto Discovery Professional Open Source™

How do you “build” a cluster? – A cluster is not statically defined ( e.g. C1 = {N1, N2, N3} ) – A cluster is identified by a name (Partition Name) – All nodes that assert to belong to a given partition name dynamically extend the cluster with this name • Dynamic composition of a cluster

Behaviour – On boot up, new cluster node detects existing group members – Group members are notified on node start/stop/failures – No configuration of topology needed.

© JBoss, Inc. 2003-2005.

10

5

Cluster Fault Tolerance – Server Side Professional Open Source™

What does “fail-over” mean on the server? Component state is always the issue! – Replicated state for stateful session beans – Replicated HTTP sessions – No replicated state for entity beans • Must use commit-option ‘B’ or ‘C’ or cache invalidation with commitoption ‘A’ (for non-critical read-mostly data) – Global cluster-wide, replicated JNDI tree (HA-JNDI)

11

© JBoss, Inc. 2003-2005.

Cluster Fault Tolerance – Client Side Professional Open Source™

Java Client (RMI) – Use of RMI smart proxies – Transparent failover of client proxies: All EJB types, RMI, JNDI

HTTP client (browser) – You need a “tool” to failover sessions – Hardware solutions • Cisco Content Switch, F5 BigIP, Alteon, etc. – Software solutions • Apache Mod_JK 1.2.x

© JBoss, Inc. 2003-2005.

12

6

Cluster Load Balancing Professional Open Source™

Load balancing for all RMI proxy types: – EJBs, RMI, and JNDI – Pluggable load balance policies • Round Robin, First Available, Random • You can write your own • SFSB, Entities: First Available (sticky) • SLSB – Round Robin (no state involved) • All Home invocations: Round Robin

HTTP LoadBalancing – Extension of Failover mechanism • Hardware solution: Cisco, etc. • Software solutions: Apache, etc.

13

© JBoss, Inc. 2003-2005.

What to Cluster? Professional Open Source™

Which components do you really need to cluster? Apache Apache AJP13 AJP13

HTTP HTTP Session Session

HTTP HTTP Session Session

SLSB SLSB

SLSB SLSB

EB EB

EB EB

Everything should be homogeneously deployed on all nodes – Optimize local calls

Why would you cluster the SLSB and EB? – If they fail, the whole node will fail anyway (JVM process failure, operating system failure or hardware failure)

© JBoss, Inc. 2003-2005.

14

7

Specific Optimisations – SFSB Professional Open Source™

With SFSB, the bean state must be replicated to other nodes after each invocation – Replication requires serialization which has a high cost

Was the state really modified? – This decision cannot be made by the container • By default, state must always be replicated – The developer can help the container • The developer will tell the container if the bean has really been modified public boolean isModified();

• If defined on the bean implementation class, this method will be called before the replication. If the bean state has not been modified, not replication occurs.

15

© JBoss, Inc. 2003-2005.

The Big Picture – HTTP Professional Open Source™

Net Net Boot Boot Load Load Balancing Balancing HTTP HTTP Failover Failover

Apache Apache AJP13 AJP13

Replicated Replicated HTTP HTTP Session Session Global Global JNDI JNDI Context Context Tomcat Tomcat

Tomcat Tomcat

Tomcat Tomcat

EJB EJB

EJB EJB

EJB EJB

Database Database

© JBoss, Inc. 2003-2005.

Commit Commit Option Option ‘B’/’C’ ‘B’/’C’

16

8

Using Clustering – Basic Steps Professional Open Source™

Use JBoss’ “all” configuration – run.bat -c all – ./run.sh -c all

The “all” configuration contains everything needed for clustering: – – – – –

Clustered beans HA-JNDI HTTP session replications Farming HA-JMS

17

© JBoss, Inc. 2003-2005.

Using Clustering Professional Open Source™

deploy/cluster-service.xml deploy/cluster-service.xml encoding="UTF-8"?> <server> <server> archives="jbossha.jar"/> <depends>JBOSS-SYSTEM:service=Naming <depends>JBOSS-SYSTEM:service=Naming <mbean <mbeancode="org.jboss.ha.framework.server.ClusterPartition” code="org.jboss.ha.framework.server.ClusterPartition” name="JBOSS-SYSTEM:service=DefaultPartition"/> name="JBOSS-SYSTEM:service=DefaultPartition"/> <mbean <mbeancode="org.jboss.ha.jndi.HANamingService" code="org.jboss.ha.jndi.HANamingService"name="JBOSS-SYSTEM:service=HAJNDI"> name="JBOSS-SYSTEM:service=HAJNDI"> 1100 name="Port">1100 <mbean <mbeancode="org.jboss.ha.framework.server.FarmMemberService“ code="org.jboss.ha.framework.server.FarmMemberService“ name="jboss:service=FarmMember,partition=DefaultPartition" name="jboss:service=FarmMember,partition=DefaultPartition">> <depends>jboss:service=DefaultPartition <depends>jboss:service=DefaultPartition DefaultPartition name="PartitionName">DefaultPartition ./farm name="FarmDeployDirectory">./farm jboss.deployment:type=DeploymentScanner,flavor=URL name="ScannerName">jboss.deployment:type=DeploymentScanner,flavor=URL © JBoss, Inc. 2003-2005.

18

9

Using Clustering – Protocol Stack Professional Open Source™

deploy/cluster-service.xml deploy/cluster-service.xml name="PartitionConfig"> --> --> /> /> <MERGE2 <MERGE2min_interval="5000" min_interval="5000"max_interval="10000" max_interval="10000"/>/> /> /> /> /> /> /> /> />

19

© JBoss, Inc. 2003-2005.

JBossCluster – Using It… Professional Open Source™

EJB Configuration jboss.xml jboss.xml <jboss> <jboss> <secure>false <secure>false <enterprise-beans> <enterprise-beans> <session> <session> <ejb-name>SLSB <ejb-name>SLSB <jndi-name>SLSB <jndi-name>SLSB True True

© JBoss, Inc. 2003-2005.

20

10

JBossCluster – Using It… Professional Open Source™

EJB Configuration jboss.xml jboss.xml <enterprise-beans> <enterprise-beans> <session> <session> <ejb-name>SFSB <ejb-name>SFSB <jndi-name>SFSB <jndi-name>SFSB True True <partition-name>MyPartition <partition-name>MyPartition org.jboss.ha.framework.interfaces.RoundRobin org.jboss.ha.framework.interfaces.RoundRobin org.jboss.ha.framework.interfaces.FirstAvailable org.jboss.ha.framework.interfaces.FirstAvailable

21

© JBoss, Inc. 2003-2005.

HTTPSession Replication Professional Open Source™

How to activate HTTP session clustering – Set your application as distributable in WEB-INF/web.xml

HTTP Session Replication Strategy: WEB-INF/jboss-web.xml <jboss-web> <jboss-web> SET SET ASYNCH ASYNCH

© JBoss, Inc. 2003-2005.

22

11

HTTP Session Replication Professional Open Source™

Replication Trigger – SET_AND_GET – SET_AND_NON_PRIMITIVE_GET – SET

Replication Type – SYNCH – synchronous – ASYNCH - asynchronous

23

© JBoss, Inc. 2003-2005.

JBossCluster Uses Client Side Proxy Professional Open Source™

HAInvokerProxy – Proxy behavior is enhanced to deal with clustered invokers – Interceptor contains list of available nodes – Interceptor chooses server to invoke on through load-balance policy

List List of of nodes nodes

Client Container

© JBoss, Inc. 2003-2005.

EJB Tx

Security

Cluster Client Interceptor

HA RMI

JMX

HA RMI

JMX

24

12

JBossCluster Uses Client Side Proxy Professional Open Source™

HARMIResponse – Client-side cluster interceptor maintains a list of nodes the MBean (EJB, JNDI, etc…) is available. – If cluster topology changes, new list is piggy-backed with invocation response to client. – HA invoker manages this.

List List of of nodes nodes

Client Container

EJB Tx

Security

Cluster Client Interceptor

HA RMI

JMX

HA RMI

JMX

25

© JBoss, Inc. 2003-2005.

Client Side Proxy for Clustering Professional Open Source™

package package org.jboss.invocation.jrmp.interfaces; org.jboss.invocation.jrmp.interfaces; /** /** ** JRMPInvokerProxy, JRMPInvokerProxy, local local to to the the proxy proxyand and is is capable capable of ofdelegating delegating to to local local and and JRMP JRMP implementations implementations */*/ public public class classJRMPInvokerProxyHA JRMPInvokerProxyHAextends extends JRMPInvokerProxy JRMPInvokerProxyimplements implements Externalizable Externalizable {{ //// Public -------------------------------------------------------Public -------------------------------------------------------protected protected ArrayList ArrayListtargets targets ==null; null; protected protected LoadBalancePolicy LoadBalancePolicyloadBalancePolicy; loadBalancePolicy; protected protected transient transientlong long currentViewId currentViewId ==0; 0;

The The HA HA version version contains contains aa list list of of invoker invoker proxies proxies

public public static static final final HashSet HashSetcolocation colocation == new new HashSet(); HashSet();

© JBoss, Inc. 2003-2005.

26

13

Client Side Proxy for Clustering Professional Open Source™

public publicObject ObjectgetRemoteTarget() getRemoteTarget(){{ ifif(targets.size() (targets.size()== ==0) 0) return returnnull; null; synchronized synchronized(targets) (targets) return returnloadBalancePolicy.chooseTarget(targets); loadBalancePolicy.chooseTarget(targets); }}

Pluggable Pluggable load load balance balance policies policies are are responsible responsible for for picking picking the the target target

/** /** **Returns Returnswether wetherwe weare arelocal localto tothe theoriginating originatingcontainer containeror ornot. not. */*/ public publicboolean booleanisLocal(Invocation isLocal(Invocationinvocation) invocation) {{ return returncolocation.contains(invocation.getObjectName()); colocation.contains(invocation.getObjectName()); }}

27

© JBoss, Inc. 2003-2005.

Client Side Proxy for Clustering Professional Open Source™

public publicObject Objectinvoke(Invocation invoke(Invocationinvocation) invocation)throws throwsException Exception {{ ////we wegive givethe theopportunity, opportunity,to toany anyserver serverinterceptor, interceptor,to toknow knowififthis thisaa ////first firstinvocation invocationto toaanode nodeor orififititis isaafailovered failoveredcall call int intfailoverCounter failoverCounter==0; 0; invocation.setValue invocation.setValue("FAILOVER_COUNTER", ("FAILOVER_COUNTER",new newInteger(failoverCounter), Integer(failoverCounter),invocation.AS_IS); invocation.AS_IS); ////optimize optimizeififcalling callinganother anotherbean beanin insame sameEJB-application EJB-application ifif(isLocal(invocation)) (isLocal(invocation)){{ return returnInvokerInterceptor.getLocal().invoke(invocation); InvokerInterceptor.getLocal().invoke(invocation); We We still still collocate collocate ifif we we are are }} local local else else {{ ////We Weare aregoing goingto togo gothrough throughaaRemote Remoteinvocation, invocation,switch switchto toaaMarshalled MarshalledInvocation Invocation MarshalledInvocation MarshalledInvocationmi mi==new newMarshalledInvocation(invocation); MarshalledInvocation(invocation); ////Set Setthe thetransaction transactionpropagation propagationcontext context mi.setTransactionPropagationContext(getTransactionPropagationContext()); mi.setTransactionPropagationContext(getTransactionPropagationContext()); mi.setValue("CLUSTER_VIEW_ID", mi.setValue("CLUSTER_VIEW_ID",new newLong(currentViewId)); Long(currentViewId)); Invoker Invokertarget target==(Invoker)getRemoteTarget(); (Invoker)getRemoteTarget();

© JBoss, Inc. 2003-2005.

28

14

Client Side Proxy for Clustering Professional Open Source™

while while(target (target!= !=null) null){{ try try{{ HARMIResponse HARMIResponsersp rsp==(HARMIResponse)((MarshalledObject)target.invoke(mi)).get(); (HARMIResponse)((MarshalledObject)target.invoke(mi)).get(); ifif(rsp.newReplicants (rsp.newReplicants!= !=null) null){{ setTargets(rsp.newReplicants); setTargets(rsp.newReplicants); currentViewId Server currentViewId==rsp.currentViewId; rsp.currentViewId; Server piggy-backs piggy-backs }} information information about about the the return rsp.response; return rsp.response; topology topology }} ////ALL ALLCATCH CATCHJUST JUSTGOES GOESON ON catch catch(java.rmi.ConnectException (java.rmi.ConnectExceptionce){} ce){}

We fail over here by taking

We fail over here by taking ////IfIfwe wereach reachhere, here,this thismeans meansthat thatwe wemust mustfail-over fail-over aa new new target target and and looping looping on on remoteTargetHasFailed(target); remoteTargetHasFailed(target); the the while while (target (target !=null) !=null) target target==(Invoker)getRemoteTarget(); (Invoker)getRemoteTarget(); failoverCounter++; failoverCounter++; mi.setValue mi.setValue("FAILOVER_COUNTER", ("FAILOVER_COUNTER",new newInteger(failoverCounter), Integer(failoverCounter),invocation.AS_IS); invocation.AS_IS); }} ////ififwe weget gethere herethis thismeans meanslist listwas wasexhausted exhausted throw thrownew newjava.rmi.RemoteException("Service java.rmi.RemoteException("Serviceunavailable."); unavailable."); }} }}

29

© JBoss, Inc. 2003-2005.

Conclusion Professional Open Source™

Understand why you want to cluster: – Fail-over? – Load-balancing?

State replication is expensive – How often do you want to send the state across nodes?

HTTP load balancing is based on: – Hardware – Mod_jk 1.2

Stand-alone client load-balancing is easy because of the smart proxies

© JBoss, Inc. 2003-2005.

30

15