Improving Scalability With Objectgrid Partitions

Processing complex business events with WebSphere eXtreme Scale, Part 2: Improving scalability with ObjectGrid partitions Skill Level: Intermediate Kenji Kojima ([email protected]) Developer IBM

12 Nov 2008 This two-part series shows how IBM® WebSphere® eXtreme Scale can be used to improve performance and scalability when processing complex business events. Part 1 introduces a traffic monitoring scenario and uses an ObjectGrid on a stand-alone server to show how the scenario works. eXtreme Scale is used to pass high-volume raw events to a prefiltering application, which significantly reduces the processing load on WebSphere Business Events. Part 2 builds on this scenario and uses a distributed Object Grid to partition the events based on the Business Events context ID. This introduces high availability and scalability, as events can be processed by multiple Business Events instances.

Introduction This article is the second in a series on how WebSphere Business Events (hereafter referred to as Business Events) and WebSphere eXtreme Scale (hereafter referred to as eXtreme Scale) can be used to improve scalability and high availability in a complex event processing solution. Part 1 used eXtreme Scale to preprocess a speed calculation, filter high-volume raw events by a given threshold, and then pass this subset of raw events, called actionable events, to Business Events. Using this design pattern can significantly increase the rate of event processing. In part 1 Business Events was running as a single instance on a stand-alone application server. eXtreme Scale was configured with a distributed ObjectGrid, but with a single partition. Realistically, as your application and system grow, you need

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 1 of 42

developerWorks®

ibm.com/developerWorks

both a highly available and a scalable operating environment. Business Events v6.1 doesn't support the existing clustering and high availability features of WebSphere Application Server (hereafter referred to as Application Server). However, we can achieve this goal by combining the existing features of Application Server, Business Events and eXtreme Scale. This article shows you how to modify the application and configuration used in part 1 to answer several questions from a scalability and high-availability point of view: • How do clustered application servers interact with distributed ObjectGrid and multiple partitions? • How do multiple Business Events instances and eXtreme Scale servers connect by using the Service Integration Bus (SIBus)? • How do we modify Business Events interaction sets and eXtreme Scale applications appropriate to the new system environment for part 2? • How do we install and configure multiple Business Events instances and connectors run on multiple nodes? The scenario described here was built using Red Hat® Enterprise Linux® ES release 4, but the artifacts supplied in Downloads can be used on either Windows or Linux. The following software levels were used: • IBM Rational Application Developer V7.0.0.7 • WebSphere Business Events V6.1 Fix Pack 1 for Linux • IBM DB2® V9.5 Fix Pack 1 for Linux • WebSphere eXtreme Scale V6.1.0.3 • WebSphere Application Server Network Deployment V6.1.0.5

Planning for scalability and high availability An event is the representation of an activity in a system. The correlation of the activity across disparate events is known as Complex Events Processing (CEP). Business Events is specifically designed to orchestrate CEP, which evaluates not only individual events but also patterns of events at run time. This powerful feature also requires careful planning when building a scalable solution with multiple Business Events instances. Part 1 of this series describes the filter used to determine which event or action patterns have occurred. The filter can be a defined series of events; based on the results of the filter, Business Events governs the interaction between events at run time. For example, if the LowSpeedEvent occurs more than nine times within a

Improving scalability with ObjectGrid partitions Page 2 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

one-minute period, the condition of the MultipleLowSpeeds filter is met. The filter also checks that there haven't been any occurrences of the LowSpeedAlert in the last five minutes. Figure 1. MultipleLowSpeeds filter

This type of a filter is considered a CEP filter because it evaluates whether an event has taken place in a given context. The MultipleLowSpeeds filter shown in Figure 1 is used in the ManyLowSpeeds interaction set shown in Figure 2. An interaction set can have only one event, which in our case is the LowSpeedEvent. Notice that the SpeedLoc.location is specified in the “Related by” field. This becomes a key that Business Events uses to detect that the event pattern has occurred. This is called a Business Events context. (In reality, a Business Events context is a field reference from an intermediate object, associated with the interaction set by the "Related by" relationship.) Figure 2. ManyLowSpeeds interaction set

Because we defined the context relationship to be SpeedLoc.location, the interaction set and filters are evaluated on a per-location basis in our scenario. As you know, Business Events evaluates not only individual events but also flows of events in a context. When configuring multiple Business Events instances for scalability, we must ensure that events in the same context are always sent to the same Business Events instance. Each subsequent event that is part of the context must be processed on the same Business Events instance with the affinity for that context. Therefore, our solution must provide context-based event routing from EJB clients to Business Events instances. To achieve EJB scalability we want to use the EJB clustering support of Application Server. Therefore, we need to install and configure Business Events on the top of Application Server. eXtreme Scale supports both scalability and high availability for partitions. Although we used a single ObjectGrid partition for the high-volume raw

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 3 of 42

developerWorks®

ibm.com/developerWorks

events store in part 1 (which was neither highly available nor scalable in its configuration), we can gain a scalable and highly-available event store with some simple eXtreme Scale configuration changes from a single partition to multiple partitions. We use this partitioning support provided by eXtreme Scale to ensure that all events with the same Business Events context are always routed to the same Business Events instance. There is no need to buy any additional products or develop any new code. Business Events v6.1 doesn't support high availability. There is a single point of failure (SPOF) with respect to the Business Events instance and its repository database. You can use a High Availability (HA) solution such as IBM High Availability Cluster Multiprocessing (HACMP) or Oracle® Real Application Clusters (RAC) to eliminate SPOF for repository database, but this is outside the scope of this article.

The solution architecture The previous section introduced considerations for developing a scalable and highly available architecture. This section outlines the solution architecture that satisfies those considerations. • EJB clustering is used to provide scalability for our stateless session EJB. • ObjectGrid key-based partitioned routing is used so that each subsequent event in the Business Events context can be stored in a specific partition. This requires that the key of Business Events context is used as the key of ObjectGrid entity model. This ensures that Business Events maintains consistency because it receives all the events in a context from the ObjectGrid listener. • When a partition is activated as a primary, the ObjectGridEventListener implementation class receives the activate shard event and starts its worker threads that listen for objects added to this partition. These threads access a partition using First In First Out (FIFO) because ObjectGrid tracks the insertion order for a partition. When an object is read from the ObjectGrid partition, the ObjectGrid listener publishes an event to the Business Events action topic for this partition on SIBus using the JMS API. Threads are always running to be collocated with a primary partition. If a partition is stopped, the threads are deactivated and started again on a primary partition on other eXtreme Scale servers. More information on this is provided in Listing 6 and Listing 7 of part 1. • Multiple Business Events instances can share a DB2 instance where the Business Events repository database is defined. Business Events

Improving scalability with ObjectGrid partitions Page 4 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

communicates with DB2 using the DB2 JDBC driver. Figure 3. Solution architecture for part 2

Next, we will map the components used in part 1 to the solution architecture above. The new solution is shown in Figure 4. The Member1 and Member2 application servers are defined as cluster members in the cluster where the TrafficAlerter and TrafficAlerterFilter EAR files are deployed. Because all of the application servers are members of a bus called WbeBus, the TrafficAlerterFilter can readily access the Business Events outbound action topic. Figure 4. Solution components for part 2

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 5 of 42

developerWorks®

ibm.com/developerWorks

ObjectGrid provides a public Java API for FIFO access to object maps. By using the ObjectMap.getNext method within the TrafficAlerterFilter.ear file, we can easily access maps like a FIFO queue. The getNextKey method returns the key of the next available map entry in the map, which can then be removed from the queue using the returned key. The TAFilterWorker.run method for using the FIFO API is shown in Listing 7 of part 1. Because we use the local ObjectGrid reference provided to the ObjectGridListener we always read from the local partition. This is the behavior we need to ensure that the same Business Events context is always sent to the same Business Events instance.

Implement changes for a multiple-partition solution Several changes are required to implement the multiple-partition solution: • Create a schema root definition for distributed ObjectGrid partitions • Change the entity model for partitioned data routing • Change the ObjectGrid transaction strategy • Map the partition ID to the JMS JNDI name • Change the configuration for multiple Business Events connectors on

Improving scalability with ObjectGrid partitions Page 6 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

multiple nodes • Configure the SIBus and add a Business Events instance Create a schema root definition for distributed ObjectGrid partitions ObjectGrid partitions are established by splitting the data described by the user-defined entity model in the entity.xml file. In a distributed ObjectGrid environment a client can access all the defined entities through the root entity. In part 1, ObjectGrid was used with distributed and stand-alone mode, and we had only one partition so we didn't need a schema root definition in the entity.xml file. In part 2, multiple partitions are deployed to multiple eXtreme Scale servers and the EJB works as an ObjectGrid remote client. This means that a request must be routed to a specific partition (possibly across the network), therefore requiring a schemaRoot definition in our objectGrid.xml file. We must also define two ObjectGrid spaces because one ObjectGrid space can have only one schema root. Recall that we have two data types that we want to store in ObjectGrid. The first type is DataPacket, used to calculate the speed of an RFID. The second type is SpeedAtLocation, which stores the speeds in the ObjectGrid FIFO queue and is read by the TrafficAlerterFilter application. Therefore we must define two object grids, TrafficAlerterOG and TrafficAlerterFilterOG, which have DataPacket and SpeedAtLocation entities defined as schema roots. See Listing 1 and Listing 2 for the TrafficAlerterOG; Listing 4 and Listing 5 for the TrafficAlerterFilterOG. Listing 1. The objectGrid.xml (TrafficAlerterOG)

Listing 2. The entity.xml file (TrafficAlerterOG) <entity class-name="com.ibm.ta.util.DataPacket" name="DataPacket" access="FIELD" schemaRoot="true"> <description>"This is the DataPacket class"

Notice that we specify numberOfPartitions="2" and numInitialContainers="2" in the

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 7 of 42

developerWorks®

ibm.com/developerWorks

objectGridDeployment.xml file in Listing 3 and Listing 6. This means that we have two partitions on the clustered Member1 and Member2 servers and we also have an asynchronous replica. Listing 3. The objectGridDeployment.xml file (TrafficAlerterOG) <deploymentPolicy xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ibm.com/ws/objectgrid/deploymentPolicy ../deploymentPolicy.xsd" xmlns="http://ibm.com/ws/objectgrid/deploymentPolicy"> <mapSet name="mapSet" numberOfPartitions="2" minSyncReplicas="0" maxSyncReplicas="0" maxAsyncReplicas="1" numInitialContainers="2"> <map ref="DataPacket"/>

Listing 4. The objectGrid.xml file (TrafficAlerterFilterOG)

Listing 5. The entity.xml file (TrafficAlerterFilterOG) <entity class-name="com.ibm.ta.util.Location" name="Location" access="FIELD" schemaRoot="true"> <description>"Location" <entity class-name="com.ibm.ta.util.SpeedAtLocation" name="SpeedAtLocation" access="FIELD"> <description>"Speed at a location" <many-to-one name="location" target-entity="com.ibm.ta.util.Location">

Improving scalability with ObjectGrid partitions Page 8 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®



Listing 6. The objectGridDeployment.xml file (TrafficAlerterFilterOG) <deploymentPolicy xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ibm.com/ws/objectgrid/deploymentPolicy ../deploymentPolicy.xsd" xmlns="http://ibm.com/ws/objectgrid/deploymentPolicy"> <mapSet name="mapSet" numberOfPartitions="2" minSyncReplicas="0" maxSyncReplicas="0" maxAsyncReplicas="1" numInitialContainers="2"> <map ref="SpeedAtLocation"/> <map ref="Location"/>

Change the entity model for the partitioned data routing The entity model used in part 1 must be changed so that ObjectGrid automatically routes a request to the correct server. Recall that we defined a unique id as a key for the SpeedAtLocation entity. In this case, ObjectGrid routes a request to a partition by using an algorithm that divides a hash value of a key by the number of partitions. This means that we can’t control the partitioned data routing in distributed and multiple partitions because the routing key is unique. In our multiple-partition solution, an event for each location must get sent to the same partition, so that the ObjectGrid listener for that partition sends the event to the same Business Events instance. To do this we define a new entity model as shown in Figure 5. A Location entity has a reference to one or more SpeedAtLocation entities. Each SpeedAtLocation entity has a reference to a single Location entity. There are bidirectional relationships between the Location and the SpeedAtLocation. The locationId of the Location entity is used as a key for ObjectGrid partition routing. As the locationId is also a Business Events context in our interaction set, an event will get forwarded to the correct Business Events instance. Figure 5. Entity model for multiple-partition solution

Figure 6. Entity model for single-partition solution

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 9 of 42

developerWorks®

ibm.com/developerWorks

Because we changed the entity model, we need to modify the sendDataPacket method (Listing 7) as follows: 1.

The ejbCreate method invokes this method to create an ObjectGrid manager, a client cluster context and get references to the TrafficAlerterOG and TrafficAlerterFilterOG. The CATALOG_SERVER_PROP should include the hostname and ORB_LISTENER_ADDRESS port of the deployment manager.

2.

Get a reference to the EntityManager for the TrafficAlerterFilterOG.

3.

Begin a transaction for the TrafficAlerterFilterOG.

4.

Find a location entity by the location ID as a key.

5.

If there is no entry with this location ID in the TrafficAlerterFilterOG, then create a new Location and SpeedAtLocation entity, associate them and store two related entities.

6.

Create a new SpeedAtLocation entity, add to the Location entity, and commit them.

Listing 7. Modified sendDataPacket method private ObjectGridManager ogManager; private ObjectGrid og; private ObjectGrid ogf; 1.

private void initialiseOG() { ogManager = ObjectGridManagerFactory.getObjectGridManager(); ClientClusterContext context = ogManager.connect(TrafficAlerterConstants.CATALOG_SERVER_PROP,null ,null); og = ogManager.getObjectGrid(context,TrafficAlerterConstants.TA_OG_NAME); ogf = ogManager.getObjectGrid(context,TrafficAlerterConstants.TAF_OG_NAME);

} public void sendDataPacket(DataPacket data) { EntityManager em = getEntityManager(); em.getTransaction().begin(); try{

Improving scalability with ObjectGrid partitions Page 10 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

DataPacket currentData = (DataPacket) em.find(DataPacket.class, data.RFID); if (currentData == null) { currentData = new DataPacket(); currentData.locationId = data.locationId; currentData.RFID = data.RFID; currentData.timeInMillis = data.timeInMillis; em.persist(currentData); em.getTransaction().commit(); } else { BigDecimal diff = new BigDecimal(data.timeInMillis – currentData.timeInMillis); BigDecimal timeInHours = diff.divide(millisInHour,10, RoundingMode.HALF_UP); BigDecimal speedBig = halfMile.divide(timeInHours, 0, RoundingMode.HALF_UP); int speed = speedBig.intValue(); currentData.locationId = data.locationId; currentData.timeInMillis = data.timeInMillis; em.persist(currentData); em.getTransaction().commit(); 2. 3. 4. 5.

6.

EntityManager emf = getEntityManagerForFilter(); emf.getTransaction().begin(); Location currentLoc = (Location) emf.findForUpdate(Location.class, data.locationId); if (currentLoc == null) { Location loc = new Location(); loc.locationId = data.locationId; loc.speedAtLocations = new ArrayList<SpeedAtLocation>(); SpeedAtLocation sp = new SpeedAtLocation(); sp.location=loc; sp.speed = speed; sp.UUID = UUID.randomUUID().toString(); loc.speedAtLocations.add(sp); emf.persist(loc); emf.getTransaction().commit(); }else{ SpeedAtLocation sp = new SpeedAtLocation(); sp.location=currentLoc; sp.speed=speed; sp.UUID = UUID.randomUUID().toString(); currentLoc.speedAtLocations.add(sp); boolean persisted = false; while (! persisted){ try { emf.getTransaction().commit(); persisted = true; } catch (EntityExistsException eee) { sp.UUID = UUID.randomUUID().toString(); } } }

} } catch (Exception e) { e.printStackTrace(); } } package com.ibm.ta.util; public class TrafficAlerterConstants { public static final String TA_OG_NAME = "TrafficAlerterOG"; public static final String TAF_OG_NAME = "TrafficAlerterFilterOG"; public static final String CATALOG_SERVER_PROP = "wstiblade9.rtp.raleigh.ibm.com:9100"; }

Change the ObjectGrid transaction strategy

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 11 of 42

developerWorks®

ibm.com/developerWorks

In part 1 we did not specify a locking strategy in the objectGrid.xml file. By default, ObjectGrid uses an optimistic lock strategy. This works fine for a stand-alone eXtreme Scale server and a single partition (with only one ObjectGrid near cache, there is no chance for a collision error). However for multiple eXtreme Scale servers, collision errors can occur as illustrated in Figure 7. 1.

The EJB client invokes a method on the Member1 server. The request contains values for the RFID, locationId and timeInMillis parameters of the DataPacket entity model. For example, the EJB client sends the parameters RFID=111 and locationId=1 at Location 1 and then the EJB commits them to the TrafficAlerterOG object grid. In the meantime, ObjectGrid keeps them in the near cache of the Member1 server.

2.

Next, a client moves to Location 2 and sends the parameters RFID=111 and locationId=2. This request is dispatched to the Member2 server by the EJB clustering function. The EJB can commit from the Member2 server because we use a distributed ObjectGrid. At this point, the ObjectGrid near cache of the Member1 server is stale.

3.

After that, a client moves to Location 3 and sends parameters RFID=111 and locationId=3. If this request is dispatched to the Member1 server, a collision error occurs when the client tries to commit it.

Figure 7. Collision error situation

See Listing 5 for more information on the sendDataPacket method used in part 1. This logic works fine when using a local ObjectGrid with an optimistic strategy, but when we have more than one eXtreme Scale server we need to specify a pessimistic strategy in the objectGrid.xml file (Listing 1 and Listing 4).

Improving scalability with ObjectGrid partitions Page 12 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

Map the partition ID to the JMS JNDI name This section shows you how we make sure that each ObjectGrid partition sends events to the same Business Events instance. The TAOGListener and TAFilterWorker classes are explained in Listing 6 and Listing 7 of part 1. The TAFilterWorker class retrieves an event from each partition and forwards it to a specific Business Events server. To do this, add a new method to the TAFilterWorker class called getWbeResourceNames (Listing 8). This method takes a partition number and calculates which JMS connection factory and topic it should use. This establishes affinity between the event and a Business Events instance. 1.

The method gets the JMS JNDI name corresponding to the ObjectGrid partition ID and assigns it to the CF_name and TOPIC_name variables.

2.

The sendEventToWBE method sends the actionable event to the Business Events instance having affinity with the current ObjectGrid partition ID.

3.

This method initializes a lookup table (map) which maps a constant comprised of a resource prefix concatenated with the partition ID to the desired resource JNDI name.

Listing 8. The TAFilterWorker.run method private String CF_name; private String TOPIC_name; public void run() { initialiseOG(); Session session = getSession(); EntityManager em = session.getEntityManager(); try { ObjectMap map = session.getMap("SpeedAtLocation"); BackingMap bMap = og.getMap("SpeedAtLocation"); 1. getWbeResourceNames(bMap.getPartitionId());

2.

while (!appStopping.get()) { em.getTransaction().begin(); Object msgKey = map.getNextKey(2000); if (msgKey != null) { SpeedAtLocation sp = (SpeedAtLocation) em.find(SpeedAtLocation.class, msgKey); if (sp.location.locationId.equals("-1")) { sendTestFinishedEvent(); } else { // now consume the message if (sp.speed < Constants.TA_MIN_SPEED) { sendEventToWBE(sp); } } // need to remove it map.remove(msgKey); } em.getTransaction().commit();

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 13 of 42

developerWorks®

ibm.com/developerWorks

} } private void getWbeResourceNames(int partitionID) { CF_name = wbeResourceMap.get("tcf"+partitionID); TOPIC_name = wbeResourceMap.get("topic"+partitionID); } public TAFilterWorker(ObjectGrid og) { super(); this.og = og; if (wbeResourceMap.isEmpty()) { initializeMap(); } } 3. private void initializeMap(){ wbeResourceMap.put("tcf0", TrafficAlerterConstants.WBE1_TCF_PROP); wbeResourceMap.put("tcf1", TrafficAlerterConstants.WBE2_TCF_PROP); wbeResourceMap.put("topic0", TrafficAlerterConstants.WBE1_TC_PROP); wbeResourceMap.put("topic1", TrafficAlerterConstants.WBE2_TC_PROP); } package com.ibm.ta.util; public class TrafficAlerterConstants { public static final String WBE1_TCF_PROP = "cell/nodes/wstiblade9Node01/servers/WBE-1/jms/WbeTopicConnectionFactory"; public static final String WBE1_TC_PROP = "cell/nodes/wstiblade9Node01/servers/WBE-1/jms/eventTopic"; public static final String WBE2_TCF_PROP = "cell/nodes/wstiblade6Node01/servers/WBE-2/jms/WbeTopicConnectionFactory2"; public static final String WBE2_TC_PROP = "cell/nodes/wstiblade6Node01/servers/WBE-2/jms/eventTopic2"; }

Change the configuration for multiple Business Events connectors on multiple nodes In our scenario, we defined two types of action connectors that can send an action to Business Events touchpoints (Figure 8). One is the file system connection for the LowSpeedAlert action to the HighwaysAgency touchpoint. Another is the message queue connection for the AllEventsProcessed action to the WXSServer touchpoint. Figure 8. Action connectors on Business Events touchpoints

Improving scalability with ObjectGrid partitions Page 14 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

Figure 9 shows the Business Events connectors topology. The JMS queue of AllEventsProcessed is used to determine when Business Events has processed all the events. The EJB client application waits to receive the last message from this queue after all the raw events are sent. In part 1, the AllEventsProcessed queue was defined on the WXSServer. In part 2, because our eXtreme Scale servers are clustered we define an additional application server for this message processing called the Test1 server; multiple connectors can share its JMS queue. Figure 9. Business Events connectors topology

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 15 of 42

developerWorks®

ibm.com/developerWorks

To define JMS resources on the Test1 server, use the WebSphere administrative console to do the following: 1.

2.

Create a bus destination called TEST_FINISHED. a.

Select Service integration => Buses and click WbeBus.

b.

Click Destinations.

c.

Click New.

d.

Select the Queue checkbox, and click Next.

e.

Specify TEST_FINISHED as the Identifier and click Next.

f.

Select Test1 as the bus member and click Next.

g.

Click Finish.

Create a JMS queue connection factory called AllEventsProcessedQCF. a.

Select Resources => JMS => Queue connection factories.

b.

Select the first node as the scope and click New.

c.

Select the Default messaging provider checkbox and click OK.

d.

Specify AllEventsProcessedQCF as the Name and

Improving scalability with ObjectGrid partitions Page 16 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

jms/AllEventsProcessedQCF as the JNDI name.

3.

e.

Select WbeBus as the Bus name and specify the messaging engine name as the Target for the connection. You can find the messaging engine name by selecting Application servers => Test1 => Messaging engines.

f.

Select Messaging engine name as the Target type and specify the host name and port number as the Provider endpoints. You can find the value of SIB_ENDPOINT_ADDRESS by selecting Application servers => Test1 => Ports and clicking OK.

Create a JMS queue called AllEventsProcessed. a.

Select Resources => JMS => Queues.

b.

Select the first node as the scope and click New.

c.

Select the Default messaging provider checkbox and click OK.

d.

Specify AllEventsProcessed as the Name and jms/AllEventsProcessed as the JNDI name.

e.

Select WbeBus as the Bus name and TEST_FINISHED as the Queue name for the connection, and click OK.

We added two properties in the trafficAlerter.props file: AllEventsProcessedQCF and AllEventsProcessedQueue. We can define compound JNDI names for JMS resources. The providerURL should include the hostname and BOOTSTRAP_ADDRESS port of the deployment manager. Listing 9. The trafficAlerter.props file providerURL=iiop://wstiblade9:9809 AllEventsProcessedQCF=cell/nodes/wstiblade9Node01/servers/Test1/jms/AllEventsProcessedQCF AllEventsProcessedQueue=cell/nodes/wstiblade9Node01/servers/Test1/jms/AllEventsProcessed

We also changed the configuration for these connectors. For the message queue connections, use the Design Data UI to edit properties so that the Provider URL matches your system environment. The port number should be the BOOTSTRAP_ADDRESS port of the Test1 server. See Figure 5 in part 1. For the file system connections, edit the properties so that the directory name doesn’t conflict with the directory name for another Business Events connector. Use the Design Data UI to change the directory path in the TAProject-partition.xml file for the WBE-2 server as shown in Figure 10. There are two TAProject-partition.xml files (one for each Business Events instance), so we need to make similar changes to the WBE-1 TAProject-partition.xml file. Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 17 of 42

developerWorks®

ibm.com/developerWorks

Figure 10. Edit properties for the file system connection

Build the topology and configure SIBus This section describes the required WebSphere profiles needed to build the solution topology. We assume you are familiar with using the WebSphere administrative console to configure application servers and J2EE applications. Because we want to build a mission-critical system that can handle high-volume raw events, we use a deployment manager profile and a custom profile (both enabled for eXtreme Scale) instead of a stand-alone profile. (In the profile management tool, choose WebSphere Extended Deployment Data Grid in the Environment Selection dialog, and then choose Deployment Manager in the Profile Type Selection dialog.) All the application servers are defined within a single cell, as shown in Figure 11. This configuration can provide central management capabilities for multiple application servers. First, we define a deployment manager profile and then create a custom profile and federate the custom profile to the cell. After the federation, we can further configure the topology by adding application servers and clusters from the deployment manager using WebSphere administrative console. Figure 11. WebSphere profiles and application server topology

Improving scalability with ObjectGrid partitions Page 18 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

We created a cluster and five application servers in a cell. The application cluster AppCluster consists of a pair of application servers: Member1 on Node 1 and Member2 on Node 2. Due to an eXtreme Scale product issue at the time of the testing, we needed to use the same profile names for the AppCluster members on both nodes. If the Member1 server fails, EJB requests are routed to the Member-2 server. The Test1 server is used by our test application so that it knows when all events have been processed by Business Events. (See Test Finished processing in part 1.) The WBE-1 and WBE-2 servers are installed with the Business Events run-time code (WBERuntimeEAR). To define this system topology, use the WebSphere administrative console to do the following: 1.

Create a new server cluster called AppCluster. a.

Select Servers => Clusters and click New.

b.

Specify AppCluster as the cluster name and uncheck the checkbox of prefer local, and click Next.

c.

Specify Member1 as the member name and the first node as the select node, and click Next.

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 19 of 42

developerWorks®

2.

3.

4.

ibm.com/developerWorks

d.

Specify Member2 as the member name and the second node as the select node, and click Next.

e.

Click Finish.

Change the classloader policy as server specific application settings. a.

Select Servers => Application servers and click Member1.

b.

Select Single as the classloader policy and click OK. Repeat this step to configure Member2.

Create shared libraries for the TrafficAlerterUtil.jar file. a.

Select Environment => Shared Libraries and select the first node as a scope, and click New.

b.

Specify TrafficAlerterUtil as the name and //TrafficAlerterUtil.jar as the classpath, and click New. Repeat this step to configure the second node.

Configure class loaders for the Member1 and Member2 servers. a.

Select Servers => Application servers and click Member1.

b.

Select Server Infrastructure => Java and Process Management => Class loader and click New.

c.

Select Classes loaded with parent class loader first and click OK.

d.

Click the created classloader.

e.

Click Additional Properties => Shared library references.

f.

Click Add and select TrafficAlerterUtil as the library name, and click OK. Repeat this step to configure for the Member2 server.

Configure the Service Integration Bus (SIBus) The SIBus configuration is very adaptable. You can have individual messaging engines (MEs) in an application server, or clustered multiple MEs for sharing workloads. You can also configure the ME to be highly available. We defined a bus member by adding all the servers to a bus called WbeBus (Figure 12). The WbeBus is created and a server is added as a member of the bus using the configure_bus.sh

Improving scalability with ObjectGrid partitions Page 20 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

script when we install and configure a Business Events server. (The WBE-1 and WBE-2 servers are added to the bus when those servers are configured.) See the WebSphere Business Events Installation and Operations Guide in Resources for information on installing WebSphere Business Events on an existing WebSphere Server instance. To define the SIBus topology, do the following: 1.

2.

Add the Test1 server and AppCluster as new members of the WbeBus using the WebSphere administrative console. a.

Select Service integration => Buses and click WbeBus.

b.

Click Topology => Bus members.

c.

Click Add.

d.

Select the radio button for the cluster, select AppCluster, and click Next.

e.

Select the radio button for the file store and click Next.

f.

Specify the log directory path and the permanent store directory path, and click Next.

g.

Click Finish.

Repeat this step for the Test1 server. Figure 12. Bus members on the WbeBus

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 21 of 42

developerWorks®

3.

ibm.com/developerWorks

By default, when you add a cluster to a bus, Application Server V6.1 does ME failover. It will ensure that only one instance of the ME is running at any given time within a cluster. Additional configuration steps are needed for each cluster member to have its own ME. To do this, we need to add the ME into the AppCluster bus member and create a static policy for the individual ME. The steps are: a.

Click Add messaging engine to add an ME into the AppCluster. Figure 13. Add a messaging engine into the AppCluster

b.

Click New to create a new static policy. Figure 14. Create a new static policy

c.

Specify the name of a static policy and click the Match criteria link to define the new static policy. Figure 15. Define the new static policy

Improving scalability with ObjectGrid partitions Page 22 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

d.

Define two match criteria in the policy. Figure 16. Define the match criteria

e.

Finally, click the Static group servers link. Add the Member1 server into the static group servers of this policy as shown in Figure 17. Figure 17. Add the Member1 server to the static group servers

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 23 of 42

developerWorks®

ibm.com/developerWorks

Repeat Step 3 to configure Member2. Install and configure the second Business Events server This section describes the step-by-step procedure for installing and configuring the second Business Events server such as WBE-2 on a different node. It describes how to run the supplied custom script files and how to define a DB2 database as a Business Events repository. 1.

Create the repository database for the WBE-2 server on the first node where you installed DB2. a.

Switch the db2 instance user ID. $ su - [userid]

b.

Create the database. We will use the name of the <BusinessEvents_DB_name> to define Business Events database settings at step 19. $ db2 create db <BusinessEvents_DB_name>

2.

Copy the db2jcc.jar file from the first node where we created the Business Events repository database to the second node.

3.

Install Business Events on the second node.

Improving scalability with ObjectGrid partitions Page 24 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

4.

5.

6.

developerWorks®

a.

Run the Business Events installer.

b.

Select Custom on the Choose Install Set dialog and click Next.

c.

Select WebSphere Business Events Technology Connectors and Database Server and click Next.

d.

Click Next on the Choose Install Folder dialog.

e.

Select IBM DB2 V9.5 and click Next.

f.

Click Choose… and specify the JDBC driver location.

g.

Enter the DB2 connection information for the first node.

h.

Next, run the Business Events Fix Pack 1 installer.

Using the WebSphere administrative console, create a new application server named WBE-2. a.

Select Application servers and click New.

b.

Select the second node, enter server name WBE-2, and click Next.

c.

Select a default template and click Next.

d.

Select a checkbox to generate unique ports and click Next.

e.

Click Finish.

Add host aliases. a.

Select Environment => Virtual Hosts and click default_host.

b.

Click Host Aliases.

c.

Click New.

d.

Specify the WC_defaulthost port number as the Port and click OK. You can find the port value for WC_defaulthost by selecting Application servers => WBE-2 => Ports.

Navigate to the <BusinessEvents_installation_dir>/config/was directory and edit the setenv.sh script file as follows: a.

Specify the existing Application Server binary directory, for example:

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 25 of 42

developerWorks®

ibm.com/developerWorks

WASHOME=/opt/IBM/WebSphere/AppServer b.

Specify the deployment manager’s hostname, for example: WASADMINHOST=wstiblade9.rtp.raleigh.ibm.com

c.

Specify the deployment manager’s bootstrap port number, for example: WASBOOTSTRAPPORT=9809

d.

Specify a profile name where you want to install the second Business Events server, for example: WASPROFILE=Custom01

e.

Specify the application server name where the Business Events run-time EAR file is to be installed. WASSERVER=WBE-2

7.

Copy the <BusinessEvents_installation_dir>/config/WbeConfig.jar file to the Application Server library directory.

8.

Run the start_was.sh script to start the WBE-2 server.

9.

Run the configure_server.sh script to configure the WBE-2 server.

10. Run the configure_bus.sh script to configure SIBus for the WBE-2 server. 11. Edit the configure_messaging.py script file as follows: a.

Add the following line to define a topic space for the WBE-2 server. topicSpaceName = topicSpaceName + '2'

b.

Add the following line to define name variables for the WBE-2 server. name = name + '2'

c.

Comment out the following line: # description = topics.get(name)

d.

Change the description for the WBE-2 server. '-description', "For WBE-2"]

12. Run the configure_messaging.sh script to configure JMS administrative objects for the WBE-2 server. 13. Specify the target messaging engine name for the queue connection factory using the WebSphere administrative console.

Improving scalability with ObjectGrid partitions Page 26 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

a.

Select Resources => JMS => Queue connection factories.

b.

Click WbeQueueConnectionFactory2.

c.

Specify the messaging engine name as Target for the connection and click OK. You can find the messaging engine name by selecting Application servers => WBE-2 => Messaging engines.

14. Specify the target messaging engine name for the topic connection factory. a.

Select Resources => JMS => Topic connection factories.

b.

Click WbeTopicConnectionFactory2.

c.

Specify the messaging engine name as Target for the connection and click OK. You can find the messaging engine name by selecting Application servers => WBE-2 => Messaging engines.

15. Repeat steps 12 and 13 to configure for the WbeQueueConnectionFactory and WbeTopicConnectionFactory on the WBE-1 server. 16. Edit the install_app.py script file. a.

Change the following line for the WBE-2 server. jdbcDriverLibrary = 'WbeJdbcDriver2'

b.

Add the following application name parameter. '-appname', "WBERuntimeEAR2", \

c.

Change parameters as follows. parameters += ['-MapSharedLibForMod', [['WBERuntimeEAR2', \

17. Run the install_app.sh script for installing the Business Events run-time EAR file into the WBE-2 server. 18. Navigate to the <BusinessEvents_install_dir>/director/bin directory and edit the connector.sh script file. We specify the from Step 2. export CLASSPATH=/db2jcc.jar 19. Next, specify the configuration properties using the Business Events properties UI.

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 27 of 42

developerWorks®

ibm.com/developerWorks

a.

Start the Business Events properties UI.

b.

Specify values for the Hostname, Instance, Username and Password fields. For the Instance field, we specified <BusinessEvents_DB_name>, which is defined in Step 1. Figure 18. Business Events Properties UI for Database Settings

c.

Click Next and specify values for the Store URL, Topic Connection Factory and Queue Connection Factory fields. For the Store URL field, we specified the WBE-2 bootstrap port. Figure 19. Business Events Properties UI for JMS Settings

Improving scalability with ObjectGrid partitions Page 28 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

d.

Click Full Configurator.

e.

Click the Common section and specify values for Subsection: Default Database Settings and Subsection: Asset Repository as shown in Figure 20. Figure 20. Business Events Properties UI for Asset Repository Settings

f.

Click the JMS section and specify values for Subsection: Topics.

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 29 of 42

developerWorks®

ibm.com/developerWorks

Figure 21. Business Events Properties UI for JMS Topics Settings

g.

Click the Logging section and specify values for log4j.logger.com, log4j.logger.event, log4j.appender.JMS1.TopicBindingName and log4j.appender.JMS2.TopicBindingName as shown in Figure 22. Figure 22. Business Events Properties UI for Logging Settings

Improving scalability with ObjectGrid partitions Page 30 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

h.

developerWorks®

Click the Server section and specify values for as.director.server.instanceName, as.director.server.ruleProcessorInstances and as.director.server.preProcessor.instances. Also specify values for Subsection: Timebased Repository Settings and Subsection: Eventflow Database Settings as shown in Figure 23. Figure 23. Business Events Properties UI for Timebased Repository and Eventflow Database Settings

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 31 of 42

developerWorks®

ibm.com/developerWorks

i.

Click File and select Save Properties.

20. Start the WBE-2 server using the WebSphere administrative console. 21. Deploy the TAProject-Partition.xml file using the Business Events Design Data tool.

Run the application on the Business Events multiple instances

Improving scalability with ObjectGrid partitions Page 32 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

To run the application on multiple instances, perform the following steps: 1.

Start a DB2 instance. $ su - db2inst1 $ db2start

2.

Start the WBE-1, WBE-2, and Test1 servers by using the WebSphere administrative console.

3.

Start the AppCluster.

4.

Navigate to the /opt/IBM/WBE/director/bin directory on the first node and start the Business Events connector-1. $ connectors.sh

5.

Navigate to the /opt/IBM/WBE/director/bin directory on the second node and start the Business Events connector-2. $ connectors.sh

6.

Run the launchClient.sh script to start the client application. The trafficAlerter.props file is described in Table 3 in part 1. We defined the deployment manager’s bootstrap port as providerURL in the trafficAlerter.props file. $ launchClient.sh ClientEAR.ear -CCclasspath=TrafficAlerterUtil.jar trafficAlerter.props

7.

After the client application is finished, navigate to the /opt/TrafficAlerts/WBE-1 directory which we specified for the file system connection in Figure 10. Check the action XML files. They should contain a sequence of odd numbers as a location id if our scenario is working well. To do this, confirm the value of the XML tag . Similarly, the action XML file in the /opt/TrafficAlerts/WBE-2 directory should contain even numbers as a location id.

Listing 10. Action XML files in the WBE-1 directory 21 30 <system>wstiblade9.rtp.raleigh.ibm.com

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 33 of 42

developerWorks®

ibm.com/developerWorks

2008-09-23T18:16:48.797-0400 Generated from IBM WebSphere Business Events 25 30 <system>wstiblade9.rtp.raleigh.ibm.com 2008-09-23T18:17:29.911-0400 Generated from IBM WebSphere Business Events 29 30 <system>wstiblade9.rtp.raleigh.ibm.com 2008-09-23T18:29:40.061-0400 Generated from IBM WebSphere Business Events

Listing 11. Action XML files in the WBE-2 directory 28 30 <system>wstiblade9.rtp.raleigh.ibm.com 2008-09-23T18:27:17.480-0400 Generated from IBM WebSphere Business Events 2 30 <system>wstiblade9.rtp.raleigh.ibm.com 2008-09-23T17:48:15.302-0400 Generated from IBM WebSphere Business Events

Improving scalability with ObjectGrid partitions Page 34 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

22 30 <system>wstiblade9.rtp.raleigh.ibm.com 2008-09-23T17:48:53.661-0400 Generated from IBM WebSphere Business Events

Performance Results To check the effectiveness of our solution, we measured the performance data for three different system topologies (patterns 1, 2, and 3). Each pattern consists of a single cell containing two nodes; each node runs on a separate machine (Figure 24, Figure 25, and Figure 26). Look carefully at Figure 26 (pattern 3). Because ObjectGrid partitions are defined as two partitions on each of two application servers, each application server has a primary partition and an asynchronous replica. As EJB requests from a client are spread over both servers by the EJB clustering function, the EJB accesses either a local or a remote partition. Stateless EJB instances aren’t always collocated with a primary partition. Accessing a local partition is very fast because it is in memory; but when the EJB accesses a remote partition (that another server is running as a primary), performance suffers because the access is across a network. Therefore, the primary consideration in a performance test should be the number of remote-access paths to the ObjectGrid partitions. This means that we shouldn’t compare the performance results of the part 2 solution against part 1 — the single partition in part 1 is always local and is neither highly available nor scalable. To correctly determine performance behavior in a distributed system environment, we chose to build the three system topologies and compare the results for the three systems. As shown in Figure 24, pattern 1 is running the TrafficAlerterFilter application and the WBE-1 server on a single node and the TrafficAlerter application is on another node. The important point to note in pattern 1 is that the EJB will remotely access the partition of the TrafficAlerterFilterOG over a network. Pattern 2 is running the TrafficAlerterFilter application and two Business Server instances across two nodes. Pattern 2 uses clustering for the TrafficAlerterFilter applications and multiple Business Events instances. In pattern 3 both the TrafficAlerter and TrafficAlerterFilter applications run in the cluster, and each is configured to use two partitions. Figure 24. Pattern 1 topology

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 35 of 42

developerWorks®

ibm.com/developerWorks

Figure 25. Pattern 2 topology

Improving scalability with ObjectGrid partitions Page 36 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

Figure 26. Pattern 3 topology

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 37 of 42

developerWorks®

ibm.com/developerWorks

We used the same code for all the patterns. We ran performance tests to change the number of total events that Business Events processed. For the purposes of the performance test, the WBERatio parameter in the TrafficAlerter.props file was set to 1 and every message was forwarded to the Business Events instances without applying the prefilter logic. The performance results are shown in Figure 27. The results show that the throughput is approximately 1.7 times better for patterns 2 and 3 than for pattern 1. This proves the effectiveness of our solution design for integrating Business Events with eXtreme Scale. Figure 27. Performance results

Improving scalability with ObjectGrid partitions Page 38 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

Summary This article describes important design considerations for building a scalable and highly available solution by using Business Events and eXtreme Scale. eXtreme Scale is used for high-volume raw events storage. A prefiltering application reduces the processing load on Business Events. Using context-based event routing, events are routed to appropriate partitions in the distributed ObjectGrid. Performance results show a significant improvement in throughput when clustering and distributed partitions are used. You learned how to implement this solution using the ObjectGrid Entity manager API, the ObjectGrid FIFO API, and the JMS API on the SIBus. You learned how to configure multiple Business Events instances and to define a DB2 database as a Business Events repository.

Acknowledgments The author would like to thank Gale Botwick, Jeffrey Garratt, and Daniel McGinnes for their careful review, comments, and feedback on this article.

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 39 of 42

developerWorks®

ibm.com/developerWorks

Downloads Description

Name

Size

Download method

Project XML file

TAProject-Partition.zip 4KB

HTTP

Deployable EAR files

deployableEARFiles.zip 111KB

HTTP

Custom scripts for Linux and Windows

customScripts.zip 5KB

HTTP

Source code for part 2

part2Source.zip 44KB

HTTP

Information about download methods

Improving scalability with ObjectGrid partitions Page 40 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

Resources Learn • Read part 1 of this article series Processing complex business events with WebSphere eXtreme Scale, Part 1: Reducing the load on WebSphere Business Events. • Learn the basic concepts of ObjectGrid in "Build a scalable, resilient, high performance database alternative with the ObjectGrid component of WebSphere Extended Deployment" (developerWorks, November, 2007). • Find details in the WebSphere eXtreme Scale and ObjectGrid V6.1 User Guide. • Read the WebSphere eXtreme Scale Information Center. • Read the Business Events Installation and Operations Guide. • Check out the WebSphere Business Events V6.1 Information Center. • Read the series "Business Event processing with WebSphere Business Events " (developerWorks, September, 2008) to learn about how Business Events interoperate with other IBM products. Get products and technologies • Download the latest Fix Packs for WebSphere Application Server. • Download the latest Fix Packs for WebSphere eXtreme Scale. Discuss • Check out developerWorks blogs and get involved in the developerWorks community.

About the author Kenji Kojima Kenji Kojima is a Developer at IBM at the WebSphere Technical Institute in Research Triangle Park, NC. He works on the integration of WebSphere Business Events with WebSphere eXtreme Scale. He previously worked as a Certified Consulting IT Specialist providing consultancy to IBM customers in Japan.

Trademarks

Improving scalability with ObjectGrid partitions © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 41 of 42

developerWorks®

ibm.com/developerWorks

DB2, IBM, and WebSphere are trademarks or registered trademarks of IBM Corporation in the United States, other countries, or both. Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States, other countries, or both. Red Hat and Red Hat Enterprise Linux are registered trademarks of Red Hat, Inc. in the U.S. and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other company, product, and service names may be trademarks or service marks of others.

Improving scalability with ObjectGrid partitions Page 42 of 42

© Copyright IBM Corporation 1994, 2008. All rights reserved.