Yawsa: Final Project Report

University of St Andrews

Service Composition with Hyper-programming SH Project Deliverable, Final Report

Angus Macdonald 4/17/2007

Angus Macdonald

Service Composition with Hyper-Programming

Abstract This report introduces YaWSA (Yet another Web Service Acronym) - a web-based system designed to simplify the orchestration and testing of Web Service compositions. The current trend in software architecture is towards building platform-independent components made available as Web Services. In enterprise computing the composition of such services is viewed as important in automating business processes, thus helping to streamline application integration. Current approaches to Web Service composition require considerable knowledge of either low level XML syntax or model based descriptions in UML. Interactive component integration and composition testing is challenging in these frameworks, as compositions must be created statically then compiled into a flow specification before they can be run. The general scarcity of public Web Services seems to highlight the need for a tool that makes their invocation and composition simpler. YaWSA provides a more accessible approach to Web Service composition, allowing users to compose services from a web-based interface rather than using a UML or XML editor. The result and structure of invocations can then be stored, allowing users to interactively compose and test Web Services. This is achieved through the application hyper-programming, a paradigm for system construction, in the creation of compositions. This report identifies the motivations behind YaWSA and discusses its implementation.

Page | 2

Angus Macdonald

Service Composition with Hyper-Programming

Declaration I declare that the material submitted for assessment is my work except where credit is explicitly given to others by citation or acknowledgement. The main text of this project report is 19,999 words long. In submitting this project report to the University of St Andrews, I give permission for it to be made available for use in accordance with the regulations of the University Library. I also give permission for the title and abstract to be published and for copies of the report to be made and supplied at cost to any bona fide library or research worker, and to be made available on the World Wide Web. I retain the copyright in this work.

Angus Macdonald.

Page | 3

Angus Macdonald

Service Composition with Hyper-Programming

Contents Abstract .......................................................................................................................... 2 Declaration .................................................................................................................... 3 Table of Figures ............................................................................................................ 8 Introduction ............................................................................................................... 10 Existing Approaches ................................................................................................................. 10 YaWSA ..................................................................................................................................... 10 Use Case................................................................................................................................... 11 Report Structure....................................................................................................................... 12

Context Survey ........................................................................................................... 13 Hyper-Programming ................................................................................................................ 14 Introduction ........................................................................................................................ 14 Hyper-Programming on the Internet.................................................................................... 15 Current Research...................................................................................................................... 16 Web Service Composition .................................................................................................... 16 Interactive Composition ........................................................................................................... 17 Web Service Modeling ............................................................................................................. 17 Existing Applications ................................................................................................................ 18 Mindreef SOAPscope........................................................................................................... 18 Yahoo! Pipes ....................................................................................................................... 18 Web Service Invocation Libraries.............................................................................................. 18 Other Technologies .................................................................................................................. 18 Apache Tomcat ................................................................................................................... 18 Java Dynamic Compiler........................................................................................................ 19 Conclusion................................................................................................................................ 19

Project Details ........................................................................................................... 20 Software Engineering Practices ................................................................................................ 20 Requirements Engineering .................................................................................................. 20 Development Methodology ................................................................................................. 20 Page | 4

Angus Macdonald

Service Composition with Hyper-Programming

Testing ................................................................................................................................ 21 System Overview...................................................................................................................... 21 Design Decisions ...................................................................................................................... 23 Server Language .................................................................................................................. 23 Web Service Invocation ....................................................................................................... 23 User Interface ..................................................................................................................... 24 Conclusion .......................................................................................................................... 25 Design Overview ...................................................................................................................... 25 Use Cases ................................................................................................................................. 26 Use Case A .......................................................................................................................... 26 Use Case B .......................................................................................................................... 28 Use Case C .......................................................................................................................... 31 System Walkthrough................................................................................................................ 34 Introduction ........................................................................................................................ 34 Step 1: User Input (Enter Service Address) .......................................................................... 34 Step 2: Interpretation of Compositions ................................................................................ 35 Step 3: Serializing Data ........................................................................................................ 44 Step 4: Invocation of Compositions...................................................................................... 47 Step 5: Web Service Responses - Results Decomposition ..................................................... 50 Step 6: Post-Invocation Storage ........................................................................................... 52 Novel Features ......................................................................................................................... 54 User Interface ..................................................................................................................... 54 Serialization of Data ............................................................................................................ 55

Testing ......................................................................................................................... 56 Unit Testing.............................................................................................................................. 56 Prototyping .............................................................................................................................. 56 Usability Testing ...................................................................................................................... 57 Functionality to Test............................................................................................................ 57 Target Audience .................................................................................................................. 57 Questions and Results ......................................................................................................... 57

Page | 5

Angus Macdonald

Service Composition with Hyper-Programming

Conclusion .......................................................................................................................... 59 Heuristic Evaluation ................................................................................................................. 59 Efficiency Testing ..................................................................................................................... 60 System Startup .................................................................................................................... 60 Proxy Generation – Single Operation Service ....................................................................... 60 Proxy Generation – Multiple Operation Service ................................................................... 60 Conclusion .......................................................................................................................... 61

Evaluation and Critical Appraisal ........................................................................ 61 Software Engineering............................................................................................................... 61 Development Model............................................................................................................ 61 Testing ................................................................................................................................ 62 Objectives ................................................................................................................................ 62 Primary ............................................................................................................................... 62 Secondary ........................................................................................................................... 63 Tertiary ............................................................................................................................... 63 Conclusion .......................................................................................................................... 63 Requirements ........................................................................................................................... 64 Functional Requirements..................................................................................................... 64 Non-Functional Requirements ............................................................................................. 65 Conclusion .......................................................................................................................... 67 Design Decisions ...................................................................................................................... 67 Server ................................................................................................................................. 67 Web Service Invocation ....................................................................................................... 67 User Interface ..................................................................................................................... 68 Comparison with Existing Systems ........................................................................................... 68 Web-based Invocation......................................................................................................... 68 Service Composition ............................................................................................................ 68 Conclusion .......................................................................................................................... 69

Conclusion .................................................................................................................. 70 Achievements of Project........................................................................................................... 70 Page | 6

Angus Macdonald

Service Composition with Hyper-Programming

Future Direction of Work.......................................................................................................... 70 Persistent Storage ............................................................................................................... 70 Flexible Service Invocation................................................................................................... 71 Description of Compositions................................................................................................ 71 Integration into Middleware................................................................................................ 72 Separate Content from Interface ......................................................................................... 73 Visualization of Compositions .............................................................................................. 73 General Comments................................................................................................................... 73

Bibliography............................................................................................................... 74 Appendices ................................................................................................................. 78 Appendix A: Maintenance Documentation ........................................................................... 79 Appendix B: Project Specification and Plan .......................................................................... 82 Appendix C: Literature Review and Design ........................................................................... 98 Appendix D: Status Report ................................................................................................... 121 Appendix E: Testing Summary ............................................................................................. 123

Page | 7

Angus Macdonald

Service Composition with Hyper-Programming

Table of Figures Figure 1: Web Services programming stack. Web Services operate over a standard network layer protocol (generally HTTP), generally being serialized in SOAP, but not limited to this. .............. 13 Figure 2: Prototype One; Command-Line Invocation. The system consists of a single component, which communicates across the network to the Web Service being called. ................................. 21 Figure 3: Prototype Two; Server-Side Compositions. A composition component is added to the server, allowing users to build up interactions between Web Services. The original ‘Web Service Invocation’ component is still responsible for invoking individual services provided by the new component. ............................................................................................................................... 22 Figure 4: Prototype three, final System; user interface added. ........................................................... 22 Figure 5: User's Conceptual Experience. The system now provides a conceptual viewpoint that is different from its actual operation; from a user’s viewpoint, they are interfacing directly with a Web Service, but, in reality, their request is passing through the system. .................................... 23 Figure 6: Data flow through the system. Starting with the client (at A) a user’s communication with a Web Service is handled through a Tomcat servlet (B) and a back-end server (C). Finally, the backend server communicates directly with the Web Service. ........................................................... 25 Figure 7: The YaWSA homepage. Users see this page when they first connect go to the website. ....... 26 Figure 8: YaWSA Screenshot. The user is given a list of available operations..................................... 27 Figure 9: YaWSA screenshot. Having chosen the 'add' operation the user can enter values for the two parameters................................................................................................................................. 27 Figure 10: YaWSA screenshot. The webpage returns the result of a user's invocation. ........................ 28 Figure 11: YaWSA screenshot. The screen displayed to the user once they have decided to invoke the 'divide' operation. ...................................................................................................................... 29 Figure 12: YaWSA screenshot. The page now allows the user to enter the address of the service they wish to use in the composition. .................................................................................................. 29 Figure 13: YaWSA screenshot. The page returned once the user has chosen to use the 'add' operation. They can now enter parameters for the add operation; the result of which will be returned to be used as a parameter to 'divide'. ................................................................................................... 30 Figure 14: YaWSA screenshot. The system displays the result of invoking the AddWS-DivideWS composition. ............................................................................................................................. 31 Figure 15: YaWSA screenshot. The user can tag an object with information so that it can be stored and used at a later date. .................................................................................................................... 32 Figure 16: YaWSA screenshot. The form given when a user chooses the add operation in AddWS. .... 32 Figure 17: YaWSA screenshot. The user is given a choice of suitable objects for this parameter. They must choose from one of these. .................................................................................................. 33

Page | 8

Angus Macdonald

Service Composition with Hyper-Programming

Figure 18: YaWSA screenshot. The user has chosen the stored object that they wish to use in the invocation. ................................................................................................................................ 33 Figure 19: Left - State of the tree once a user has chosen to invoke the DivideWS service. ................ 36 Figure 20: Function Wrapper. The format of the data structure used to represent references to Web Services in composition. ............................................................................................................ 36 Figure 21: Tree structure now that the user has chosen an operation. The form (right) is the result of traversing this tree. .................................................................................................................... 37 Figure 22: Tree structure updated to reflect the user's decision to use a Web Service Reference as a method of input. ........................................................................................................................ 39 Figure 23: Tree structure now that the user has selected the AddWS service, and the add operation. . 40 Figure 24: UML Inheritance Diagram for Tree. Illustrates how the tree structure has been implemented in YaWSA. ........................................................................................................... 41 Figure 25: View of the DivideWS-AddWS invocation tree as implemented in YaWSA ......................... 42 Figure 26: Flattening/Restoration of Tree Structure. A user’s input on the webpage is serialized in the form shown, making it possible to reconstruct its original structure on the server. ....................... 45 Figure 27: Representation of a composition client-side (left) and server-side (right). This figure shows how node IDs can be used to place values into the appropriate place on the server. The YaWSA screenshot (left) shows the IDs of text-boxes in italicized red text. In reality these IDs are not visible to the user. ..................................................................................................................... 46 Figure 28: Form Data sent from Client. The serialized structure of the form data sent to the server. Each form field is stored, by including the ID of that field, and its value..................................... 46 Figure 29: Data flow through the system for an AddWS-DivideWS composition. The client submits their composition at A, which is sent to the system at B. This is passed onto the back-end server at C, where the composition is traversed, and the Web Service at D invoked. ................................. 47 Figure 30: Invocation tree, with all values entered. This tree is ready to be invoked. ......................... 48 Figure 31: State of the DivideWS-AddWS invocation tree after AddWS has been invoked................... 49 Figure 32: Object Store Structure. The object store is a map, containing key->value pairs of type>map, where the map contains stored objects of the given type. ................................................. 52 Figure 33: Node ID Replacement. When an existing composition (Tree B) is added to a new composition (Tree A) the node IDs for the existing composition must be updated to reflect their new position in the tree.............................................................................................................. 54 Figure 34: Atomic (left) and Composite (right) Services. Composite services are externally identical to atomic services, but encapsulate some kind of Web Service composition.................................... 71

Page | 9

Angus Macdonald

Service Composition with Hyper-Programming

Introduction The current trend in software architecture is towards building platform-independent components made available as Web Services. These services, defined in open protocols, are self-describing, and aim to ease the integration of components on disparate computers. In enterprise computing the composition of such services is viewed as important in automating business processes, thus helping to streamline application integration. There have been considerable efforts in industry and academia to this end, with flow specification languages such as BPEL4WS (Andrews, et al., 2001) becoming common in service-oriented architectures. Existing research has focused on model-based approaches to composition (Skogan, et al., 2004)(Baresi, et al., 2001) though these are limited in their accessibility, as they require knowledge of model based descriptions in UML and several distinct toolsets to create and invoke compositions.

Existing Approaches The academic and commercial focus on Web Services has produced numerous systems facilitating their invocation. Of the websites that allow services to be invoked, most offer limited functionality. For example, Microsoft ASMX files allow invocation of operations with primitive parameters, but are not capable of invoking services with arbitrary data types or arrays. This makes the testing and invocation of these services difficult. To be able to thoroughly test Web Services a more advanced tool is needed; one that can test all operations. No toolset currently allows for web-based composition of Web Services, though sites such as Yahoo! Pipes (Yahoo!, 2007) have shown the potential for such an approach1. Web Service composition is still inaccessible for the average user Existing tools are aimed at enterprise computing2, neglecting the needs of small companies and individuals who represent a large potential market for Web Service consumption. The general scarcity of public Web Services seems to highlight the need for a tool that makes their invocation and composition simpler. Tim O’Reilly argues that Yahoo Pipes is “a milestone in the history of the internet … *with+ enormous promise in turning the web into a programmable environment for everyone” (O'Reilly, 2007). While Pipes has made great progress in feed orchestration, it is not capable of Web Service invocation, and consequently ill-equipped to handle the move towards a semantic web (Berners-Lee, et al., 2001). This vision relies on a large number of publicly available Web Services, yet the current complexity in 1

2

Yahoo Pipes allows for the web-based composition of RSS feeds, not Web Services. For example, Borland Together Developer supports the creation of business processes in BPEL4WS, a flow specification language. Other toolsets offer similar approaches, generally using modelling languages such as UML to support composition.

Page | 10

Angus Macdonald

Service Composition with Hyper-Programming

using these services has meant that few are available. Accessibility must become a key concern, if Web Services are to reach their potential. To move significantly towards this goal, a web-based tool for service composition is needed, to give impetus to the creation and use of open Web Services.

YaWSA This report introduces YaWSA (Yet another Web Service Acronym) - a system designed to simplify the orchestration and testing of Web Service compositions through a web-based interface. YaWSA is the realization of the original project specification (Appendix B, page 82), which required the creation of a system “permit[ting] Web Services that have been deployed … to be invoked and composed from a standard Web Browser”, and more specifically “the incorporation of hyperprogramming technology permitting arbitrary objects from multiple address spaces to be passed to the services”. This sets YaWSA apart from many contemporary systems in allowing the invocation of services with arbitrary types, arrays, and enumerations. YaWSA provides a more accessible approach to Web Service composition than enterprise flow specifications, allowing users to compose services from a web-based interface rather than through a UML or XML editor. The system also allows the result and structure of invocations to be stored, thus providing a means for the interactive testing and construction service compositions. There is a need for such a system because existing approaches focus on commercial flow specifications, overlooking the need for service composition on the internet. Current approaches are either too complex or ill equipped to be used by Web developers, as integration and composition in existing systems is difficult. This report shows how YaWSA attempts to rectify these deficiencies, and presents an analysis of areas where further work is needed.

Use Case The original project specification defined a use case which will be used throughout this report as a motivating example. It is reprinted below: A company provides two web services; AddWS, which adds two numbers (given as parameters) then returns the result, and DivideWS that divides one parameter from another then returns the result. An employee wishing to find the average of two numbers has to invoke AddWS and DivideWS in sequence, creating an individual web services client for each task. The proposed system helps this employee in two ways. Firstly, to invoke these services the user doesn’t have to create a web services client – they simply give the system a link to the services description file. Having done this the employee can now chain the services by piping Page | 11

Angus Macdonald

Service Composition with Hyper-Programming

the output from AddWS into DivideWS as a parameter, removing the need for the results to be copied manually. If this is a task that has to be performed frequently the employee can save the newly created composite Web Service and return to use it when required. There are three pieces of distinct functionality that this use case covers: 1.

Service Invocation. How Web Services will be invoked by the system. This could be handled by an existing Web Service library, or by creating a component to handle serialization and derserialization into SOAP.

2.

The representation of compositions on the server prior to and during invocations. The system has to allow for an appropriate level of flexibility, while keeping the data format manageable and accessible.

3.

The method of displaying information on the webpage. Both Java Applets and HTML are options here. The key issue here is the difficulty in serializing and deserializing data to and from the webpage, and the system’s method of handling this.

These issues, and the solutions to them, are discussed in the project details section on page 20.

Report Structure This report provides an account of the problem being solved, the work done towards this goal, and an analysis of the resultant system. It is divided into five sections. The Project Details section presents an overview of the system, discussing the main achievements of the project along with the main elements of the design and novel features of the implementation. A use case is used to clarify these points and to illustrate the reasoning behind, and need for, particular aspects of the implementation. The Testing section details the steps taken to ensure reliability and quality within the system, and introduces the results of usability testing, used to evaluate the system’s user interface. YaWSA is assessed against the original system objectives and requirements (Appendix B, page 82) in the Evaluation section. This section presents a critical analysis of the system, with respect to related work in the public domain. The report finishes with a Conclusion, which summarises the achievements of this project, and reflects on significant accomplishments and remaining weaknesses. This summary is used as a basis to discuss future directions for the project, and possible avenues of research. Finally a set of Appendices contain previously published deliverables including the project specification, project plan, literature review and design. Also included is a status report summarizing the state of the delivered software and a summary of the testing strategy used during development.

Page | 12

Angus Macdonald

Service Composition with Hyper-Programming

Context Survey This section is based on the context survey and literature review contained within the project’s specification and literature review (Appendices B & C, page 82). It is included here to introduce some of the domain specific terminology used throughout this document. Web Services A web service is any service available over the Internet that uses a standardized XML messaging system, is self-describing, and is not tied to any one operating system or programming language (Cerami, 2002). Web Services differ from existing service oriented architectures by being defined through open standards, which means that developers are not obliged to use proprietary applications. The Web Services architecture consists of a set of protocols, each used for a distinct purpose. Figure 1 gives examples of these protocols and the functions that they perform. BPEL4WS

SERVICE FLOW

PUBLICATION & DISCOVERY

WSDL

SERVICE DESCRIPTION LAYER

SOAP

XML-BASED MESSAGING LAYER

HTTP, FTP, SMTP, etc.

NETWORK LAYER

Figure 1: Web Services programming stack. Web Services operate over a standard network layer protocol (generally HTTP), generally being serialized in SOAP, but not limited to this.

The bottom three layers of this stack are required in order to have interoperable web services, with the remaining two providing desirable but optional functionality. The following sections describe the protocols that have become standards for the service description layer and messaging layer. Web Service Description Language (WSDL) WSDL (Christensen, et al., 2001) is the standard for the service description layer of the Web Services stack, allowing descriptions of the methods provided by particular Web Services. These services are defined by abstract method definitions, transport protocols employed for communication with the service, and the location of the service in the form of a URL. While WSDL is in some ways limited (e.g. it isn't able to describe transactions), it has become the de facto standard for the service description layer, with higher layers in the stack utilizing it. SOAP SOAP (Box, et al., 2000) is a message layout specification that describes a model for passing queries and responses in XML. It originally stood for Simple Object Access Protocol, but later specifications dropped the acronym.

Page | 13

Angus Macdonald

Service Composition with Hyper-Programming

Like WSDL, SOAP remains a W3C recommendation, but is the most common protocol in use at the messaging layer. It's only direct competitor is XML-RPC, but Representational State Transfer (REST) is SOAPs most widely used challenger. Not a specification, but an architectural style, REST is often used without being realized or the technology understood (Burnham, 2004). Examples of REST web services include Google's Ad-sense network and Amazon's Affiliate network which only require a few lines of HTML to call.

Hyper-Programming Introduction Hyper-programming is a paradigm for system construction, where a source program (a hyperprogram) contains both text and links to objects. Contrast this with traditional programs that are limited to referring to purely textual data (Kirby, et al., 1992). Hyper-programs contain denotations to data that will be resolved to the actual data prior to execution. This resolution is made possible through three items contained within these denotations (Kirby, 1992): 

Data Item: The value to be resolved. In Web Services composition this corresponds to the data returned by a service.



Access Path: This is a description of the position of a data item. The access path of a Web Service is its URI.



Access Specification: Contains the access path of an item along with a description of its expected type. The Service Description layer of the Web Services Architecture performs this function with WSDL.

The first implementation of a hyper-programming system was created in the Napier88 persistent programming environment (Morrison, et al., 1989), with a Java implementation created later (Zirintsis, et al., 1994). A major motivation for providing such a system is to allow the application programmer to compose programs interactively, selecting items to be incorporated into their programs (Farkas, et al., 1992)(Kirby, et al., 1992). Users should be able to navigate the value space, selecting data items for inclusion to their source. In doing this the complexity is reduced, as access specifications have no need to be written for data items used by the program. This enables type checking to be performed earlier at compile-time, ensuring that all items can be accessed at run-time (the access specification can change). If incorporated into the web services architecture, hyper-programming could help reduce the need to write complex process specifications, by allowing services to be composed interactively.

Page | 14

Angus Macdonald

Service Composition with Hyper-Programming

Hyper-Programming on the Internet Hyper-Programming requires the presence of an external value space where bindings can be constructed during program composition. While this is typically provided by a persistent store, a file system or other mechanism such as the World Wide Web is equally valid (Morrison, et al., 1999). A proposal for hyper-programming on the internet was first discussed by Morrison et al. in 1999, but the technology required to extend the paradigm was inadequate at the time. There were significant technical difficulties, including how to integrate data from external web sources in a typed computation, and how to integrate typed data with the HTTP protocol. Since then the standardisation of XML and the creation of Web Services standards such as WSDL have helped to alleviate these problems, though there is no accepted solution for making the potential failure of references tolerable. A hyper-programming implementation for the internet, HyperWeb (Fox, et al., 2004), has been developed and attempts to solve a number of these problems through a model of web application program safety that aims to ensure referential integrity. Other research outwith hyper-programming has focused on adapting existing internet fault tolerance techniques, such as load balancing and data replication (Anderson, 1996)(Aghdaie, et al., 2001)(Aghdaie, et al., 2002), to Web Services. A client-transparent implementation of such a system by Aghdaie & Tamir works on a three-tier architecture, where server failure is detected by a front-end server and requests are routed to a backup server. The scheme ensures that requests being processed at the time of a crash are completed through use of TCP’s error control protocols. Asynchronous JavaScript + XML (AJAX) AJAX refers to the XMLHttpRequest technology in JavaScript that allows asynchronous communication from web-pages, outwith the normal confines of HTTP page requests. The recent explosion in the number of AJAX web pages would not have been possible without the AJAX and DOM libraries that have abstracted away from the intricacies of JavaScript. This section looks at some of the most prominent libraries available, as they could form a basis for the system’s interface. AJAX libraries can be broadly split into three categories: those that abstract over commonly used JavaScript functionality; those that provide functions for developing widgets and visual effects in web pages; those that make asynchronous communication with server languages (e.g. Java) easier by wrapping the XMLHttpRequest object. Prototype Packaged as a 45kb JavaScript file, Prototype (Protopage, 2005) aims to ease the development of dynamic web pages. It provides various programming shortcuts to commonly used functions such as XMLHttpRequest, but requires users to program higher-level visual interfaces commonly associated

Page | 15

Angus Macdonald

Service Composition with Hyper-Programming

with AJAX applications. As a result a number of other frameworks have been built on top of Prototype, providing programmers with more support for interface design. script.aculo.us script.aculo.us is one of a number of toolkits using Prototype. It offers support for various effects and controls such as drag-and-drop lists, visual effects, and AJAX interaction. At the time of writing it is one of the most popular AJAX libraries available (Gottipati, 2006).

Current Research Web Service Composition Web Services Composition is the chaining of a set of appropriate web services to produce an application. Researchers in the field see great potential in reducing overheads for enterprise application integration, by automating the integration of components (Srivastava, et al., 2003). Two approaches have emerged to describe the functionality of web services (beyond the provision of WSDL), extending into descriptions of compositions - semantic annotation identifies what a service does, while functional annotation describes how a service behaves (Srivastava, et al., 2003). Functional Annotation (e.g. BPEL4WS) Functional annotation is used in Web Service composition to express how a service behaves. Services using this approach are expressed within flow specifications that define the order in which messages are exchanged between services. There are an excess of Web Services standards vying for standardization at present (LaMonica, 2004), however the Business Process Execution Language for Web Services (Andrews, et al., 2001) is currently gaining most ground amongst flow specifications. Formed from a collaboration between IBM and Microsoft, BPEL4WS superseded both corporations’ previous attempts at a flow language; WSFL (Laymann, 2001) and XLANG (Thatte, 2001) respectively. The result of combining features from both specifications is a language which is very complex (Staab, et al., 2003). Despite this it is fast becoming a standard among functional annotation languages. Semantic Annotation (e.g. DAML-S) Semantic Annotation languages specify pre-conditions and the effects of services, using standard ontologies. While this technique hasn’t found the kind of commercial support that functional annotation languages have, it has been widely discussed within the semantic web community (Berners-Lee, et al., 2001). Currently semantic annotation efforts have focused on describing web services using the Resource Description Language (RDF), and using the DARPA Agent Mark-up Language [DAML] (Ankolenkar, et al., 2001) to provide a sufficiently expressive ontology to represent the properties of web services.

Page | 16

Angus Macdonald

Service Composition with Hyper-Programming

Comparison Semantic annotation, if fully realized, takes most of the work of composing services away from human operators. In this sense functional annotation protocols are limited, as they can only describe how a service will behave, leaving creating compositions as additional work. Despite this, semantic annotation protocols are largely unproven, so remains a need for tools facilitating manual composition.

Interactive Composition Several efforts are underway to create a method of interactively composing semantic Web Services. These raise a number of issues relevant to both semantic and functional compositions. Kim and Gil (Kim, et al., 2004) found that, even for a small number of services, users would benefit from the assistance of tools that allowed them to specify complete and correct pathways in compositions. This underlines the importance of guiding users – first by showing available compositions, and then preventing invalid invocations. Wenjun et al. (2005) break web service composition into three stages: designing process, binding activities with concrete Web Services, and assigning data flow in the process. In assigning data flow it has to be assumed that all services in a flow use coincident message types, so that the output of one service is of the same type as the input of another. This assumption is flawed, as it is common for two messages to be subtly heterogeneous, despite both representing similar semantics (Wenjun, et al., 2005). Many composition frameworks simply rule out communication between services with heterogeneous but similar types. Others such as SEWSIP (Semantic Based Web Service Integration Platform) include a message transforming service that converts the output from one service to be compatible with the input of another service. Extending this concept to allow another intermediate state would create an additional layer to Web Service composition. This has not been covered in any existing Web Services research.

Web Service Modeling There have been various efforts to create applications that visualize web services and compositions of web services; most focusing on description using UML (Baresi, et al., 2001)(Johnston, 2005). Numerous other attempts at visualizing web services are either defunct (WSUI) or early in development [KWSDL + Kung, WSGUI (Web061)]. The primary benefit of graphically modeling web services is that the structure of compositions can be separated from specific composition languages (such as BPEL4WS), and converted based on user preference.

Page | 17

Angus Macdonald

Service Composition with Hyper-Programming

Existing Applications Mindreef SOAPscope Of all the current web based Web Service tools, Mindreef SOAPscope (Min06) is the most complete. Users can invoke any web service provided they can provide the corresponding WSDL file, even when parameters are non-scalar values. The site allows users to customize HTTP headers and SOAP sent to the service, and provides comparison and analysis tools for WSDL files. Yahoo! Pipes Released in February 2007, Yahoo Pipes is a realization for early interface designs for this project, applied to RSS feeds not Web Services. Registered Yahoo users can create compositions from RSS feeds, get requests, and user input. The site’s interface is of particular interest as it uses various drag and drop AJAX components, and has been praised by numerous respected Web 2.0 commentators as being innovative (O'Reilly, 2007)(Techcrunch, 2007).

Web Service Invocation Libraries A number of libraries are available that assist in the creation of Web Services and Web Services Clients. This section looks at two such programs. Apache Axis A part of the Web Services project at the Apache Software Foundation, Axis (Apache, 2006) is an implementation of the SOAP submission to W3C. It was originally based on IBM’s SOAP4J library but has since been redesigned, and versions now exist for Java and C++. Axis is the most complete Java implementation of SOAP, allowing programmers to create Web Services and client proxies for Web Services that handle the generation and transport of SOAP messages. .NET Framework Microsoft provide support for sending SOAP via HTTP(S) in the .NET framework (Microsoft, 2006), and give IDE support through Visual Studio .NET. These tools provide similar functionality to Apache Axis.

Other Technologies This section describes a number of technologies relevant to the YaWSA implementation of the project. Apache Tomcat Tomcat (Apache, 2005) is a web container that implements the Servlet and JSP specification from Sun Microsystems. It can function as a standalone Web Server or in combination with another, such as an Apache HTTP Server. Its suitability for integration with Java programs has made it a popular program to the extent that it became a top-level Apache project in September 2005.

Page | 18

Angus Macdonald

Service Composition with Hyper-Programming

As the server-side section of the system will be written in Java, Tomcat provides useful functionality for integration with a dynamic webpage. Java Dynamic Compiler The Java Dynamic Compiler, developed at the University of St Andrews, allows java files to be compiled and used by a Java program at runtime (Kirby, et al., 1998) (Kirby, 2006).

Conclusion There are a number of systems (such as SOAPscope) which fulfill the basic criteria of this project. However none of these systems allow for the interactive composition of Web Services, or consider the possibility of an intermediate data store, as set out in this project. The system allows interactive composition of services. At present there is no other software in the public domain that achieves this, as business processes are written by hand, or with less interactive toolsets. No other system considers the temporary storage of data for use among Web Service invocations. This is, in part, a result of the desire for automation of processes, making such a system unattractive. However a system providing a data store gives advantages in testing – where services are being configured and linked – and provides a useful tool where no pre-set composition will suffice. Such a system would make the process of creating compositions considerably easier.

Page | 19

Angus Macdonald

Service Composition with Hyper-Programming

Project Details YaWSA is an adventurous and well planned realization of the system described in project design documents (Appendix C, Page 98). This section presents an outline of the system, broken down into five areas: 

Software Engineering Practices (page 20). A discussion of the software development practices used in the project.



System Overview (page 21). A brief introduction to the overall structure of the system.



Design Decisions (page 23). A summary of decisions made before work on YaWSA began. These decisions include the choice of programming language for the server and interface.



Design Overview (page 25). An overview of the system structure, incorporating design decisions from the previous section.



Use Cases (page 26). The section introduces three use cases that are used to showcase YaWSA’s interface and provide an example of typical interactions with the system.



System Walkthrough (page 34). A detailed examination of the YaWSA implementation. This section first presents an overview of the entire system, before looking at individual components in more detail.



Novel Features (page 54). A summary of the features that distinguish YaWSA from comparable existing systems.

Software Engineering Practices Requirements Engineering Robert Glass noted that “requirement deficiencies are the prime source of project failures” (Glass, 1998). Consequently the initial focus of work on the project was in developing adventurous but achievable requirements. These were validated using checks for consistency, completeness and realism, as advocated by Sommerville (2004). Once finalized, these requirements were used as the basis of development on the system. Development Methodology Development models are integral to the software engineering process, in helping to guide development, and compensate for risks. Accordingly, the choice of development model for YaWSA was well-considered and reasoned. The waterfall model was considered due to its close match to course deliverables, but was deemed too inflexible, being unable to cope with a number of risks identified in the system plan (Macdonald, 2006). Ultimately Extreme Programming (Wells, 2006) was chosen, as it factors risk well, while allowing for the prototyping necessary for end of semester deliverables. In recent years it has become

Page | 20

Angus Macdonald

Service Composition with Hyper-Programming

one of the most prominent agile development methodologies, by promoting simplicity and feedback through test-based development. The system plan (Appendix B, page 82) made Extreme Programming integral to the success of the system, by proposing the creation of operational prototypes for required deliverables and internal deadlines; and introducing plans for the regular testing process. Testing Extreme Programming was chosen partly as a result of its emphasis on test-driven development. However, this alone was not sufficient for testing of the system, and so other complementary strategies were chosen. The Nielsen-Norman law states that “usability is quantifiable” (Endres, et al., 2003). In view of this a set of usability tests were designed to help identify the effectiveness of the proposed interface. In addition to this and unit testing, program profiling was planned to ensure the system met certain speed requirements. These points are discussed further in the testing chapter on page 55, and the model itself is evaluated on page 61.

System Overview Initially, the system’s design was split into three key areas to aid the prototyping effort (described in Testing, page 56) and to focus development at crucial stages. These areas were: 1.

Web Service invocation. The initial prototype allowed users to invoke arbitrary services from a command line interface. Figure 2 (below) shows the state of system at this stage in prototyping. The user interacts with the system’s command line interface to invoke a service: they provide the address of the service they wish to invoke, and then the system gives them a series of options allowing them to select an operation and enter values for its parameters. System Command Line Interface User

Web Service Invocation

Web Services

Figure 2: Prototype One; Command-Line Invocation. The system consists of a single component, which communicates across the network to the Web Service being called.

At this point the system consists of two components – a temporary interface (to be replaced later), and a module capable of making calls across the network to a given Web Service.

Page | 21

Angus Macdonald

2.

Service Composition with Hyper-Programming

Composition. The second prototype added the ability to compose Web Services. This required extending the system to allow compositions to be built up before being invoked. Figure 3 illustrates the progression from the first prototype. System Command Line Interface User Composition Representation

Web Service Invocation

Web Services

Figure 3: Prototype Two; Server-Side Compositions. A composition component is added to the server, allowing users to build up interactions between Web Services. The original ‘Web Service Invocation’ component is still responsible for invoking individual services provided by the new component.

Now, instead of a single invocation, the system can handle multiple invocations within a composition, by allowing users to build up interactions between services. To invoke an interaction the system traverses its representation of the composition, making a call to the Web Service Invocation component every time a Web Service is to be called in the composition. The result of this call can then be used for further invocations. 3.

User Interface; the final prototype. Work on this component concerned the creation of a usable interface for composition. Figure 4 shows this interface as being outside the system, as it resides within the client’s Web Browser - unlike previous prototypes where the user interacted with a command line interface in the system. Users now access the system indirectly, through their Web Browser. System response User Interface

Composition Representation

User

Web Service Invocation

Web Services

Figure 4: Prototype three, final System; user interface added.

The final prototype (Figure 4) is the first to provide a suitable interface to the end user. This introduces a novel conceptual viewpoint, around which the system is based. The user’s conceptual experience is of a direct communication with the Web Services they are invoking (through the interface) - though they are working through the system, which is acting as a silent intermediary. Figure 5 illustrates this model below. Page | 22

Angus Macdonald

Service Composition with Hyper-Programming

User Interface User response System

response Web Services request

Users Conceptual Model Actual Communication

Figure 5: User's Conceptual Experience. The system now provides a conceptual viewpoint that is different from its actual operation; from a user’s viewpoint, they are interfacing directly with a Web Service, but, in reality, their request is passing through the system.

YaWSA supports this conceptual view of the system, as it provides a useful abstraction from the underlying intricacy of structuring compositions. Existing systems fail to do this, as they often require transformation from a modeling language (such as UML) to a flow specification language (such as BPEL4WS) before this specification can be used in a separate process to run a composition. This makes composition unnecessarily difficult, as they are inaccessible to the average user.

Design Decisions Before development began a number of critical design decisions were made. These are discussed here, and then evaluated along with the rest of the system on page 67. Server Language The server (termed the ‘system’ in Figure 5) could be implemented in either Java or C#. Both languages have support for Web Service invocation, though C#’s is better integrated into the main language specification. However, the developer has over three years of experience in Java, and a Java solution would be more portable (C# server programs can only be run on the Microsoft IIS server3). As a result of the issues of portability and experience, Java was chosen to implement the back-end server solution. This choice was critical, as all other design decisions had to be compatible with a Java back-end solution. Web Service Invocation Web Services could be invoked from a Java back-end in one of two ways: via a Web Service library (which handles serialization), or a custom-made component, tailored for this system. The latter option would be preferable as it would give the solution a flexibility not offered by existing libraries 3

Linux implementations have been created, but are not supported by Microsoft.

Page | 23

Angus Macdonald

Service Composition with Hyper-Programming

that are not aimed at dynamic invocation. However, due to the tight schedule imposed by course deadlines it was not possible to implement a bespoke solution, so Apache Axis (Apache, 2006) – a Web Service library – was chosen. The decision to use Apache Axis over other Web Service invocation frameworks was made as a result of Axis’ ability to parse WSDL and generate proxies for individual web-pages. This helped to focus development on core requirements, rather than issues relating to message description and serialization. It also ensured that compositions were strongly typed server-side, as invocations are handled using Java classes. By representing proxies and their parameters as Java objects they were intrinsically strongly typed, ensuring the system is capable of differentiating between heterogeneous types, and can validate user input to services. However, this comes at a cost, as use of the framework requires a series of computationally expensive reflective operations to be performed server-side. These operations are necessary to find the operations provided by a service, and to perform the invocation. Another problem is the inability to obtain parameter names from the proxies, as it is not possible to access them using reflection. An alternative to this approach would be using lower level libraries such as WSDL4J and SOAPJ4 which were the initial basis of Apache Axis - or another toolset such as the Web Services Invocation Framework (WSIF). These would have required the creation of a program to parse the contents of WSDL files, and create code for serializing data into SOAP, meaning the typing of values would have to be considered. User Interface The choice of interface was critical to the system’s effectiveness, as it is the only component visible to the user. The system specification stated that the interface had to be accessible from a website, limiting the range of available options. As Java was chosen as the language of the back-end, two approaches were considered: an AJAX based HTML page, or a Java Applet. The eventual decision to use an AJAX based front-end to the system over a more standard approach (such as Java SWING) was not taken lightly, but it has been justified by the final systems’ utility and accessibility. AJAX presented a more adventurous task, as issues related to the serialization of messages to and from the Web Browser introduced a measure of complexity to the implementation. However, AJAX implementations are generally more accessible than Java SWING equivalents, as they require no more than the standard JavaScript functionality of a modern browser, rather than a Java installation. Indeed, recent statistics show that around 94% of internet users have a JavaScript enabled browser (W3Schools, 2007), most of which would be capable of rendering the YaWSA site.

Page | 24

Angus Macdonald

Service Composition with Hyper-Programming

Given the choice of AJAX a decision was taken to use the Apache Tomcat servlet container as an intermediary between the server and the user’s browser. Specifically the DWR servlet (an opensource AJAX library) would be used to facilitate asynchronous communication on the system. Conclusion These design decisions were critical to the development of the system, as they defined the languages which could be used during development, thus limiting the range of available tools to those offered supported in those languages.

Design Overview Having decided on the tools that would be used to implement the system, the design was updated accordingly (shown in Figure 6). This diagram adds a new element, the tomcat servlet (B), and encapsulates the invocation and composition of services behind the back-end server (C). Having made the decision to use AJAX and a Java back-end it was initially decided that all server-side functionality should reside within a Tomcat servlet, as this would make communication with the client simpler. However, due to bugs in Apache Axis it became necessary to move proxy generation into a separate Java application (C)4. System

B

AJAX

A

Tomcat Servlet Client

DivideWS AddWS

divide()

Java RMI

SOAP

add()

Back-end Server

C

D

Figure 6: Data flow through the system. Starting with the client (at A) a user’s communication with a Web Service is handled through a Tomcat servlet (B) and a back-end server (C). Finally, the back-end server communicates directly with the Web Service.

The following sections walk through the system to illustrate the reasons behind this structure.

4

When attempting to dynamically instantiate arbitrary proxy objects within a servlet environment Axis periodically fails to

recognize its current environment, causing an error in the Internationalization code. This problem proved to be deep-rooted, and could not be fixed without altering the Axis source code, making the system less portable as a consequence.

Page | 25

Angus Macdonald

Service Composition with Hyper-Programming

Use Cases This section introduces three use cases that are used to walk through YaWSA’s functionality. It also helps to illustrate how users may build up and invoke compositions using the web-based interface. To recap, a user has access to two services - AddWS and DivideWS – which allow them to add two numbers and divide two numbers respectively. Using these services YaWSA should allow the user to: Use Case A Use the AddWS service to find the result of adding 25 and 175 (page 26, below) Use Case B Use AddWS and DivideWS together (in a composition) to find the average of 36 and 48 (page 28) Use Case C Store the result and structure of the invocation above, to re-use them in another composition (page 31). These use cases are devised to cover the three main areas of the system’s functionality: service invocation, service composition, and object storage respectively. This section goes through each case to illustrate a typical interaction with the YaWSA interface. Use Case A A user must invoke a simple service in this use case. The AddWS service returns the result of adding two parameters, in this case 25 and 17. This invocation takes place in eight stages: 1.

User connects to the system via A, the YaWSA homepage. Through this interface they provide the system with the address of the AddWS service, as shown below in Figure 7.

Figure 7: The YaWSA homepage. Users see this page when they first connect go to the website.

2.

The user’s request is submitted via Tomcat (B), to the backend server (C).

3.

Using Apache Axis, the server (C) generates proxy classes to access this chosen service, and performs reflection on these to find the operations provided by the service. These are returned

5

An arbitrary choice on numbers, used to illustrate the systems functionality.

Page | 26

Angus Macdonald

Service Composition with Hyper-Programming

to the user as options, allowing them to choose the operation they wish to invoke from the form shown below (Figure 8).

Figure 8: YaWSA Screenshot. The user is given a list of available operations.

This page shows that AddWS has one operation, add, which takes two integers and returns a double. 4.

The user chooses to invoke the add operation.

5.

The server (C) receives this choice and starts building a server-side representation of the user’s composition. Initially, this representation is used to return a webpage allowing the user to enter values for the two parameters, as shown by Figure 9 below.

Figure 9: YaWSA screenshot. Having chosen the 'add' operation the user can enter values for the two parameters.

Page | 27

Angus Macdonald

Service Composition with Hyper-Programming

It is important to note that each update to this page is asynchronous, so the webpage never has to be reloaded – the only elements to be updated are those whose server-side representation has changed. 6.

Seeing the new page (Figure 9) the user enters the values 25 and 17, and then presses the Invoke Service button.

7.

Only now, at step 7, the system communicates with the AddWS Web Service (D). This is done through the proxy class (created in step 3) residing on the server (C).

8.

The result, 42, is then returned to the user via the proxy classes on C, and displayed on-screen as shown by Figure 10 below.

Figure 10: YaWSA screenshot. The webpage returns the result of a user's invocation.

This page provides some other functionality – such as the ability to store the result of invocations – that is discussed in Use Case C (page 26). Use Case B This use case extends its predecessor by introducing service composition. A user wants to find the average of two numbers, 36 and 48, by adding them with AddWS and dividing the result by two with a new service, DivideWS. 1.

The user begins by entering the address of DivideWS on the YaWSA homepage (as in Figure 7), and then deciding to use the divide operation. This action returns the following page:

Page | 28

Angus Macdonald

Service Composition with Hyper-Programming

Figure 11: YaWSA screenshot. The screen displayed to the user once they have decided to invoke the 'divide' operation.

2.

To perform this use case, the user must tell the system to pipe in the result of an addition as the first parameter of divide. They select WS Ref from the Method of Input options shown in Figure 11 (above). This tells YaWSA that they wish to use the result of another Web Service as the method of inputting a value for this for this parameter, and results in the following update to the page:

Figure 12: YaWSA screenshot. The page now allows the user to enter the address of the service they wish to use in the composition.

Page | 29

Angus Macdonald

Service Composition with Hyper-Programming

As Figure 12 shows, the user is given space to enter the address of the service that will provide the double object (the parameter required for the divide operation). 3.

The user enters the address of the AddWS service and selects the add operation (as in Figure 8). This produces a page allowing the user to enter parameters for the add operation, and the second parameter for the divide operation (the first being provided by the result of add). Figure 13 shows this page (below).

Figure 13: YaWSA screenshot. The page returned once the user has chosen to use the 'add' operation. They can now enter parameters for the add operation; the result of which will be returned to be used as a parameter to 'divide'.

This update means that the result of invoking the AddWS service will be used as the first parameter in the invocation of divide. 4.

The user can now enter parameters for the services (36 and 48 for add, 2 for divide) and press the Invoke Service button. The system invokes AddWS first, using the result (84) in the invocation of DivideWS. The result is returned to the user in the following screen:

Page | 30

Angus Macdonald

Service Composition with Hyper-Programming

Figure 14: YaWSA screenshot. The system displays the result of invoking the AddWS-DivideWS composition.

The final result of the composition is displayed, but not the intermediate invocations such as the result of invoking AddWS. Use Case C This use case starts where the previous two cases finished – with the system showing the result of an invocation. YaWSA allows users to store values returned by Web Services, and the structure of compositions used to get those values. It provides this functionality below the result of an invocation, as shown in Figure 14 (above). The user has the option of tagging an object or composition with a name and description, but they can bypass this step using the Quick Add function. In this case the system automatically tags the item with relevant information. 1.

This use case begins with the user attempting to store the result of the composition in Use Case B. From the page shown in Figure 14 they begin by clicking the show button next to the Show Value in Object Store title, which gives them the following form:

Page | 31

Angus Macdonald

Service Composition with Hyper-Programming

Figure 15: YaWSA screenshot. The user can tag an object with information so that it can be stored and used at a later date.

The values they enter in this form are important, as they will help the user identify the stored object at a later date. 2.

The user enters a name and description of the object then presses the ‘Yes, Store Result’ button. The system returns a message stating that the object has been successfully stored.

3.

To use the stored value the user goes back to the YaWSA homepage and selects the divide operation from DivideWS (as in Figure 7 and Figure 8 for AddWS). This produces the following form:

Figure 16: YaWSA screenshot. The form given when a user chooses the add operation in AddWS.

Page | 32

Angus Macdonald

4.

Service Composition with Hyper-Programming

To use the stored value the user must change the ‘Method of Input’ from the current setting (User Input) to ‘Object Ref’, indicating that they wish to provide a reference to a previously stored value. Selecting this option produces the following form:

Figure 17: YaWSA screenshot. The user is given a choice of suitable objects for this parameter. They must choose from one of these.

5.

The system gives the user a list of objects (see Figure 17 above) that can be used for this parameter. Having chosen an object the page is updated again, as shown below.

Figure 18: YaWSA screenshot. The user has chosen the stored object that they wish to use in the invocation.

6.

The user is now in a position to invoke the service as in previous use cases. They enter a value for the second parameter and press the ‘Invoke Service’ button, completing the use case.

These use cases are the basis for the structure of the next section. However, unlike this section, it goes through a walkthrough of the system from the perspective of the system, not the user.

Page | 33

Angus Macdonald

Service Composition with Hyper-Programming

System Walkthrough Introduction This section describes the issues involved in the design of YaWSA by stepping through Use Cases B and C. As before (Use Case B, page 28) a user wishes to use two services – AddWS and DivideWS – to find the average of two numbers. From the system’s perspective there are six distinct stages in facilitating composition from a webpage. These are: Step 1. User Input (page 34). User enters the address of the Web Service they wish to invoke. Step 2. Construction of Compositions (page 35). The system presents the user with available options (such as ‘choose operation and ‘enter parameters’), allowing them to build up a composition. While they do this the system maintains a representation of the composition on the server. Step 3. Serializing Data (page 44). User input for an operations parameters must be serialized before being sent across the network from the browser to the Tomcat servlet. Step 4. Invocation of Services (page 50). Once the server has received the user’s input it can begin invoking services sequentially. This step deals with the ordering of invocations in a composition, and the method of communicating with the Web Services being invoked. Step 5. Post-Invocation Display (page 50). When a Web Service returns the result of an invocation the system must display this to the user. Step 6. Post-Invocation Storage (page 52). A user may want to re-use the result of a service invocation, or the composition which produced this result. The system stores these values, making this possible. The following section expands on each of these steps, in the process describing the YaWSA implementation. To the right of each heading is a diagram, showing the areas of the system involved in the given step. Step 1: User Input (Enter Service Address)

Client

Tomcat

WS

Server

Use Case: User enters the address of the DivideWS service, and submits this to the server.

The process of composing services begins on the client’s browser with the YaWSA homepage (as in Figure 7). This page allows users to enter the address of a service they wish to invoke – the root service in their composition. When a user submits this address into a form on the webpage, it is sent asynchronously to the tomcat servlet and then onto the server. On the server Apache Axis is used to generate proxy classes for the given service. These proxy classes will later be used to invoke the service, but the user must first select the operation they wish to use. To do this the system uses linguistic reflection on the proxy classes to get the names, required parameters, and return types of the service’s operations. Page | 34

Angus Macdonald

Service Composition with Hyper-Programming

These details are then returned to the user via tomcat, allowing them to choose the operation they wish to invoke. In the meantime the proxy classes are cached. Users are assigned a globally unique identifier (GUID) that is used as a session ID, identifying them in calls to the server. Because the system doesn’t use conventional servlet functionality (most communication is asynchronous), it is not possible to use standard HTTP session functionality. Instead, YaWSA implements its own session handler, mapping a user’s GUID to an Interaction object, and storing all data related to the users current interaction in this object. This includes the current server-side form of a user’s composition. Step 2: Interpretation of Compositions

Client

Tomcat

WS

Server

Use Case: The user selects the operation they wish to invoke (divide, an operation of DivideWS), and is given a form allowing them to create a composition that starts with a single call to DivideWS. For example, a user invoking the DivideWS service might choose to get the input of one parameter from another Web Service. The webpage updates, allowing them to enter the address of this secondary service. It will update again when they enter this address, and when they choose the operation they wish to invoke from the service. Challenge: While the user is creating this composition, the structure and content of their web form is constantly being updated to give new, more relevant, options. The server must be able to represent users’ compositions in a way that allows it to be easily expanded, altered, and subsequently invoked. The system maintains a representation of a user’s composition on the server, which is continually updated to match the client-side state of the composition. This server-side representation is updated every time the user makes a decision, by sending each choice asynchronously to the server. The biggest challenge in creating YaWSA was in finding a method of representing compositions on the server that was both efficient and sufficiently flexible to meet the requirements of the system. A tree structure was chosen. When modified, this structure automatically generates HTML allowing the user to make further updates to the composition. These updates are then reflected in the tree, where the process begins again. This tree structure uses the concept of hyper-programming, as each reference maintains a data item (value returned by the service), access path (location of the service), and access specification (service description). This gives flexibility in linking services and values in compositions. Use Case: Our user wishes to create a composition where the result of AddWS is used as the first parameter in DivideWS. Through this composition our user wishes to find the average of two numbers: 36 and 48. They have entered the address of the service they wish to invoke (DivideWS). In step 1 the user began creating a composition by selecting to invoke the DivideWS service. When they made this choice the service began to create a tree – the server-side representation of the user’s Page | 35

Angus Macdonald

Service Composition with Hyper-Programming

composition. Initially this tree has one node, the root node, which contains a reference to DivideWS. This signifies that DivideWS is to be invoked, and as it is on the root node, the value returned by this service will be returned to the user. Figure 19 (below, left) shows the tree structure at this point. While this structure is maintained on the server, the user’s webpage is updated, allowing them to select the operation they wish to invoke. The relevant part of this webpage is shown to the right of Figure 19 below. This HTML was generated by traversing the tree structure on the left. INVOKE

Function Wrapper Service Wrapper

Wrapper DivideWS

divide

Operation to be called

Figure 19: Left - State of the tree once a user has chosen to invoke the DivideWS service. Right – HTML returned to the user as a result of this traversal.

In its current state the tree contains a reference to DivideWS within a function wrapper. This wrapper contains references to the information required to invoke a Web Service. It allows the server to specify a Web Service6, the name of an operation in that service, and any parameters to be used in a call to that operation (as illustrated by Figure 20).

Web Service

Operation Name

Parameters

Figure 20: Function Wrapper. The format of the data structure used to represent references to Web Services in composition.

However, at this stage in our use case the function wrapper has no link to any parameters, as the user hasn’t chosen the operation they wish to invoke. During a traversal of the server-side tree the system notes that that no operation has been selected, so it returns HTML allowing users to chose from one of the available operations (as shown in Figure 19).

6

References to Web Services are stored within service wrappers – objects that cache proxy objects. This provides

an interface that the system can use to make invocations, while ensuring that the underlying method of invocation (Axis proxy objects) remains distinct from the rest of the system. The same service wrapper can be used in multiple places within an invocation, as it maintains no state.

Page | 36

Angus Macdonald

Service Composition with Hyper-Programming

Use Case: The user now selects the divide operation. Having made this choice the tree on the server is updated. It is then traversed so that the system can generate HTML to return to the user’s webpage. As the tree now contains the name of the operation to be invoked, child nodes representing the parameters required to invoke the service can be added. By default these nodes allow users to directly input values into textboxes on the webpage. As a result of this change to the tree (seen on the left of Figure 21 below) a server-side tree traversal returns an HTML form to the user, allowing them to enter parameters for the divide operation. This webpage form is shown on the right of Figure 21 below. INVOKE

Parameter(s) “divide”

Wrapper DivideWS

divide

Double

Double

Figure 21: Tree structure now that the user has chosen an operation. The form (right) is the result of traversing this tree.

The above figure shows the relevant part of the page returned to the user - Figure 11 on page 29 shows the full page at this stage. This page is the client-side representation of the tree structure shown in Figure 21 above. The challenge in this is keeping these representations synchronized so that the server knows what to expect once the user presses the ‘Invoke Service’ button. Recap: The user has chosen to invoke the divide operation in DivideWS. In doing so the system has created a server-side representation of their composition that specifies the service being invoked and the required parameters. This representation is being used to update the form on the client’s webpage, which allows them to add to their composition. The DivideWS service can now be invoked without any further updates to the tree. The user can enter values in the provided textboxes and press the ‘Invoke Service’ button. Our user, however, wishes to make a composition with the AddWS service. As we are now at step 2 of Use Case B (see page 28) the user is now in a position to link AddWS to the composition.

Page | 37

Angus Macdonald

Service Composition with Hyper-Programming

To do this they must select from one of the ‘Method of Input’ buttons on the form (see Figure 21, right). These allow the user to select how they wish to provide a value for the given parameter. The following methods of inputting values are available: 

User Input. This is the default option. It allows users to enter values for the parameter directly into textboxes on the webpage. In our use case the user could enter two numbers in the textboxes provided to calculate the result of dividing one from the other.



Web Service Reference. This allows users to provide a reference to another Web Service, in effect creating a composition. The value returned by invoking this service will be used as a parameter in the invocation of DivideWS. Our user wants to use this option, to use the result of adding two numbers (in the AddWS service) in our composition.



Object Reference. This allows users to use a previously stored value. The user will be presented with a list of available objects of the correct type. For more on this option, see page 52.



Composition Reference. This allows users to add a previously stored composition to the current composition. This is discussed on page 53.

Use Case: Our user chooses ‘Web Service Reference’ for the first parameter, as they wish to use the result of AddWS as the first parameter of DivideWS. The server-side tree is updated as a result of the user’s input. The user has chosen to provide a reference to another Web Service rather than enter the value for a parameter, so the first parameter in the tree now points to a Web Service reference instead of the object reference that was there before. Figure 22 illustrates this update (left), and the resulting HTML form returned to the user (right). The form in the user’s webpage now allows them to enter the location of the service they wish to invoke – AddWS in our use case.

Page | 38

Angus Macdonald

Service Composition with Hyper-Programming

INVOKE

“divide”

Wrapper DivideWS

divide

INVOKE

Double

Figure 22: Tree structure updated to reflect the user's decision to use a Web Service Reference as a method of input.

In this example the parameter nodes in the server-side tree (one for each parameter in the operation) contain references to another Web Service (AddWS) and to user input (the parameter will be entered directly on the form) respectively, but the system is capable of more. Parameter nodes can also contain references to stored values or compositions. The representation of each of these differs on the server, and consequently on the client. In Use Case B the user has chosen a Web Service reference as a method of input, meaning the user’s webpage allows them to specify the name of a service that will return the value required for another service invocation. The only requirement for parameter nodes is that they return the data type required for the given parameter. So, for example, to use a Web Service result as input to the DivideWS service the selected service must return a value of type Double. The newly created Web Service reference is treated in the same way as any other on the tree, including the root service. Consequently the user must go through the same options that they went through at the outset: they provide the address of the AddWS service, then choose the add operation when asked. The server-side tree is updated through each of these choices, and is shown below (Figure 23) in its final state.

Page | 39

Angus Macdonald

Service Composition with Hyper-Programming

INVOKE

“divide”

Wrapper DivideWS

divide

Double

INVOKE

Wrapper AddWS

“add” add

Figure 23: Tree structure now that the user has selected the AddWS service, and the add operation.

As with every previous step, the HTML returned to the user’s webpage has been generated from a tree traversal. To complete our use case the user needs to enter the numbers 36, 48, and 2 into the appropriate text-boxes, then press ‘invoke service’. The ensuing process of serializing this data is discussed in Step 3: Serializing Data (page 44). To prevent excessive refreshing of the user’s webpage the server only calls the traverse function is only called at the level of the highest node that has been modified. For example, if a user decides to change their method of input from entering parameters on a web form to piping the result of another service, the system will traverse the tree only from the ParameterRef being changed. Step 3: Serializing Data (page 44) looks at how user input is serialized and sent across the network when they press ‘Invoke Service’, with service invocation discussed in Step 4: Invocation of Compositions (page 47). Before this, the Java implementation of the tree structure is discussed. Implementation Up until now this document has looked at the server-side implementation of composition trees in an abstract manner. The concrete implementation of this structure follows these abstract designs closely, making full use of Java’s object-oriented capabilities. This method of representing compositions has been named the Web Services Interactive Composition Model (WSICM). Page | 40

Angus Macdonald

Service Composition with Hyper-Programming

The abstract class Node provides generic tree operations that apply to every node in its tree. Seven classes extend Node, each covering a specific aspect of a composition, ranging from references to Web Services (WebServiceRef), to values from the object store (StoreRef). Figure 24 (below) shows Node and the sub-classes used to create the server-side composition tree.

Figure 24: UML Inheritance Diagram for Tree. Illustrates how the tree structure has been implemented in YaWSA.

There are seven sub-classes to Node, each representing a different aspect of a composition. So, for example, when the user decides to invoke the divide operation (as in Use Case B) the system creates two ParameterRef objects – one for each parameter required to invoke the service. When they decide that the ‘Method of Input’ for a parameter should be a Web Service reference (as in Figure 12) the system creates an instance of the WebServiceRef class that will become the child of a ParameterRef object in the tree. This can be seen in Figure 25 below, which shows concrete implementation of the final composition in Use Case B. The abstract version of this tree can be seen in Figure 23.

Page | 41

Angus Macdonald

Service Composition with Hyper-Programming

RootNode

WebServiceRef

ServiceWrapper

“divide”

“DivideWS”

Reference to proxy class ParameterRef

WebServiceRef

ServiceWrapper

ParameterRef

“Double”

“add”

“Double”

ObjectRef

“Double”

ObjectRef

double

“AddWS” 2

Reference to proxy class

ObjectRef

“int”

36 ObjectRef

“int”

48

Figure 25: View of the DivideWS-AddWS invocation tree as implemented in YaWSA

Of particular note in this diagram is the representation of the Double parameter ‘2’ in the diagram. Unlike the two ObjectRef objects of type int, Double is a complex type. As complex types may have multiple constructors, they will have children once the user has chosen the constructor they wish to use. In these circumstances an ObjectRef will not contain a value (this only happens for primitive types) but will point to children that will later be used to instantiate the complex object it represents. While Figure 25 shows some of the functionality supported by the system, it omits other areas. For example, the system uses ArrayRef as a node in the tree when an array is a parameter to a service. This is required as arrays are distinct from other parameters in that their size cannot be determined. For completeness, the following classes can be used in a server-side tree: 

WebServiceRef: A reference to a Web Service. o

Initially this will allow the user to select the operation they wish to invoke.

o

Once an operation has been chosen, the system creates child nodes (of type ParameterRef) that represent parameters required for invocation.



ParameterRef: Used as the children of WebServiceRef objects – they represent the parameters required to invoke a service. This is done to abstract away from the method of input for that parameter. For example, a ParameterRef node may have another WebServiceRef as its child, indicating that the result of that invocation will be used as a parameter to the parent node.

Page | 42

Angus Macdonald 

Service Composition with Hyper-Programming

ObjectRef: Represents any value or object that is required to invoke a Web Service. For example, with the AddWS service, two ObjectRef objects would be required – one for each primitive parameter. ObjectRef objects can also be used to represent parameters of other objects; complex types that require constructors for instantiation are an example of where this is applicable.



ArrayRef: A subtype of ObjectRef, this represents a special case. It is used for array parameters, and differs from normal objects as the length of an array cannot be predetermined.



StoreRef: Represents an item held by the system within the object store.



CompositionStoreRef: A subtype of StoreRef, this is a specialized store item. This stores compositions, not objects.

Of the remaining classes in Figure 24, Interpreter is used to access the tree, while Generator is used by Node and its subclasses to generate HTML code to be displayed to the user. As with Web Service references, other types of node maintain the hyper-programming approach of the tree structure by providing methods for accessing their data item (value), location (for example, in the object store), and specification (type) for a given reference. This allows a great amount of flexibility in constructing compositions, as nodes can be placed in the tree in nearly any order, irrespective of their type. For example, parameters can be references to store items, Web Services, or user input. Tree Traversal To make use of these compositions efficiently the server must provide functionality to access data within the tree with a minimum of method calls. The Node class provides a number of methods designed with this in mind. Of particular importance are the following: 

traverse(): Performs a post-order traversal of the tree, creating an HTML form from the contents of each node. Each type of node overrides the traverse method, as they all maintain a different structure dependant on their purpose. A node will return HTML code allowing the user to further modify its state, and reverse their actions.



isComplete(): Used to ascertain whether the tree is in a state ready for invocation. If no more user input is required to get the tree into a state where it can successfully invoke a service, this will return true; otherwise false. To achieve this method performs a post-order traversal of the tree. If all of a node’s children are complete then that node is complete; if a node is a leaf node, the method must check whether a value has been entered (as with object store references), or could be entered when the form is submitted (as with primitive input).



invoke(): Performs a post-order traversal constructing objects and invoking services where possible. A more detailed description of this method is given in step 4 (page 47).

Page | 43

Angus Macdonald

Service Composition with Hyper-Programming

To prevent excessive refreshing of the user’s webpage the server only calls the traverse function is only called at the level of the highest node that has been modified. For example, if a user decides to change their method of input from direct input to a Web Service reference the system will traverse the tree only from the ParameterRef being changed. Step 3: Serializing Data

Client

Tomcat

WS

Server

Use Case: Once the user has created their composition, and entered the required parameters, they invoke it by pressing the ‘Invoke Service’ button. Challenge: The system must be capable of serializing the user’s parameter input in such a way that it can be sent to the backend server, reformed and placed into the invocation tree when it reaches the server. Data Transfer Data is sent asynchronously between client and server using the Direct Web Remoting7 AJAX library. This library handles this communication through its own Tomcat servlet, allowing Java objects to be serialized and used client-side (in JavaScript). Data Scraping As form details on the client are submitted asynchronously it is not possible to use a standard submit operation to obtain all form details. This problem is further compounded by the inability to determine the form structure prior to invocation. The solution is to use a JavaScript function which iterates recursively through HTML elements, and returns the contents of all form elements in the tree, matched to their respective element IDs. As AJAX forms become more prevalent on the Internet, a function of this type is likely to become necessary for more applications, making this implementation useful outwith the YaWSA system. Of the open source solutions currently available none of those found are recursive, meaning they are only capable of finding top level form elements, and not those found within an underlying tag. Serialization Maintaining the structure of submitted form data is a requirement of serialization. If this were not met it would not be possible to place values in the correct place on the server-side composition. To this end YaWSA makes use of the server’s tree structure to create unique IDs for each element. This means that, when submitted, form data, sent with the corresponding ID, can be accurately reconstructed server-side.

7

Direct Web Remoting (DWR), version 1.1.4: http://getahead.org/dwr

Page | 44

Angus Macdonald

Service Composition with Hyper-Programming

Encoding the Data Data is encoded using the following technique, ensuring it can be restored server-side. Letters in Figure 26 represent textbox values and options in the form, with the ID being that assigned in the elements HTML tag. Webpage Structure

Reformed Structure

A [id=1] B [1]

[A, 1], [B,11], [C,12], [D,121], [E,122]

C [2] D [1]

A [id=1]

Encoded Format

B [1]

E [2]

C [2] D [1]

E [2]

Figure 26: Flattening/Restoration of Tree Structure. A user’s input on the webpage is serialized in the form shown, making it possible to reconstruct its original structure on the server.

The root letter, A, represents a parameter of the Web Service. A’s children are the parameters required to instantiate A, and so on. When the form is submitted an ID is generated for each node in the tree based on its position. For example, C is the second child of A, meaning it is given the ID ‘12’, where ‘1’ is the ID of A. This process continues for every element, until the entire form is encoded. It is then sent across the network where it can be reformed by reversing the process. Data is encoded into an array-based format. Each value in the web form is transmitted, along with the ID given to that form element. Use Case: A user submits a form containing the values 36, 48, and 2. These values are sent along with their corresponding ID to the server, which uses the ID to place values in the correct place in the composition. Figure 27 (below) shows another screenshot from YaWSA (left), this time annotated with the ID assigned to each textbox. This ID isn’t visible to the user, but is stored within the HTML ID tag for the respective textbox element. To the right of this diagram is a segment of the server-side representation of the composition. Again, this is annotated with the ID given to each individual node in the tree.

Page | 45

Angus Macdonald

Service Composition with Hyper-Programming

ID: 0

ID: 1

INVOKE

Double ID: 0 2

ID: 1

ID: 0 36

48

Resolved ID: 01

Figure 27: Representation of a composition client-side (left) and server-side (right). This figure shows how node IDs can be used to place values into the appropriate place on the server. The YaWSA screenshot (left) shows the IDs of text-boxes in italicized red text. In reality these IDs are not visible to the user.

As this example demonstrates, only the node IDs are required to find the correct position for a value on the server. For example, consider the process to find the position of the second parameter (ID = 01) in the server-side tree. The first number in the ID is 0, so the tree is traversed to the root node. The second number is 1, so the traversal goes to the second child of the current node. As the ID is complete, the traversal has found the related node. The value (in this case 48) can now be placed into the tree. In view of this, IDs and values are sent to the server together in an array-based format (shown below; Figure 28). The array being sent contains more arrays, each with two elements. The first element is the ID of an element, while the second is the corresponding value.

ID

00

01

10

Value

36

48

2

Figure 28: Form Data sent from Client. The serialized structure of the form data sent to the server. Each form field is stored, by including the ID of that field, and its value.

Page | 46

Angus Macdonald

Service Composition with Hyper-Programming

This structure was chosen over other, more defined, approaches, due to its simplicity; messages sent between client and server are serialized in JavaScript and deserialized in Java, so minimal approaches require less serialization. This serialization is the first of three points where data must be sent across the network, as Figure 29 illustrates. Building on previous diagrams, this shows data flow during an invocation based on the use case. System

B

A AJAX

Tomcat Servlet

Divide: Num 1

Num 2

Client

Add: Num 1

31

Num 2

53

2

Invoke

84 2

Java RMI

DivideWS divide()

SOAP

AddWS

Back-end Server add()

D

C

Figure 29: Data flow through the system for an AddWS-DivideWS composition. The client submits their composition at A, which is sent to the system at B. This is passed onto the back-end server at C, where the composition is traversed, and the Web Service at D invoked.

Each of the three areas of communication is handled differently: A to B

Data is encoded (client-side) in the array based format described earlier, and then sent asynchronously, using AJAX, to the Tomcat servlet.

B to C

Tomcat does some minor error checking then passes the message, unaltered, onto the back-end server (C). This communication is handled over Java RMI, with a well defined interface used.

C to D

Finally, during the invocation of a Web Service, the back-end server sends a SOAPencoded request to the Web Service, returning the response through the servlet and back to the client at A.

Step 4: Invocation of Compositions

Client

Tomcat

WS

Server

Use Case: The back-end system now has all the parameters required to invoke the composition of the AddWS and DivideWS. It will now perform the invocation.

Page | 47

Angus Macdonald

Service Composition with Hyper-Programming

Challenge: The tree must be invoked in sequence to ensure services have the parameters they require. During service invocation the system must reflect on the proxy classes to instantiate them and use the resulting objects to make a Web Service call. Now on the server, the process of invoking compositions can be broken down into two areas: from a coarse-grain, how services are invoked from the perspective of an encompassing composition; from a fine-grain, how individual services are invoked across the network using Apache Axis. This section first discusses the coarse-grain view – composition invocation. Composition Invocation When executing a composition, services are invoked sequentially by a post-order traversal of the composition tree. Consider again our use case: Use Case: A user wishes to find the average of two numbers through a composition of DivideWS and AddWS. This example begins with a complete invocation tree, as shown below. The user’s input is now on the server. Invocation of a Service

INVOKE

Method to be called Service in Wrapper Parameter(s)

Wrapper

“divide”

divide

DivideWS

INVOKE

Double

2

Wrapper AddWS

add

“add” 36

48

Figure 30: Invocation tree, with all values entered. This tree is ready to be invoked.

The invocation process is described below. Note that all of these steps take place on the server, which is solely responsible for interacting with the given Web Services. The result of this invocation is the only information returned to the client, ensuring the user’s conceptual model is sustained. Page | 48

Angus Macdonald 

Service Composition with Hyper-Programming

First we perform a traversal of this tree, eventually reaching the value ‘36’. This is a primitive 8

value, so it remains untouched , along with the other child node ‘48’. These are complete, as they are leaf nodes (so incomplete children are not a concern), and the values don’t require instantiation. 

Now all of the children of the AddWS node are complete, so we can invoke the Web Service. 9

The value returned by this service is stored in place of the reference to AddWS . At this stage the tree looks like this: Invocation of a Service

INVOKE

Method to be called Service in Wrapper Parameter(s)

Wrapper

divide

“divide”

DivideWS

Double

Double

2

84

Figure 31: State of the DivideWS-AddWS invocation tree after AddWS has been invoked.



Following this update we are now at the children of the DivideWS invocation. The Double value ‘84’ was returned by another service, so has already been instantiated. This node is complete.



The other parameter, containing ‘2’, requires instantiation. Double is not a primitive type, so must be instantiated as a Java object. All child nodes of Double are present and complete, so the object can be instantiated.



All parameters are now complete, so the DivideWS operation can be invoked. Following this invocation the value ‘42’ is returned to the root node, signaling that invocation is complete.

8

Primitive types don’t need to be instantiated, so they are always leaf nodes in the tree. Complex objects require

instantiation through a constructor in Java, and may require more than one parameter for instantiation. As a result complex objects in the tree have parameter child nodes in the same way as Web Service references. 9

The actual implementation differs slightly from this account, as the reference to AddWS is not removed; it is

marked as complete and has the return value stored within it. This allows for later re-use of the composition, but makes the composition’s diagrammatic representation unnecessarily complex.

Page | 49

Angus Macdonald

Service Composition with Hyper-Programming

The root node is a special sub-type of Node, which deals with providing this response to the end-user. Before this can happen the response must be decomposed into a readable format. Service Invocation The system uses Apache Axis (Apache, 2006), a Web Services library for Java, to handle invocations to Web Services. Using Axis, the system generates Java proxy classes from the WSDL file provided by the user. These classes are used to provide the user with a list of operations to choose from (done prior to Step 2: Interpretation of Compositions, page 35), and are required here to actually perform the invocation. In both steps YaWSA must use linguistic reflection to establish, and make use of, the operations provided by the proxy classes. Web Service libraries often use heterogeneous naming conventions in the WSDL files they create; proxy classes reflect these differences, as they use names and information provided in WSDL. The system can deal with proxy classes in various formats, notably proxy classes: 

generated from an Apache Axis Web Service, or a Microsoft .NET Web Service



generated from WSDL files of type ‘document’ or ‘rpc’



containing a single, or multiple, operation(s)



containing operations with no parameters, one parameter, or many parameters



containing operations with parameters that are: o

primitive types

o

complex types

o

enumerations

o

arrays

While this is not an exhaustive list, it shows the system’s flexibility in handling proxy classes. Once proxy classes have been dynamically compiled they are cached in a ServiceWrapper object. This allows the proxies to be used multiple times within the same composition (or at a later point), ensuring the same reflective operations are not performed more than is necessary, as they are computationally expensive operations. Step 5: Web Service Responses - Results Decomposition

Client

Tomcat

WS

Server

Use Case: The composition has been invoked and the system now has a result to return to the user.

Challenge: Returning a primitive result is relatively simple; however the system must be able to return arrays and arbitrary complex types, knowing only that the returned value is a Java Object. A Web Service’s response to an invocation must be returned to the user. This is a trivial task for primitive types as the returned value can be immediately obtained, and then sent from the server via

Page | 50

Angus Macdonald

Service Composition with Hyper-Programming

Tomcat to the browser. However, arbitrary complex types, arrays, and enumerations are more difficult. Apache Axis returns Web Service responses as Java objects, meaning, for example, that an object of type Person would be returned in a Java object, Person. Obtaining the contents of these classes is relatively simple if the system is expecting an object of that type prior to system creation. However, for arbitrary complex types the system cannot determine the structure of objects returned by Web Services, meaning a generic method - capable of getting information from any object - had to be created. The YaWSA implementation of this method is able to print out the contents of any Java class which extends java.lang.Object. It performs a recursive reflective decomposition of the object, returning its contents formatted in HTML. The following pseudo-code summarises the function, named toDescriptiveHTMLString, below. Input:

obj: object to be deconstructed

Output:

HTML formatted string of object contents

Variables:

arrLen: array length objField: array of fields in this object fieldLen: number of fields in object

toDescriptiveHTMLString(obj): 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18.

If obj = null Return