Xml-enabling Your Lotus Notes Application

XML-enabling your Lotus Notes application Skill Level: Introductory Branavan Ganesan IBM software developer IBM

07 Feb 2003 This tutorial shows how the team at developerWorks used the XML capabilities of Lotus Domino to extract data and transfer it to DB2. The same technique can be used to transform and transfer data to any other relational database engine.

Section 1. Getting started About this tutorial This tutorial shows how we at developerWorks, working with the IBM eServer Developer Domain team, used the XML capabilities of Lotus Domino to extract data and transfer it to DB2. The same technique can be used to transform and transfer data to any other relational database engine. This tutorial goes hand-in-hand with "Managing and moving data using Java technology, XML, DB2, and Lotus Notes," which describes how to use forms, views, agents and page design elements to facilitate XML representations of Domino data. It also covers how to invoke an external Java servlet to read and transform the data in XML format. You should at least skim the article before proceeding with this tutorial. In this tutorial, you'll get a hands-on approach to the design techniques discussed in "Managing and moving data using Java technology, XML, DB2, and Lotus Notes". This tutorial shows you how to gather publish-ready metadata documents for articles, surface them as XML, and invoke an external Java servlet from within Notes that reads, transforms, converts the information, and updates DB2 accordingly. Because this tutorial will focus on the Domino portion of this exercise, we won't cover the external servlet in any detail. XML-enabling your Lotus Notes application © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 1 of 16

developerWorks®

ibm.com/developerWorks

Getting the Notes database In this tutorial, we'll be enhancing a Domino application. If you are an old hand at Notes, and want to hone your skills on XML enabling a Domino app, you could create a test database and build these design elements. If you are unfamiliar with Notes or just don't feel like creating the forms, views, and so on, download the demo database called xmlcptut.nsf, which has the base elements that we'll develop in this tutorial. You will be using the JDK 1.1 that ships with Domino to develop the agent within Domino. If you are feeling adventurous, you might want to try your hand at an external servlet that acts on the agent. If you need the Java 2 SDK, 1.3.0, you can download it from http://java.sun.com/j2se/1.3.0_01/index.html. This tutorial does not cover the coding of the external Java servlet.

Initial setup Now that the database is downloaded, let's quickly check that everything is in place. • Place the database on a Domino server. Be sure to include your name or a group that includes you in the access control list (ACL). You will need a minimum of designer level access to this database on the server. • If you do not have internet passwords assigned at your company (or on this server), be sure to add 'Anonymous' as type person with a minimum of reader level authority in your ACL. This is essential to access our documents over the HTTP protocol. • Browse to the XML view in the database using the URL http:/[hostname]/[dir]/xmlcptut.nsf/xmlview?OpenView. Substitute 'hostname' with the hostname of your Domino server. Substitute 'dir' with the path where you have placed the xmlcptut.nsf database. If everything has been set up correctly, you should see the rudimentary Domino view containing the three documents that are set up to publish as XML documents.

Section 2. The content management form Introduction XML-enabling your Lotus Notes application Page 2 of 16

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

Before we dive into the data transfer function, it's a good idea to understand exactly what the data looks like. In this chapter we'll review: • The content management form • The fields that constitute the metadata of an article object • The fields that support work management in this form

The content management form 1.

Open the xmlcptut.nsf database in Domino Designer.

2.

Open the Content submission form.

Figure 1. The content submission form

There is no rocket science here. The first two sections, "General document information" and "Document searchable information," contain facts about the record. The last section, "Document tracking information," contains fields that support and facilitate work management.

XML-enabling your Lotus Notes application © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 3 of 16

developerWorks®

ibm.com/developerWorks

Data fields The data fields in this form describe the characteristics of an article that is published on the production Web site. Collectively, this constitutes the metadata for a given article. docID

The 32-character Notes ID of the document

docTitle

The title of the document

docAbstract

An abstract that describes the article

activeURL

The URL of the live article

docTargetDateLive

The date the article moves to production

hardwarePlatform

The hardware platforms this article pertains to

docType

The kind of document this article is

contentTopic

What topics this article falls under

Work management fields Carrying out this same exercise for the work management fields yields the following: docStatus

The state of the current record in the editorial workflow

serverStatus

A field used to determine whether this is a new document, an updated document, or an obsolete document that is flagged for removal

publish_flag

A simple flag field to mark a document as publish-ready or not

What is of most importance to us is the publish_flag field. Any document with a publish_flag value of 0 is deemed production ready. Now that we have an overview of the content management form, we'll proceed to the XML display form, which takes data created and edited using the content management form and displays it in XML-ready format. Using a form that users are familiar with in the Domino environment helps keep the back-office work a non-factor when determining your solution. Users familiar with this form will not need any training, or changes, for you to implement your data transfer solutions.

XML-enabling your Lotus Notes application Page 4 of 16

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

Section 3. Setting up the XML display form Displaying an XML document In this chapter, you'll set up a display form that takes a Notes document and displays it as an XML document. Once the form is set up, you will modify the fields so the information displays as text when shown in the display form.

The XML display form 1.

Open the xmlcptut.nsf database in Domino Designer.

2.

Open the XML Display form form.

Figure 2. The XML display form

We will modify three fields here: doc_abstr, last_mod_date, and doc_content_topic. The goal is to code the field values so they are interpreted correctly both by the Domino translation engine and by a parser that is expecting a well-formed XML

XML-enabling your Lotus Notes application © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 5 of 16

developerWorks®

ibm.com/developerWorks

document. Note the following: • All the fields in the form are set to be computed for display. This is to avoid data overwrites during an accidental save when viewing documents using this form. • The second line is a processing instruction with a reference to the stylesheet stored as a page design element within the database.

We will briefly look at that later in this tutorial. • The Treat document contents as HTML option in the form properties is checked. This is to ensure the Domino translation engine does not add extraneous HTML tags when rendering the document over the browser.

The abstract field The abstract field contains a general blurb about the article. It is likely to contain one or more reserved characters, in particular & and '. These must be replaced with the necessary escape characters. To do this, remove the quotes in the doc_abstrfield and code it as follows: tempvar := @ReplaceSubstring(docAbstract ; "&" :""":; "&":"""); tempvar

The function @ReplaceSubstring takes the sourceList, which in this case is docAbstract, and replaces all the values in the fromList (&, ") with the values in the toList (&, ').

The modified date We will do two things with the modified date field: • The XML file will be read as text, with an encoding of UTF-8. Therefore, a date/time datatype will not be recognized. We change the datatype to text in the display formula. • Our database engine is configured to accept dates in the mm-dd-yyyy format. XML-enabling your Lotus Notes application Page 6 of 16

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

Enter the following code into the formula field in the last_mod_date field: tempvar := @ReplaceSubstring(@Text(lastmodifieddate); "/" ; "-"); tempvar

In this case, the function @ReplaceSubstring takes the sourceList, which in this case is lastmodifieddate, and replaces values in the fromList (/) with the values in the toList (-). The one twist is that we change the datatype of the date field before manipulating it.

The content topic The doc_content_topic field is used to categorize all of the records in the database into topics pertinent to users of the library. This field is set up as a keyword field with a predefined set of values based on a pre-determined taxonomy. The issue here is that any given document can pertain to one or more topics. This is solved in Domino by allowing this field to accept multiple values. To render this in XML, however, each of the values has to be tagged separately. For example, the two values Server architecture and Parallel computing need to display as follows: 614708929906D77186256B7B005C919D FCD74466386C241386256B7B005E3AC5

Note the additional snag. In our DB2 tables, the content topics are stored in lookup tables, so we need to supply the unique id, which will function as the foreign key into the lookup table. The unique id is obtained using @DbLookup. Following is the code for the content topic field: shortval := @DbLookup("" : "Cache"; ""; "Content Topics"; contentTopic;4); @If(@IsError(shortval); @Success;@Failure("Error")); shortval = @If(contentTopic = ""; ""; shortval); topicval := "" + shortval + "" + @NewLine; topicval

We are beginning to get a handle on things. We now have a form that will display a Notes record as an XML document, but we have more work to do before it can be tested.

XML-enabling your Lotus Notes application © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 7 of 16

developerWorks®

ibm.com/developerWorks

Section 4. XML views Introduction Now that we have the display form set up, we'll set up the necessary views to support it. In this chapter we will look at the main work management view, and build a view that facilitates the servlet call to Domino.

The two views The two supporting views function as: • The search view, used by the servlet call to locate the document it is supposed to read in as XML. • The working view, which allows editors to manipulate which documents are to be published, updated, deleted, and so on. There is a third utility view that acts as a DBLookup view, and provides the ability to substitute content topic value with the corresponding unique id. We won't examine it in this tutorial because it's standard Domino fare. Since we are using the form to display data as XML, the views will not be used to perform this function. But, they will serve as necessary elements to form and manipulate document collections.

The search view First, we'll build the search view, or doc display view, used by the servlet. 1.

Open the XML Capabilities Tutorial database in Domino Designer.

2.

Build a new view and call it "XML Document view" with an alias of "servletview."

3.

Set up the 'View selection' formula as follows:

SELECT Form = "Content Submission Form" & (progressStatus = "Completed" | progressStatus = "Archived") & publish_flag = "0"

XML-enabling your Lotus Notes application Page 8 of 16

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

This formula corresponds to the status fields in the content submission form, and ensures that only documents intended to be published appear in this view. 1.

Under Design-View-Properties, click the Treat view contents as HTML box. This ensures that the Domino translation engine does not include unwanted HTML tags when rendering this view.

2.

Set up the "form formula" for this view to show documents using the XML display form by entering "XML display form".

Note that this view and the editorial working view could be collapsed by making the unique id the first sorted column in the view. However, conventional wisdom shows that collapsing utility functions and work management functions in the same view can cause complications during later development and enhancement efforts. To test this view: 1.

Open the database using the Domino client and go to the XML document view.

2.

Open a document by double clicking on it.

3.

You should see the document opened up in the the display form, showing the data in the record in XML format.

Good work! Now that this view is set up, you have the essential utility elements necessary to access Notes data in XML format both within the Domino environment and from any external call over HTTP.

Working view It's now time to take care of the end users. Users of the database do not really care that you are using Java and XML-based technologies to transfer and manipulate their data. They just want a database that is reliable and easy to use. Creating design elements to support this is essential for a successful application. For this application, there are a few pre-built views to support end user work management. These are by no means exhaustive, but are there to show the types of supporting views that can be built. As developers of Domino applications, these views should be familiar to you. Let's take a quick look at the view that shows publishable documents. This view lets the end user control which documents are going to be moved to be published. 1.

Open the "Docs ready to publish - XML view" in Domino Designer.

XML-enabling your Lotus Notes application © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 9 of 16

developerWorks®

ibm.com/developerWorks

2.

Open a document by double clicking on it.

3.

Note that the view selection formula contains the condition progressStatus = "Completed" & publish_flag = "0".

The progressStatus field and publish_flag field give end users two ways to facilitate work management and to control which documents are moved for publication. Showing additional values such as the title and content type give the documents more meaning.

Section 5. Building XSLT stylesheet page elements Introduction This section of the tutorial takes a quick look at how to build and store Extensible Stylesheet Language for Transformations (XSLT) stylesheets that transform your XML documents and store them along with the data. XSLT intricacies and frustrations constitute a whole subject area unto itself. While it is not covered extensively in this tutorial, investigation into XSLT is a must for any developer doing serious work involving XML.

A stylesheet page element Let's create a simple stylesheet that acts on the XML document surfaced by our now-polished display form. We will do a variation of the "Hello World," and a simple text message that shows the document title. 1.

Open the database in Domino Designer.

2.

Go to the page design elements by clicking on Pages in the left navigation area. You should see a pre-created page element called articletransform.xsl.

3.

Click on New Page and create a new page element.

4.

Go to page properties and call this hello.xsl. Be sure to check Treat contents as HTML for this element to ensure that unwanted HTML tags are not added by the Domino translation engine during the rendering

XML-enabling your Lotus Notes application Page 10 of 16

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

process.

Showtitle.xsl Type the following code into the body of the page element: <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:fo="http://www.w3.org/1999/XSL/Format"> <xsl:output method="html"/> <xsl:template match="/"> <xsl:apply-templates select="/articleinfo/tables/fields"/> <xsl:template match="/articleinfo/tables/fields"> <xsl:text>Title: <xsl:value-of select="papertitle"/>

Save and close the page element. This stylesheet can now be called from our XML document by using the processing instruction:
XML-enabling your Lotus Notes application © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 11 of 16

developerWorks®

ibm.com/developerWorks

type="text/xsl"?>

The transformation result Running a transformation against any article record using the processing instruction in either an XML-aware browser or an XML IDE, such as XMLSpy, looks like this:

An Easter egg In addition to the stylesheet we just went through, I've included another stylesheet as a page design element in the demo database. We used this stylesheet in the project to take the data from the XML document we used and generate an SQL statement. It is provided as is.

Section 6. The calling agent Introduction We will finish this tutorial by looking at the section of a Java agent that invokes an external servlet. Information about the XML document is passed in the call. Using this technique allows the inclusion of code outside the Domino environment in the overall application solution. We will concentrate on the following aspects of the agent: • Because the invocation occurs over HTTP, it is stateless. Therefore, we will pass as parameters the URL of the XML document that is to be acted on, the unique id of the document (for logging and error tracking purposes), and an identifier to facilitate the location of application-specific information that is kept server side by the servlet. • Two URLs will be constructed -- one to identify the XML document and

XML-enabling your Lotus Notes application Page 12 of 16

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

one to make the invocation. • The invocation will be placed within a loop that acts on a document collection. The collection will comprise all records marked as ready for promotion to a production environment.

The agent To speed things along, the shell of the agent has been pre-coded. Much of this is self evident. We will add the method that generates the Notes URL, and will add the portion of the code that invokes our external servlet. Notice that the while loop that holds the servlet invocation has been set up in the main method. The document collection is generated by using the AgentContext.getUnprocessedDocuments() method. This way, the collection can change from selected documents to all the documents in the view, depending on the setting you use for the Which document(s) should it act on property. Because this code is executed from an agent, a check is added to ensure that the publish_flag value is not 1. An agent can potentially be invoked from any view against any document in the database. Last but not least, note that prior to invoking the servlet, the status fields and modified date value are changed within the agent. Doing this automatically within the agent gives users a data value showing what has been promoted and when.

The generateURL method 1.

Open the database in Domino Designer.

2.

Go to the agents design elements and open the PublishCode agent.

3.

Add the following method to the agent before the main method:

private String generateURL(String uniqueID, Database thisdb, View viewName, Session s) { String fullURL= new String(); try { String dbPath = new String ( thisdb.getFilePath().replace( '\\' , '/' ) ); Vector aliases = viewName.getAliases(); String view = new String((aliases.elementAt(0)).toString()); //Use alias in order to accommodate view name change if necessary Name sname = s.createName(thisdb.getServer()); String server = new String( sname.getCommon()); fullURL = new String ("http://" + server + ".ibm.com/" + dbPath + "/" + view + "/" + uniqueID); //substitute the ibm.com with your domain } catch(Exception e) {

XML-enabling your Lotus Notes application © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 13 of 16

developerWorks®

ibm.com/developerWorks

e.printStackTrace(); } return fullURL; }

Invoking the servlet Now let's move to the code that invokes the servlet. We have already generated the URL of the XML document with the aptly named generateURL() method. Now we will generate a second URL, this time the URL of the calling servlet, feed it the three parameters that we have built, and then invoke it. Enter the following code under the comment in the main method: URL servletURL = new URL("http://hostname.ibm.com/servlet/DataTransfer?notesURL=" + url + "UNID="+ uniqueid + "propID=IDENTIFIER"); URLConnection uc = servletURL.openConnection(); BufferedReader in = new BufferedReader( new InputStreamReader(uc.getInputStream())); String inputLine; while ((inputLine = in.readLine()) != null) System.out.println(inputLine); in.close();

This code makes use of the URL and URLConnection classes. Once the connection is set up, the page contents are read by an input stream. We make use of a BufferedReader, as it has a very helpful readLine() method.

Activating the agent In our example, the "Activating the agent" property is set to "Manually from agents list" to remove "(PublishCode)" from the actions list. The agent is then activated via the "Publish" action button in the "Docs ready to publish - XML view". This precaution is taken so users do not have the ability to run the agent against documents from other views in the database. If such a precaution is not necessary, then this step need not be taken. Allowing the agent to be invoked from the actions list allows users greater flexibity in terms of which views to use when identifying documents that need to be promoted.

XML-enabling your Lotus Notes application Page 14 of 16

© Copyright IBM Corporation 1994, 2008. All rights reserved.

ibm.com/developerWorks

developerWorks®

Section 7. Wrapup Summary This tutorial has shown you how to collect Domino documents and surface them as XML documents. We reviewed the data that's in the Notes forms, built views and a stylesheet to handle the data, and then looked at a Java agent that invokes an external servlet. With this information in hand, you should be able to apply these techniques to your own Notes documents.

XML-enabling your Lotus Notes application © Copyright IBM Corporation 1994, 2008. All rights reserved.

Page 15 of 16

developerWorks®

ibm.com/developerWorks

Resources • Read the article that accompanies this tutorial, "Managing and moving data using Java technology, XML, DB2, and Lotus Notes." Download the demo database called xmlcptut.nsf, which has the base elements that we'll develop in this tutorial. There are innumerable resources to guide you on your path to mastering XSLT. Listed below are some resources that provide basic groundwork, along with some that are specific to using the XML-enablement capabilities of Lotus Domino. • Exercising XML in Domino Designer (Lotus Developer Domain, January 2001) • "Publishing XML Data using Notes/Domino 6" (Lotus Developer Domain) • Take the tutorial, "Manipulating data with XSL" (developerWorks, October 2001). • Take the Introduction to XML tutorial (developerWorks, August 2002)

About the author Branavan Ganesan Branavan Ganesan is a Lotus Domino developer who remembers when Domino was just a pizza franchise. He has been working with the developerWorks development team for the last two years and had contributed to the content management system and the publishing environment of developerWorks. He holds a very useful Bachelors degree in Comparative Religion from Haverford College and a Masters degree in South Asia Regional Studies from the University of Pennsylvania. When he is not playing foosball or attempting to work, he plays tablas and practices surviving triathlons. You can reach Bran at [email protected]. This tutorial highlights the XML enablement of Domino, but is based on a collaborative project between the developerWorks infrastructure team and the eServer Developer Domain (eSDD) team. The members of the team are Jay Allen, Barry Busler, Elizabeth Dunnagan, Tim Janikowski, Keith Purcell, Adam Reeves.

XML-enabling your Lotus Notes application Page 16 of 16

© Copyright IBM Corporation 1994, 2008. All rights reserved.