More Phase Iii Final Report

MoRÉ: Mobile Reference Environment Rapid Prototyping, Spring 2006 Phase 3 Report May 12, 2006 Editors: Brian Ellis, Jae Choung Ha, Megan Hyland, Aashni Shah

HCI Group Brian Ellis Anand Gopalkrishnan Yong Woo Rhee Aashni Shah Wen Shu Tang Lu Daniel Zinow Search Group Tabreez Govani Melanie Haskell Sachin Kulkarni Kevin Smith Capturing Experience Group Jae Choung Ha Leon Mai Jonathan Tran Platform Group Mohammad Ahmad Megan Hyland Zack Menegakis Bryon Smith Adam Wolbach

– Implementation and Integration

RPCS – Spring 2006

1

– Implementation and Integration

RPCS – Spring 2006

Table of Contents 1

Introduction......................................................................................................... 4 1.1 Purpose ........................................................................................................... 4 1.2 Background..................................................................................................... 4 1.3 Group Members .............................................................................................. 5 1.4 Overview ........................................................................................................ 6

2

Conceptual Design............................................................................................... 7 2.1 Problem Definition.......................................................................................... 7 2.1.1 Problem Scenario .................................................................................... 7 2.1.2 Key System Requirements......................................................................10 2.2 Initial Solution Concepts ................................................................................12 2.2.1 Table of Selected Technologies ..............................................................12 2.2.2 Visionary Scenario .................................................................................12 2.3 Conceptual Design .........................................................................................13 2.3.1 Solution Scenario Selection Criteria .......................................................13 2.3.2 Product Design Specifications ................................................................14 2.3.3 Search ....................................................................................................14 2.3.4 Experience Capture ................................................................................14 2.3.5 Platform .................................................................................................15

3

System Tutorial and Usage Scenario .................................................................15 3.1 Summary of Integrated User Interactions........................................................15 3.1.1 User Interaction Issues and solution based on Input Device ....................15 3.1.2 User Interaction Issues and Solution Based on System and User .............17 3.2 Usage Scenario...............................................................................................17

4

Implementation and Integration........................................................................22 4.1 Unifying User Interaction...............................................................................22 4.1.1 User Interaction Design and Tutorial ......................................................22 4.1.2 User Studies Results or Walkthroughs ....................................................25 4.2 Search ............................................................................................................27 4.2.1 Functionality ..........................................................................................27 4.2.2 Experimental Measurements...................................................................29 4.2.3 Software Architecture.............................................................................33 4.2.4 Software Modules and Status..................................................................34 4.3 Experience Capture ........................................................................................38 4.3.1 Functionality ..........................................................................................38 4.3.2 User Studies Results ...............................................................................39 4.3.3 Experimental Measurements...................................................................40 4.3.4 Software Architecture.............................................................................42 4.3.5 Software Modules and Status..................................................................43 4.4 Platform .........................................................................................................49 4.4.1 Functionality ..........................................................................................49 4.4.2 Component Specifications and Pictures ..................................................49 4.4.3 Hardware Architecture............................................................................58

2

– Implementation and Integration

RPCS – Spring 2006

4.4.4 Software Architecture.............................................................................60 4.4.5 Hardware on Body..................................................................................61 4.4.6 Hardware Modules and Status ................................................................65 4.5 Conclusions....................................................................................................67 4.5.1 Summary of Key Design Issues ..............................................................67 4.5.2 Lessons Learned .....................................................................................71 5

Project Management ..........................................................................................73 5.1 Implementation and Integration Phase Results................................................73 5.1.1 Summary of Logbook Hours...................................................................73 5.2 Summary of the Entire Project........................................................................75 5.2.1 Task Dependency Chart..........................................................................75 5.2.2 Summary of Logbook Hours...................................................................75 5.3 Suggestions for Improving the Class ..............................................................78

6

Glossary of Terms ..............................................................................................79

7

Appendix.............................................................................................................80 7.1 Hardware Specification Tables .......................................................................80 7.2 Task Dependency Chart .................................................................................87 7.3 List of Figures................................................................................................88 7.4 List of Tables .................................................................................................88

3

– Implementation and Integration

1 1.1

RPCS – Spring 2006

Introduction Purpose

Mechanical systems have followed two parallel trends in the last few decades: they have become exponentially more complex, and have become increasingly common in various and myriad unexpected environments, many of which are far less controlled and accessible than a typical office. This reality has forced technicians who are assigned the task of repairing these machines into a dilemma: how does one access all the information one needs to repair a system with thousands of parts, while remaining mobile enough to service machinery that is in an inconvenient or unconventional location? The traditional solutions to this problem have been ad-hoc, ranging from reams of paper manuals to (more recently) laptop computers, and all have generally involved a fervent hope that the reference material one needs is present and can be searched through before one dies of starvation. Clearly, a better solution is required that takes into account the nature of mobile repair and allows technicians to find a balance between adaptability and well-preparedness. Further, an optimal solution would itself be adaptable to other purposes: the reasons that one might need access to information on the go are innumerably many, and the current tools — including laptops, personal digital assistants, and iPods — fall short. Laptops, with their full-size keyboards, delicate screens, and often dubious portability, simply aren’t agile enough; PDAs are streamlined for input, not output, and often lack the screen size and real estate to efficiently display large amounts of textual and graphical information; iPods suffer from the screen size problem as well, and lack the software and hardware extensibility necessary to cope with the informational needs of professionals (such as an Internet connection and the ability to display data that are not strictly textual). The MoRÉ platform was thus developed as an alternative interface designed specifically for individuals needing quick, ready access to large amounts of data while eschewing traditional keyboard-and-mouse input models that would compromise portability and efficiency.

1.2

Background

The Rapid Prototyping for Computer Systems class at Carnegie Mellon was contacted in early 2006 by Dick Martin of Inmedius Software, Inc. to design and prototype a mobile system that would allow United States Navy repair technicians maintaining F/A-18 Hornet fighter aircraft to more efficiently conduct their repairs. Initially, it was hoped that the class would have access to the repair procedure and training manuals for the aircraft themselves; due to issues of timing and material availability, however, the only material available as of the date the class was to begin were training and reference manuals for the IETM (Interactive Electronic Technical Manual) software system, made by Boeing, which is itself a software environment designed to allow repair technicians maintaining F/A-18s to more efficiently conduct their repairs.

4

– Implementation and Integration

RPCS – Spring 2006

This extra degree of separation from the repair process provided a unique challenge: it was unlikely that a repair technician would need a mobile unit to access information about the IETM, since the IETM itself could only be accessed from a stationary computer (or, at best, a semi-mobile laptop). In designing a mobile environment for accessing the IETM documentation, however, it was hoped that the problem could be generalized and that the mobile unit could pave the way for the replacement of the very system it had been created to document. This somewhat ironic arrangement has come to be referred to as a “meta-problem”, since the immediate goal of the project was to solve a problem with a system designed to solve a different problem, with the longer-term goal of solving the more fundamental problem itself.

1.3

Group Members

The MoRÉ development group was composed of four major teams, plus a fifth that was assembled from the other four during the final phases of development and testing. The teams, their responsibilities, and their members were as follows: Search Team The Search team was responsible for creating and deploying a search engine and interface that could be used effectively under the constraints imposed by the mobile unit’s input devices, as well as organizing and indexing the data that were to be made available through the mobile platform. Members: Brian Ellis Anand Gopalkrishnan Tabreez Govani Melanie Haskell Sachin Kulkarni Kevin Smith. Experience Capture Team The Experience Capture team was responsible for exploring ways of leveraging the experience of previous users of the system to improve the experience of current users, and examining the behavior of users to tailor their experience to their particular needs. Members: Jae Choung Ha Leon Mai Aashni Shah Wen Shu Tang Jonathan Tran

5

– Implementation and Integration

RPCS – Spring 2006

Platform Team The Platform team was responsible for determining, assembling, and manufacturing the hardware equipment needed to realize the mobile unit, as well as providing a software interface between the hardware devices and the software running on each device. Members: Mohammad Ahmad Megan Hyland Zack Menegakis Bryon Smith Adam Wolbach Mechanical Engineer: Jules Henry Human-Computer Interaction Team The Human-Computer Interaction team was responsible for defining the initial project specification and hardware and software requirements, baseline and visionary scenarios, exploring and incorporating prior research, user interface design of software and hardware interfaces, and the content of the final prototype demonstration. Members: Brian Ellis Anand Gopalkrishnan Yong Woo Rhee Aashni Shah Wen Shu Tang Daniel Zinzow Renderer Team The Renderer team was responsible for creating a framework that could display the information contained in the manuals to the user in a consistent and visually accessible manner, and for integrating the various software components together into a cohesive system. Members: Brian Ellis Tabreez Govani Jae Choung Ha Melanie Haskell Jonathan Tran Wen Shu Tang

1.4

Overview

The MoRÉ system provides an efficient, portable means of accessing reference material or any other dense, inter-related set of informational units. It can be used for extended periods of time without requiring the presence of an electrical outlet, does not necessitate a keyboard or any other means of text entry (such as handwriting recognition or an onscreen keyboard), leaves both hands free, and has many other advantages over any other common means of accessing such information. Its software interface is streamlined

6

– Implementation and Integration

RPCS – Spring 2006

to facilitate searching and leverages the inherent connections and relationships between different pieces of data to assemble all relevant details together, resulting in a smaller requirement of effort in order to find a particular piece of information. Most importantly, it is eminently scalable to address the “meta-problem” of mobile information retrieval in a professional context.

2 2.1 2.1.1

Conceptual Design Problem Definition Problem Scenario

To demonstrate the current manner in which aircraft maintenance tasks are performed, a "baseline scenario" was presented following an imaginary but typical technician through the steps of using the present-day Automated Maintenance Environment to perform a simple maintenance procedure. Consider the example of a maintainer named Mark. Mark is a rigger, a specific kind of technician whose responsibilities include entering observed problems with a plane into the Automated Maintenance Environment's Debrief system. Today, Mark is waiting out on the tarmac for an F/A-18 to land. Once it does, he will remove a small card from the nose of the plane and carry it back to the operations building, where he will place it into the Data Acquisition System (DAS). The DAS unit will generate a report detailing all the problems the card detected while it was inside the plane. Each of these problems is referred to as a "BIT discrepancy". Once the DAS report has been generated, Mark switches over to the Debrief system to examine the discrepancies and file work orders for the problems revealed in the report. He goes through the steps of loading the data into the Debrief system and finds himself on the Discrepancy Review screen (Figure 1). Mark is fairly new to his job, and his training process, though thorough, was a great deal of information to process all at once, and Mark soon finds himself confused. In particular, he finds the label for a pop-up menu in the lower right-hand corner of the screen – "Verified/Observed" – entirely inscrutable. Mark spies a binder on the desk claiming to be a reference manual, and starts flipping through it, hoping to find an answer. Many pages later, he finally arrives at the right chapter and section and sees a picture of the Discrepancy Review screen (Figure 2). To his dismay, however, he finds that the label for that pop-up menu is not the same in the reference manual as it is on his screen. The reference manual calls it "Create Work Order". Needless to say, this does little to alleviate his confusion, so he decides to consult the training manual – a document with which he is already familiar with from his orientation instead. He brings up the training manual, which is an interactive Web site, on his computer screen and begins clicking his way through the various chapters, pages, demonstrations, quizzes, and other content. He can only move one page or section at a time due to the design of the manual, which was intended to act as a tutorial rather than a

7

– Implementation and Integration

RPCS – Spring 2006

reference, so getting to the right chapter, section, and subsection takes some time and perseverance. Eventually, he comes across the picture he is looking for and reads the description for the Verified/Observed label (Figure 3). The training manual (Figure 3) seems to imply that the pop-up menu does indeed create a work order, but after the inaccuracy of the reference manual, Mark is no longer so sure he can trust the material to be up-to-date. As a last resort, Mark gets up from his desk and hunts down the resident expert on Debrief, a technician named Ray who has been on the job for many years. Mark describes the problem to Ray, who responds almost immediately, "Oh, yeah, they changed that text last year. It does the same thing though. Just always set that to 'YES' unless the BIT code is wrong." Mark thanks Ray for the help and goes back to his desk, realizing all at once that his confusion, and the failure of the manuals to relieve it, has cost him 35 minutes he could otherwise have been using to do his job.

Figure 1: Discrepancy Review Screen in Debrief System

8

– Implementation and Integration

RPCS – Spring 2006

Figure 2: Reference Manual Explanation

9

– Implementation and Integration

RPCS – Spring 2006

Figure 3: Training Manual Explanation

2.1.2

Key System Requirements

To fulfill the needs to meet the design, some basic functional requirements will be presented to appropriately address the problem at hand. For example, we have seen that any system containing small parts that could fall into a plane or get lost are inherently dangerous, and therefore infeasible. As a corollary to this rule, the system must also not protrude from the mechanic's body, lest some component get caught on a piece of equipment. Being tethered to a jet fighter as it prepares for take-off is not a pleasant experience by any stretch. Furthermore, one must also consider the harsh conditions in which mechanics often find themselves working. Any system designed for those conditions must therefore be rugged enough to withstand both high and low temperatures and rough environments, including blowing sand and rain. Similarly, a mechanic may be called upon to work under the glare of the desert sun or in a dimly lit hangar within minutes of each other, so the system must adapt to various lighting conditions without interfering with usability. In the interest of unobtrusive portability, any large, flat screens or keyboards – even those that might attach to the body or limbs of the mechanic cannot be used, especially since they may very well get damaged. A keyboard as input mechanism would be infeasible as mechanics often times use heavy work gloves. Having to remove one's gloves to interact with the system would place an unjustifiable burden on the mechanic to find a place to put them. On the other hand, any input or output devices incorporating an auditory component as an alternative must take into account the high ambient noise conditions usually present near jet aircraft, and function satisfactorily with ambient noise levels up to 85 dB or higher.

10

– Implementation and Integration

RPCS – Spring 2006

The software platform for an integrated interactive manual must also satisfy certain functional requirements. Since a technician cannot be expected to type or enter search terms every time they wish to see more information on a topic that is detailed in the page they are currently looking at, the software must support dynamically generated links between various types of content. For example, a page in the reference manual that mentions a certain acronym may link to a page in the glossary that explains the meaning of that acronym, and several pages in the training manual each of which also mention that acronym. The manuals must also be clearly labeled for easy referencing for the technicians. In other words, the user must be able to easily distinguish which of the two manuals he or she is currently reading from. For those cases where a technician needs to know about a concept or component that is not reflected on the page he or she is currently perusing, the system must also support some kind of open-ended search capabilities whereby a technician can enter certain keywords to find relevant content. The software must also capture the experience of the users in order to help future users to have an easier time finding what they need. In a sense, the software must recognize the browsing experience of past users and give leads to the direct results (what they were trying to find) to future users. This would facilitate the browsing experience of the users. Lastly, in order to take advantage of shared experience and expert practices, the system must enable users to leave annotations on particular pages that expand or clarify the manual content there. The physical system as a whole must also conform to other requirements, some of which are not as obvious from the visionary scenario but all of which are important to the satisfaction of the user. The entire system must be portable, including the display; this requirement constitutes the primary endorsement for a head-mounted display. The typical shift length of an aircraft technician is eight hours, with a half-hour lunch break halfway through. Since the technician cannot necessarily drop whatever he or she is doing and return to the hangar to recharge the unit, the battery life of the system must be at least four hours. This assumes "hot-swapping" capabilities, whereby depleted batteries may be changed out for charged ones without the device losing its state. If such capabilities are not feasible, the minimum battery life requirement becomes eight hours. Lastly, but still importantly, the device must be usable by everyone who might have a need for it. Just as it must be operable with heavy work gloves from inside the cockpit of a plane, so too must it be usable from a desk equipped with a keyboard and external monitor. Maximizing the usability of the device will also maximize its utility.

11

– Implementation and Integration

2.2 2.2.1

RPCS – Spring 2006

Initial Solution Concepts Table of Selected Technologies

Below is a list of selected technologies used in the final implementation of the design. The following sections address user scenarios for use of the design, incorporating each of these features, as well as more in-depth descriptions of the requirements, helping to explain why these technologies were chosen for the final prototype. Table 1: Selected Technologies System Requirement Selected Technology Mobile Computer Sony Vaio-U8G User Input Device Custom Dial Device Earphones David Clark H10-13.4 Headphones David Clark H10-13.4 USB Connections Targus USB 2.0 Mini Hub Head Mounted Display Liteye HMD Audio Connector Sound Professionals USB Audio Converter Browser Custom Browser based on Firefox

2.2.2

Visionary Scenario

The initial visionary scenario from Phase I follows an imaginary but typical technician through a routine day of using the manual system. The scenario gives enough detail to show how it must work to provide the technicians with what they need, but not so much detail as to explain every nuance of the system. Our Technician, Mark, is standing aside while an F/A-18 comes down on the runway. After the pilot gets out, he does his initial routine tasks and finds a pen lying in the floor of the plane during his inspection. Knowing that foreign objects in the plane can cause damage to sensitive systems, he enters this into the Debrief system as he heads back to the hangar. While heading back, though, a sand storm starts up. Gladly, Mark's equipment is ruggedized against the elements, but the same cannot be said of the $60 million aircraft. Worried that the blowing sand might cause damage to the plane while Mark has the canopy open, he decides to move it into the hangar. As he brings the system into the plane, it adjusts to the artificial lighting. Now Mark can begin doing the debriefing on the plane. Unfortunately he is unsure of how to enter the discrepancies he noticed and thus consults the reference manual. The reference manual's explanation of entering non-BIT discrepancies is rather confusing, so Mark decides to compare it with the training manual by following a dynamic link. The training manual's detailed explanation quickly clears up his confusion, and to help any novice in the future, Mark adds an audio annotation to the reference manual. This annotation appears on the page as a selectable element.

12

– Implementation and Integration

RPCS – Spring 2006

Having finished doing the debrief Mark moves on to repairing the plane by bringing up the list of repair tasks. Mark finds the correct repair procedure and starts going through the steps. Now however, he needs to keep a constant watch on the hundreds of controls in the cockpit so as not to accidentally change anything. Mark is very glad that he does not have to removes his gloves to flip any pages. After the repair is done he powers up several systems of the aircraft to test the changes made. Doing so causes a loud whine that fills the hangar, but Mark is still able to continue working with no problem. He is able to issue commands and receive feedback from his repair system without any hindrance.

2.3 2.3.1

Conceptual Design Solution Scenario Selection Criteria

Several of the functional requirements for the prototype hardware are derived almost directly from the visionary scenarios. For example, we have seen that any system containing small parts that could fall into a plane or get lost are inherently dangerous, and therefore infeasible. As a corollary to this rule, the system must also not protrude from the mechanic's body, lest some component get caught on a piece of equipment. Being tethered to a jet fighter as it prepares for take-off is not a pleasant experience by any stretch. The visionary scenario also illustrates the harsh conditions in which mechanics often find themselves working. Any system designed for those conditions must therefore be rugged enough to withstand both high and low temperatures and rough environments, including blowing sand and rain. In the interest of unobtrusive portability, any large, flat screens or keyboards – even those that might attach to the body or limbs of the mechanic -- cannot be used, especially since they may very well get damaged. A keyboard is also not usable with heavy work gloves on, and having to remove one's gloves to interact with the system would place an unjustifiable burden on the mechanic to find a place to put them. So we decided to add a scrollable device to be attached to the chest of the technician for input and also use a high intensity microphone and headphone for the purpose of audio input. The software platform for an integrated interactive manual must also satisfy certain functional requirements. Since a technician cannot be expected to type or enter search terms every time they wish to see more information on a topic that is detailed in the page they are currently looking at, the software is made to support dynamically generated links between various types of content. Lastly, in order to take advantage of shared experience and expert practices, the system must enable users to leave annotations on particular pages that expand or clarify the manual content there. The annotations must be easily accessible within the same page and while the content is being displayed. Furthermore, the annotation facilitates collaboration within the aircraft maintenance workers to facilitate. In other words; more experienced maintenance workers can share their experience with others through annotations. Thus, a single pane was designed to support audio annotations and also have it updated in the search links for other users to hear and learn.

13

– Implementation and Integration

RPCS – Spring 2006

The typical shift length of an aircraft technician is eight hours. Since the technician cannot necessarily drop whatever he or she is doing and return to the hangar to recharge the unit, the battery life of the system must be at least four hours. Thus, our system is intended to address the worst case scenario of when a worker is out on the field for the entire duration of his or her shift. This assumes "hot-swapping" capabilities, whereby depleted batteries may be changed out for charged ones without the device losing its state. So to counter this problem a unique swap controller was designed to alternate between three different set of battery packs.

2.3.2

Product Design Specifications

Each of the four major design groups helped to research and design a module of the overall system prior to integration. Design choices and specifications for the final integrated product are addressed in depth in Section 4.

2.3.3

Search

The Search group was responsible for mappings between the procedure and the training manual and to ensure that searching through the manuals is easy and straightforward. Depending on the input from experts and learning from users’ responses and usage patterns (data provided by capturing experience), specialized links and annotations were incorporated into individual pages. As such, the search group was responsible for integrating all these components and presenting them in a unified view to the users. This included retrieving relevant information from different functionalities and formatting both manuals and finally rendering them into our custom browser to be presented to the user. In addition, the limited input capabilities of the handheld means that we had to tailor our methods accordingly. Thus, the scope of search group was pretty wide, including interacting with the Hardware/Platform Group for the specific input interfaces, with the Capturing Experience Group for experts’ data, annotations etc and with the HCI Group to ensure that the users are able to effectively use the capabilities provided by our group.

2.3.4

Experience Capture

Based on the visionary scenarios presented in Phase I and Phase II, ideas were formed about how to capture a user’s experience. First, it was felt that knowledge from experts would be greatly helpful to a user who has a question. The user may be a novice, or perhaps an experienced technician that has switched to an unfamiliar area of maintenance, etc. Though this, the Expert Contact Information module was formed. This component will make available enough information to contact the expert, such as title, years of service, and contact information. Another idea from the visionary scenario was that if a user was lost and confused in dealing with the manual for a long time, then finally figures out the solution, they should be able to provide their experience to other users. In this way they can help save their colleagues time. To solve this, audio annotations will be available for users. Through these, a user can add their own input to the manual, visible as links to files, so that other 14

– Implementation and Integration

RPCS – Spring 2006

users can learn and save time using them. The general category of task recognition was proposed from the beginning. However, since robust task recognition is very difficult, a simplified model is used for this prototype. The Cycling Detection module does task recognition by identifying when a user is cycling. It is assumed that if a user is cycling they are probably lost, so in a situation where they are cycling, the page renderer will highlight help in the side pane. It was noticed early on in the project that navigating through the manuals was not easy; they were large and often confusing. Smart Links shorten the time it takes a user to find the information they are looking for. Links will be generated either through user’s previous usage experience or through indexed search results to facilitate browsing experience. Finally, to enable many of these features, there is a User Login page to appropriately record information from the user to enable some the features from capture experience and help user in search for the content he or she is looking for. Further detail on all these modules is provided in the Functionality section.

2.3.5

Platform

Based upon the Visionary Scenario, the Platform team was responsible for designing and obtaining the necessary hardware to create a mobile computing unit and to enable peripherals capable of collecting user input and delivering outputs such as display, and speech annotations. With these requirements in mind, the Platform team identified different technologies and specifications for the design, which was determined to include a mobile computer, dial input device, headphones, microphone, and head mounted display unit. In order to implement the hardware architecture and to interact with the Software development groups, a new dial device was designed to accommodate all of the new searching features. A mobile computer running on the Windows XP platform made the integration with the Software teams smooth. In addition, to meet the minimum system time requirements, and to maximize user use, the group decided to implement a hot-swap controller, enabling a user to utilize the runtime of several attached batteries without having to physically switch the battery in the system. This increased time will add to usability of the system, and add to the simplicity of use. Furthermore, all components can be attached to a wearable suit composed of belts and holsters and will not interfere with the mobility of the user.

3 3.1 3.1.1

System Tutorial and Usage Scenario Summary of Integrated User Interactions User Interaction Issues and solution based on Input Device

By far the most challenging UI issue in a mobile reference environment is the lack of a character-based text input device. Much of the conventional wisdom about navigating

15

– Implementation and Integration

RPCS – Spring 2006

dynamic content assumes the existence of a keyboard or similar technology; if the choices are too many to enumerate, the obvious fallback measure is to allow (some might say “force”) the user to enter search terms directly. While this approach does enjoy a certain ease of implementation, most modern attempts at information organization and retrieval at least supplement text-based entry with some kind of relevance engine — it could be that the user does not know precisely what he or she is looking for, and the system could suggest pieces of information that have historically been relevant to the concept at hand. Suggesting likely next steps does more than just save keystrokes; it can demonstrably improve the quality of results and increase user satisfaction. That being as it may, however, dropping support for text entry altogether is a very dangerous move. Content, especially on the Internet, is typically presented as a graph of associations between units of information, and text-based search provides the ability to teleport to any node in that graph immediately. Remove that functionality and what remains is traversal, which may be appropriate for small maps but becomes a complex proposition for larger, less well-organized systems. The potential problems are manifold: for one, traversal assumes the existence of a starting point, and the only nodes available from that starting point are those directly adjacent to it. This is almost guaranteed to be a small subset of all available nodes, and worse yet; it will probably be a set of closely related nodes that have relatively few external connections. In the worst case, this leads to another problem with traversal: the possibility of nodes or entire sub graphs that are entirely disconnected from the rest of the system. Put more plainly, if nothing links directly to a page, there is no way to access that page without text entry. We can see, then, why removing text-based search is a risky move: it takes away our safety net and requires us to be very careful with our relevance algorithms. If we systematically miss a relevant result, we effectively ensure that that page will never be viewed unless it is directly linked to by another relevant page. In the case of the manuals hat are viewed on the mobile units, these direct links are few and far between at best, and entirely absent at worst (as in the reference manual). We can mitigate this effect by such tactics as setting the starting point of the traversal to a “table of contents” page that actively links to every possible page, but this solution can be high-maintenance and may constitute a special case; it is therefore essential that search results be either exhaustive which can itself be problematic from a user interface point of view, as no one wants to sort through a hundred or more search results), or highly relevant at all times. With such high stakes on relevant search results, it would be ill-advised to design a system without some means of detecting a breakdown in quality and compensating, either by showing more results that had not yet been visited or by broadening the scope of the search to include pages the user might not have realized were important. The experience capture subsystem is a step in this direction, and while a robust and adaptive mechanism for tuning search results based on past history could very easily be a project in and of itself, it is hoped that features such as cycle detection will help to make up for the lack of text-based input.

16

– Implementation and Integration

3.1.2

RPCS – Spring 2006

User Interaction Issues and Solution Based on System and User

An important design requirement of the MoRÉ system is to allow the user maximal control over his or her information context. Particularly, the system must avoid whisking a hapless user away to parts unknown while his or her original question gets buried under a ponderous browser history. Even mainstream browsers have dealt with this problem in their own time; the solution was, at first, multiple windows, and then later multiple tabs, but even before this browser took care not to change context without a good reason. Imagine if the only way to go to a Web site was to click a link somewhere on the current page that would display a new page, blank except for a text field in the middle, in which you could enter the address of a new Web site to visit. While the example may seem silly, in a system with a limited range of possible user gestures this approach might easily become a design reality. To prevent this, MoRÉ borrows a page from mainstream browsers and displays a sidebar in which search results can be displayed. This encourages the user to click terms about which they may be unfamiliar or confused: in many cases, the titles of the search results may themselves serve to alleviate the user’s uncertainty, and in any event there is no obligation to click on one: investigating a search result does not change the information context. Similarly, the user may always choose to preserve their previous context as well as entering the new one by opening the link in a new tab; this mimics the real-world operation of opening two manuals side-by-side. The use of multiple panes also permits preservation of context by providing a mechanism whereby the user can switch their focus to another goal without losing their progress on the first. For example, a user could scroll halfway down a lengthy page, click on a keyword link to bring up search results in the search pane, then switch to the search pane and open one of the results in a new tab. The user could then simply switch back to the original content pane and continue reading the document, secure in the knowledge that their search result page will be waiting for them once they are finished. The content view would remain in its original state, preserving such properties as position on the page, highlighted keywords, etc. The next section talks about how the user would interact with the dial and the system.

3.2

Usage Scenario

Once the user has logs in and arrives at the Main page of the browser there are multiple functionalities he could perform using the dial. Below is a short description of these functionalities of the dial and its corresponding effect on the browser:

1. Scrolling Between Links: You can scroll between the different smart links by just turning the rotating piece on the dial. Turning the dial clockwise will scroll down the page while anti-clockwise will scroll

17

– Implementation and Integration

RPCS – Spring 2006

up the page. This will allow you to swap between different smart links on the particular page with ease. Dial Usage:

Desired Effect:

2. Clicking on a Smart-Links: You can click on a smart link by just pressing on the center piece. This clicking can help you bring up the search results on the right pane or can take you to a different page itself entirely when clicking on a search result. In case of annotations it would bring up the player to hear the voice annotations left behind. Dial Usage:

18

– Implementation and Integration

RPCS – Spring 2006

Desired Effect: (clicking on the smart link “Work center” brings up results on the search pane)

3. Switching Between Panes: You can use the pane switch button to swap between different panes in the browser window. This gives you and option to scroll through links in a different pane of the browser. Dial Usage:

Desired Effect: (Clicking on the pane switch button twice moves the highlight from “Work center” in the main window to the first results in the search pane)

19

– Implementation and Integration

RPCS – Spring 2006

4. Bringing Up Links in New Tab: By holding down both the left and right tab button the user can bring a desired search result in a new tab entirely. Both the tab button should be held down simultaneously on the desired search result. Dial Usage:

Desired Effect: (holding down and releasing both the tab buttons on the first search result of the search pane brings a new tab with the search result)

5. Switching between the Tabs: The dial gives you the option of switching between the tabs using the left and right tab buttons. The right tab lets you tab to the right, while the left allows you to tab to left. This allows you to switch between multiple tabs thereby providing quick easy reference to certain content. Dial Usage:

20

– Implementation and Integration

RPCS – Spring 2006

Desired Effect: (holding either of the left or right lets you switch tabs)

6. Using the Mouse cursor (optional): The dial gives you and option to move the mouse pointer or cursor of windows to either point to the desired location or move the cursor to close the browser. This feature of the dial can be used by move the cursor either in a 2-dimentional plane similar to the use of a joystick. Dial Usage:

7. Using the windows left Mouse Click (optional): After the mouse cursor has been used to move the cursor to a desired location you can use the windows mouse left-click option by holding down and releasing the pane switch and select button. The click happens only upon simultaneous release of both the buttons. Dial Usage:

21

– Implementation and Integration

4 4.1 4.1.1

RPCS – Spring 2006

Implementation and Integration Unifying User Interaction User Interaction Design and Tutorial

The user will use the dial as the main input device to interact with the user interface of the manual integration software. The software provides four panes in which contents of the manuals, options for the screen, results of user’s search queries, and annotation recording interface are presented. The dial’s components include a wheel, a select button, a pane switch button, two tabbing buttons, and a pointer knob. Each component of the user interface and the dial are functionally connected to support the user’s process to achieve his/her goal. 4.1.1.1 User Interface

Figure 4: User Interface The main pane in Figure 4 displays the contents of a manual that the user had requested for. Each word of the manual is a link that displays the related materials in the search results pane upon selection. However, some of the common words such as the, and, and of are not highlighted as a link since they do not represent or mean a specific topic related to a maintenance task. This pane provides the user the capability of scrolling as the contents may exceed a length that cannot be displayed at once on the screen. The next pane in Figure 4, positioned on the top right section of the screen, provides two options: the option for closing the current window and the other option for marking

22

– Implementation and Integration

RPCS – Spring 2006

whether the problem was solved. When multiple tabs had been created, the user can close the current tab by clicking on the “Close this window” option. In the case of which only one tab is present, this option will be grayed out and won’t be accessible to the user. This pane is also used to indicate whether the material (contents of the manuals or expert’s annotation) solved the user’s problem. The user can select this link if his/her question was solved by the materials provided by the interface in result of his/her search query. The large pane on the right section of the window in Figure 4 is reserved for displaying the search results, definitions of key words, and smart links related to the topic displayed in the main pane. To prevent cluttered information in this pane, not all search results are displayed, but only the most relevant links are provided by the interface. The interface provides the number of results returned, the origin of the link (reference or training manual), and the title of the link. The definitions section mainly serves to provide what the abbreviations mean, such as IETM, which refers to Interactive Electronic Training Manual. Following section displays the smart links, which are the links to topics that are related to the page displayed in the left. If an expert have had left an audio annotation for the topic, it is also displayed in this pane. The information of the annotation including the time it was recorded, the name of person who left the annotation, and the length of the audio are also provided here. The user can select an annotation to refer to an expert’s knowledge on the topic to solve a problem. The bottom right pane in Figure 4 provides an interface for recording an audio annotation relating to the topic. When a user desires to leave his/her knowledge that may be useful in helping other users to solve their problems, he/she can select the record button and record an audio message. Upon selection, the interface will display how long the it has been recording and allow the user to stop recording by once again selecting the same button, which now displays as a stop button. 4.1.1.2 The Dial

23

– Implementation and Integration

RPCS – Spring 2006

Figure 5: Dial Prototype The most dominant feature of the dial, seen in Figure 5, is the wheel. By turning the wheel clockwise/counter-clockwise when worn, the user can move the selection box to the next/previous link accordingly. The wheel scrolls up/down the page when the selection box reached the top/bottom of the displayed screen but not the actual top/bottom of the pane. When the box reaches the top/bottom link on the pane, continuously turning the wheel will move the selection box back to the bottom/top link of the pane. The inner area of the ring of the wheel (Figure 5) functions acts as the select button. After moving the selection box to a desired link by turning the wheel, the user can select the link by pressing any section of the area. This button will also allow the user to click on different buttons positioned in the interface. The center knob within the select button (Figure 5) has the capability to move the cursor on the screen in eight-direction movement. However, this knob is not to be used as a primary controller for the interface. The large button placed on the upper section of the dial (Figure 5), is the pane switch button. By pressing this button, the user can move the selection box to the next pane in the current tab. For example, clicking this button while the selection button is in the search results pane will move the selection box to the record annotation pane. The two tab buttons on each side of the dial (Figure 5) have two functions. Pressing the left/right tab button when worn by the user will open the tab accordingly. Second functionality of the tab buttons is that when they are pressed (squeezed) together simultaneously, a new tab is created with the materials related to the selected link.

24

– Implementation and Integration

4.1.2

RPCS – Spring 2006

User Studies Results or Walkthroughs

The following contains a scenario that walks through how the dial and computer interface interact with each other and how this system is used by the user. The user in this example is Mark, a maintenance worker at an airfield. He goes out into the field to fix the problems with the airplanes. When done with his work, Mark returns to the office to download all of the information on his device to the computer. Mark has just finished a day’s work at the airfield. As he heads back to the office, he wants to upload all of the procedures that were completed, part information that was collected, and bookmarks made. To follow the steps of doing this correctly, Mark goes to the Uploading section of the reference manual. Mark scrolls through this document, which he does by rotating the wheel of the dial. Rotating the wheel causes a highlight on the screen to move from link to link on the screen. Everything is making sense until Mark reads “Uploads should only be performed when running Work Center on a workstation.” Mark doesn’t really work with the workstation too much, so he is kind of confused about this “Work Center.” To find out more, Mark rotates the wheel until the highlight is on “Work Center.” To do a search on Work Center, Mark presses down on the center button that is located within the wheel. The search results appear in the pane on the right side that is just below the “Solved Problem” pane. To get there Mark clicks down on the pane button with his finger to move the highlight from the main pane to the “Solve Problem” pane and finally to the “Search Results” pane. In the search results, Mark sees a link to the Work Center part of the reference manual. He decides that this result looks like it would be helpful and he rotates to the wheel until “[reference] Work Center” is highlighted. Thinking that he might want to compare the information about Work Center from the reference manual with the current screen, as there are more mentions of this Work Center on this page, Mark squeezes both tab buttons into the dial. This action causes the reference section on Work Center to appear in a new tab. After scrolling a bit through the page, Mark now understands that the Work Center is the software on the workstations (desktop computers) that is what grabs the data from his wearable computer and transfers it to the workstation. To go back to the page on Uploading, Mark presses in on the left tab button which brings him back to the first page. He reads further along the page and realizes that there are a lot of mentions of how the Work Center needs to be running for many things to happen. Mark is not sure how to know if the Work Center is running and he definitely does not want to attempt to upload just to fail. With this confusion in mind, Mark clicks on the right tab button to move back to the Work Center page. Reading further along on this page, Mark finds out that to tell if the Work Center is running he just needs to look at the desktop of the workstation and look to see if the Work Center software is open. If it is open, then it is running. Feeling confident he now understands this whole Work Center thing, Mark clicks on the pane button to move to the “Close Window/Solved Problem” pane. He rotates the wheel until “Solved Problem” is highlighted. Mark then presses on the center button located in the wheel to indicate that this current page solved his problem. After he clicks on the

25

– Implementation and Integration

RPCS – Spring 2006

button, a red check mark appears next to “Solved Problem”. Mark then continues to rotate the dial until “Closed Window” is highlighted. With this highlighted, Mark presses the center button within the wheel to close this tab. Now Mark is left with the page about Uploading and with no tabs in sight and no “Close Window” button (since the first page cannot close at all). Not wanting anybody else to have to go through the same struggle as he did, Mark clicks on the pane button to move to the “Annotation” pane. With the highlight on top of the record button, Mark presses the wheel down and the record button turns to a green stop button and the length of the recording also appears Mark goes ahead and explains what a Work Center is and how to know when it is running. When he is finished, Mark presses down on the dial to stop the recording. A note that an annotation was made by Mark on April 2, 2005, with a length of 2 minutes appears underneath the search results and smart links. Mark finishes reading about the uploading process and then actually performs the process. With that done, Mark moves onto the next section “Downloading Portable Computer to Work Station.” This section describes how to download any annotations made and problems solved. To do this, though, Mark needs to attach the portable computer to a dock. This involves taking the portable computer out of the protective case attached to his leg, then open up the computer connector, and finally to place this computer in a certain orientation into the dock. Mark knows how to take the portable computer out of the case attached to his leg but everything afterwards he has no clue. To figure out more about the portable computer and how to attach it to this dock, Mark rotates the wheel so the word “Portable Computer” is highlighted at which point he clicks down on the center button within the wheel to bring up search results for this concept. There are so many search results so Mark takes a look at the Smart Links and sees “Portable Computer and Dock Station.” Mark rotates the dial to highlight this, and presses down on the dial to bring this information on the screen. This page is very helpful, showing and explaining the parts of the computer. Even with the visuals and explanations, Mark is still confused as to how to open the connector and he does not want to damage the controller so he decides to look for more help. He sees that there is an annotation, so he clicks down on the pane button to the “Search Results” pane, rotates the dial to the play button and clicks down on the wheel. The annotation plays and informs Mark that the connector is covered by a cover that just needs a hard push to open and that the computer is durable enough not to be damaged. Mark feels confident that he can open this connector without any problems now. With his problem solved, Mark clicks on the pane button until the highlight is in the “Close Window/Solved Problem” pane. He then rotates the wheel until “Solved Problem” is highlighted at which time he clicks on the center button within the wheel to indicate

26

– Implementation and Integration

RPCS – Spring 2006

that the current page solved his problem. A red check mark appears by “Solved Problem” also. To go back to the “Downloading Portable Computer to Work Station” page, Mark rotates the wheel to highlight “Close Window” and presses down on the dial to close the current page. Mark is now back to the “Downloading Portable Computer to Work Station.” From there he continues to read the instructions and follow them, successfully downloading all of the annotations and solved problems to the Work Station.

4.2 4.2.1

Search Functionality

Figure 6: MoRE Web Browser Interface The search subsystem provides the user with their sole means of navigating through the training and reference material. This reliance on the search mechanism is due to the lack of a keyboard or other text-entry interface on the mobile unit; the user may only select from the links that are presented to him or her at any time, and these links must therefore always permit the user to access the largest possible subset of the manuals without causing information overload. In general, this is accomplished by intelligently deciding what is relevant to a particular page through the use of smart links, as well as providing a list of topics relevant to specific important terms, called keywords, on the page. When a user wishes to learn more about a topic related to the page he or she is currently viewing in the web browser, he or she can click on a smart link to be taken to a page related to the current page in some way, or can click on a keyword to see a list of search results for that keyword in another pane. Because clicking on a smart link causes the user’s primary focus to change to another page, it is therefore most appropriate for “nearmisses”, cases where the user has found a page related to the answer they are seeking but

27

– Implementation and Integration

RPCS – Spring 2006

not the exact page they wanted. Clicking on keywords, by contrast, is non-destructive to the context: that is, the user stays on the same page and simply sees a list of pages related to the keyword that was clicked. The search subsystem also provides access to annotations, editions or additions to a page that are input from the mobile unit itself (usually by way of an audio input, producing a sound file that can be played back by others). These annotations are stored and associated with their respective pages by the search framework. Care has been taken to ensure that the search subsystem is as adaptable as possible. Although there is currently no way of performing free-text searches (again due to the lack of a keyboard or other text input device), the search framework provides low-level functionality to permit such searches. If future versions of the reference environment support speech recognition or some novel form of character-based input, supporting such input should be straightforward and will not require the content of the manuals to be reindexed.

Figure 7: Search Functionality The search subsystem is comprised of several components, which can be seen in Error! Reference source not found.. The browser, as depicted in Figure 6 and explained

28

– Implementation and Integration

RPCS – Spring 2006

above, acts as the user’s interface to the MoRÉ system, and in addition to the browser, there are a number of other components, transparent to users, which provide the functionality previously described. As can be seen from Figure 7, user requests must pass through a number of components before presenting the user with the requested information. Every request must pass through the web server and then the controller, which passes the request to the experience capture logic unit before passing the request to the renderer. The request may be for a new manual page, new search results, new smart links, or a new browser tab among other things, and the renderer must generate the necessary response as determined by Error! Reference source not found.. After receiving the updated view, the web server can display the results to the user’s browser.

4.2.2

Experimental Measurements

For the search subsystem, the first experimental method consists of gathering data about the manuals and using the data to design and run the next set of tests. The Indexer and Search Engine are the components involved in this refinement process. Tests are performed with only the training manual since the training manual provided is complete and the procedures manual provided only includes a few sparse examples. The results of the tests performed are used as if they are manual independent due to the belief that the general content formatting and sentence structure is similar. The second method of testing is performed by looking at how browsers currently implement different functionality to see how much the team’s implementation can leverage and customize current implementations. 4.2.2.1 Training Manual Statistics Provided by Indexer The Indexer successfully captured a multitude of information regarding the training manual. It cataloged information about all 1,176 training manuals pages provided. Such information about the pages included page length and page path. The primary task of the Indexer was to store information about the words on a page. A summary of the training manual statistics can be found in Table 2. Table 2: Test Results for Indexer Tests Test Result Number of Pages 1,362 Number of Instances of Words 96,614 Number of Unique Words 3,868 Occurrences of “the” 4,965 Occurrences of “maintenance” 1,279 Top 5 Words With Most Instances the, to, of, maintenance, and Number of Words that Only Appear Once 884 Number of Words that Only Appear On One Page 947

29

– Implementation and Integration

RPCS – Spring 2006

Number of Words that Appear on 10 Pages or Less Percentage of Strong Words Percentage of Javascript Words Time to Index Entire Training Manual

Number of Keywords

2,578 9,5% 21.6% ~5 minutes locally ~15 minutes remotely 612

Approximately 96,614 instances of words were added to the index table in our database and about 3,868 unique words. The types of information collected about each instance of a word include the word, the page it was found on, the offset into the page, whether or not it was found bolded, in a title, or strong, and whether or not it was found in javascript. The word “the” was the most common with 4,965 instances, and the word “maintenance” came in fourth with 1,279 instances, appearing on 386 pages. About 884 words only appear once ever, 947 words only appear on one page, and 2,578 different words appear on no more than 10 pages. The first thing that these results indicate is that the amount of data that we are dealing with is large enough that we have to be careful with what we store and how we store it because the database will live on our mobile device. In our database, we use page_ids as references instead of the path to the page, and the field sizes aren’t extraneous. Additionally, we put extra time in creating intelligent schemas to avoid redundant information while still keeping performance up. Also, what this data indicates is that the distribution of words in the training manual is extremely exponential, with a few words appearing ubiquitously and the majority of words appearing on very few pages. Figure 8 shows this trend more specifically. The distribution of the words helped in selecting thresholds for word importance, page importance, and how relevant certain words are in determining the content of the page. These concepts together form the basis of selecting keywords to create search links on within each page. Keyword generation tests are addressed in the following section.

30

RPCS – Spring 2006

Frequency vs. Number of Words with Frequency 1800 1600 1400

training manual

Amount of words with given frequency in

– Implementation and Integration

1200 1000 800 600 400 200 0 0

1000

2000

3000

4000

5000

Frequency of words

Figure 8: Graph of Frequency of Words to Number of Words with Given Frequency The last section of Indexer statistics gathering was to see if previous hypotheses about the importance of formatting on a page would prove to be a strong classifier of the importance of a word on a page. Nearly 9.5% of words contained a bold or title markup, and 21.6% were found in javascript. Since only a fraction of the words are bold or in a title, these words should be weighted more heavily. Words appearing in javascript are usually associated with specific instructions, so they should be given priority if the user is searching for instructions. But since about one fifth of the training manual’s words appear in javascript, the additional weighting should be only slight because the javascript condition is not too unique. The data that was gathered from the Indexer helped the search team to adjust the Search Engine ranking parameters in hopes of providing more relevant results to the user. Additionally, the data has helped the queries performed by the Search Engine to focus on performance of search by taking advantage of the way the data is structured. Actual tests on the Search Engine relevancy were not done since no user testing took place and the result pages returned are difficult to analyze in their raw form. The next steps of Search Engine and Indexer tests would include actual relevancy tests. 4.2.2.2 Keywords Results Provided by Indexer The keywords’ testing that was performed provided an evolution of the idea of how keywords should be captured. Given that the central requirement of a keyword was a word that alone would be a useful search term for the user, the idea of how to find a useful word is what these tests sought to accomplish. The first idea of how to find a keyword was to use all words that appeared on only a small fraction of documents (less than 11 pages). If the word didn’t appear too often, then maybe a search on that word would be useful. This algorithm was implemented and tested. Almost two-thirds of the different words in the training manual were on no more than 10 pages. These words account for almost 9.7% of the total instances of all words. If all these keywords were highlighted, than about one-tenth of all words would be

31

– Implementation and Integration

RPCS – Spring 2006

highlighted within a page on average. This may be a good set of words that are useful to search on for the user. The words that were added to the keywords table using the above criteria were looked at. Unfortunately, words like “very,” “who,” and “regardless” appear in the list of keywords along with more technical terms. While the user could search on these types of common English words and get a small set of results which might seem relevant, these words hardly describe the content they are found in and wouldn’t be suitable words to link the content among pages. Therefore, it was decided that these types of words should be eliminated. The next test we ran was an algorithm to remove common English words from the set of keywords to further refine the set. Comparing frequencies of words in the manuals to words found in more typical English text, the test was able to select words that appeared significantly more frequently in the manuals than they did in the English text. These words differentiate the manual from typical English text, and thus keywords should be contained in this set. This optimization resulted in a keyword list half the original size (about one-twentieth of the total instances of all words) and upon inspection, the words in the keyword list seemed fairly descriptive to the content they might be found in. Besides further tweaking of thresholds, the basic algorithm of selecting a keyword was chosen as a result of these tests. 4.2.2.3 Browser Feature Testing Results Since the search team is building a customized browser using XML User interfaces Language (XUL), and the Mozilla Browser supports this type of customization, functionality in Mozilla was examined. Specifically, functionality dealing with the response to user actions such as Tab, Enter, Space, Click, and Scroll was tested. Tabbing in Mozilla moves the focus from one hyperlink to the next. When it reaches the end of the hyperlinks for a particular frame, the focus is shifted to the next frame, and tabbing starts moving through hyperlinks in the new frame. When this is complete for all frames, tabbing then moves through different elements in the toolbar area. The functionality that is sought from a customized browser is that tabbing moves through hyperlinks in a particular frame, and when the end of the frame is reached, the next Tab press resets the focus to the first hyperlink of the current frame. Additionally frame switching other than tabbing is not mapped to a particular key currently. A customized browser would need to assign a key to this particular operation. The Space button acts as a page-down, while the Enter key acts as a click on the currently selected hyperlink or a form submit, depending on how the page is coded. A click on a link follows the link; a click on anywhere but a link does not follow a link but may change the current frame focus if the click is in a frame that does not have focus. Lastly, scrolling using the scroll bar or a mouse scroll wheel scrolls down the page in a smooth fashion. When tabbing through the links of a frame, if a link receives focus and it is not currently visible, the page will scroll until it is visible.

32

– Implementation and Integration

RPCS – Spring 2006

Based on the behavior that the customized browser should have and these observations of how the Mozilla browser currently operates, the search team will implement slightly different behavior for tabbing, the space bar, and the Enter key. Additional button press messages will be handled in order to add frame switching, left and right tab window switching, and controlled tab creation. 4.2.2.4 Renderer Feasibility Tests Another type of test that was performed was checking the feasibility of central Renderer functions. Specifically, a Customized Browser/Renderer prototype unit was developed using Coacoa under Mac OS. Coacoa allowed the implementation to have Document Object Model (DOM) control over the pages in the training manual, and thus control enough to find, highlight, focus on, and allow selection of keywords on the page. The prototype demonstrated tabbing through keyword links on the page, switching panes, and navigating through content. The prototype test demonstrated the feasibility of basic Renderer functionality and provided confidence that customized browsing, as well as keyword detection, highlighting, and selection could be implemented in our system on a general level.

33

– Implementation and Integration

4.2.3

RPCS – Spring 2006

Software Architecture

The search software is split into three main sections as shown in Figure 9. These sections are the web browser, local web server and the database.

Figure 9: Software Architecture Web Browser: This is a graphical user interface developed in XML user interface language (XUL). This is the part of the system that the user interfaces with. It was custom designed to handle input from the click wheel mouse however it will work with any standard input as well. Local Web Server: The local web server is composed of many different components. 1. Controller: Handles user login and sessions. Also parses HTTP requests from the browser and forwards them to the appropriate modules. 2. Renderer: Dynamically builds the pages in both manuals by adding in the navigation, panels, keyword links and annotations that are specific to the MoRÉ system. 3. Indexer: parses both manuals (XML and HTML) and populates the database with

34

– Implementation and Integration

4. 5. 6. 7. 8.

RPCS – Spring 2006

relevant information about the keywords and pages. It is highly configurable thus allowing for the constraints that determine keywords to be adapted to any system. Annotation Engine: Deals with playing and storing voice annotations (see section 4.3 Capturing Experience for more details) Proxy: Can be used with conjunction with a central server in order to facilitate seamless transfer of files to the client. Capturing Experience Module: Tracks usage of the system and users’ experiences. (See section 4.3 Capturing Experience for more details) Search Engine: Allows searching in both manuals. The engine pushes all the logic unto the database to improve speed. Database Access Controller: Provides a wrapper class to the database access to eliminate long quoted SQL queries in code. Currently configured for use with either CVS or MySQL databases but easily adapted to different database types.

Database: A MySQL database which stores all keywords, annotations, smart links, and any other relevant information.

4.2.4

Software Modules and Status

4.2.4.1 Indexer: Implementation of the indexer is complete and it has been extensively tested. Special care has been taken to make it highly configurable. It is very easy to specify what tags are important and to specify actions to be taken for those tags. It can parse XML as well as HTML. Titles, words in JavaScript, words in bold, etc., are extracted from the manuals. Positions of words within the page are recorded as well. After the detailed architecture of the entire system was designed, the work on indexer began. Exact details of what data is needed for search engine, was analyzed and accordingly the indexer were designed. Care was taken to ensure that data from the manuals is represented at a sufficient level of abstraction without overwhelming the other components with excessive metadata. The cases of a word being: - in the title of the page - in bold - in larger font are all considered equal and represented in the database using a single column so that other components depending on the indexer find it easier to extract the relative importance of the word for that page. On the other hand, a word being in JavaScript (this usually happens when point-by-point instructions are given) is represented separately. The Capturing Experience group wanted a separate titles table and hence the indexer fills that table in as well. The well-defined database schema acted as an effective interface between various components of the Search group as well as the Capturing Experience group.

35

– Implementation and Integration

RPCS – Spring 2006

4.2.4.2 Search Engine: The search engine has been completely implemented and thoroughly tested. It has been extensively tweaked to improve the accuracy of the results. Various configurable parameters were fine tuned until a near-optimal configuration was obtained. It takes into consideration a number of factors before deciding on the rank of a page for a given word. The factors considered are as follows: - Word count i.e. how many times the word occurs on the page - Title/bold/larger font i.e. a higher weight is given if the word occurs in bold or is in the title of the page or has a larger font - Higher weight if the word occurs in JavaScript - For multiple word queries, proximity is considered Separate considerations are given depending on whether the search query/string is a single word query or a multiple word query. The search logic has been carefully optimized for single word queries and they run significantly faster due to ease of query generation and use of precompiled statements. In case of multiple word searches, proximity of the words has been considered as an important metric. If the words occur closer in a certain page, then that page receives a higher weight. All these metrics are finally combined using weighted addition to compute the rank of pages and the result returned in decreasing order of rank. In order to satisfy space constraints, creation of new tables was avoided. To increase speed of execution, all the operations are pushed to the database. Each search operation results in only one query being fired on the database and the results returned. Hence, no extra computations are to be done in the program itself. Different groups needed different number of results (Search group needed 10 results while Capturing Experience preferred 3 results). Hence provisions had to be done to make the number of results to be returned as a parameter which can be set prior to the invocation of search operation. Alternatively, appropriate number of results can be chosen from the list returned. The search engine has been made flexible in that regard. 4.2.4.3

Proxy:

The proxy has been divided into three broad parts: - Fetching a file from local server, if present, else fetching it from the remote server - Taking care of automatic updates as well as forced updates - Storing recorded annotations on the main server File fetching has been done. The module checks if the file is present locally. If it is unable to find it locally, it fetches it from the centralized server. This fetching is particularly useful when the user clicks on an annotation and a sound file has to be played.

36

– Implementation and Integration

RPCS – Spring 2006

Automatic updates are of two types: fetching of updated pages of manuals and fetching annotations. Manual page fetch is typically done on a daily basis. Annotation fetching is done every hour. The times were chosen depending on the size of data that may have to be transferred and the frequency of updates. They are configurable, however. Forced updates will fetch immediately without waiting for the hourly updates. Binary log comparisons are used to minimize bandwidth used when transferring updates. Implementation of updates is complete, but hasn’t been integrated with the rest of the system for want of time. However, considering the clear interface designed, it is expected to be fairly straightforward. Currently, the updates have to be run manually and the client database synchronized with the central database. Storage of annotations has been done in close association with the Capturing Experience group since they were responsible for recording, storing and naming the annotations. Interfaces in terms of the database schema were defined to make the two activities relatively independent, and that proved to be a good idea.

4.2.4.4 Renderer: The renderer is responsible for dynamically generating pages. Depending on the user request, certain parts of the page may be changed. For example, a query to search for a keyword may result in only the search pane getting modified. Thus, the renderer has to rebuild the page and feed it to the browser for display. The renderer adds links to both the manuals and avoids adding any code in HTML tags or CSS or JavaScript. The renderer displays three panes on the right side of the page: -

Navigation pane Link pane Annotations pane

The navigation pane allows the user to close a window and indicate whether a page was helpful in solving the problem. This input is fed to the Capturing Experience group’s module. The link pane displays the search results as well as the static and dynamic smart links. The annotations pane allows the user to record annotations. The renderer modifies the resolution of the content so that objects and pages can be easily viewed on the HUD. The images in the manuals have a resolution which is fit for a desktop computer. Hence, this had to be explicitly taken care of. Since the HTML of the manuals is horrible, a prototype was developed (using Cocoa) and depending on the lessons learned, the final design was decided. The renderer is a central component which calls the methods of other classes, fetches all the data it needs and dynamically builds the pages.

37

– Implementation and Integration

RPCS – Spring 2006

The renderer has to interface extensively with Capturing Experience group’s modules. For example, the smart links generated by CE will be fetched by the renderer and it has to take actions if a user is found to be cycling. A clean interface was designed between the renderer and CE modules. 4.2.4.5 Controller: Controller distinguishes between different user actions like ‘search for a word’, ‘play an annotation’, ‘login’ etc., and routes the request to the appropriate module. It allows users to login and stores the session information in a Java bean so that the information can be retrieved at any time. The controller has wrapper classes for databases tables and queries. This makes developers’ life easier by not having to worry about connection establishment or other related issues. The controller also sends the request to the Capturing Experience Black box so that every request is reflected in history of pages accessed and cycling detection and similar other functions can be performed.

4.2.4.6

Customized browser:

Due to unconventional requirements, a customized browser was needed. It takes input from renderer and displays the page on screen. Also, it is also responsible for accepting user inputs and passing them on to the controller to handle them appropriately. The user inputs come from the input dial. The browser supports custom navigation like switching between panes and tabs. The browser was created using XUL (XML User Interface Language) with the Mozilla browser. The browser takes care that the rendered pages are displayed properly. Though the browser was designed to work with the dial, it works with any standard input device (mouse, keyboard, etc.) The browser is easy to install and runs on virtually any machine. Collaboration with the Platform group was needed for the browser since inputs from the dial had to be received. The signals sent on a particular button press were fixed and the hence integration went on smoothly.

4.3 4.3.1

Experience Capture Functionality

This section details the functional level of Capturing Experience Components. More implementation details are available in the Software Modules section. The goal of Capturing Experience Group is to capture the experience of aircraft maintenance workers to assist them as well as to develop tools that will guide and help users as they use the Procedural and Training Manuals. The Capturing Experiencing 38

– Implementation and Integration

RPCS – Spring 2006

Group came up with 6 specific components that will be beneficial to the users. The 5 components are Annotations, Cycling Detection, Expert Contact Information, Static and Dynamic Smart Links, and User Login. Annotations are commentaries that users will input for a manual page. Users using voice recording manually input annotations. Along with the actual recording, the user name, timestamp of the annotation will be displayed on the page. The recording is saved compressed in GSM format, because a raw wave would be too large. After compression, it is stored in a database. Another module known as the renderer will display the newly formed web page, which will include the annotation. This information will be sent to the central server eventually so other users may see it in their browser. Annotations can be of variable length; the user can start and stop recording at anytime. There is, however, a maximum file size of 5MB so that the user does not make ridiculously long annotations. The Cycling Detection module identifies when a user is cycling between pages. The definition of cycling is 3 instances of one URL and 2 instances of a different URL. For example, these two examples below would be identified as cycling: (1) Page1, Page2, Page1, Page2, Page1 (2) Page1, Page2, Page3, Page2, Page1, Page2 Once a user is identified as cycling between two pages, those two pages are stored in a temporary data structure until the user is finished with what he/she is looking for. When the user clicks on the “solved my problem” link, the pages that the user cycled between along with the page that solved his/her problem will be stored into the local database. Then future users will then be given the stored answer link when and only when the user is cycling between the appropriate pages. The Expert Contact Information module was designed to provide a list of other users that a user can contact for help if their question cannot be answered by the manuals. Since there are security issues involving the actual names of the aircraft maintenance, fake names were substituted for real users on the login page. However, our client gave us some titles that would be realistic and usable. The user name and title will first appear at a login page, which will be the first thing displayed when a user desires to open a manual up. Once the user chooses their name, their information will be logged throughout a session. Along with having their names and titles on the login page, their information may appear on certain pages in which they are experts. With their information, background and contact information, other users will know who and how to contact someone if they need help. Dynamic Smart Links are dynamically generated links that will help users find what they are looking for. It learns from other users, and is semi-automatic. A path is determined as the URLs the user visits starting from the first page of the procedural manual until they have found what they are looking for. Users specify when they find what they’re looking for by pressing the “Solved my problem” link on the page. The paths of future users will be compared to the paths from previous ones. The future users will be given links to potential solutions, which will be displayed on the right pane. Through User Login, modules will know which using is looking through the manual, and be able to record the

39

– Implementation and Integration

RPCS – Spring 2006

appropriate information. This is also important so that the identity of the user who records an annotation is known. To login, users will first scroll through a displayed list until they find their category. This makes another column appear with user names. The user once again scrolls until they have found their name. Static Smart Links are links that will always be on a page that will never change (until the manual version is updated). They are links that correlate URLs from the procedural manual to the training manual. These links are automatically generated at index time using the titles and headings of each page, which are stored in a database in the local server. Considering that there could be many results for a certain page, the best results are displayed on procedures manual pages. For now we limited the links to top 3 results from the search index. Results are based on number of hits per page.

4.3.2

User Studies Results

Due to the entire system not being complete until the day before the presentation, most modules did not have any user testing done. However, here included are some proposed methods of user testing for future purposes. For annotations, a user should try to record an annotation. On the customized browser, an important criterion would be to see if they could find and use the record link. A user would determine whether the recording process had any difficulties and whether the annotation played the recording. The user would also determine if the annotation was helpful or irrelevant to the page also to see the amount of annotations recorded over a period of time. These tests would be run over a variety of users ranging from experts and novices in the system. For cycling detection, users did not necessarily consider the sequence they went through as cycling between two pages. The users tried out different definitions of cycling to see if the definition of cycling matched to their expectations, however this was not fully resolved. The definition that had the most hits of what a user defined as cycling is if a user has been to a page 3 times and another page 2 times in a sequence of 10 URLs. However, this user study was only done with 2 people that are novices of the manuals. With more user testing, a more accurate depiction would be met. Another idea that was suggested was to change the name of the module to “Hunting Detection” to determine when a user is trying to find certain information as compared to determining if they are lost in the system. For expert contact information, user testing can be conducted by having users browse the manual to see if the contact information for each page was relevant to what was on the page. For dynamic Smart Links, users were not able to determine whether a Smart Link was relevant to what they were looking for until they selected them, especially since link titles were often the same among two or three links. Link titles were extracted from the pages that the links refer to. So if multiple pages had the same title, which is common in pages in the training manual, links to those pages had the same title as well.

40

– Implementation and Integration

RPCS – Spring 2006

For static Smart Links, user testing can be conducted using subjects who are intimately knowledgeable with both manuals so that they can identify static links that draw strong relations from those ones that do not. Furthermore, users can replace those links that draw relations with the ones that do and attempt to fix them in the database.

4.3.3

Experimental Measurements

Due to the entire system not being complete until the day before the presentation, most of the components of capturing experience were not tested with the complete system. However, if testing was done on the components in Phase III, they were usually done apart from the complete system. For displaying and playing annotations, there were a number of things to be sensitive to. On the displaying side, it was necessary to be sure that a page displayed annotations only if a page had annotations. A quick check on the main pane of the browser was required initially to also make sure that the format: the link, the user id, and timestamp, was being properly displayed in a page with an annotation. To check the correctness and usability of playing annotations, it was necessary to check that the correct annotation was played when the annotation link was clicked, and that it was possible to close the Windows Media Player window once playing had finished. It was also necessary to ensure that the system was configured so that the player would open without any intermediate (such as whether to open the file or save it) messages, and that the volume is adjusted properly. See the “Software Modules and Status” section on annotations for more details on these configurations. In terms of recording annotations, it was important to check that recording started and stopped when the user instructed the system to, and that the file lengths reflected that. It was also necessary to check that the filename was correct, the file’s path was correct, and the user and timestamp, which were both used in the filename, were accurately saved. Lastly, there was a check to see that the file ended up in the directory it should have, and that the proper fields were updated in the database. Cycling Detection module was tested extensively using different test cases. Before the renderer was fully functional a simple main function was implemented. In the main function test cases were used to see if the cycling detection was correctly implemented. Considering the input into the cycling detection module was the URL as a string type, strings were used in the main function. These were the test cases ran for cycling detection: 1.) A.com, B.com, C.com, B.com, A.com, B.com, C.com, D.com, F.com - To see if the module detected that the user is cycling between B.com and A.com and if B.com, A.com and F.com was stored in the “cyclingdetection” table in the database 2.) A.com, B.com, C.com, B.com, A.com, B.com, C.com, D.com, F.com (4 times) - To see if the rank of F.com incremented 3.) A.com, B.com, C.com, B.com, A.com, B.com, C.com, D.com, G.com - To see if G.com was stored in the same row as the previous cycled pages (B.com, A.com) under a new column in the “cyclingdetection” table in the database and not in a new row

41

– Implementation and Integration

RPCS – Spring 2006

4.) A.com, B.com, C.com, B.com, A.com, B.com, C.com, D.com, G.com (3 times) - To see if the rank of G.com incremented 5.) A.com, B.com, C.com, B.com, A.com, B.com, C.com, D.com, H.com - To see if G.com was stored in the same row as the previous cycled pages (B.com, A.com) under a new column in the “cyclingdetection” table in the database and not in a new row 6.) A.com, B.com, C.com, B.com, A.com, B.com, C.com, D.com, H.com (5 times) - To see if the rank of H.com incremented 7.) A.com, B.com, C.com, B.com, A.com, B.com, C.com, D.com, Evict.com - To see if the lowest ranked Smart Link was deleted and Evict.com was added in its place 8.) J.com, K.com, L.com, M.com, N.com, D.com, B.com, J.com, K.com, J.com - To see if the queue evicts J.com and K.com because the sequence is 12 deep 9.) J.com, K.com, L.com, M.com, L.com, M.com, O.com, M.com, Z.com - To see if a new row was created with M.com, L.com and Z.com stored in the database The same sequences were performed when the integrated system was functional enough for cycling detection to be tested. The cycling detection module completed each test successfully. In order to test the dynamic Smart Links, the users simply browsed through the manual while viewing various topics. The URLs of the pages viewed were tracked manually, and the problem-solved link was selected at various points. Since the data is stored in the local database, the database records were verified against the expected data, including the weights of edges between pages. Pages in the manuals were viewed again to verify that the expected Smart Links were displayed. The local web server was also reloaded to test that data persisted between uses. After the initial testing and debugging phase, all the tested features behaved as expected. Expert Contact Information was not tested extensively due to the fact that it had dependencies with many other components. However, these would have been the approaches taken for testing. Having people test out the annotation module would have tested the displaying of his/her name with a user’s annotations. Along with this, the actual titles of users would have been tested to see if they match and correlate with the page. The testing of the static Smart Links is fairly straightforward since it depends solely on accessing the database and appropriately using the search function. Links are stored in the database beforehand using the search function. When static Smart Links are needed from a page, a simple lookup is performed to return the page ids and their respective URLs. This process of accessing the pages has been successfully tested since it only involves wrapper functions to connect and retrieve data from the database. A series of previously entered data were pre-populated in the database and retrieved successfully using the static link module. These links then serve as output of the capture experience module, which is then passed to the renderer.

42

– Implementation and Integration

RPCS – Spring 2006

The pre-population of the database through extensive search of all pages has not been tested. This is mainly due to the tight schedule of integration where certain aspects of the software system were given more priority due their immediate need during the demonstration. Populating the database using the StaticLinkUpdater function would allow the static smart link module to function in a more complete manner.

4.3.4

Software Architecture

In the capturing experience portion of the software architecture, there are six main components: the audio annotation maker, the Cycling Detection sub module, the static Smart Links generator, the dynamic Smart Links generator, and the User Login. These components are contained within the capturing experience module of the client web server. The only exception is the user login, which is handled at the top-level entry point upon receiving requests from the user.

Page Request Handler New Annotation? Yes

No

History Tracking Annotations

Cycling Detection

Dynamic Smart Link Generation

Smart Links Synthesizer

Static Smart Links

Renderer Figure 10: Diagram of Software Architecture for Capturing Experience

43

– Implementation and Integration

RPCS – Spring 2006

When the user attempts to view a page in one of the manuals, the web browser sends an HTTP Get request to the web server on the client. Upon receiving this page request, it notifies the capturing experience module. This page is then stored in an internal data structure to track the history of page views of the user. Both the cycling detection sub module and the dynamic Smart Link sub module use the resulting data structure. The cycling detection sub module uses its heuristics to determine if the user is cycling based on the recent history before passing on any additional links as help. The dynamic Smart Link generator uses the history and its own processing to determine related links based on past usage. The receipt of the message that the user has solved his problem is merely a special case of a page request since all requests are sent through the web browser. The resulting dynamic Smart Links are intelligently combined with the statically generated Smart Links stored in a database and any additional help from the cycling detection module. The result is then passed on to the renderer for display. The audio annotation maker is also part of the capturing experience module. Based on the request sent from the browser, the capturing experience module detects whether the user is attempting to record a new annotation. The custom browser interacts with the user directly via a side pane. When a request for a new annotation is made, the page request handler dispatches to the audio annotation maker, which inserts the annotation into the database for retrieval by the renderer. The login page is a special page that, when viewed by the user, creates a cookie which is sent along with each subsequent request to the web server. Data is stored along with the annotations when they are created to allow the renderer to display which user created each annotation. The Static Smart Links database is populated by the static Smart Links generator. Before being deployed for the first time or whenever the pages of the manuals are updated, this sub module is executed once after the search indexer is run.

4.3.5

Software Modules and Status

As discussed in the functionality section, the Capturing Experience Group has 5 main modules. The 6 components are Annotations, Cycling Detection, Expert Contact Information/User Login and Static and Dynamic Smart Links. 4.3.5.1 Annotations 4.3.5.1.1 Displaying and Playing Annotations When a user arrives at a manual page, the renderer calls annotation methods to query the database and return information about the annotations on a page using the page id, which is generated for every page in the manual. If there are no annotations, nothing is returned, and the page will not appear differently to the user. If there are annotations, the annotation method returns the user who recorded it, the time it was recorded, the path and filename. This information is displayed at the bottom of the main page as a bulleted html list, ordered with the most recent annotation shown first. The word “Annotation”, in an example below, is an html link which links directly to where the annotation file is stored on the web server.

44

– Implementation and Integration

RPCS – Spring 2006

Figure 11: Screenshot of a Training Manual Page with Annotations When the annotation link is clicked in the Moré system, the annotation plays in Windows Media Player. Windows Media Player can then be closed by using the joystick and click capabilities on the dial to navigate to the close button in the top right corner of the Windows Media Player. While the above describes how it is played in the current system, this could also be configured in different ways. For example, the installationdefault setting for the browser when clicking a link to an audio file is to ask whether to open it or save it to disk. This was configured to always open without asking. Another way that playing could be done differently is to play the file in another player. To do this, the other player, such as Winamp, would have to be installed, and then configured such that it is the default player. As of the final demo, displaying and playing worked properly. They were included in the final demo. 4.3.5.1.2 Recording Annotations A major change that occurred in Phase III was how recording would be started and stopped. In Phase II, a Java GUI button was used. When the button was pressed, methods would be called to start or stop the threads that did recording as appropriate. In Phase III, this was revised into a link in the annotation pane of the browser window, because of the difficulty and complexity of integrating this Java button version into Moré. When the link is pressed, the Capturing Experience controller calls the Java annotation recording modules, which spawns a new thread to start recording. When the link in the browser is pressed again, the thread is interrupted and recording stops. This gives the user the ability to control the length of their annotation. There is also a maximum file size of 5MB to both prevent overly long recordings, and as a safeguard to have recording

45

– Implementation and Integration

RPCS – Spring 2006

stop by itself in case the user forgets to stop it. Information about the annotation, most notably—the page id, the user id, the time it was recorded, the path and filename, are saved into to the database. The annotation itself is compressed and saved into a .GSM format, which in tests reduced the file to about 10 times that of its original raw wave format. Future work in a system with multiple users would include functionality to upload these annotations to a central server on an interval basis, so that they can be shared with other users. The non-integrated version of the annotation modules worked as expected on a Moré student’s home machine. However, the desktop machine in the lab that was being used (and that had the configured web server and database) for integration could not support the recording Java code because it did not have audio input devices required by internal libraries. This was likely because the machine did not have a sound card, or other devices that supported similar functionality. Thus, it was not possible to test integrated recording modules on the machine that was used for integration. When the web server, database, and the class files were migrated to the Vaio for final testing, the recording module unfortunately did not work reliably. In some installations it worked properly, recording from the microphone in the headset for a user-controlled time, then compressing, naming, and saving the module properly and storing appropriate information in the database. However, there were also times when it did not work, halting in the section of the code where it searches for recording devices. At the end, the reason for this was uncertain, as the device configuration used in the Java code matched one that the Vaio did support, according to a tool from Sun, which was tested on the Vaio. One reason suggested was that since the code was being compiled on the machine that did not support the recording modules, which were then copied onto the Vaio, this may have caused corruption in the class files. There was not a member knowledgeable about how Java compiles and links to clarify whether this was possible or not. 4.3.5.2 Cycling Detection The Cycling Detection module is fully functional. It uses the recent history of pages viewed by the user to attempt to determine when the user is confused or needs additional help. It uses heuristics that recognize certain patterns in the recent history. The definition of cycling as of now is whenever a user has a sequence that includes three instances of one page and two instances of another. This is a good but not completely accurate indication that the user is cycling or switching between viewing the two pages. The module was implemented in a manner that easily allows modifications to the definition of cycling. A concern that came up was in the scenario that a user has a long sequence of URLs he/she visited. To avoid generating a false positive of cycling in this case, only a small constant number of pages, such as the last ten viewed, are taken into consideration. Furthermore, the history is not stored between different user sessions. The module uses a queue to store URLs that the user has gone to. The size of the queue is currently 10, but may be increased or decreased depending on feedback from future user testing. Before the module adds a URL into the queue, it checks the URL to see if there are instances of itself in the queue. If there are 2 instances of it, then it checks to see if a

46

– Implementation and Integration

RPCS – Spring 2006

second URL, which is different from the first, is already stored in the queue. The idea behind this is to see when the user has gone between two pages a couple of times. Here is a diagram that shows this process:

Input: URL2

Sees the user has already gone to this URL twice before

URL1

0

URL2

1

URL3

2

URL2

3

URL1

4

Input: URL2

Sees the user has gone to another URL twice

URL1

0

URL2

1

URL3

2

URL2

3

URL1

4

5

5

6

6

7

7

8

8

9

9

Figure 12: Diagram of How Cycling Detection stores the URLs Once a user has cycled between two pages, those two pages are temporarily stored in another data structure until the user presses the “solved my problem” link. When that is clicked, all the URLs in the data structure along with the URL the user was on when he/she clicked on the “solved my problem” link are stored in a table named “cyclingdetection” in a local database. The URL the user was on is stored under the Smart Link column. Every different page that a user has cycled between is stored in a new row in the database. There is a limit of three Smart Links per cycled page. Each Smart Link has a rank which will determine which one will be replaced if a new Smart Link is added when there are already three Smart Links stored for given cycled pages. Here is an example of what happens internally when a user browses through certain sequences of pages in the training and procedural manuals. Sequences a user went through: - A.com, B.com, C.com, B.com, A.com, B.com, H.com - A.com, B.com, C.com, B.com, A.com, B.com, G.com - A.com, B.com, C.com, B.com, A.com, B.com, K.com - A.com, B.com, C.com, B.com, A.com, B.com, H.com - A.com, B.com, C.com, B.com, A.com, B.com, G.com - A.com, B.com, C.com, B.com, A.com, B.com, Z.com In the first sequence, A.com and B.com are stored in a row under column name “cycledpage1” and “cycledpage2”. Also H.com is stored as a Smart Link under column name “smartlink1” and the rank is stored as 1 under column name “rank1”.

47

– Implementation and Integration

RPCS – Spring 2006

In the second sequence, G.com is stored under the column name “smartlink1” and the rank is stored as 1 under column name “rank2”. In the third sequence, K.com is stored under the column name “smartlink1” and the rank is stored as 1 under column name “rank3” In the fourth sequence, rank under column name “rank1” will be incremented to 2. In the fifth sequence, rank under column name “rank2” will be incremented to 2. In the sixth sequence, K.com is replaced with Z.com because it has the lowest rank and the rank will be stored as 1 under column name “rank3”. As the user cycles through the same pages as the ones in the database, the appropriate Smart Link will display in the right pane under the search results with the other Smart Links. 4.3.5.3 Dynamic Smart Links The dynamic Smart Links generation module is fully functional. It stores its internal graph data structure – the minimum amount of data needed to generate Smart Links – into the local database. Whenever a problem is solved and the graph is updated, the database is also updated. Upon instantiation, the data structure is loaded from the database. So if synchronizing between the local database and the central server database is implemented in the future, Smart Links will be shared seamlessly between users.

Figure 13: Screenshot of a Training Manual Page with Smart Links

48

– Implementation and Integration

RPCS – Spring 2006

4.3.5.4 Expert Contact Information This module mainly consists of getting information from the client about what information is useful to have, and making it available in the system. Due to security reasons, the client could not provide actual user data. Consequently imaginary users were created. The data used are: user First and Last names, Titles, years in service and contact information. All this information is stored in the database to be used by the browser. Also, a login page was created using html which would be the first page displayed for the user. He/she would need to choose their name to continue on to the manuals. 4.3.5.5 Static Smart Links The static smart links are intended to provide the users with alternative suggestions of possible related pages in the other manual. This comes especially in handy when the dynamic link module cannot provide the user with suggestions due to lack of training on the module. Applying a search function on the current page of the manual produces the static smart links. In particular, the title of the current page is used as input for the search; the search function then returns a series of links based on these indexed titles. These links would point the user to related pages in the other manual. This process of producing static smart links is done offline, through an updater that would index through all the pages and use their titles to perform search functions. The search would return related page ids and three of them will be converted into their respective URLs and stored into the database. Thus, within the database, there will be three URLs (generated by search) correspond to one URL. After the database is populated with correlations between the manuals, static link look up can be carried out in an online fashion when the user is running the More System. A simple lookup of the current page id is performed in the database, which would in turn retrieve up to three related links. These links are then ordered in an array structure and used as output for the static link module. The implementation of this module is complete and integrated with the rest of the system.

4.4 4.4.1

Platform Functionality

The functionality of the mobile computing unit is to exist as a hardware unit, capable of delivering the manuals and experience captured to the user. The unit must also support gathering spoken input annotations to be added to the manuals, user input to navigate the pages of the manuals, display the computing units screen to a head-mounted display, and to allow the user to hear spoken annotations added to the manuals. This high-level functionality was achieved using a Sony Vaio mobile computing unit, a Liteye HMD for display, a David Clark headset and earphones and a custom-made dial input device. The Dial is intended to be a pure input device that allows the user to control the mobile

49

– Implementation and Integration

RPCS – Spring 2006

computer while in many different orientations and possibly wearing gloves. Functionality for the Dial will include the ability to left click, scroll through menus and links on a page, navigate the screen with a joystick, move through tabs, create tabs, and switch page frames. The mobile platform that MoRÉ runs on is a fully-functional computing unit, with primary and secondary input methods, displays, and audio outputs. The Sony Vaio mobile computer runs the Windows XP Professional operating system and can run any ordinary Windows application. The platform can take audio recordings via a microphone attached to the composite Head-Mounted Display / David Clark Headset or a secondary microphone in the USB audio adapter, and playback recordings through the David Clark headset or a speaker internal to the Sony Vaio. The platform can visually display the MoRÉ application through the primary Head-Mounted Display on an 800x600 SVGA screen, or also display on the included LCD attached to the Sony Vaio at 640x480 VGA, 800x600 SVGA (native), and 1024x768 XGA resolutions. The platform can also receive input primarily through the custom Dial, as well as via a touch-screen, buttons and a virtual keyboard on the Vaio itself.

4.4.2

Component Specifications and Pictures

4.4.2.1 Dial The Dial electronics design is based on a Freescale Semiconductor MC908JB16DWE microcontroller with the following features: • • • • • • •

• •

• • • •

High-performance M68HC08 architecture Fully upward-compatible object code with M6805, M146805, and M68HC05 families Low-power design; fully static with stop and wait modes 6-MHz internal bus frequency 16,384 bytes of on-chip FLASH memory with security feature 384 bytes of on-chip random access memory (RAM) Up to 21 general-purpose input/output (I/O) pins, including: o 15 shared-function I/O pins o 8-bit keyboard interrupt port o 10mA high current drive for PS/2 connection on 2 pins (with USB module disabled) o 1 dedicated I/O pin, with 25mA direct drive for infrared LED (32-pin package) o 6 dedicated I/O pins, with 25mA direct drive for infrared LED on 2 pins and 10mA direct drive for normal LED on 4 pins (28-pin package) Two 16-bit, 2-channel timer interface modules (TIM1 and TIM2) with selectable input capture, output compare, PWM capability on each channel, and external clock input option (TCLK) Universal Serial Bus specification 2.0 low-speed functions: o 1.5Mbps data rate o On-chip 3.3V regulator o Endpoint 0 with 8-byte transmit buffer and 8-byte receive buffer o Endpoint 1 with 8-byte transmit buffer o Endpoint 2 with 8-byte transmit buffer and 8-byte receive buffer Serial communications interface module (SCI) Dual clock generator modules (CGM) (32-pin package) In-circuit programming capability using USB communication or standard serial link on PTA0 pin System protection features: o Optional computer operating properly (COP) reset o Optional Low-voltage detection with reset o Illegal op-code detection with reset

50

– Implementation and Integration

• • • • •

RPCS – Spring 2006

o Illegal address detection with reset Master reset pin with internal pull-up and power-on reset IRQ interrupt pin with internal pull-up and Schmitt-trigger input 32-pin low-profile quad flat pack (LQFP) and 28-pin small outline integrated circuit package (SOIC) Features of the CPU08 include the following: o Enhanced HC05 programming model o Extensive loop control functions o 16 addressing modes (eight more than the HC05) o 16-bit index register and stack pointer o Memory-to-memory data transfers o Fast 8 × 8 multiply instruction o Fast 16/8 divide instruction o Binary-coded decimal (BCD) instructions o Optimization for controller applications o Third party C language support

The MC908JB16DWE microcontroller is a 28-pin SOIC surface-mount device and is clocked via 12MHz microprocessor crystal. The MC908JB16DWE microcontroller supports in circuit programming (ICP) through its USB interface port and is powered from +5VDC direct from host USB interface. The MC908JB16DWE microcontroller performs all sensing, control and supervisory and communications functions. Pin interfaces and functions are contained in the Appendix. (USB & SCI Communications Ports) Two communications port interfaces are provided to the microcontroller; Universal Serial Bus (USB) and non-buffered RS-232 Serial Communications Interfaces (SCI). Both USB and SCI communications ports are available on connectors J2 and J1 respectively with pinouts, as described in the Appendix. (RKJXM Rotary Encoder/8-Position Joystick Switch) A specialized multi-purpose Rotary Encoder / Joystick component (RKJXM Series manufactured by ALPS) is utilized to provide mechanical detente and encoding of rotational dial, and 8-position joystick with pushbutton. The RKJXM is a low cost rotaryencoder/joystick with quadrature output at 15 pulses per revolution and 8-position momentary joystick with pushbutton. The RKJXM is a through-hole PCB mountable device and is based on mechanical switch closure. Filter and conditioning circuitry is required to effectively eliminate contact-bounce and EMI/RFI pickup. All switch and encoder signals are buffered from EMI/RFI injection and contact bounce using discrete low-pass RC filter elements. (DHC3 Series Pushbutton: Left, Right and Middle User Input) The Dial device includes three user pushbutton inputs that may be programmed for any standard mouse or keyboard function. DHC3 Snap-Action pushbuttons manufactured by Cherry Switch Corporation are utilized for this function. The DHC3 pushbuttons are SPDT contacts with only the normally open contact used. When a switch is depressed, a low-level appears at the corresponding microcontroller pin. In non-depressed state, a high level appears at the corresponding microcontroller pin via internal or external pull-up

51

– Implementation and Integration

RPCS – Spring 2006

resistor. All switch and encoder signals are buffered from EMI/RFI injection and contact bounce using discrete low-pass RC filter elements. (STAT1 & STAT2 LED User Supervisory Indicators) Two Super-Bright user supervisory LED indicators are implemented on the Dial PCB. These LED’s are intended for direct user output device for indication of various device status or feedback functions. The specific function of each LED is determined via microcontroller programming. One blue and one red LED are included on the component-side of the Dial PCB, the local of these LED’s facilitates user viewing through transparent mechanical housing about the middle user pushbutton location. The red and blue LED’s are individually controller via microcontroller pins PTD0 and PTD1 pins respectively. Microcontroller pins PTD0 and PTD1 must be driven low in output LED-drive configuration mode to turn on LED’s. See Microcontroller Pin Function & Description table for additional operational information. (Microcontroller Reset) A PCB mounted momentary pushbutton switch is implemented to allow user-initiated microcontroller reset function during device programming development. This reset pushbutton is not intended to be used during normal Dial field operation. Features • Powerful MC908JB16DWE microcontroller • Miniature 80% surface mount PCB implementation • Very low-cost implementation • Intrinsic Safety elements included on power and serial communications interfaces • High-degree of manufacturability • Simple & reliable design • Powered via Host USB port • Low-power Fully programmable user input/output device

52

– Implementation and Integration

RPCS – Spring 2006

Figure 14: Dial Final Product 4.4.2.2 Vaio Ruggedized Enclosure The Enclosure was meant to house the processing unit in a safe environment from heat, poor weather, dust, and any other interactions with the physical environment. It was implemented in less than 0.1 inch thick aluminum, measuring 7.75 x 6.38 x 2.38 cubic inches with a foam padded interior to protect the Vaio and allow heat to be dispersed from the unit. In addition, the enclosure was designed to fit entirely inside the carrying holster, as shown in Section 4.4.5.

53

– Implementation and Integration

RPCS – Spring 2006

Figure 15: Vaio Ruggedized Casing 4.4.2.2 Hot-Swap Controller The Hot-Swap Controller (HSC) is based on the non-standard Sony Li-Ion Smart Battery Pack. At the heart of the HSC design is a Freescale MC68HC908SR12 microcontroller. The microcontroller monitors the standard Serial Module Bus (SMB) communications interface between the Sony Vaio U PC and the smart battery pack to determine battery state and charge capacity. Solid-state power and signal switches are implemented on the HSC that isolate or connect the PC to selected battery pack under microcontroller command and supervisory. The HSC may then make intelligent decisions about battery status and switch between pack/HSC assemblies as appropriate. The HSC also includes circuit implementations to independently monitor battery pack current and cell voltage, in addition provisions for shared internal power bus between HSC devices and interrupt request bus to synchronously communicate events between devices. The HSC also included specialized power switching circuitry based on a dual-MOSFET switch and analog amplifier that limits current during pack switching. (MC68HC908SR12 Microcontroller) The Hot-Swap Controller (HSC) electronics design is based on a Freescale Semiconductor MC68HC908SR12 microcontroller with the following features: • • • •

• • • • • • • • • •

High-performance M68HC08 architecture Fully upward-compatible object code with M6805, M146805, and M68HC05 Families Maximum internal bus frequency: o 8-MHz at 5V operating voltage o 4-MHz at 3V operating voltage Clock input options: o RC-oscillator o 32kHz crystal-oscillator with 32MHz internal phase-lock-loop 12k-bytes user program FLASH memory with security1 feature 512 bytes of on-chip RAM Two 16-bit, 2-channel timer interface modules (TIM1 and TIM2) with selectable input capture, output compare, and PWM capability on each channel Timebase module 3-channel, 8-bit high speed PWM (125kHz) with independent counters and automatic phase control Serial communications interface module (SCI) System Management Bus (SMBus), version 1.0/1.1 (Multi-master IIC bus) 14-channel, 10-bit analog-to-digital converter (ADC), with auto-scan mode for 4 channels Current sensor with programmable amplifier Temperature sensor (–20°C to +70°C)

54

– Implementation and Integration

• • • • • • •

• •

RPCS – Spring 2006

IRQ1 external interrupt pin with integrated pullup IRQ2 external interrupt pin with programmable pullup 8-bit keyboard wakeup port with integrated pullup 31 general-purpose input/output (I/O) pins and 2 dedicated pins: o 31 shared-function I/O pins o Two dedicated analog input pins Low-power design (fully static with Stop and Wait modes) Master reset pin (with integrated pullup) and power-on reset System protection features o Optional computer operating properly (COP) reset o Low-voltage detection with optional reset o Illegal opcode detection with reset o Illegal address detection with reset 48-pin low quad flat pack (LQFP) and 42-pin shrink dual-in-line package (SDIP) Features of the CPU08 include the following: o Enhanced HC05 programming model o Extensive loop control functions o 16 addressing modes (eight more than the HC05) o 16-bit Index register and stack pointer o Memory-to-memory data transfers o Fast 8 × 8 multiply instruction o Fast 16/8 divide instruction o Binary-coded decimal (BCD) instructions o Optimization for controller applications o Efficient C language support

The MC68HC908SR12 microcontroller is a 48-pin low quad flat pack (LQFP) surfacemount device and is clocked via 32KHz microprocessor crystal. The MC68HC908SR12 microcontroller supports in circuit programming (ICP) through its SCI interface port and is powered from 3.3VDC direct from host battery pack power interface. The MC68HC908SR12 microcontroller performs all sensing, battery control, supervisory and communications functions. Microcontroller pin interfaces and functions are shown in the Appendix. (SMB & SCI Communications Ports) Both Serial Module Bus (SMB) and Serial Communications Interface (SCI) communications port interfaces are provided to the microcontroller. The SCI port is implemented as user development interface and the SMB port is dedicated to smart battery communications. The SCI communication port supports standard RS-232 and includes a level translator and full RS-232 signal levels. The RS-232 translator automatically enters a low-power shuts-down mode when valid RS-232 signal levels are not detected at the host interface connector. RS-232 serial communication interface is available on connectors J4 with the following pinout, as shown in the Appendix. The Serial Module Bus (SMB) channel 0 communications port is dedicated to smart battery communications between the Vaio U PC and the battery pack. The HSC SMB port may communicate directly with the battery pack (when Vaio U PC electrical interfaces are disabled) or utilized to monitor SMB communication between pack and PC. SMB serial communication interface is available on connectors J1, J3, J5 and J6 with the pinouts available from the main schematic or main connector tables.

55

– Implementation and Integration

RPCS – Spring 2006

(Battery Pack Current Monitor) The MC68HC908SR12 microcontroller includes an analog section specifically designed to monitor battery charge and discharge current. This function allows an optional means for the HSC to manage and assess battery state. Battery current measurement is facilitated by a low-side current-sense resistor, the voltage from this sense resistor is monitored via microcontroller OPIN1 (BATT_AMP pin. See MC68HC908SR12 microcontroller datasheet (section 14 Analog Module) for additional details and information on battery charge and discharge monitor.

(HSC Temperature Monitor) The MC68HC908SR12 microcontroller includes an analog section specifically designed to monitor ambient temperature. This function allows an optional means for the HSC to manage and assess battery state. Temperature measurement is facilitated by a semiconductor temperature sensor internal to the MC68HC908SR12 package, the voltage from this temperature sensor is monitored via microcontroller ADC. See MC68HC908SR12 microcontroller datasheet (section 14 Analog Module) for additional details and information on temperature monitor. (Battery Pack Voltage Monitor) Battery pack voltage may be monitored by the MC68HC908SR12 microcontroller ADC and additional conditioning, switching and scaling circuitry implemented on the HSC. It is important to note that battery pack voltage and Vaio PC supply voltages are measured separately and can differ depending on whether pack is currently providing power to the Vaio PC. If an alternate pack is currently supplying current, the battery pack and PC voltage will differ, but if the bidirectional MOSFET power switch is turned on; battery pack and PC voltages will be identical. Battery pack voltage is monitored at MC68HC908SR12 ADC pin ATD4, This pin reflects scaled voltage supplied from battery pack connected to HSC independent from Vaio PC voltage. Note PTD6 (VMON_SW) must be high logic level to enable analog voltage feed to this pin. Battery voltage appearing at microcontroller ADT4 pin is scaled by implemented discrete circuitry and may be characterized using following equation.

ADT 4(VDC ) = BATT (VDC ) * 0.4 See MC68HC908SR12 microcontroller datasheet section 15 (Analog to Digital Converter) for addition details and information. (Vaio Supply Voltage Monitor) Vaio U PC supply voltage may be monitored by the MC68HC908SR12 microcontroller ADC and additional conditioning, switching and scaling circuitry implemented on the HSC. It is important to note that battery pack voltage and Vaio PC supply voltages are measured separately and can differ depending on whether pack is currently providing power to the Vaio PC. If an alternate pack is currently supplying current, the battery pack

56

– Implementation and Integration

RPCS – Spring 2006

and PC voltage will differ, but if the bidirectional MOSFET power switch is turned on; battery pack and PC voltages will be identical. Vaio U PC supply voltage is monitored at MC68HC908SR12 ADC pin ATD3, This pin reflects scaled voltage supplied to the Vaio U PC independent from battery pack voltage. Note PTD6 (VMON_SW) must be high logic level to enable analog voltage feed to this pin. Vaio U PC supply voltage appearing at microcontroller ADT3 pin is scaled by implemented discrete circuitry and may be characterized using following equation.

ADT 3(VDC ) = Vaio U PC (VDC ) * 0.4 See MC68HC908SR12 microcontroller datasheet section 15 (Analog to Digital Converter) for addition details and information. (Dual MOSFET Power Switch) A specialized dual MOSFET based bidirectional power switch has been implemented on the HSC. This power switch consists of dual series common-drain configured power MOSFET’s. The power switch includes a specialized analog circuit that monitors the voltage appearing on the Vaio U PC supply (when power switch is off) and matches the monitored voltage at the common drain connection. This function prevents dramatic current spikes in the event MOSFET is switched on when battery and Vaio supply voltages differ significantly (e.g. when a fresh battery pack is switched on over a depleted pack). MC68HC908SR12 microcontroller pin PTD1 (PWR_SW) controls power switch on/off function. High logic level on microcontroller PTD1 output pin enables power switch between battery pack and Vaio PC. Low logic level disables power switch between battery pack and Vaio PC. (MOSFET Communications & Signal Switch) Multiple MOSFET based bidirectional switches have been implemented on the HSC. These low-current communications and signal switches are realized using standard bidirectional analog switches. MC68HC908SR12 microcontroller pin PTD0 (COM_SW) controls communications and signal switch on/off function. High logic level on microcontroller PTD0 output pin enables communications and signal switches between battery pack and Vaio PC. Low logic level disables communications and signal switches between battery pack and Vaio PC. (Power Bus & 3.3VDC Regulator) When multiple HSC and battery pack assemblies are implemented within a given system. It is desirable to minimize power draw by placing HSC and battery pack in low-power standby mode. The Sony battery pack automatically enters low-power mode when the pack is disabled (i.e. either not electrically connected to the Vaio PC or not enabled by the HSC microcontroller PTD3 output pin (/BAT_ON). The HSC on the other hand must be placed in low-power mode through software action, however with the battery power switch tuned off; the microcontroller would not be powered. It is also important that the microcontroller is ready at all times to resume operation (from low-power mode) within milliseconds to immediately supply power to the Vaio U PC in the event another

57

– Implementation and Integration

RPCS – Spring 2006

HSC/pack assembly is either user removed or becomes sufficiently low in charge capacity. In order to maintain microcontroller power at all times; a power bus is implemented that connected common power between all HSC’s within a given network. This power bus is active with battery terminal voltage from any battery pack currently switched on and supplying power to the Vaio U PC. Safety rectifiers are included at on each HSC implementation to prevent current-flow on power bus between discrete battery packs. With the power bus always powered; all HSC’s within a given network are provided power from a single enabled battery pack. HSC/pack assemblies not currently powering Vaio UPC would enter low-power states ready to resume programming upon bussed /IRQ request. 3.3VDC voltage regulation is implemented on each HSC, this regulated and conditioned voltage is derived from the common power buss battery terminal voltage. Most HSC systems are directly supplied voltage from the 3.3VDC regulator implementation. (System-Wide Bussed IRQ) A bussed interrupt request signal is implemented between all HSC’s within a given system. The bussed /IRQ is interconnected between all HSC devices in system and driven low when any HSC IRQ_EN (PTD2) is driven high. The /IRQ bus is directly connected to each HSC microcontroller /IRQ1 (/IRQ) pin, and may be used to provide synchronous signaling between HSC devices in an interconnected system. The primary use of this bussed interrupt request is to signal sleeping microcontrollers to wake-up from lowpower modes and resume power management functions (e.g. when a given HSC determined that its battery pack is sufficiently low in charge capacity, the /IRQ bus is used to signal fresh HSC/pack assemblies to take over as primary power supply). Note the microcontroller /IRQ1 (/IRQ) pin is used to monitor the /IRQ bus, when an HSC microcontroller initiates an /IRQ bus request; the PTD2 output pin is used. High logic level on microcontroller PTD2 pin initiates system-wide /IRQ to all connected HSC devices. Logic low on PTD2 pin disables all /IRQ interrupts. PTD2 pin should only be pulsed high for short durations when an /IRQ is to be initiated system-wide. (Microcontroller Battery ON/OFF Switch) The battery pack may be switched on and off under microcontroller control independent of electrical power and communications connection with Vaio U PC. Microcontroller PTD3 (/BAT_ON) output pin is utilized for this function. Low logic level on this pin turns on battery pack connected to HSC. High logic level on PTD3 pin turns off battery pack connected to HSC. This function is required when battery pack and HSC are not currently supplying Vaio PC with power; for minimum power draw when pack is not currently being used as Vaio U PC primary power source. This function may also be used to allow HSC microcontroller interrogation of battery to determine status and charge state via USB communications port. (Crystal Oscillator)

58

– Implementation and Integration

RPCS – Spring 2006

A 32KHz ceramic resonator is used for primary microcontroller time-base. Discrete termination and loading elements are implemented on the HSC PCB in support of resonator operation. The HSC microcontroller includes internal PLL implementation which multiplies the 32KHz clock to derive desired bus frequency. (/LED_RED & /LED_BLU LED User Supervisory Indicators) Two Super-Bright user supervisory LED indicators are implemented on the Hot-Swap Controller PCB. These LED’s are intended for direct user output device for indication of various device status or feedback functions. The specific function of each LED is determined via microcontroller programming. One blue and one red LED are included on the component-side of the Hot-Swap Controller PCB; the local of these LED’s facilitates user viewing through transparent mechanical housing. The red and blue LED’s are individually controller via microcontroller pins PTA4 and PTA5 pins respectively. Microcontroller pins PTA4 and PTA5 must be driven low in output LED-drive configuration mode to turn on LED’s. See Microcontroller Pin Function & Description table for additional operational information. (Microcontroller Reset) A PCB mounted momentary pushbutton switch is implemented to allow user-initiated microcontroller reset function during device programming development. This reset pushbutton is not intended to be used during normal Hot-Swap Controller field operation.

4.4.3

Hardware Architecture

The MoRÉ Platform Architecture consists of a combination of custom hardware components and premium commercial devices. The system is centered around a computationally powerful, yet small in size and lightweight computer, the Sony VaioU8G. The requirements of the MoRÉ software demand a computer capable of running the Microsoft Windows operating system, the ability to interface wirelessly with a remote server, have the means of presenting audiovisual information, and interface with a variety of custom and commercial devices. The Sony Vaio-U8G meets these and other functional necessities, and with a long battery life of 4-5 hours, light weight of only 1.2 lbs. and size of only 6.57 x 4.25 x 1.03 inches. There are three main pieces of hardware: the integrated headset for display/audio, the Sony Vaio holster and enclosure, and the Dial and its harness. The Vaio interfaces with two other primary devices, the combined audiovisual headset and the custom Dial pointing device, through a small Targus USB 2.0 Hub, which gets its power from the Vaio. The hub sends and receives all user data to and from the Vaio through a single USB cable, and uses three out of its four I/O communications ports for interfacing with the various other devices. One of these three connections is used to power the Liteye HeadMounted Display (HMD), and the other two connections are used as data links for the Sound Professionals USB Audio Converter, and the Custom Dial Pointing Device (Figure 16).

59

– Implementation and Integration

RPCS – Spring 2006

The Audiovisual Head Unit consists of the Liteye Head-Mounted Display and David Clark H10-13.4 headset. As mentioned earlier, the HMD gets its power from the hub through a USB cable, and it receives the signal for the user’s visual computer display through a VGA cable that is connected directly to the Vaio. The Liteye Head-Mounted Display is physically attached to the David Clark headset, which provides the user with audio information from the software, and provides the capability for the system to receive voice data from the user, if allowed for. This earphones/microphone headset plugs into 1/8 inch stereo audio ports on the Sound Professionals USB Audio Converter. The David Clark headset provides the added bonus of hearing protection from high volumes of noise, from 23 decibels, to be specific. The small USB audio converter, powered through the hub, is capable of converting raw stereo audio signals into a standard audio data stream interpreted by the operating system. And lastly, the custom-made Dial Pointing Device also interfaces with the system via a USB cable connected to the hub. As described in this report, the Dial’s unique functionality is utilized by the custom browser that displays the manual and other relevant information.

60

– Implementation and Integration

RPCS – Spring 2006

Figure 16: Hardware System Architecture The Vaio interfaces with two other primary devices, the combined audiovisual headset and the custom dial pointing device, through a small Targus USB 2.0 Hub, which gets its power from the Vaio. The hub sends and receives all user data to and from the Vaio through a single USB cable, and uses three out of its four I/O communications ports for interfacing with the various other devices. One of these three connections is used to power the Liteye Head-Mounted Display (HMD), and the other two connections are used as data links for the Sound Professionals USB Audio Converter, and the Custom Dial Pointing Device.

4.4.4

Software Architecture

The software in the dial is very complicated. It is not a simple task to program a microcontroller correctly and this task is made more difficult by the fact that this microcontroller is a USB microcontroller. In our microcontroller the first chunk of software that had to be mastered was the in-circuit programming portion (ICP). This software is done in assembly language and is built upon existing icp code provided by the

61

– Implementation and Integration

RPCS – Spring 2006

manufacturer. Understanding how the microcontroller ICP architecture works was the most challenging part, once that was figured two lines of assembly had to be modified so that the assembly code jumped to the correct memory location upon being plugged in and also could have its memory erased and then reprogrammed if the code had to be changed. The second code chunk that had to be written was the USB code. This code sends USB messages from the microcontroller to the PC. This code was also very difficult because of the complexity of the USB code. Making this code was very difficult so we relied on existing USB code for a similar microcontroller to speed our progress. Making this code work was more challenging that first glance because the memory mapping was different for the two microcontrollers. The last chunk of code was user code which consisted a big polling while loop that checked to see which buttons are being pressed and then combines that with the USB code to send keyboard messages to the PC. Once the microcontroller had been programmed to conform to Human Interface Device (HID) USB standards, it was able to communicate without having to write additional drivers with the Windows XP operating system running on the Vaio. This enabled the Dial to communicate with the custom browser developed by the Software teams, which runs on top of XP, as shown in Figure 17.

ButtonPolling Loop

Windows Virtual Keyboard

Moré Application

User

User Code USB Protocol Library

USB Host Controller Cable

Kernel Level

Kernel

Dial

Vaio

Figure 17: Dial Firmware Architecture

4.4.5

Hardware on Body

After completing the Hardware and Software design phases, design engineering and implementation for the wearables of the system, including implementation and user testing. The integrated headset covers most of the head, including the right eye and both ears. It is relatively light. A more in-depth description of the design is contained below.

62

– Implementation and Integration

RPCS – Spring 2006

Figure 18: Integrated HMD and Headphones The Vaio/holster wraps around the waist and can be attached to either leg. The product weighs several pounds together. It also encumbers that leg somewhat by sticking out several inches from the body.

Figure 19: Vaio Holster

63

– Implementation and Integration

RPCS – Spring 2006

The Dial’s harness wraps around the chest with two belts, placing the Dial directly on the user’s chest which allows for equal usage for left and right-handed users.

Figure 20: Dial Harness 4.4.4.1 Audiovisual Head Unit Human-Interface Considerations The design of the Audiovisual Head Unit needs to correct for imperfections in the physical interaction between the Liteye HMD and the David Clark headset. Both devices were designed independently from one-another at different companies, and were not intended to be used in conjunction with any other apparatus on the head. The Platform team would have preferred alternative designs from both companies, but such substitute styles were either not in stock or did not exist. The problems encountered were as follows. For one, both devices, in their unmodified form, were competing for physical space above the right ear. The headset needed significant volume surrounding the ear, and the HMD protruded from its head-mount, directly above the right ear. The second problem was that the two devices had trouble being mounted to the head at the same time. There was either difficulty in seeing the display screen, or physical discomfort to the user. The simple, if non-ideal, solution to these problems was to detach the display screen from its base, and mount it directly onto one of the protruding screws on the headphones. This provides an interim solution to this problem, until a more sophisticated HMD can be obtained from the company Liteye.

64

– Implementation and Integration

RPCS – Spring 2006

Figure 21: HMD and Headphones - Before and After Views 4.4.4.2 Power Savings versus Power-up Times in Various Modes of Operation The Platform team was able to complete preliminary studies on the effects of putting the system into various power-saving modes, as opposed to fully turning off and restarting the system. The Sony Vaio U8G comes with similar power-saving features that are on laptops on the market, such as “stand-by” and “hibernate”. In stand-by mode, the Vaio will shut down most of its systems, including the hard drive and LCD display, and stop sending any signals to and from the microprocessor and wired or wireless communication channels. It will, however, maintain some power on the system board, and keep power to its RAM. This allows for a speedy time in returning to where the user stopped working with the system. Depending on which applications are running, and which resources the Vaio needs in returning to work with these applications (hard drive data or establishing a wireless network connection), this resume time can range between 5 and 9 seconds. Further studies collecting data as to how much power-savings this mode provides need to be conducted, but initial estimates indicate that up to 10%, or higher, of a fully-charged battery could be expended over the course of an 8-hour workday, in providing this capability. The second power-saving mode is called “hibernate”. Upon receiving the signal for hibernation, Windows begins storing pertinent data to the hard disk, primarily consisting of all information actively being used in memory, such as Kernel memory and information stored in RAM. And when the user wishes to resume use of the system, Windows reads back this stored information and writes it to the appropriate memory locations. In order to use this mode, the Vaio must have sufficient hard disk space available to store this data from memory, usually ranging in the hundreds of megabytes. However, this power-saving mode provides complete savings of the battery, as no energy is spent in trying to keep any Vaio systems on. No component requires power in this mode, as all data necessary to continue work, is permanently stored on the hard drive. Other than needing adequate room on the hard drive, the Vaio also requires additional time to fully resume operations after hibernation. Depending on how much information needs to be read back by the operating system and rewritten to memory, resuming work 65

– Implementation and Integration

RPCS – Spring 2006

after hibernation will usually take between 8 and 15 seconds, but sometimes longer. If a reasonable means of allowing for hibernation during a normal workday can be developed, then this method of power-saving is well-worth the wait. 4.4.4.3 Usability Tests: Webbing Setup Time Introductory studies on the way that individual users attach the various components to their bodies have also been performed. There are three major webbing units in this system, including the Audiovisual Head Unit, Dial Strap, and Han Solo Belt, so named for the movie character who wore a quasi-belt-gun-holster. The head unit is put on in the exact same manner as any typical pair of headphones, and the display is adjusted into place with the hands. The dial strap is put on over the head, and then tightened with the Velcro belt across the stomach. And finally, the Han Solo belt secures the Vaio and surrounding ruggedized enclosure to the leg with two Velcro straps, but allows the weight to be carried by the subject’s waist, via the belt. In this way, the lightweight Vaio is locked in-place on the leg, but the user is not discomforted by its weight since it hangs from the waist. Test subjects were given these three components, and asked to put them all onto their bodies, and plug in all of the devices into the appropriate ports. The system was already turned on, and the appropriate software was already functioning. These users were not asked to startup the system, place the Vaio into the ruggedized enclosure, nor asked to use the devices, other than to put them on. These subjects had never worn the system before, and they were only given a picture and a brief description of how the system should be worn. It was found that new users took on average just over 3 minutes to complete the task. And as these users continued to try putting on all of the webbing components in More iterations, the times were reduced to a little less than 2 minutes. Thanks to the simplified design of only having three major articles of webbing, easily recognizable I/O ports for all of the devices, and having these ports housed in one easy place on the waist, setup times for both veterans and novices are minimized.

4.4.6

Hardware Modules and Status

As of the end of the Spring 2006 semester, the status of the Platform system is as follows: The Dial is a completed prototype as of the end of the Spring 2006 semester and provides keyboard input to any host to which it is connected via the USB protocol. The hardware contained within the designed Dial enclosure is pictured below. Additional information about the hardware connections and pins is available in the Appendix.

66

– Implementation and Integration

RPCS – Spring 2006

Figure 22: Dial PCB Component-side and Solder-side Views The Enclosure is a completed prototype as of the end of the Spring 2006 semester and both secures and protects the Sony Vaio unit, as well as allowing heat to escape and placing the unit directly on the thigh. Pictures of the completed design are shown above. The Hot-Swap controller has completed the hardware design phase and is in the process of implementation. Layout and PCB production have been completed as of the end of the semester (Figure 23). The Hot-Swap Controller Printed Circuit Board (PCB) was fully implemented using Cadence ORCAD Schematic Capture CIS and Layout Engineers Edition. The PCB is a double sided, 62mill thickness G10 circuit board with full ground planes on both component and solder surfaces. Additional information describing the microcontroller and connections of the Hot-Swap hardware are contained in the Appendix. Future work to improve upon the existing prototype includes the completion of the firmware design of the Hot-Swap Controller.

Figure 23: Hot-Swap PCB Component-side and Solder-side Views

67

– Implementation and Integration

RPCS – Spring 2006

Key issues that remain include no preexisting hot-swap firmware for the microcontroller used, as well as no battery clips to attach to the batteries and holster belt. Figure 24 contains a prototype for the proposed Hot-Swap Controller, which would contain three batteries to be located on the small of the back. When programming for the controller is completed, the physical design of the casing and on-the-body testing can be completed for the design.

Figure 24: Proposed Hot-Swap Controller Design

4.5 4.5.1

Conclusions Summary of Key Design Issues

4.5.1.1 Capturing Experience There were many design issues that came up in Phase III. One was to implement our modules according to when and how the controller and renderer call the capturing experience modules. Depending on the sequence of calls, the capturing experience modules had to be adjusted. Also the modules had to be designed in a way that they would be easily altered depending on the user testing we were planning on doing. For cycling detection, determining the definition of cycling was an issue that came up often along with having links that are useful to the user. Also an issue came up that in the initial use of the Moré application, the dynamic smart links would be the same links as the cycling detection ones. Only when more users use the system and the ranking of smart links in the two modules differ will there be separate links. The main design issues with dynamic Smart Links were to allow users to shortcut to useful pages in the manuals without much burden. Based on experience, if users have to do too much for a feature to function properly, they will probably not use it at all. So heuristics were devised to attempt to infer when users had found useful pages, which was the main issue in creating automatically generated Smart Links. For example, one 68

– Implementation and Integration

RPCS – Spring 2006

heuristic was that when a user spends a long time viewing a single page without navigating away, it is likely that the user found useful information. Or by mapping pages into a topic hierarchy based on the table of contents, one heuristic was to infer that the user had found what he was looking for when he navigated to a page that was about a different topic or in a different sub tree in the hierarchy. However, to maintain a simplified approach for a rapid prototype, a link was used to require manual user input to specify when he found what he was looking for. This does not bring much burden onto the user, but it allows for dynamically generated Smart Links that are otherwise created completely automatically. 4.5.1.2 The Dial In order to facilitate fast design and implementation cycle, a direct to PCB approach was taken. This approach eliminates the typical ‘breadboard’ engineering prototype cycle using large pinned devices, and instead implements a prototype PCB based on final design component selection and footprint requirements. Direct to PCB prototyping is an effective and efficient engineering methodology that includes the following benefits: • Forces thorough and competent design practices in schematic-entry phase • Matures detailed design in schematic-entry phase • Allows use of single set of procured components • Prototypes circuit, mechanical and PCB layout designs in single engineering phase • Allows analysis and testing of matured design in single engineering phase • Lowers risk of design anomalies The direct to PCB engineering approach often requires only a single revision cycle to produce error-free circuit design and PCB implementations. Component Selection The use of miniature surface-mount (SMD) electronic components was pursued early in the engineering design process. SMD components offer the following benefits: • Extremely space efficient • Readily available (industry standard) • Low-cost • Easy to hand solder • Facilitate low-noise characteristics • Allow efficient miniature PCB layout • Lower cost of PCB production and assembly USB Interface One key design issue was the port interface of the Dial. We assumed from very early on that the Dial would be USB-compatible, as USB ports are widely available and fairly universal in new computers, have mountains of background information available, and

69

– Implementation and Integration

RPCS – Spring 2006

have a strong development community. The Sony Vaio computer has one USB port, but since several More devices use USB for data and power, the solution was to utilize a miniature, unpowered USB hub. The choice to conform to the USB interface influenced the electrical engineering by leading to the choice of a Freescale USB microcontroller to run the Dial. It also influenced the software engineering by allowing the Dial to conform to the USB Human Interface Device (HID) class of devices, which encompass keyboards, mice, joysticks, and other More exotic input devices. This was a natural fit given the goal of the Dial to function as a rugged, orientation-independent input device. Later, the design decision to emulate a USB keyboard was made when it was discovered that working USB keyboard code existed for a close relative of our Freescale microcontroller. As a result, the Dial is simply a functional USB keyboard that sends specific characters when different functions are performed – the characters are interpreted by the MoRÉ application to become meaningful. Microcontroller Selection Various microcontrollers were considered for use for dial implementation. These included microcontrollers from various manufacturers; Atmel, PIC, Analog Devices, Freescale (Motorola), Cypress and Mitsubishi. The final microcontroller selection was weighted and based on the following ordered requirements: • USB Hardware Facilities • I/O Pin Count and Functionality • Availability of Driver Software Libraries • Rich Programming Environment & Infrastructure • Product Availability • Package Configuration • Cost • Low-Power The Freescale (Motorola) 8-bit HC08 based MC68HC908JB16JDW microcontroller was selected for final design implementation due to built-in USB module, highly-flexible digital I/O, vast software library availability, ease of programming and support, surface-mount 28-pin package, extremely low power and cost. Peripheral Component Selection One of the most important aspects of Dial design simplification was the selection of specific peripheral components. These peripheral components include the following: • Integrated low-cost miniature joystick, pushbutton and quadrature encoder device • Vertical and horizontal miniature tactile pushbutton • Super-bright SMD LED’s Passive RC Filter Elements Passive RC filter elements were included between all switch, quadrature-encoder and microcontroller inputs. These passive elements eliminate electrical noise generated from mechanical switch contact-bounce, prevent EMI/RFI injection, protect microcontroller

70

– Implementation and Integration

RPCS – Spring 2006

CMOS digital inputs from static-discharge damage and reduce software overhead associated with signal conditioning. Filter elements are implemented using discrete SMD components and include nominal time constants of 1.0ms. 4.5.1.3 Hot Swap Controller In order to facilitate fast design and implementation cycle, a direct to PCB approach was taken. This approach eliminates the typical ‘breadboard’ engineering prototype cycle using large pinned devices, and instead implements a prototype PCB based on final design component selection and footprint requirements. Direct to PCB prototyping is an effective and efficient engineering methodology that includes the following benefits: • Forces thorough and competent design practices in schematic-entry phase • Matures detailed design in schematic-entry phase • Allows use of single set of procured components • Prototypes circuit, mechanical and PCB layout designs in single engineering phase • Allows analysis and testing of matured design in single engineering phase • Lowers risk of design anomalies The direct to PCB engineering approach often requires only a single revision cycle to produce error-free circuit design and PCB implementations. Component Selection The use of miniature surface-mount (SMD) electronic components was pursued early in the engineering design process. SMD components offer the following benefits: • Extremely space efficient • Readily available (industry standard) • Low-cost • Easy to hand solder • Facilitate low-noise characteristics • Allow efficient miniature PCB layout • Lower cost of PCB production and assembly Microcontroller Selection Various microcontrollers were considered for use for Hot-Swap Controller implementation. These included microcontrollers from various manufacturers; Atmel, PIC, Analog Devices, Freescale (Motorola), Cypress and Mitsubishi. The final microcontroller selection was weighted and based on the following ordered requirements: • I/O Pin Count and Functionality • Rich Programming Environment & Infrastructure • Product Availability • Package Configuration • Cost • Low-Power

71

– Implementation and Integration

RPCS – Spring 2006

The Freescale (Motorola) 8-bit HC08 based MC68HC908SR12CFA microcontroller was selected for final design implementation due to highly-flexible digital I/O, vast software library availability, ease of programming and support, surface-mount 48-pin package, extremely low power and cost. Circuit Protection Elements Due to the high-potential for electrostatic-shock damage and EMI/RFI injection associated with a pluggable device like the Hot-Swap Controller, passive circuit protection elements were included on all electrical interfaces to external components. Protective elements are based on simple series resistor and fast shunt-zener diode combinations. High current paths are protected with shunt-zener diode elements only. Modular Design Approach Hot Swap Controllers are designed in a manner consistent with flexibility and modularity. A given system implementation may serially cascade as many Hot-Swap Controller nodes as desired to extend battery capacity. Each Hot-Swap Controller is identical in hardware design and includes interface connectors to enable serial interconnectivity between nodes, primary load and charging facilities. Low Power Mode Hot Swap Controller electronics design includes provisions to enter standby low-power mode under microcontroller command during periods when controller is not the primary power source. In this mode the Hot Swap Controller draws only micro-amps of battery current effectively preventing primary battery drain during idle periods. Hot Swap Controllers share a common interrupt request signal that may be used to wake-up controller from a standby state.

Interface Compatibility Hot Swap Controllers are designed for direct interconnection between the Vaio PC and the primary battery pack. In order to achieve this requirement, standard battery connector interfaces are included for both Vaio and battery pack interface points. The standard battery connectors are manufactured by Tyco Electronics (formerly Amp Inc.), and are strategically located to enable efficient sandwiching between PC and battery pack.

4.5.2

Lessons Learned

The main challenge faced by the HCI group throughout the project is devising an application interface that would work along with a custom input device (the dial). Many parameters had to be considered in great detail for the design, this includes: real estate of the screen, easy accessibility of the buttons and most importantly the location and manner of triggering the buttons such that they are intuitive to users when interacting with our custom software interface.

72

– Implementation and Integration

RPCS – Spring 2006

The main lesson learned throughout the project is the lack of user study tests which are crucial to a good UI design. User testing is particularly important when one is trying to design a customized system for a very small target of end users. Unfortunately, we had no access to aircraft maintenance workers nor did we have hands on experience with tasks that they would perform. Much of the design was carried out based on the intuition of the group members however the access to user subjects and perhaps even test environment under which the prototype would be deployed would greatly help in creating a better design. Throughout the project, in particular for Phase II and III, the HCI team had a difficult time in identifying our roles as HCI. Although each the members were very involved with their assigned sub-system, many of us were unsure of our role as HCI members. Perhaps, this can be clarified for future classes so The experience with coming up with a script was also somewhat challenging. For MoRÉ, the system was not fully integrated to have a detailed draft of the script at an earlier stage. Many details in the script that were planned for the demo had to be revised to adapt to the working version of the system. Perhaps for future classes, scripting the demo could closely related to integration and become a simultaneous process running parallel with integration. Many lessons were learned the hard way in Phase III by the Capturing Experience Group. Assuming components would be integrated easily was a bad assumption make. Each of the components worked individually, however after the initial integration a lot of problems arose. Also, Phase III showed how important testing is. Whether it is airplanes or software, testing and observing expected results is the only way to know whether a prototype works. No matter how careful one is, inevitably there will be bugs or other flaws. Something that was needed but never done was to list issues that arose of each module. Many problems were overlooked and forgotten. More specifically, when the recording module was being implemented in Java in Phase III, part of the code necessarily required the hardware to be able to record using a set of configurations. Since getting this to work on a personal machine did not require a great amount of tweaking, because settings, configurations, and hardware setup could be modified freely, it was forgotten until Phase III, when it became a problem. In sum, it was learned that it is important to record things that may be a problem in the future, and also if there are things that are dependent on an environment, which may be different from the one it is being implemented and tested on. Also, better coordination between different groups is key to a successful prototype. For example, it was/is important to make sure everyone is not only using compatible versions and also for all modules that had a dependency on another group’s module to understand what will be inputted or outputted from each module. It is definitely something to watch out for in the future when it may be much harder to transition.

73

– Implementation and Integration

RPCS – Spring 2006

One of the hardest-hitting lessons learned by the Platform group was that it turned out to be basically impossible to write USB firmware from scratch, so it is important to choose microcontrollers and chips that have available, working code bases. This wasn’t factored in when the electrical engineering portions of the Dial and Hot-Swap Belt were done, and almost became a critical issue with the system as there wasn’t enough time to write code from scratch. Even with months of planning and scrutinizing of plans in the project, it is really amazing how a single minor problem can raise the possibility of disaster. The problems that the platform team found in getting the dial to work as planned were so simple, yet required so much effort to overcome. Although there was code readily available for our PCB board, it took a great deal of time to locate it and to edit and conform it to our purposes. And it took a number of our team members to complete the task. This hindered us in another aspect of the project, which was the programming of the hot-swap controllers. So a lesson that can be learned from this is that the proper research needs to be allocated as soon as possible to clarifying assumptions made in the first few steps of the project. And if at all possible, this research (in this case, the locating of proper code) should come even before the creation of designs and plans of specific components. Of course, it was difficult to do so in such a short period of time, but it is nevertheless the biggest lesson that I have taken away from the class. Another lesson is that project engineering design and implementation approach methodologies were highly-successful with realization of revision A PCB hardware. Both Dial and Hot-Swap controller electronics designs, PCB-layout and component assembly phases were completed within schedule and functioned as per design specification without circuit revisions. Fundamentally the fast-prototype direct to PCB approach was reinforced as a viable and efficient means to design and implement custom electronics for rapid-prototype projects.

5 5.1 5.1.1

Project Management Implementation and Integration Phase Results Summary of Logbook Hours

Table 3: HCI Group Task Administrative Tasks Class Coding Group Meeting Research Design Engineering

Phase 3 Worklog Dan Yong Brian 1:00 1:00 3:30 15:00 16:00 13:00 7:30 12:00 1:30 30:15 4:30 39:00 22:30

74

Anand 2:00 11:00 37:00 11:10

Wen Shu

Aashni

17:00 21:30 12:00

4:00 5:00 23:30 4:00

Total 5.00 66.30 41.00 67.00 38.45 61.30

– Implementation and Integration

User Interface Design User Testing Preparing Specs Purchasing Activities Writing Reports/Presentations Staff/Leaders Meeting Architecture Design Working with client Total

11:00

RPCS – Spring 2006

5:30 6:00 10:30

6:00 3:00 13:00

24:00

6.00 8.30 6.00 92.30

11:00 23.00

0:45

54:00

2:30

92:30

1:00 66:00

3.15 1.00

92:10

Table 4: Search Group Phase 3 Worklog Task Melanie Sachin Administrative Tasks 8:00 Architecture Design Class 11:30 Coding 37:00 Data Migration and Restoration 8:00 Design Engineering Field Testing Group Meeting 10:00 Preparing Specs Purchasing Activities Research Staff/Leaders Meeting User Interface Design Writing Reports/Presentations 17:00 Total 91:30

64:00

Kevin

52:00

Tabreeze 1:30 12:00 26:30 8:00 11:00 4:30 3:30 6:00 5:00 78:00

Table 5: Capturing Experience Group Phase 3 Worklog Ha, Jae Mai, Leon Tran, Jonathan Administrative Tasks 3:45 Architecture Design Class 5:30 Coding 74:30 74:25 48:20 Design Engineering 1:30 Field Testing 7:00 Group Meeting 8:35 Preparing Specs Research Staff/Leaders Meeting 1:25 10:00

75

Total 8:00 1:30 23:30 63:30 16:00 11:00 14:30 3:30 6:00 22:00 169:30

– Implementation and Integration

RPCS – Spring 2006

User Interface Design

-

-

-

Writing Reports/Presentations Total

10:00

6:30

14:45

84:30

83:50

87:40

Table 6: Platform Group Phase Task Adam Megan Administrative Tasks 4:00 1:15 Architecture Design 1:00 Class 13:40 14:00 Coding 31:00 18:00 Design Engineering 9:00 Field Testing Group Meeting 5:30 4:00 Preparing Specs Purchasing Activities Research 3:00 3:30 Staff/Leaders 2:00 Meeting Working with client 1:00 Writing 23:00 18:30 Reports/Presentations Total 93:10 60:15

5.2 5.2.1

3 Worklog Zack Mohammad 3:00 24:00 12:50 34:00 12:30 2:00 10:00 3:00 12:00 12:15 3:00 -

Bryon 9:00 12:00 4:00 10:30 -

Total 5:15 4:00 73:30 83:00 33:30 2:00 26:30 12:00 32:15 -

4:00

0:30 -

11:00

1:30 57:30

76:45

56:20

47:30

323:50

Summary of the Entire Project Task Dependency Chart

The Task Dependency Chart shows the different tasks allocated per group along the timeline of Phases 2 and 3. It has been added to the Appendix Section of the report. The groups tried their best to stick to the Chart’s timeline as it was possible. One major change in the chart is the time allotted for integration, which was greatly shortened due to time constraints in preparing for the final presentation.

5.2.2

Summary of Logbook Hours

Table 7: HCI Group Semester Worklog Task Dan Yong Brian Administrative Tasks Class Coding Data Migration &

1:30 20:45 -

40:10 76

4:30 38:30 33:30 -

Anand 2:00 26:30 37:00 4:00

Wen Shu 0:30 44:00 33:00 -

Aashni

Total

23:30 7:00 -

8:30 192.45 110.30 4:00

– Implementation and Integration

Restoration Field Testing Group Meeting Research Design Engineering User Interface Design Preparing Specs Purchasing Activities Writing Reports/Presentations Staff/Leaders Meeting Architecture Design User Testing Working with client Total

37:30 21:45 38:45

RPCS – Spring 2006

42:00 14:30 39:00 11:00 9:30 6:00 31:30

18:30 22:30 1:00 39:30

5:00 30:10 3:00 39:30

0:45 1:00 1:00 4:00 2:00 10:00 6:00 1:00 131:00 194:40 163:00 172:40

Table 8: Search Group Semester Worklog Task Melanie Sachin Kevin Tabreeze Administrative Tasks 8:00 4:05 Architecture Design 4:00 1:30 Class 35:35 30:40 26:40 Coding 62:00 28:50 54:30 Data Migration and 8:00 8:00 Restoration Design Engineering 6:15 11:00 Field Testing Group Meeting 21:30 16:45 18:30 Preparing Specs 1:05 3:30 Research 4:30 14:40 34:00 Staff/Leaders 5:30 2:45 Meeting User Interface 1:00 Design Writing 26:00 20:00 22:30

77

31:00 1:00 4:30 26:00 7:00 2:30 2:00 151:30

Total 12:05 5:30 92:55 145:20 16:00 17:15 56:45 4:35 53:10 8:15 1:00 68:30

23:30 7:00 34:00 1:00 96:00

5:00 182.00 43.75 61.3 15.30 13.30 6.00 206.35 13.45 4.30 18.00 2.00

– Implementation and Integration

Reports/Presentations Total

166:35

RPCS – Spring 2006

129:05

180:10

475:50

Table 9: Capturing Experience Group Semester Worklog Ha, Jae Mai, Leon Tran, Jonathan Administrative Tasks 4:15 1:00 15:30 Architecture Design 0:45 Class 13:40 20:10 22:40 Coding 116:30 105:25 55:45 Design Engineering 4:00 1:30 2:30 Field Testing 7:00 Group Meeting 23:30 19:25 44:00 Preparing Specs 11:30 1:50 Research 23:00 24:00 2:05 Staff/Leaders Meeting 3:45 2:25 8:20 User Interface Design

-

-

0:20

Writing Reports/Presentations Total

41:35

29:25

35:35

241:45

203:20

196:20

Table 10: Platform Group Semester Worklog Task Adam Megan Zack Mohammad Administrative Tasks 23:45 8:55 6:00 3:30 Architecture Design 1:00 3:00 Class 35:55 39:45 54:00 35:15 Coding 31:00 23:30 45:50 Design Engineering 9:00 28:00 Field Testing 22:00 Group Meeting 15:35 12:10 13:00 11:45 Preparing Specs 1:00 2:30 Purchasing Activities 0:30 12:00 Research 15:45 16:15 22:00 21:35 Staff/Leaders 5:30 1:00 Meeting Working with client 1:00 0:30 Writing 35:00 38:00 14:15 17:30 Reports/Presentations Total 174:00 139:35 174:45 138:55

78

Bryon 35:00 10:00 73:20 19:00 1:00 3:00 48:30 0:05

Total 42:10 4:00 199:55 110:20 110:20 22:00 71:30 4:30 15:30 124:05 6:35

33:00

1:30 127:45

222:55

851:10

– Implementation and Integration

5.3

RPCS – Spring 2006

Suggestions for Improving the Class

To improve RPCS, we would suggest emphasizing very early in the course how important it is to look down the road at every step – as a group, we wasted a lot of time choosing one microcontroller when there was much more code available for another from a nearby grad student. This could have saved us a ton of time and headaches programming. The main issue here is that the amount of programming required and the difficulty of implementation was not considered during the hardware implementation phase, because students have little experience in completing end-to-end projects such as in this course. Instructors and Teaching Assistants should encourage students to begin software research throughout the hardware design phase so that these considerations are taken into account before the design is complete and it is too late. Fortunately, the group was able to devote enough time and research into finding base code and developing a new implementation. Other groups may not be so fortunate if not guided early.

79

– Implementation and Integration

6

RPCS – Spring 2006

Glossary of Terms

MoRÉ – Mobile Reference Environment IETM - Interactive Electronic Training Manual DAS – Data Acquisition System Vaio – Sony Vaio Mobile Computer HID – Human Interface Device ICP – In-circuit programming

80

– Implementation and Integration

7 7.1

RPCS – Spring 2006

Appendix Hardware Specification Tables

Tab le 11: MC908 JB 16DW E Microco ntro ller P in Fu nct ion & De scr ip tio n Microcontroller Legend Description Configuration Function Pin 1

VSS

Microcontroller negative supply potential External oscillator input

N/A

Tied to PCB GND

2

OSC1

N/A

OSC2

External oscillator output

N/A

4

VREG

N/A

5

VDD

6

PTD0

Microprocessor 3.3VDC regulated output Microcontroller positive supply potential STAT1

Used with external 12MHz crystal and related termination elements Used with external 12MHz crystal and related termination elements Used to bias USB D- signal for USB low-speed interface host detection Tied to PCB +5VDC

3

7

PTD1

STAT2

Output

8

PTD2

ENCA

9

PTD3

ENCB

10

PTD4

Not Used

Input: Note active pullup resistor must be enabled Input: Note active pullup resistor must be enabled Input

11

PTE1

ENCA

12

PTE3

D+

Input: Note active pullup resistor must be enabled I/O

13

PTE4

D-

I/O

14 15 16 17

PTC0 /IRQ PTC1 PTD5

TxD Interrupt Request RxD Not Used

Output Input –active low Input Input

N/A Output

81

Red LED drive. Drive this pin low to turn on red LED. Note LED drive port option must be configured Blue LED drive. Drive this pin low to turn on blue LED. Note LED drive port option must be configured Rotational encoder quadrature A signal. This signal is routed to both PTD2 for level detection and PTE1 for interrupt detection Rotational encoder quadrature B signal. This signal is routed to both PTD3 for level detection and PTE2 for interrupt detection This pin must be configured as input Rotational encoder quadrature A signal. This signal is routed to both PTD2 for level detection and PTE1 for interrupt detection USB Data+ differential communications data USB Datadifferential communications data CMOS level RS-232 Transmit Not Used CMOS level RS-232 Receive This pin must be configured as

– Implementation and Integration

RPCS – Spring 2006

18

PTA7

LF

Input: Note active pullup resistor must be enabled Input: Note active pullup resistor must be enabled Input: Note active pullup resistor must be enabled Input: Note active pullup resistor must be enabled

19

PTA6

MID

20

PTA5

RT

21

PTA4

PUSH

22

PTE2

ENCB

23

PTE0

Not Used

24

PTA3

DIRD

Input: Note active pullup resistor must be enabled

25

PTA2

DIRC

Input: Note active pullup resistor must be enabled

26

PTA1

DIRB

Input: Note active pullup resistor must be enabled

27

PTA0

DIRA

Input: Note active pullup resistor must be enabled

28

/RST

Microcontroller Reset

I/O – active low

Input: Note active pullup resistor must be enabled Input

82

input Dial left user pushbutton. This pin is normally high and driven low when the user right pushbutton is depressed Dial middle user pushbutton. This pin is normally high and driven low when the user middle pushbutton is depressed Dial right user pushbutton. This pin is normally high and driven low when the user right pushbutton is depressed Dial right joystick depression pushbutton. This pin is normally high and driven low when the user joystick pushbutton is depressed Rotational encoder quadrature B signal. This signal is routed to both PTD3 for level detection and PTE2 for interrupt detection This pin must be configured as input Joystick direction D input. This pin is normally high and is driven low when the user joystick is depressed in the D direction Joystick direction C input. This pin is normally high and is driven low when the user joystick is depressed in the C direction Joystick direction B input. This pin is normally high and is driven low when the user joystick is depressed in the B direction Joystick direction A input. This pin is normally high and is driven low when the user joystick is depressed in the A direction This pin is connected to an external momentary pushbutton that generates an active-low microcontroller reset when depressed

– Implementation and Integration

Tab le 12: J1 & J2 S CI Molex Part # J2 USB Interface Pin Number 1 2 3 4 Molex Part # J1 SCI Interface Pin Number 1 2 3 4

RPCS – Spring 2006

and USB In terf ace 53048-0410 Signal Legend +5VDC DD+ GND

Signal Description USB Power Data + Data USB Ground

53048-0410 Signal Legend +5VDC RxD TxD GND

Signal Description SCI Power TTL Level Receive Data TTL Level Transmit Data SCI Ground

Tab le 13: MC68 H C90 8SR 12 Microcon tro ller P in Func tio n & De script ion Microcontroller Legend Description Configuration Function Pin 1

PTC3

Not Used

Input

2 3

NC PTD0

No Connection COM_SW

Output

4

VDD

Microcontroller positive supply potential

N/A

5

OSC1

External oscillator input

N/A

6

OSC2

External oscillator output

N/A

7

VSS

N/A

8

PTD1

Microcontroller negative supply potential PWR_SW

9

/IRQ1

/IRQ

Input

Output

83

This pin must be configured as input with active pullup disabled to prevent power draw No Connection High logic level on this pin enables communication between Vaio PC and battery pack. Low logic level disables communication between Vaio PC and battery pack Tied to PCB +3.3VDC Voltage Regulator. This voltage is regulated from the common system power bus and is present whenever a battery pack (from any HSC) is present in system Implemented with external 32KHz crystal and related termination elements Implemented with external 32KHz crystal and related termination elements Tied to PCB GND High logic level on this pin enables battery power between Vaio PC and battery pack. Low logic level disables battery power between Vaio PC and battery pack System-wide interrupt request. This pin is connected between all HSC devices in system and driven low when any HSC IRQ_EN

– Implementation and Integration

RPCS – Spring 2006

10

PTD2

IRQ_EN

Output

11

/RST

Microcontroller Reset

I/O – active low

12

PTD3

/BAT_ON

Output

13

SDA0

SDA0

I/O

14

SCL0

SCL0

I/O

15

SDA1/TxD

Tx/SDA1

I/O

16

SCL1/RxD

Rx/SCL1

I/O

84

(PTD2) is driven high. High logic level on this pin initiates system-wide /IRQ to all connected HSC devices. Logic low on this pin disables all /IRQ interrupts. This pin should only be pulsed high for short durations when an /IRQ is to be initiated system-wide. This pin is connected to an external momentary pushbutton that generates an active-low microcontroller reset when depressed Low logic level on this pin turns on battery pack connected to HSC. High logic level on this pin turns off battery pack connected to HSC. This function is required when battery pack and HSC are not currently supplying Vaio PC with power. For minimum power draw from idle pack. SMB Serial Data pin channel 0. This pin is connected to the battery pack SDA communications pin and used to communicate with battery or monitor SMB communications between Vaio and battery pack. SMB Serial Clock pin channel 0. This pin is connected to the battery pack SCL communications pin and used to communicate with battery or monitor SMB communications between Vaio and battery pack. This pin is dual function: RS-232 TxD when a valid RS-232 voltage level is detected at the RS-232 RxD connector pin. And SMB port SDA otherwise (SMB communications between HSC devices). Note PTD7 input (RDY) signal is high logic level when valid RS-232 voltage level is detected at the RS-232 RxD connector pin. This pin is dual function: RS-232 RxD when a valid RS-232 voltage level is detected at the RS-232 RxD connector pin. And SMB port SCL otherwise (SMB communications between HSC devices). Note PTD7 input (RDY) signal is high logic level when valid RS-232 voltage level is

– Implementation and Integration

RPCS – Spring 2006

17

PTD4

Not Used

Input

18

PTD5

Not Used

Input

19

PTD6

VMON_SW

Output

20

PTC0

Not Used

Input

21

PTC1

Not Used

Input

22

PTC2

Not Used

Input

23

PTA7

Not Used

Input

24 25

NC PTD7

No Connection RDY

Input

26

PTA6

Not Used

Input

27

PTB6

Not Used

Input

28

PTB5

Not Used

Input

29

PTB4

Not Used

Input

30

OPIN1

BATT_AMP

Input

31 32

VSSAM PTA0

GND Not Used

NA Input

33

PTC7

Not Used

Input

34

ATD1

Not Used

Input

35

VREFL

VREFL

Input

36

VREFH

VREFH

Input

85

detected at the RS-232 RxD connector pin. This pin must be configured as input This pin must be configured as input High logic level on this pin enables battery voltage and PC voltage monitors at ATD3 & ATD4 ADC pins. Low logic level on this pin disables battery voltage and PC voltage monitors at ATD3 & ATD4 ADC pins. This pin must be configured as input This pin must be configured as input This pin must be configured as input This pin must be configured as input No Connection Logic high on RDY pin indicates valid RS-232 voltage level is detected at the RS-232 RxD connector pin. SDA1/TxD and SCL1/RxD mode is determined by logic level on RDY pin. With RDY high: RS-232 mode should be invoked, otherwise USB mode should be configured. This pin must be configured as input This pin must be configured as input This pin must be configured as input This pin must be configured as input This pin is used by the microcontroller internal amplifier to monitor voltage across the battery current sense resistor for detection of battery charge and discharge current Analog Module Ground Reference This pin must be configured as input This pin must be configured as input This pin must be configured as input ADC VREF Low reference. Connected to GND ADC VREF High reference.

– Implementation and Integration

RPCS – Spring 2006

37

ATD3

PC_V

Analog Input

38

ATD4

BAT_V

Analog Input

39

PTA2

Not Used

Input

40

PTA4

/LED_RED

Output

41 42

NC VSSA

No Connection VSSA

43

VDDA

VDDA

44

PTC6

Not Used

Input

45

PTC5

Not Used

Input

46

PTC4

Not Used

Input

47

PTA5

/LED_BLU

Output

48

CGMXFC

CGMXFC

NA

Tab le 14: J4 S CI I nt erface Molex Part # 53048-0410 J4 RS-232 SCI Interface Pin Number Signal Legend 1 No Connection 2 RxD 3 TxD 4 GND

Connected to VDD via RC LPF Vaio PC supply voltage monitor. This pin reflects scaled voltage supplied to PC independent from battery pack voltage. Note PTD6 (VMON_SW) must be high logic level to enable analog voltage feed to this pin. ADT3 Voltage = Voltage at PC * 0.4 Battery pack supply voltage monitor. This pin reflects scaled voltage supplied from battery pack connected to HSC independent from Vaio PC voltage. Note PTD6 (VMON_SW) must be high logic level to enable analog voltage feed to this pin. ADT4 Voltage = Battery V * 0.4 This pin must be configured as input Red LED drive. Drive this pin low to turn on red LED. Note LED drive port option must be configured No Connection Analog Ground Reference. Connected to PCB ground Analog supply voltage. Connected to regulated +3.3VDC via RC LPF. This pin must be configured as input This pin must be configured as input This pin must be configured as input Blue LED drive. Drive this pin low to turn on blue LED. Note LED drive port option must be configured External filter interface pin

Signal Description No Connection RS-232 Receive Data RS-232 Transmit Data RS-232Ground

86

– Implementation and Integration

RPCS – Spring 2006

Tab le 15: Pr imary Con nector Pinout s Vaio U PC Connector J1 Interface Molex: 6-1447143-0 Pin Legend Type Description 1 GND_PC NA Battery Negative Potential 2 /EN_PC I/O Battery Pack Enable Pin. Ground this pin through 1K to turn on battery pack 3 /OK_PC I/O This pin is low when battery pack fuses are OK 4 SCL0_PC I/O Battery pack & PC SMB communications Serial Clock pin 5 SDA0_PC I/O Battery pack & PC SMB communications Serial Data pin 6 VBATT_PC NA Switched Battery Positive Potential Inter-Board Interface Connectors J5 & J6 Molex: 22-05-7105 Pin Legend Type Description 1 /EN_PC I/O Battery Pack Enable Pin. Ground this pin through 1K to turn on battery pack 2 /OK_PC I/O This pin is low when battery pack fuses are OK 3 SCL0_PC I/O Battery pack & PC SMB communications Serial Clock pin 4 SDA0_PC I/O Battery pack & PC SMB communications Serial Data pin 5 SDA1 I/O Inter-Module SMB communications Serial Clock pin 6 SCL1 I/O Inter-Module SMB communications Serial Data pin 7 /IRQ_BUS I/O Inter-Module Interrupt Request Bus 8 GND NA Battery Negative Potential 9 VBATT_PC NA Switched Battery Positive Potential 10 PWR_BUS NA Inter-Module Power Bus Battery Pack Connector J3 Interface Molex: 6-1447143-9 Pin Legend Type Description 1 GND_BATT NA Battery Negative Potential 2 /EN_ BATT I/O Battery Pack Enable Pin. Ground this pin through 1K to turn on battery pack 3 /OK_ BATT I/O This pin is low when battery pack fuses are OK 4 SCL0_ BATT I/O Battery pack & PC SMB communications Serial Clock pin 5 SDA0_ BATT I/O Battery pack & PC SMB communications Serial Data pin 6 VBATT_ NA Switched Battery Positive Potential BATT

87

– Implementation and Integration

7.2

RPCS – Spring 2006

Task Dependency Chart Figure 25: Task Dependency Chart

88

– Implementation and Integration

7.3

RPCS – Spring 2006

List of Figures

Figure 1: Discrepancy Review Screen in Debrief System................................................ 8 Figure 2: Reference Manual Explanation ........................................................................ 9 Figure 3: Training Manual Explanation..........................................................................10 Figure 4: User Interface .................................................................................................22 Figure 5: Dial Prototype.................................................................................................24 Figure 6: MoRE Web Browser Interface........................................................................27 Figure 7: Search Functionality .......................................................................................28 Figure 8: Graph of Frequency of Words to Number of Words with Given Frequency ....30 Figure 9: Software Architecture.....................................................................................33 Figure 10: Diagram of Software Architecture for Capturing Experience ........................43 Figure 11: Screenshot of a Training Manual Page with Annotations ..............................44 Figure 12: Diagram of How Cycling Detection stores the URLs ....................................46 Figure 13: Screenshot of a Training Manual Page with Smart Links ..............................48 Figure 14: Dial Final Product.........................................................................................52 Figure 15: Vaio Ruggedized Casing...............................................................................53 Figure 16: Hardware System Architecture .....................................................................60 Figure 17: Dial Firmware Architecture ..........................................................................61 Figure 18: Integrated HMD and Headphones .................................................................62 Figure 19: Vaio Holster .................................................................................................62 Figure 20: Dial Harness .................................................................................................63 Figure 21: HMD and Headphones - Before and After Views..........................................64 Figure 22: Dial PCB Component-side and Solder-side Views ........................................66 Figure 23: Hot-Swap PCB Component-side and Solder-side Views ...............................66 Figure 24: Proposed Hot-Swap Controller Design..........................................................67 Figure 25: Task Dependency Chart ................................................................................87

7.4

List of Tables

Table 1: Selected Technologies......................................................................................12 Table 2: Test Results for Indexer Tests ..........................................................................29 Table 3: HCI Group Phase 3 Worklog ...........................................................................73 Table 4: Search Group Phase 3 Worklog........................................................................74 Table 5: Capturing Experience Group Phase 3 Worklog ................................................74 Table 6: Platform Group Phase 3 Worklog.....................................................................75 Table 7: HCI Group Semester Worklog .........................................................................75 Table 8: Search Group Semester Worklog .....................................................................76 Table 9: Capturing Experience Group Semester Worklog ..............................................77 Table 10: Platform Group Semester Worklog ................................................................77 Table 11: MC908JB16DWE Microcontroller Pin Function & Description .....................80 Table 12: J1 & J2 SCI and USB Interface ......................................................................82 Table 13: MC68HC908SR12 Microcontroller Pin Function & Description ....................82 Table 14: J4 SCI Interface .............................................................................................85 Table 15: Primary Connector Pinouts.............................................................................86

89