Integrating Analytical Cots Software Into Operational Environments

Integrating Analytical COTS Software into Operational Environments: A Practical Guide Ted Driver Analytical Graphics, Inc.

Overview This paper outlines lessons learned during the integration of a COTS software analysis tool, AGI's Navigation Tool Kit (NavTK), into a military operational environment. NavTK replaced contractor-specific software in the GPS Operations Center (GPSOC), which focuses on GPS receiver operational issues for the U.S. Air Force 2nd Space Operations Squadron (2SOPS). Using NavTK, GPSOC analysts produce routine, daily reports and graphs as well as analyze specific scenarios.

Why COTS? Although contractor-created software can be highly customized to solve particular operational problems, COTS software offers several advantages. Dozens, if not hundreds, of individuals develop COTS software, whereas a contractor-developed tool may be built by only a handful of people. This extra developer set behind the COTS tool provides expertise that benefits both the software and the end user. Furthermore, a contractordeveloped tool frequently has little room for expansion if the problem changes in scope, and enhancing it can take a significant portion of the contract budget. Therefore, precious resources are shifted from solving operations problems into software development. Along with that, contractor-developed software typically either belongs to the contractor or is not delivered as part of the contract. Therefore, when the contract is up, operations may lose a contractor—and its operational software—to a contract recompete, whereas the COTS tool remains. Because COTS provides continuity, operators become accustomed to it and can produce faster and better analysis products.

Specific Issues with COTS usage Operations groups planning to integrate COTS software into an operational environment for the first time will face several issues including: • Operations always comes first • Hardware considerations • COTS software customization • Shadow operations • Operational procedures • System setup and software installs • Network permissions, user profiles, common data files, and regularly updated data

1

• • • • •

COTS software licensing Operator/analyst training Decision criteria for operations cutover to the new software New requirements and the software update process New release integration and test

Operations always comes first This point cannot be overstated. Every operator will tell you, “No matter what happens, I cannot interrupt Ops." A typical operations environment is a buzz of activity, and stopping the center for a week to install, configure, and test new software is not an option. Installation must be done without causing interruptions, which slows transition to a crawl. So, if you look at the schedule for integrating COTS software and wonder why it's moving to the right, just look at the title of this section.

Hardware considerations Initially, the operational environment might not have enough hardware to support a COTS tool, since ops centers are frequently configured to strictly support operations and have few excess processors available. Therefore, you should look at the current hardware configuration and a plan to upgrade it if possible. Sometimes, contractor-provided software must run by itself on specific hardware. In this case, a complete hardware replacement might be necessary. Fortunately, COTS software typically runs on modern machines with popularly available operating systems. Computers are relatively inexpensive and can be purchased in quantity for substantial discounts. When buying hardware to support COTS tools, never buy the minimum or even the recommended hardware configuration. Upgrading to a superior configuration will be well worth the extra cost, because upgrading computers at this time will minimize future disruptions of operations.

Operational simulation environment One way to lower the risk of installing COTS in an operational environment is to purchase hardware and set up a “laboratory” of sorts. The COTS tool can then be installed, configured, and used as a training site prior to actual operations. As a side benefit, all future upgrades can be made in the simulation environment, assuring the software works properly. Don't be fooled, though; operational simulation environments will at least double hardware and software expenses, and maintenance costs can be high. But the knowledge gained without interrupting operations can be priceless.

COTS customization While COTS software significantly reduces overall contract costs for operations, some customization is generally necessary to meet specific needs of each environment. The amount will depend on the operations performed and the operators’ skill levels. If the operational environment is mainly an analysis shop with few repetitive tasks, customization can be minimal, since COTS tools are best in these situations. On the other hand, if many of the same products are produced daily, more customization is likely, since a COTS tool cannot know what daily products are necessary, and should be 2

configured to allow as few keystrokes as possible for the operator performing repetitive tasks. Daily product generation can become tedious, and after an operator becomes accustomed to the process, attentiveness may drop. This problem can be compounded when the process involves multiple steps, so minimizing steps in daily procedures decreases the chance for error. A second reason to customize for differing skill levels is that highly skilled operators may create scripts to automate the tool’s behavior or for defining complex reports on several criteria from multiple data sources. Typically, these customizations will be created by individual operators and used for their own purposes. The usual sharing of scripts and analysis reports may occur and will allow other skilled operators to become more fluent with the tool. This is the lowest-cost customization—it will happen naturally over time, as each operator defines his or her own special tools. Less-skilled operators may use customizations developed by others. Usually, this will be a simple interface sometimes referred to as the "One Big Button" approach, since it does much of the work with a single button. Ideally, this would completely eliminate human error—assuming the customization took every possible operational case into account. Three key points to note: A skilled set of individuals conversant in the ops environment must create the custom analyses, reports and graphs; extra people cost extra money; and the customization requires extra time for development and testing. Typically, several cycles of revisions occur to customize engineering and report/graph creation, ops review, ops comments and changes, reengineering, ops re-review, etc. Obviously, skilled operators will cost more than less-skilled, but the money saved on labor may well be spent on customizing software, additional training, and a loss of productivity.

Shadow operations Typically, running real operations from the operational simulation environment will be difficult at first. To test the new software and hardware, operations must be performed on both old and new tools. This is called shadow operations or shadow ops. The operator on duty or someone next to him or her performs the same duties on both systems until everyone becomes comfortable with the new system. Though the operators would have been trained in the use of the tool by this point, shadow ops provides direct experience in running the tool day-in and day-out. Implementing shadow ops requires additional hardware, which can cause problems if space or budgets are limited. Configuration issues must also be addressed—will the new hardware be included on the current network? Will it be on a new network? How will shadow ops be run? By doubling the work of the current operator? Adding more people to the ops schedule to handle the work? How will all of the operators become qualified on the new system? A good shadow ops plan must address all these issues.

Operational procedures Most operational environments have procedures that describe in detail how to produce a particular analysis or output. In transitioning to a COTS tool, these procedures will have to be updated. The high-level processes that describe required elements for a particular 3

product may be generic enough that they do not need to be changed, but low-level procedures describing steps to create the product will have to be altered. If the desired products are those that the COTS already creates internally, the COTS help system may have tutorials or step-by-step procedures in place. Otherwise, processes will have to be created from scratch. The basics can be developed by running the tool on the operational simulation environment and generating the procedures there. A final review by the operations crew in shadows ops is still mandatory, and redlines will arise in ops. These procedures must be completed prior to transitioning to the new tool. One obvious reason is that once operators use the new tool, the procedures serve as a reference to the necessary steps, but the development of them also helps the operators become more familiar with the tool.

System setup and installs If you're going to use an operational simulation environment, most of the following can be worked out there and applied when the operational environment is installed. If not, you will have to take the system install and setup a step at a time. Typically, COTS software installations are simple: Put in the CD and let it run. An install usually requires the installer to be an administrative user. One of the first things to test is how the software works when a typical user logs on; do they have administrative rights? If not, does the software allow them to run, create, and save needed data? This is critical —if users can't write required data, they will not be able to use the tool. Few system administrators will be willing to change the operator's privilege level to allow administrative privileges, so additional changes may be required before the tool can be used for the first time. If the operational environment is collaborative, a shared network directory may have to be established and the tool must be configured to use it. Settings within the tool, itself, may have to be adjusted to allow for operations-specific situations. All of these points should be thought out in advance so operators do not have to make these decisions on a day-by-day basis . Having a complete plan in place will ease the transition. Additionally, having an operational simulation environment will make these steps much easier on operations. All of the kinks can be worked out in advance and lead to a much smoother transition when the environment is configured in ops.

Details Details referring to the following topics are discussed: • Network issues • Centralized directories • User profiles • Common data files • Regularly updated data

Network issues The network in which the operational tool functions may have limitations that cause it to malfunction. Though rare, these instances may occur. For instance, server permissions

4

may prohibit an operator from retrieving a file off the FTP server and cause an error in the tool. Therefore, be sure to check ALL operational functions prior to transition. If the simulation environment can be set up to mimic the operation network, this may be of some help in diagnosing problems. The bottom line, however, is the software has to work in the operational environment. Network speed is another issue. Does this environment transfer a lot of data across the network? Do file servers have sufficient processing capacity? Matching network speeds when switches are used is also important. If several machines are connected with different switches, and one is lower speed than the others, network traffic will bog down at the bottleneck.

Centralized directories A centralized directory structure may be required as a source for all shared application data. Consider the location of the centralized directory: Will the hardware support it? Be sure it is not on a machine that will be doing long calculations. Also, make certain every operator has permission to read or write the data as necessary. The following sections further outline typical behavior for centralized directories.

User profiles In a Microsoft Windows networked environment, operators may or may not have roaming profiles in which the logon profile (including My Documents contents, application settings, screen layout, etc.) will be copied from a central directory each time they logon to a machine. This configuration can have unintended consequences. If the COTS tool places any user specific data in the “My Documents” area, the data will be copied each time the user logs onto a different computer. If the tool places a lot of data in that user area, logging on to the computer can take extra time. This can be worked around using one or both of the following methods: 1. Disable roaming profiles for operators. 2. Change the definition of the User Home area for the tool. This option will allow the network manager to have the operators’ data stored on each machine in a location not associated with the user's profile. The down side of this is that the operator, when traveling from machine to machine, will not have identical copies of the data on each machine. This can be overcome by creating the user area for the tool on a shared network drive, allowing the user to have all shared data available from any machine and not have the data copied as part of the logon profile. One possible downside is that the tool may take slightly longer to initialize as it may have to read data across the network instead of from the local drive to startup. Also, if the network drive is not available, the tools startup and operation may be hindered. These situations should be tested prior to transition.

Common data files COTS analysis tools commonly have a data set that can be used by everyone on that computer. To make this available to users on a centralized file system, it must be moved to a central area. The tool's configuration options will likely have a setting to define a 5

centralized location for the data. If not, the tool manufacturer will be able to define this setting.

Regularly updated data This kind of data is used to update models that change as a function of time. These models may include inertial-to-fixed coordinate transformations, ionospheric, and tropospheric models, and satellite positioning data. Other regularly updated items might include analysis models produced by the tool manufacturer and software patches. To access to this data, the operator must use the Internet. Prior to transitioning, operators must have a complete understanding of the types of regularly updated data the tool requires and what happens to the fidelity of the outputs if the data is not updated. If the operator is in a classified environment, a connection to the Internet will be ruled out. Further procedures will have to be put in place to download the data from an unclassified location and then brought to the classified environment and placed in the appropriate directories. The execution frequency of this procedure will depend on the analysis being done and how often the external data is updated. Downloading and installing software patches into an operational environment by operators is highly discouraged. An operational environment must use a known, configuration managed suite to produce consistent, repeatable, and correct results. If one operator installs a software patch from the Internet and another unknowingly uses the software and achieves different results, the analysis is not repeatable, and the operator does not know why. Having the administrator download the patch and the operators review all of the release notes and upgrade issues will ensure all are aware of the changes expected from the release.

Licensing The first rule of thumb in operational environments is, “If it gets in the way get rid of it.” That applies to licensing. Different COTS tools will have different mechanisms for licensing—some have dedicated licenses for a specific machine, some dedicate licenses to a specific user, and others may have a network-based licensing scheme where a user logs onto a network and gets a license issued by a network license server. In this case, when the maximum number of licenses is reached, no one else can use the tool. When choosing which licensing scheme to go with, be sure to test different configurations. At first glance, a network license server may seem logical; a user can sign on to any machine and use the tool. This removes the machine dependence on licensing and even if only one license is purchased, it can be served to any machine on which the tool is installed. The risk in this approach is the network server. What machine will it be on? Is it on an operational machine used daily? How often is it rebooted? Does the license server still work after its rebooted? What about users who have licenses checked out when the license server is rebooted; are they unaffected? These questions must be balanced against having a computer-specific license. How inconvenient is it to have the tool on just one or two computers using a node-locked license—given that there will be no license server issues? Licensing protects the COTS developer from unauthorized use 6

of their software. In any operational environment, this necessary constraint must be kept as invisible as possible—if it's in the way, the operators will get rid of it. The hassle of licensing issues interrupting operations could make or break the operators’ desire to use the tool.

Operator training Before the tool can be used operationally, every operator should be fully trained on all aspects of the tool that operations will use. A contractor or an individual who is assigned to handle operator training may be involved. The COTS developer will probably have training as well, but this will likely be more general and not operations-focused. The person who trains the operators should be familiar enough with operations and the new tool to answer any questions about how the tool will be used operationally. Aside from the technical content, which will be operation specific, the training logistics must be discussed. Operators do not have much time for additional training. This can be resolved through either filling in the operator's position while the operator is in training or asking the operator to attend training outside of working hours. Trying to train someone on a new tool while operations are on-going is not wise. Operations will be interrupted, and the student will not be in an environment conducive to learning. In fact, the operator may get involved with an operational situation and stop the lesson altogether! The training may be conducted in the simulation environment mentioned previously. If available, it could be the best environment. COTS vendors can sometimes be asked to create specialty training for a particular group. They can come to the operations site and train with lessons geared toward the work performed at the operational site, but not usually with the level of operational knowledge required. This maybe the correct choice if the schedule is tight and operations specific training is not possible.

Specialty features The tool training must be covered in enough depth to allow the operator to use it on a daily basis, including all operations that may be performed during the shift. In addition to daily operations tasks, special analysis tasks may also be required, which will necessitate the use of additional tool features. While training for these features should also be given, it will not necessarily be required prior to the transition of the tool for operations. One person who is capable of performing the additional analysis may be enough until a more detailed training class can be developed. Once in use, the on-the-job training that is inherent in the operations environment may be all that's needed for advanced features and usage.

Recurring and new release training After operations transitions to the tool and all operators are familiar with it, recurring training may be necessary. If an operator leaves for an extended time, a refresher course may be required. Another regular training event may be required when new software releases are delivered. These releases will have new features that may help the way operations are done or make it easier to analyze operational issues. Training operators on

7

these features can be done before or after installing the upgrade unless the upgrades are significant and require operators to rework operational steps.

Conditions for final cross-over to operations No set way exists to determine when an operational group is ready to switch from the legacy tool to the COTS tool. This decision is operation-dependent and must be made by the operations lead. Typically the following conditions will be met first: 1. Operators have all been using the tool and are comfortable with it for a sufficient amount of time (weeks rather than days, possibly months depending on the nature and type of operations). 2. The tool is stable. This means there are no crashes; all operations specific enhancements have been installed; and tested multiple times by all operators in all aspects of operations. 3. Any missing features have been discussed, and a plan for implementation must be made. As operators use the tool, they will find features that they would like, and they may not be able to easily distinguish a “need” from a “want.” The operations lead must understand what is required and what can be moved to the next software release. This is important for two reasons: First, operations must move to the new tool in a timely manner. When the plan to convert is in place, following through rapidly is essential. Adding new features adds development, test, Ops checkout time, and introduces risk into operations. Also, the COTS vendor has to follow schedules for new releases, and the more he or she caters to a specific customer, the more time is lost for supporting new features and other customers. The main point is that nothing should be new to the operators. They should be very familiar with the tool and become bored while performing shadow operations with it and the legacy tool. When the above three conditions are met and the users can work with either the legacy or COTS tool interchangeably, then the cutover decision can be made easily.

8