15179_erecon_software_development_at_hospital_corporation_of_america-1553395398.pdf

W16110

ERECON SOFTWARE DEVELOPMENT AT HOSPITAL CORPORATION OF AMERICA

Leanne Rayo wrote this under the supervision of Kenneth J. Klassen, solely to provide material for class discussion. The authors do not intend to illustrate either effective or ineffective handling of a managerial situation. The authors may have disguised certain names and other identifying information to protect confidentiality. This publication may not be transmitted, photocopied, digitized or otherwise reproduced in any form or by any means without the permission of the copyright holder. Reproduction of this material is not covered under authorization by any reproduction rights organization. To order copies or request permission to reproduce materials, contact Ivey Publishing, Ivey Business School, Western University, London, Ontario, Canada, N6G 0N1; (t) 519.661.3208; (e) [email protected]; www.iveycases.com. Copyright © 2016, Richard Ivey School of Business Foundation

Version: 2016-02-08

At 10:00 a.m. on July 28, 2014, the eRecon technical team at Physician Services Group (PSG) in the Hospital Corporation of America (HCA) gathered in the main conference room, just as it did every Monday morning. The goal of the meeting was to review the project plan and update task completion for the team’s two-year undertaking — a six-phase, in-house development of a custom web-based accounting software called eRecon. On this morning, as the team members began to walk through each task and update the percentage completed to date, they grew increasingly concerned about obtaining critical source data that they required from the human resources (HR) department. They were approaching the deadline for that task, and it was unclear what the consequences would be of not obtaining the data on time and what impact the delay would have on the next phase’s completion deadline of December 8, 2014. There was a large degree of uncertainty surrounding the process of completing the task. The project managers, Frank Lafferty and Leslie Ray, arranged a meeting to analyze the project plan and promised to report back to the team with the results of their analysis. It was crucial that they determine how the delay in obtaining HR data would affect the project and the 125 end users who were patiently waiting on eRecon’s completion. HCA AND PSG

HCA, a publicly traded company founded by physicians in Nashville, Tennessee in 1968, was one of the largest healthcare companies in the United States. HCA owned and managed hospitals, surgery centres, and physician offices in 20 states across the country. Due to numerous mergers and transactions, it became a publicly traded company on three separate occasions between 1969 and 2011. As of 2014, the company was traded on the New York Stock Exchange with the ticker “HCA.”1 In 2013, HCA reported US$34.2

1 HCA Management Services, L.P., “Our http://hcahealthcare.com/about/our-history.dot.

History,”

HCA

Healthcare,

accessed

December

6,

2015,

Page 2

9B16D002

billion in revenues and a net income of US$1.56 billion (see Exhibit 1).2 HCA was comprised of over 2,000 subsidiaries, one of which was Physician Services Group (PSG). PSG employed approximately 13,000 people and was designed to bring business solutions to medical providers through ownership or management of physician’s offices, giving doctors the ability to focus on the most important part of their work: treating patients. HCA was patient focused, and the culture of the organization (including its subsidiaries) was primarily driven by principles of ethics and compliance. One of the most pertinent laws to every healthcare company in the United States was the Health Insurance Portability and Accountability Act (HIPAA) privacy law.3 The rules established within this law were carefully followed in every business decision made by employees at each level of the organization, without exception. Every employee was trained annually on the privacy law. A major component of the law was a directive on the use of protected health information (PHI). PHI included identifiable demographic and genetic information, among other personal details relating to the physical or mental health of an individual, health care provided to that individual, and any payments received from the individual. HCA implemented an information protection program in order to enforce and educate employees on the privacy law. As part of this program, employees were emailed informational bulletins on a regular basis that focused on one specific issue at a time. All employees, whether corporate or hospital based, were responsible for the protection of PHI and any other information that could be considered confidential (e.g., social security numbers). PROJECT MANAGEMENT FOR SOFTWARE DEVELOPMENT

Lafferty and Ray were both educated in project management and proficient in the use of Microsoft Project. However, neither one had any prior practical experience in project management for software development. To ensure that they had a handle on creating the project plan, Ray had reached out to a more experienced colleague in the information technology department at the beginning of Phase 1, six months earlier in January 2014. In those meetings, Ray had discovered a few key details. Software developers were generally resistant to providing time estimates on tasks. The industry standard was to triple developers’ estimated schedule time4. Recommended best practice was to never schedule a task’s duration for less than one day; any tasks estimated at less than one day were rounded up. Managing a software project was a complicated endeavour. Before anything could begin, project managers and the team of individuals involved in the project had to first agree on what approach they would take for the software development life cycle (SDLC). The SDLC outlined a series of steps that were necessary for the process of creating an application or software. Most commonly, these steps included analyzing, defining, designing, planning, building, testing, and deploying. There were various different types of SDLCs; the two most popular were the “waterfall” and the “agile” methods. The waterfall approach to software development was a traditional methodology that identified the various process steps as discrete and to be completed in logical order. Projects using the waterfall method were carefully and extensively planned. It was imperative that all parties strictly followed the plan. Testing occurred once all coding was complete. 2

All figures in US$ unless otherwise stated; US$1 = CA$1.08 on July 28, 2014. U.S. Department of Health and Human Services, “Rules and Regulations 45 CFR Parts 160 and 164,” Federal Register Vol. 78, no. 1, January 25, 2013, accessed December 6, 2015, www.gpo.gov/fdsys/pkg/FR-2013-01-25/pdf/2013-01073.pdf. 4 The durations pertinent to this case have already been calculated accordingly. 3

Page 3

9B16D002

In contrast, the agile approach to software development was an incremental design process. Rather than following discrete process steps in sequential order, agile developers worked in increments, coding small portions of the application and testing each part along the way. End users had several opportunities to provide feedback and change their requirements. Unlike with the waterfall approach, planning was much less extensive when using this method. Agile development was ideal for a project that required rapid production but lacked a specific vision with respect to the final product. The executive steering committee selected the waterfall method for the development of the eRecon project. A team of 10 individuals, known as the business team, worked with Ray to define requirements for each phase of the application. Once they signed off, the technical team worked on designing the application and discussed the various technical elements. Following the design discussions, the gruelling process of obtaining schedule estimates from the developers began. After building the project plan, the developers would begin coding the application on a development server. Once a set of code was ready, the work could be published (or pushed) to a quality assurance (QA) server to be tested. Following successful testing, the code would be pushed to a production server, and the application would be live for the accountants to begin using it. Corey Young was eRecon’s primary developer on Microsoft’s .NET framework. He was a senior developer who had been with the company for five years. Samuel Mendoza was the developer responsible for database management. Mendoza was hired as a developer in the spring of 2014. Both individuals brought over 10 years of experience to the project and were well respected in their field. Although it was evident during Phase 1 of the project that both men were uncomfortable with the waterfall methodology, Young appeared especially resistant to the concept. It was unclear to Lafferty and Ray where the resistance was coming from, but as project managers, their job was to obtain estimates and build a project plan for each phase. The steering committee depended on their plans to identify the deadlines for completion and effectively communicate them to the accounting department. eRECON

eRecon was a web-based application designed for accountants and managers of PSG to prepare, review, and document balance sheet reconciliations and complete necessary certifications on a monthly basis (see Exhibit 2). The application created several efficiencies for the accountants at PSG. It eliminated the need for spreadsheets and manual processes and thereby reduced the chances of errors, omissions, and other risks. In addition, the entire review and approval process was conducted within the software, so date and time stamps were recorded and retained for future reference. The system offered real-time status reporting that significantly improved visibility for managers, allowing them to make better decisions and lead more effectively. Finally, eRecon included a robust reporting module. Prior to eRecon, the accountants prepared balance sheet reconciliations in Excel or other tools and printed the reconciliation cover sheet and all supporting documentation. Each reconciliation was then submitted to the appropriate reviewer who would review it for accuracy and physically sign the cover sheet. At the end of each month, all reconciliations were compiled by facility, using the company identification (CoID) code. A cover sheet summarizing the accounts in each CoID was printed and placed on top of each pile of reconciliations. The packages were then submitted once more to the appropriate reviewer, who checked to see that all reconciliations were completed and contained the appropriate level of review. After the final sign-off, the reconciliation packages were filed for future retrieval in physical filing cabinets. After one year, the files would be boxed up and sent for storage at Iron Mountain, a secure records management company, where they were retained for seven years.

Page 4

9B16D002

Initial discussions pertaining to the eRecon project began as far back as November 2012. However, the team did not begin documenting requirements until October 2013. Around that same time, three Structured Query Language (SQL) servers were purchased at a cost of approximately $6,000 each. The three servers were to be used as the development, QA, and production machines. They would house the back end of the application, which included databases, tables, and stored procedures (SPROCS) that were critical for providing data to the front end of the application (e.g., general ledger balances and user names), and capturing and archiving data entered on the front end (e.g., reconciling items and review notes). Initial estimates put the project on a two-year completion deadline. The project was divided into six distinct phases to more effectively manage the development and rollout to the 125 potential users. Phase 1 was denoted “global approvals.” This phase included the creation of the project’s home page and the global approvals page, which was only accessible by directors and the vice-president (VP) of financial services. The concept of global approvals was exciting and unique to PSG. Effectively, management figured out a way to reconcile and review over 12,000 of the 35,000 general ledger accounts with the click of a button that still satisfied internal and external audit requirements (see Exhibit 3). To do this, low-risk groups of accounts were identified by management. Periodically, management reviewed the current groups and added more groups of accounts that were determined to be low risk. As long as a source document was available and could be generated electronically, the account was eligible to be classified as globally eligible (i.e., low risk). Every month, each low-risk account would be tested (in the back end of the software) to determine if it qualified for global approval for that fiscal period or whether it would have to be traditionally reconciled. The software checked a number of criteria, most of which were related directly to HCA accounting practices. For example, the account could not have a variance between the general ledger (GL) balance and the source balance. Global approvals were completed by the VP on the 11th day of each month, the month-end-close day of the company. At PSG, the financial books for the prior month were closed in 10 calendar days. Therefore, by the 11th day, the GL account balances were final; no further transactions (i.e., journal entries) would occur after that day. Phase 2 of eRecon impacted the larger audience of end users and was labelled “reconciliation and filter.” The goal of this phase was to create one additional page called reconciliation. This page would include three primary sections on one screen (see Exhibit 4). The section on the top left was where the user chose the filter, the top right contained the filter results, and the bottom displayed the reconciliation. The filter allowed users the flexibility to navigate to their assigned work by populating a list of CoID accounts based on the parameters set by the user. Each filter selection impacted what options would be available in the remaining filter fields. For example, if users selected their own name, only the CoIDs and accounts assigned to that user in the database would be populated in the CoID and account filters. The filter results listed records containing the CoID number, account number, and GL balance. The number of records returned was based on the query executed by the “apply filter” button. Although only nine records were displayable in the results area at one time, users could scroll to see more rows. A total record count located below the results gave users an indication of how many records were returned by their filter. Filter results were sortable by clicking the column headers. The reconciliation section displayed detail of a reconciliation (which was prepared at the CoID account level). Header information, including the account number, name, period end (PE) date, and GL and source balances, was populated by the system. A details button located in the header opened a pop-up window that included requirements for that account, according to company policy, and spaces for users to enter helpful

Page 5

9B16D002

information and key contacts. Beneath the reconciliation header was an area for users to add reconciling items and to conduct their reconciliation. Each reconciliation item line included a PE date, description, amount, and note field. The amount was critical to calculating the unreconciled variance. The unreconciled variance was required to be zero in order for a CoID account to be considered completely reconciled. The formula for the unreconciled variance was as follows: Unreconciled Variance Variance Sum of Reconciling Items where, Variance GL Balance Source Balance In this phase, users only had the ability to save or print data. Workflow and approvals, however, were not within the scope of this phase. Phase 3 was intended to replace an existing reconciliation tool called the “post close review guide” (PCRG), so this phase was known as the PCRG. It was the biggest phase in terms of user interface development. The PCRG was a Microsoft Access application (both front end and back end), primarily used for recording completion and approval of reconciliations and reporting. In this phase of eRecon, enhancements to the reconciliation page were necessary, in addition to the creation of two new pages: PCRG and “monthly financial certification.” Users were able to prepare reconciliations and submit them for review. Reviewers subsequently reviewed and then chose to approve or reject the reconciliations with notes explaining what actions were needed. Once preparers responded to those notes and resubmitted them, reviewers checked the reconciliations again and then approved them. At this point, the CoID account was considered completed. PCRG was a review of the balance sheet accounts at a CoID level; this could not happen until all accounts that rolled up to a CoID were approved. Once all CoIDs in a division were approved on the PCRG screen, managers were able to perform reviews at a division level on the monthly financial certification page. Upon the directors signing off on every division, the entire workload within the application was considered complete for the month, and it was locked to any further changes (see Exhibit 2). Phase 4 of the application development was called the “support store.” This phase included adding functionality to the reconciliation page, allowing preparers to attach documents to reconciliations as supporting documentation. Reviewers would then have everything they needed in one place to approve the reconciliation. Phase 5 was called “journal entries.” For this phase, the business team wanted the ability to create a journal entry directly from a reconciliation item line. Just by specifying that they wanted to create an entry and then entering the offsetting CoID and account, they would let the system know to populate the two-line record and upload it to the general ledger, once approval was received. Phase 6 was classified as the “home page and enhancements” phase. The home page, already created in Phase 1, was enhanced to include a dashboard that consisted of an individualized summary of a user’s assigned accounts with corresponding status of each account. One example of the enhancements written into this phase included automatically attaching supporting documentation to the support store for some accounts.

Page 6

9B16D002

PHASE 2 — RECONCILIATION AND FILTER

Phase 1 of eRecon, global approvals, was completed on June 6, 2014, when it was successfully rolled out to production. The VP of finance and the directors were the only end users impacted by the rollout. The next working day, development of Phase 2 began. There were two primary components to this phase: (1) SQL server database setup and (2) Microsoft .NET user interface development. Mendoza was assigned primary responsibility for SQL database tasks, and Young was in charge of the user interface. Young was assigned to the eRecon project on a full-time basis, while Mendoza worked on eRecon-related tasks for approximately two-thirds of his work week. SQL Server Database Setup

The SQL server database setup was critical to Phase 2 of eRecon development. Before Young could push the application’s user interface to QA for testing, Mendoza had to complete the “shared” and “security” databases. Although it was important that these two databases were both completed prior to eRecon, the actual order in which they were completed did not matter. The team set a goal of completing one of the databases by the end of August and the other database by the end of September, so Mendoza decided to work on the shared database first. Shared Database

The shared database contained four main tables: “company master,” “account balances,” “general ledger account list,” and “journal entry transactions.” For each of these tables, it was necessary to prepare the source data (two days), develop the optimal load (two days), and finally load and validate the data in the development environment (two days). Once the four tables had been validated, they would all be pushed to QA for testing at the same time. Pushing the shared database to QA would require five or six days, which included creating and submitting a work ticket to the systems group in the information technology and services (IT&S) department (one day), pushing to QA (one to two days), validating performance and data (two days), and verifying the load (one day). The tasks had to happen sequentially. The IT&S department was responsible for pushing to QA. Essentially, this meant that Mendoza provided these workers with scripts as a part of his work ticket that they would then execute on the QA server. Segregating duties in this fashion was important for change control, which was enforced by audit. IT&S scheduled all pushes for Tuesdays only. Although only one day was actually necessary to complete the push, it could take up to two days for the systems group to execute the scripts, depending on its current workload. It was not uncommon for the technical team’s requests to get delayed by one day because eRecon was not considered a high-priority application by IT&S. While Mendoza was assigned to validation, best practices called for a separate individual to verify data. For the shared database, Mendoza’s manager, Michael Craft, would accept responsibility for verifying the data. Craft was available to work on the project for 20 per cent of his work week. Once the database was tested in the QA environment, it could be pushed to production. The tasks included in this push were the same tasks required to push to QA, so it would take the same amount of time (five or six days). At the July 28 project meeting, the team confirmed that the shared database was 100 per cent complete, and they were ready to begin work on the security database.

Page 7

9B16D002

Security Database

The Security database included two large tables — “employee information” and “employee fact” — in addition to several other small tables, categorized as system or base tables for the purposes of project planning. However, before any work could begin on the tables, a few items needed to be addressed. First, the technical team, which included Lafferty, Ray, Young, Mendoza, Craft, and Daniel Mayer, needed to finalize table-naming conventions. They could do this in one day. Second, Mendoza would merge the existing security tables. He also needed one day to do this. In the meantime, Young needed to gather the minimum technical requirements for the security database. This was specific to the filters section of eRecon and was not expected to take longer than one day. Once requirements were gathered and the tables were merged, Mendoza would create the views and SPROCS necessary for eRecon to operate properly. This could be completed in three days. The process for completing the two primary security tables (employee information and employee fact), plus the system or base tables, was the same for each one. It did not matter in what order these tables were completed. The first step for each table was to identify, obtain permission for, and extract the pertinent raw data from the appropriate sources. Mendoza was responsible for completing this for the employee fact table and for the system or base tables, which would take two days each to complete. Daniel Mayer, one of the managers on the technical team, was responsible for getting the raw data from HR for the employee information table. He was committed to the project for 50 per cent of his time, and was a valuable asset to the team overall; the group was relying on him to obtain this data from HR. At the time of the July 28 meeting, Mayer reiterated that the task was more complicated than it appeared because the team would need to work through a significant amount of bureaucracy to gain access to the system that contained the information. Mayer had started working on it several months earlier to get a head start, but even with the approval of the chief executive officer, it was not yet complete. He was confident, however, that if he went back at it by the next Monday (August 4), he would have access to the database with the source data within 10 to 15 working days. After obtaining the raw data for each table, the technical team was then responsible for defining the scope and the business rules of the load. At minimum, this would take two days for each table but could take as long as one full work week. Software called SQL Server Integration Services (SSIS) was being used for all extract, transform, and load processes. This was necessary to load the millions of records of data that were required into the security and shared database tables. The next step in completing the security tables was to build, validate, and test the SSIS packages. This would take Mendoza three days per table. Once all packages were tested, a push to QA could occur. The tasks would be exactly the same as the push to QA for the shared database. Once that was completed, the content could be pushed to production. The steps for this process looked identical to the shared database push to production. Once it was pushed to production and validated, the security database would be considered successfully completed. Microsoft .NET User Interface Development

One of the most exciting properties of the eRecon application was the web-based front end. Applications with a Microsoft .NET front end were more appealing to users because they provided a more sophisticated user experience than other platforms (e.g., Microsoft Access) and gave developers more flexibility in design. Overall, they were more appealing and better functioning. The user interface would be primarily completed by Young. For purposes of project planning, he organized the development work into five sections: interface, account grouping, batch process, security, and archive. After the five sections were

Page 8

9B16D002

complete, a push to QA would occur, with at least one month for user acceptance testing, before the application could be rolled out to the production environment for the end users. Young decided to create the front end using the model-view-controller (MVC) design pattern, which was commonly used by software engineers. For eRecon, the controller was a web browser, the model was Hypertext Markup Language (HTML), and the view was SQL code. In order to complete the MVC for the interface section, Young needed between eight and 12 days but could not begin until the security database was complete. Once that database was completed, he would need to persist with the data (four days) and complete the client scripting (10 to 15 days). Unit testing would then be complete. Young required a minimum of five days and a maximum of seven days to complete it for the interface section of development. To complete the account grouping section (which could begin as soon as the shared database was complete), Young needed 10 to 12 days. The batch processing section would be completed by Mendoza, and it did not require any prior work to be completed. This included creating a process for the reconciling item carry over process and updating the current account creation process. The batch process section would take 10 days to complete. The security section, which could not be started until the security database was completed, would entail an MVC that took Young approximately two days, and coding the service and repository, which would take Young three to five days. Then, unit testing would take 1.25 days for the security section. Young also needed to complete the archive section, although there were no prior tasks for that. This included creating the archive table (one day) and modifying the SSIS package to copy or clear data (one day). Once all five sections were complete, the push to QA could begin (using the same steps as in previous pushes). The push to QA required Young to submit a work ticket to the systems group in IT&S (one day). On the next Tuesday, the group would complete the push. Before the month-long user acceptance testing could begin, Ray needed to produce the test plan and test scripts, which would be distributed to the testers. This would take her one to two weeks to complete. She could begin working on this as soon as Young submitted the work ticket to push the application to QA. Ray was in charge of coordinating the user acceptance testing during the month, gathering feedback and addressing security issues. Once the testing was completed, the application’s front end could be pushed to production. The process was similar to the push to QA. After Young submitted a work ticket, the systems group would push to the production environment (on a Tuesday). The application would be briefly tested (one day) by Young and Ray and then finally rolled out to the end users. CONCLUSION

Lafferty and Ray were concerned about Mayer’s ability to obtain the HR data that the team needed to complete the security database. It was evident that several tasks pertaining to the user interface depended on this information to be complete. They all believed that it would not be much longer, but as was often the case in a corporate environment, when working with another department, there were too many uncertainties to be able to estimate accurately how long a process would take. The two project managers needed to determine how much slack they actually had built in for this task to be completed without delaying the Phase 2 finish goal of December 8, 2014. Accounting would begin reconciling their November balances on December 9. Therefore, even the slightest delay would postpone the rollout to the next month. Fortunately, Lafferty and Ray knew that they had some flexibility in the development tasks. Anything other than the actual pushes (to QA and production) and testing could be scheduled in increments and split up over several different work sessions; they could even be done over a number of days or weeks. The team’s reputation and future phases of eRecon depended on a successful completion of Phase 2.

Page 9

9B16D002 EXHIBIT 1: INCOME STATEMENT CONSOLIDATED INCOME STATEMENTS FOR THE YEARS ENDED DECEMBER 31, 2013, 2012, AND 2011 (US$, except per share data)

Revenues before the provision for doubtful accounts Provision for doubtful accounts Revenues Salaries and benefits Supplies Other operating expenses Electronic health record incentive income Equity in earnings of affiliates Depreciation and amortization Interest expense Losses (gains) on sales of facilities Losses on retirement of debt Legal claim costs Gain on acquisition of controlling interest in equity investment Termination of management agreement

2013 $38,040 3,858 34,182

2012 $36,783 3,770 33,013

2011 $32,506 2,824 29,682

15,646 5,970 6,237 (216) (29) 1,753 1,848 10 17

15,089 5,717 6,048 (336) (36) 1,679 1,798 (15)

13,440 5,179 5,470 (210) (258) 1,465 2,037 (142) 481

175

Income before income taxes Provision for income taxes Net income Net income attributable to non-controlling interests

31,236 2,946 950 1,996 440

30,119 2,894 888 2,006 401

(1,522) 181 26,121 3,561 719 2,842 377

Net income attributable to HCA Holdings, Inc.

$1,556

$1,605

$2,465

$3.50 $3.37

$3.65 $3.49

$5.17 $4.97

445,066 461,913

440,178 459,403

476,609 495,943

Per share data: Basic earnings per share Diluted earnings per share Shares used in earnings per share calculations (in thousands): Basic Diluted

Source: HCA Holdings, Inc., “10-K: For the Fiscal Year Ended December 31, 2014” EDGAR Online, accessed December 6, 2015.

Page 10

9B16D002 EXHIBIT 2: eRECON WORKFLOW

Source: Company files.

EXHIBIT 3: GLOBAL APPROVAL SUMMARY

Source: Company files.

Page 11

9B16D002 EXHIBIT 4: RECONCILIATION AND FILTER PAGE

Source: Company files.

EXHIBIT 5: ANNUAL SALARY DATA Title 

Source: Company files.

Starting Annual Salary Range  (in US$) 

Project Manager 

$60,000 to $70,000 

Developer 2 

$60,000 to $70,000 

Senior Developer 

$70,000 to $80,000 

Manager  

$80,000 to $90,000