Breaking The Lock

Breaking the Lock 1. The first few months… I was hired to head the BCG company’s effort to setup a modern Decision Support System. The Executive VP had done his homework, or at least he thought he did, and wanted to build a Data Mart with real data and organize a proof of concept between the usual suspects vendor companies. At the onset this looked like a classical scenario: done that, been there, I was game.

The bar for what the Proof of Concept would achieve kept getting raised. We were going to build a real life Data Mart and have each software vendor build reporting applications with data from the Data Mart that we would be able to compare with the data published in the existing reporting system.

So, I sunk my head into understanding the available data elements and their relationships, identifying the existing data structures, mapping meaningful reports, starting a dimensional design, etc..

My project was assigned a bright, competent, young Indian “C” programmer who worked in the billing group since the data I needed would be sourced from specific billing tables. I was thrilled. I encountered a few difficulties creating the logical design. I decided to start the work to build the Data Mart with the tables I knew we would need no matter what and while that work would progress I would resolve my design dilemma.

The first warning sign came in the form of an apology from the programmer assigned to the project and a “heads up” from the Application Development Director. MY developer would not be able to continue working on my project for a week because of an urgent billing related hot issue.

Being a patient, understanding and accepting kind of guy, I said: “no problem”. After a few days, earlier than planned, my developer was ready to resume work on the Data Mart. And then it happened again, and again and again.

Then I realized that all the developers in the Department were actually assigned to a minimum of four simultaneous projects of various sizes.

Eventually we completed the Data Mart, the vendors came and did the Proof of Concept. But by then, the Executive VP’s interest for the whole initiative had waned. The company was in Chapter 11 and the Executive Team was busy trying to find a buyer for it.

In December, the Director of Application Development asked me if I would be willing to work as a Project Manager on general IT projects. I enthusiastically accepted.

Two months later, the executive management of the company decided to downsize. I was to layoff four developers who were reporting to me. The worse part was laying off Ms. Wang. I loved Ms. Wang. Ms. Wang is a sixty something Chinese lady who made a specialty of maintaining critical legacy applications. She was slow, no doubt about it but her code had few bugs. Ms. Wang did not understand English easily, neither did she understand the absurdity of the security procedures that a manager is supposed to follow when doing a massive layout: no contacts with other fellow employees.

So, the Director of Application Development and myself took her in his office and tried slowly to explain to her what was happening and to answer her questions until she understood. We were both crying and Ms. Wang was hurt and angry. I felt like I was firing my own Grand-Mother.

Rumors were rampant. Even before the lay-offs Fear was running high. Now, the question on everyone’s lips now was: I am next ? Soon after, the Executive Management decided to set a firm deadline to the top 4 most critical IT projects underway. The projects were to be completed by April 15th and two of those key projects were mine and they had barely started. The developers attitude was: What happens on April 15th ? What is the point for me to work long hours on these projects to deliver them by April 15th if after that I am going to be laid off. Two key developers decided to leave. The company pulled all the stops to keep them on board. For one of them it worked. The other decided that it was time for him to move on. Moral was low among developers, even after a pep talk by the COO and the CEO. 2. The Boss Dr. Santosh Anand who had the title of CIO ran the Department. Dr. Anand was the company’s first IT employee shortly after the company was founded. It was Dr. Anand first job upon graduating from a prestigious Indian University.

Dr. Anand managerial style can be characterized as “benign dictatorial”. Dr. Anand ‘s day is shared between executive meetings, long periods of time spent coding specific software components and design review meetings with specific developers. Dr. Anand is generally a mild mannered individual except when he looses his temper. At such times, Dr. Anand favorite sentence is: “This is how it is and if you do not like it, feel free to look for a job somewhere else.” Based on the uniqueness of the tools necessary to run the company and on a desire to save money on expensive specialized packages, Dr. Anand has had a long-standing policy of expanding the efforts necessary to build custom software than buying commercial software packages. As a result, the company had never bought any significant commercial software package. 3. The IT Department

The Software Development Group was dominated by Dan Thomas. Mr. Thomas spent 12 years working as a software engineer at one of the US Army labs, where military style command was the preferred mode of operation. Mr. Thomas’s favorite put down for non-conforming developers was: “this is no rocket science”. This would be followed by accusations of building over-complicated, theoretical code directly copied from academic books with no contact with day-to-day business reality. As senior developer/architect in the IT Department, developers felt compelled to follow Mr. Thomas’ software design instructions, even though they might be riddle with inefficiencies and devoid of any event logging and/or error handling and recovery capabilities. The IT Department day-to-day reality was dominated by feverish efforts to meet deadlines imposed by executives. Project commitments are made by IS Managers before detailed requirements are gathered and before a requirement analysis/design is performed. In this context, somehow, the IT Department delivered 90% of its projects on time. Further analysis revealed that most of the projects delivered were short to medium duration projects with levels of effort below the 100-man hour level. Often, these types of projects were modifications to existing legacy applications with minor bottom line impact. Commitments for Enterprise projects were treated in a similar fashion as smaller projects: IS Managers would commit to a delivery deadline imposed by executives before detailed requirements were gathered and before a Software Engineer could work through an analysis and a design of the project requirements. Typically, imposed deadlines on Enterprise projects had no relationship with the actual amount of work necessary to meet the deadlines. And typically, deadlines on large enterprise projects came and went without being met. Over time, some large enterprise web based projects were delivered although none of them ever met the expectations of their customers. Software Engineers usually worked in isolation on smaller projects. Dan Thomas frowned upon developers that paired up to work a design or a specific issue as a team. When software development teams were created to tackle enterprise projects, they tended to operate in silo, isolated from other project teams working on interdependent projects. Soon, an attitude of “us vs. them” developed, further fueled by rampant finger pointing and public put-downs by project managers from interdependent projects. Things were turning ugly.

4. Methodology and Technical Environment. Having attempted to understand part of the Production Database data structures, I realized that no shared knowledge repository existed; no organized knowledge sharing practices existed and little to no application documentation was available. Although, software developers cooperated willingly together, knowledge of business rules, legacy database tables, and specific application process existed only in the head of specific software engineers. No process existed to evaluate software development practices and tools. No organized software development training program or continuous education curriculum was offered to developers. No differentiation of roles between developers existed, nor was any career path defined or offered. Software engineers were expected to be their own business analysts, their own designer, their own coder and their own testers for their projects.

Typically, a scope document was written and approved by stakeholders. Often, but not always, end users would be in charge of providing requirements for the proposed system. Software Developers rarely provided Design and Analysis documents. Often end-users would drastically change requirements mid-course through a project development phase and no Scope Change-Control process existed. Design reviews and code reviews occurred sporadically.

Two technical environments were in use: Testing and Production. Some developers developed and tested their code on their own box, others on the test server. Prior to deployment in the production environment, new software went through unit testing and variable levels of internal customer acceptance testing on development servers. Test plans were seldom written as part of the application design documents and no automated testing tools were available. Testing was done by each individual coder or by other developers. No formal testing group was in place. No code version control was in place. Typically it was assumed that the latest and greatest code was the code in production.

Managers justified the expediency of organization’s mode of operation by the urgency and the pressure put on them by the Company’s business executives.

In an effort to minimize computing costs, Santosh Anand bought used servers and “offthe-shelf” Intel based servers and, a few years back, had decided to move from UNIX to the Linux platform. With that decision, Java and JSP were the new logical choice as preferred development language. Legacy applications were written using C and PowerBuilder. It is important to notice that new functionalities were continuously added to existing legacy application. In spite of Dan Thomas’s best efforts to build Java based applications, core business functionalities were provided by legacy applications. 5. Impact of the culture and mode of operation 1. Most individual software developers chose to simply submit to Dan’s design decisions even though they disagreed with those decisions. The rationalizations given for this position included: Expediency: considering the pressures for delivering the application, the proposed design is the shortest road to meeting the imposed deadline. -

Let’s not rock the boat: this is how it is being done here.

Other software developers chose to comply while resenting that decision. These individuals mentioned that they knew the proposed designs were fundamentally flawed and felt powerless in affecting the situation in the absence of any manager or any forum in the department where their point of view would be heard. Finally, another developer chose to avoid open conflict with Thomas by concealing the fact that the applications he built did not include any of Thomas’s design recommendations. 2. Because of the speed and reliability with which applications were built in PowerBuilder, legacy applications kept growing and expanding further delaying and rendering more complex the task of replacing client-based legacy applications by more modern and efficient web based applications. 3. Occasional trainings to new technology were organized. However, because of the continuous expansion of legacy applications, developers working with legacy technology were not getting an occasion to practice and improve their newly acquired skills. 4.

Rumors were running rampant among developers.

5. Turnover rate among developers was high. About a third of the staff had to be renewed over a typical year.

6. Most large web based Enterprise projects failed totally or did not deliver the expected functionalities and the delivered applications were unstable and inefficient.

Over time, the culture of the IT Department had become a mix of free-for-all, rigid, and fear-based culture paying little attention to methodology, new concepts, practices or tools. The chaos created by the absence of consistent methodology and processes was compensated by software developers’ willingness to sacrifice inordinate amounts of their personal and family time to get applications in production. Although managers routinely criticized existing practices, they also held strongly to the belief that a culture change was not possible. In other words, we were stuck. 7.

The Black Sheep

Sandeep, a thirty something Indian developer had joined the IT Department a few months before. Sandeep was a friend of mine, he had been unemployed for a while and since the department had to replace two Java developers, I had submitted his resume for consideration and he was hired. I knew that Sandeep had a deep commitment to and knowledge of Object Oriented Programming concepts and a mind attentive to new more efficient Open Source tools whenever they would become available.

Soon conflict arose between Dan Thomas and Sandeep on various critical technical subjects. Sandeep refused to use the Dbconnector class that Dan had build because of its blaring inefficiencies and lack of error logging, trapping and recovery. Sparks were flying during impromptu technical discussions the two would have. Since Sandeep was not reporting to me, I felt there was nothing I could do. Sandeep first project was a complete re-write of a high visibility reporting system involving very complex business logic that he totally rethought and delivered on time at the satisfaction of the client. Sandeep was trouble because he was not scared to challenge the existing status quo. 7. The Opening The Opening came fast and unexpected.

As I recall it was a Tuesday morning. Santosh Anand, Dan Thomas, myself and a couple of the Java developers of the department we were together sitting in the IT meeting room discussing the need for a single set of entity objects that everyone would agree on and use. We had just discovered that several different Java classes written by different developers for different applications represented the same particular database entity !!!!!???!!!!

Nothing surprising considering what we now know of the shape the department was in.

I turned my head to face Santosh and with one glance at each other’s eyes, I knew he was OK with what I was going to say: I proposed that all the department’s Java developers meet together once every week and discuss together for 1h design or technical issues that face them as a group. We discussed what time the meeting needed to happen and we landed on the hour between eight and nine in the morning. Santosh gave his approval to the meeting and said he would participate to it as well. I would chair and facilitate the meeting. I made sure that the voice of each developer who would attend that meeting would be heard.

Dan Thomas was resistant but complied. For months thereafter and even now he still complains that this meeting was a total waste since, according to him, that hour is the most productive hour of a developer’s time. Developer’s only gripe was with how early in the morning the meeting was. 8. The Aftermath

So we met every week, week after week. We started by tackling the design of these database entity classes and little by little, all the technical issues that had been carefully ignored and that had been weighing on the efficiency and the progresses of the developers started to come to the surface.

Soon I kept a spreadsheet with a running list of the issues that were raised during the meeting. I made my agenda to consistently bring up the top issues from the weekly developers meeting to the managers’ meetings and to my meetings with Santosh so that decisions would be taken and that issues would find resolution.

Consciously I had proposed the meeting as a venue to discuss particular technical topics. Actually I had created a safe forum where developers could bring real issues, related to object design or to the inconsistencies in the technical environment, work out their disagreements with Dan Thomas and Santosh and not be fired. For the first time, Santosh heard directly from the developers what worked, what did not work and what was needed. To Santosh’s credit he did not seem attached to making sure that any inefficient artifacts that he had created be kept in use. In other words, we worked on a level playing field. 1.

The Outcome

The first major outcome was the announcement of Dan Thomas’ resignation. Dan had found another job and he was leaving. Dan’s departure opened up a major security issue, since his name and password was used as a generic user name/password for a number of major applications. A small group of developers and myself spend a number of feverish days before and after Dan’s departure plugging that security hole. * It became OK not to use some of Dan’s Java classes that were build into a number of important applications and that were inefficient and functioned heretically. * Efforts were made to move all code into a single code repository that provided version control. Four technical environments (Unit, System, CAT, Production) with their respective servers were put in place with defined boundary rules and gatekeepers. * Santosh and the IT Managers group focus on improving software development methodology started paying off. Managers were sent to methodology training, a requirements management tool was acquired. * I designed and co-facilitated a meeting where the methodology framework was presented and developers broke out in smaller groups to identify the process they thought were relevant. Developers took ownership defining methodology steps they thought were appropriate. * Developers regained a sense of empowerment. * Moral among developers improved. * The executive team decided that the company needed to get out of the business of building its own proprietary software and concentrate on maintaining existing applications and acquiring tools available on the market.

* Taking into account the lessons of the failed initial Business Intelligence/Data Mart proof of concept project, I recommended the selection of an inexpensive Business Analytics tool that does not require the building of a Data Mart to function. That tool provides complex analytical reporting after a few days of efforts. Santosh Anand understood its benefits and how it fit the needs and resources of the organization and bought the tool. For the first time, business stakeholders have a rapid deployment, web base, sophisticated business analytics tool at their disposal. They are delighted. 2.

Lessons Learned or: What does it take to be an “Inside Change Agent ?

· First, I am going to assume that you are willing to spend the time necessary to understand the environment to the point that you can describe its implicit norms and explicit boundaries. You cannot just walk-in, make pronouncements and walk out. You need to get to know the people, live their lives from the inside until you can name the unquestioned assumptions that support the set of roles, structures, tools, people, budgets assembled together make this organization. You might get there quickly, but the occasion to break the lock might take a while to show up. · Until the point that you see an occasion to create an Opening in front you, you might feel like stuck. No surprise: you are just mirroring where the organization is at. Keep doing what you are doing. · Trust you guts. If you see an opportunity to create an opening, make it happen. Don’t spend time agonizing on whether it is THE Opening or not. You will not necessarily know the impact of what you are doing until much later. · Don’t pre-judge the form that the opening needs to have. It will be an activity, whatever the activity might be, shared by all interested parties that will open up communication channels and build trust. · In the meetings you chair, separate decision making from facilitation. The decisions can be taken by consensus or based on technical expertise, you need to stay focused on the process of the meeting, i.e.: facilitation. This point is actually very important. By separating decision making from facilitation: o You allow decision makers to listen to what the people who are doing the work have to say. o You ensure that no one is shut out of participating in the discussion and that everybody’s voice is heard and respected. With Gratitude.