CM Crossroads Webcast Series Challenges and Characteristics of Enterprise Continuous Integration Moderator Patrick Egan Publisher – CM Crossroads and Agile Journal
Speakers Paul Julius – Willowbark Consulting Anders Wallgren – Chief Technical Officer Electric Cloud
© CMC Media 2008
Challenges and Characteristics of Enterprise Continuous Integration
www.electric-cloud.com
Introduction In the past… Teams spent weeks or longer verifying that applications worked Time spent „integrating‟ was tortuous, lead to integration „storms‟ of broken builds Developers would have to track down bugs introduced months earlier The answer to this painful process is Continuous Integration
October 9, 2008
Slide 3
CI: The Beginning Continuous Integration (CI) dictates that, as soon as changes are made, they are integrated and run through a rigorous build and test cycle Build engineers began using various CI tools, sometimes without management approval Once these tools became known, they were embraced by developers who wanted and needed the instant feedback on the health of their builds As teams and applications grow over time, many enterprises find themselves with CI servers spread across the company October 9, 2008
Slide 4
Challenges With the proliferation of multiple standalone systems, next step is to investigate the idea of consolidation Consolidated approach will help: Teams to reuse procedures and tools Better utilize hardware Free personnel from manual, low value tasks
October 9, 2008
Slide 5
Challenges in Real World Enterprise CI No centralized build-test-deploy infrastructure
Difficulty scaling the environment Testing and resource administration increase in complexity Hardware Utilization difficulties
Barriers to sharing and reusing scripts Difficulty tracking builds and cross-project reporting
October 9, 2008
Slide 6
S.T.A.R The challenges of Enterprise CI require a CI solution that is: Scalable
Turnkey Automatic
Rapid
October 9, 2008
Slide 7
Scalable Has many aspects
Must scale by: Number of projects Size of an individual project Number of users of the system Number of machines in the build cluster Complexity of workflow Enterprise CI is fundamentally about many projects and many users
October 9, 2008
Slide 8
Turnkey Enterprise CI has to be turnkey. The number of builds, number of users, and the reality of 24/7 operations make this imperative Turnkey implies that from the very beginning, Enterprise CI must be useable. It must be trivial to add scripts to the system, without requiring a rewrite.
October 9, 2008
Slide 9
Automatic CI system must be automatic, builds must be launch-able with a single click, but it must also be compatible with existing scripts Three forms of automatic: On demand, On stimulus and On the clock On Demand: Builds should be a single click away On Stimulus: Typically caused by changes in the SCM system On the Clock: Replacement for ‘cron’
October 9, 2008
Slide 10
Rapid As the system scales to mulitple projects, and the number of builds increase, builds have to be rapid. Hours long builds are a deal breaker Reducing build times from hours to minutes typically is done using parallelism Enterprise CI systems should be able to work in tandem with build parallelism software
October 9, 2008
Slide 11
Other Properties Three other properties are important to Enterprise CI:
Auditing
Virtualization
Reporting
October 9, 2008
Slide 12
Customer Example Problem Medium sized Product company 10‟s of disparate existing builds Varied build infrastructure Nearly impossible to identify a “good” build
Solution Unified Dashboard for all builds Consistent build infrastructure Templated, resuable build infrastructure
October 9, 2008
Slide 13
Next Steps It’s easy to get started with an open source CI system But as your needs change and your company grows you have to evaluate the effectiveness of this solution.
October 9, 2008
Slide 14
Build vs. Buy Key question then, should we use an open source tool, build our own, or buy a commercial tool?
October 9, 2008
Slide 15
Four Key Areas As we mentioned four key areas to review
Fast Builds Software keeps growing = longer builds
On the Clock Builds Scheduled Builds The classic „nightly build‟
On-demand Builds Ad hoc, as needed builds
On-Stimuli Builds Driven by changes in SCM
October 9, 2008
Slide 16
Build Your Own?
Investment
Seems pretty straightforward…
Automated build and test for one build tool, one machine
Scheduled jobs
Features Trigger builds after check-ins October 9, 2008
Slide 17
Build Your Own?
Investment
Just a few finishing touches
Automated build and test for one build tool, one machine
Scheduled jobs Web-based access to results
Features Trigger builds after check-ins October 9, 2008
CruiseControl gives you this for Java/Ant
Notifications via e-mail/RSS Database of build results
Slide 18
Build Your Own?
Investment
Multiple servers, concurrency
Notifications via e-mail/RSS
Automated build and test for one build tool, one machine
Scheduled jobs Trigger builds after check-ins
October 9, 2008
Run multiple builds simultaneously
Web-based access to results
Database of build results
Resource pooling, load balancing
Multiple jobs run Features on one resource
Resource selection criteria
simultaneously
Multiple servers, remote invocation
Run job steps in parallel Single job can use multiple resources
Slide 19
Build Your Own?
Investment
Better monitoring and control
Monitor system status
Notifications via e-mail/RSS
Automated build and test for one build tool, one machine
Scheduled jobs Trigger builds after check-ins
October 9, 2008
Run multiple builds simultaneously
Cancel jobs
On-demand job invocation via Web
Web-based access to results
Database of build results
Resource pooling, load balancing
View partial results of builds in progress
Multiple jobs run Features on one resource
Resource selection criteria
simultaneously
Multiple servers, remote invocation
Run job steps in parallel Single job can use multiple resources
Slide 20
Build Your Own?
Investment
Better reporting
Monitor system status
Notifications via e-mail/RSS
Automated build and test for one build tool, one machine
Scheduled jobs Trigger builds after check-ins
October 9, 2008
Run multiple builds simultaneously
Cancel jobs
On-demand job invocation via Web
Web-based access to results
Database of build results
Resource pooling, load balancing
View partial results of builds in progress
Multiple jobs run Features on one resource
Resource selection criteria
Customizable reports Trend reports Cross-product reports
simultaneously
Multiple servers, remote invocation
Run job steps in parallel Single job can use multiple resources
Tools for extracting data from logs Annotate builds after completion
Resource utilization reports Slide 21
Build Your Own?
Investment
Better error handling
Monitor system status
Notifications via e-mail/RSS
Automated build and test for one build tool, one machine
Scheduled jobs Trigger builds after check-ins
October 9, 2008
Run multiple builds simultaneously
Retry steps after errors/ failures
Detect resource failures
Cancel jobs
On-demand job invocation via Web
Web-based access to results
Database of build results
Resource pooling, load balancing
View partial results of builds in progress
Multiple jobs run Features on one resource
Resource selection criteria
Customizable reports
Logging and error reporting
Timeouts for runaway job steps
Trend reports Cross-product reports
simultaneously
Multiple servers, remote invocation
Run job steps in parallel Single job can use multiple resources
Tools for extracting data from logs Annotate builds after completion
Resource utilization reports Slide 22
Build Your Own?
Investment
Developer access to production builds Shared access by multiple teams LDAP authentication
Priorities
Role-based access control Notifications via e-mail/RSS
Automated build and test for one build tool, one machine
Scheduled jobs Trigger builds after check-ins
October 9, 2008
Monitor system status Resource pooling, load balancing
View partial results of builds in progress
Run multiple builds simultaneously
Retry steps after errors/ failures
Detect resource failures
Cancel jobs
On-demand job invocation via Web
Web-based access to results
Database of build results
User impersonation, password management
Handle multiple independent groups
Multiple jobs run Features on one resource
Resource selection criteria
Customizable reports
Logging and error reporting
Timeouts for runaway job steps
Trend reports Cross-product reports
simultaneously
Multiple servers, remote invocation
Run job steps in parallel Single job can use multiple resources
Tools for extracting data from logs Annotate builds after completion
Resource utilization reports Slide 23
Build Your Own?
Investment
Developer access to production builds Shared access by multiple teams LDAP authentication
Notifications via e-mail/RSS
Scheduled jobs Trigger builds after check-ins
Database of build results
October 9, 2008
Parameters
Monitor system status Resource pooling, load balancing
Retry steps after errors/ failures
Detect resource failures
Cancel jobs
On-demand job invocation via Web Multiple jobs run Features on one resource
Control environment for jobs
Conditional steps
View partial results of builds in progress
Run multiple builds simultaneously
Web-based access to results
Modular, composable process steps
Extensibility
Priorities
Role-based access control
Automated build and test for one build tool, one machine
User impersonation, password management
More programming features
Resource selection criteria
Customizable reports
Logging and error reporting
Timeouts for runaway job steps
Trend reports Cross-product reports
simultaneously
Multiple servers, remote invocation
Run job steps in parallel Single job can use multiple resources
Tools for extracting data from logs Annotate builds after completion
Resource utilization reports Slide 24
Build Your Own? It Never Ends
Investment
Developer access to production builds Shared access by multiple teams LDAP authentication
Notifications via e-mail/RSS
Scheduled jobs Trigger builds after check-ins
October 9, 2008
Web interface for editing processes Command-line interface
Portable across hardware/OS platforms
Modular, composable process steps
Extensibility
Parameters
Monitor system status Resource pooling, load balancing
Retry steps after errors/ failures
Detect resource failures
Cancel jobs
On-demand job invocation via Web Multiple jobs run Features on one resource
Control environment for jobs
Conditional steps
View partial results of builds in progress
Run multiple builds simultaneously
Web-based access to results
Database of build results
User impersonation, password management
SCM integration, bill of materials
Priorities
Role-based access control
Automated build and test for one build tool, one machine
Support multiple languages and build tools
Resource selection criteria
Customizable reports
Logging and error reporting
Timeouts for runaway job steps
Trend reports Cross-product reports
simultaneously
Multiple servers, remote invocation
Run job steps in parallel Single job can use multiple resources
Tools for extracting data from logs Annotate builds after completion
Resource utilization reports Slide 25
Build Your Own You could try that
Before you do think about… How much time do you have to write all that code? How you are going to manage 200+ builds per day? How are you going to get the build down from 4 hours to 15 minutes? How are you going to manage hardware failure in your build system?
Build Managers now control the heartbeat of software engineering They need something more than a creaky Perl script
October 9, 2008
Slide 26
Build Your Own Fast builds Buy lots of SMP hardware and try out GNU Make parallelism or manually parallelize the build.
Scheduled builds Use cron for that
On demand builds Build an intranet page, integrate it yourself with the current build and source code management system
Stimuli builds Build ad-hoc script attached to source code management system
October 9, 2008
Slide 27
Open Source Fast builds distcc, ccache, GNU Make -j
Scheduled Builds cron, CruiseControl
On-demand Builds Apache Continuum
Stimuli Builds CruiseControl has some support SourceControl plug-in
October 9, 2008
Slide 28
Commercial Build speed tools Like ElectricAccelerator
Build visualization and troubleshooting tools Like ElectricInsight
Build management tools Like ElectricCommander or CruiseControl
Together these tools enable build managers to build, package, test, and deploy software And they enable Agile development
October 9, 2008
Slide 29
Stepping Stones Builds up 18 months little by little
Four stepping stones
October 9, 2008
Slide 30
Stepping Stones (1) Step 1: Put full builds in the hands of developers
Set up an internal web page that allows a developer to fire off a build The developer should be able to select: Which branch to build… … or the location of his personal source tree Whether to run unit tests or not
The build runs and the engineer gets an email with the result
Start a new rule: developers run on demand builds before they commit
October 9, 2008
Slide 31
Stepping Stones (2) Step 2: Buy those lava lamps
Set up the full build system to run more than once per day Set up a system the feedback to the entire team whether the full build succeeded red/green lava lamps or a flat panel screen mounted high enough for engineering to see and a web page showing live build status
Scheduled builds start to be the heartbeat of engineering October 9, 2008
Slide 32
Stepping Stones (3) Step 3: Speed up the build
Look into tools that can speed up your build Set an under-60 minute goal for a full build and smoke test Announce to the team that multiple integrations can now happen per day, during the day These fast builds begin to enable agile development
October 9, 2008
Slide 33
Stepping Stones (4) Step 4: Fully automate
Integrate your fast builds with your source code management system When the source code changes run a stimuli build and test it Now the lava lamps really matter Now the build is the heartbeat of engineering Start referring to yourself as the Build Automator not the Build Manager
October 9, 2008
Slide 34
When to buy Look at $$$ build tools when Existing builds are too slow; developers complain Existing builds are frequently broken Existing builds are blocking implementation of new techniques (e.g. Agile/XP) Development teams are spread across the country or the globe
Other drivers are Company mergers Implementation of new target platforms A rapidly growing code base
October 9, 2008
Slide 35
Summary No one builds their own: Editor Compiler Debugger SCM
Most people don’t build their own: Issue tracker Test system
Why DIY the build system?
October 9, 2008
Slide 36
For More Information Electric Cloud 408.419.4300
[email protected] www.electric-cloud.com
Willowbark Consulting 312.560.5351
[email protected] www.WillowbarkConsulting.com
October 9, 2008
Slide 37
Questions and Answers Please post your questions now using the “Ask a Question” box on left side of the screen
View other Webcasts in the Series at www.cmcrossroads.com/wc
© CMC Media Inc. 2008