Acceptance Test Driven Development Naresh Jain
[email protected]
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 1
Tutorial Schedule • Part discussion, part exercise • Starts light and fluffy • You’ll get your hands dirty by the end • Questions welcomed all the time copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 2
Welcome
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 3
Welcome • Continuum
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 3
Welcome • Continuum • Hopes and Concerns
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 3
Getting Started Exercise
Warmup Scenarios
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 4
Getting Started Exercise
Warmup Scenarios • Going out for Movie (THX sound and Digital projection)
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 4
Getting Started Exercise
Warmup Scenarios • Going out for Movie (THX sound and Digital projection)
• Going out for meal (one veg.)
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 4
Getting Started Exercise
Warmup Scenarios • Going out for Movie (THX sound and Digital projection)
• Going out for meal (one veg.) • Going shopping ($50)
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 4
Getting Started Exercise
Warmup Scenarios • Going out for Movie (THX sound and Digital projection)
• Going out for meal (one veg.) • Going shopping ($50) • You are a party of 5. copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 4
What are Acceptance Tests?
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 5
What are Acceptance Tests?
Criteria for Completion • Written by the “Customer” • Given to developers
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 6
What are Acceptance Tests?
Communication • Forces customers, testers, and developers to work together
• creation • implementation • execution • maintenance copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 7
What are Acceptance Tests?
Feedback • A source of data • A tool for project management
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 8
Data From Acceptance Tests Total ATs
Failing ATs
Passing ATs
90 72 54 36 18 0 1
2
3
4
5
6
7
8
9
10
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 9
Acceptance Tests: A Critical Piece of Agile
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 10
Acceptance Tests: A Critical Piece of Agile
Traditional Approach 1 May
1 Jul
1 Sep
1 Nov
Analysis Design Implementation DFD
ERD DD
ST
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 11
Acceptance Tests: A Critical Piece of Agile
The Agile Aproach May 1
Jul 1
Sep 1
Nov 1
Analysis Test Implementation Design
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 12
Acceptance Tests: A Critical Piece of Agile
The Agile Approach
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13
Acceptance Tests: A Critical Piece of Agile
The Agile Approach Acceptance Criteria
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13
Acceptance Tests: A Critical Piece of Agile
The Agile Approach Acceptance Criteria
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13
Acceptance Tests: A Critical Piece of Agile
The Agile Approach Iteration
Acceptance Criteria
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13
Acceptance Tests: A Critical Piece of Agile
The Agile Approach Iteration
Acceptance Criteria
Automated Acceptance Tests
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13
Acceptance Tests: A Critical Piece of Agile
The Agile Approach Iteration
Acceptance Criteria
Automated Acceptance Tests
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13
Acceptance Tests: A Critical Piece of Agile
The Agile Approach Iteration
Acceptance Criteria
Automated Acceptance Tests
Automated Acceptance Tests copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13
Acceptance Tests: A Critical Piece of Agile
The Agile Approach Iteration
Acceptance Criteria
Automated Acceptance Tests
Automated Acceptance Tests copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13
Acceptance Tests: A Critical Piece of Agile
The Agile Approach Iteration
Acceptance Criteria
Automated Acceptance Tests
Automated Acceptance Tests
Acceptance Criteria
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13
Acceptance Tests: A Critical Piece of Agile
Criteria for DONE • Every story must have at least one Acceptance Test
• A story is not DONE until it passes it’s Acceptance Tests
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 14
Acceptance Tests: A Critical Piece of Agile
Manual Acceptance Tests
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 15
Acceptance Tests: A Critical Piece of Agile
Manual Acceptance Tests Manual Acceptance Tests
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 15
Acceptance Tests: A Critical Piece of Agile
Manual Acceptance Tests Manual Acceptance Tests
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 15
Acceptance Tests Are Automated
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 16
Acceptance Tests: A Critical Piece of Agile
The Button • How often would you press it?
• When would you press it?
• Who would press it?
•
Testers, Developers, Managers, Customers, Spectators, etc. copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 17
Acceptance Tests: A Critical Piece of Agile
Automated Tests Require: • High level, yet, precise language • Unambiguous detail • “Examples” • Interaction with the system copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 18
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 19
Criteria for DONE
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 19
Criteria for DONE +
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 19
Criteria for DONE +
Automated
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 19
Criteria for DONE +
Automated
Executable Specification copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 19
Acceptance Tests: A Critical Piece of Agile
Executable Specification • A new paradigm for testing • Puts quality first • Removes ambiguity from requirements copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 20
Who Writes Acceptance Tests?
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 21
Who Writes Acceptance Tests?
The Customer • Yeah right! Then who? • The Customer Role • Stake holder • Business Analyst • Quality Assurance • Product Owner • Developer
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 22
Who Writes Acceptance Tests?
Tests Get Technical • The “Customer” may need technical help to write tests
• Developers and QAs are technical • Pair test authoring copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 23
Who Writes Acceptance Tests?
Business Rules Get Fuzzy • Sometimes developers need help understanding tests
• Customers know business rules • Pair test implementation copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 24
Who Writes Acceptance Tests?
Business Decisions are Hard To Make
• Sometimes, the “customer” will be challenged to make decisions
• The Man ($$$) can help • Do you see a pattern of communication? copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 25
Exercise #1
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 26
Exercise #1
The Login Test • Write a test plan, in plain text, for the business rules of logging in.
• Web application • User credentials are stored in relational database
• Successful login redirects to “Welcome” page
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 27
Writing Good Acceptance Tests
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 28
Writing Good Acceptance Tests
Login Test Possibilities 1. Direct browser to URL for login page
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 29
Writing Good Acceptance Tests
Login Test Possibilities 1. Direct browser to URL for login page
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 29
Writing Good Acceptance Tests
Login Test Possibilities 1. Direct browser to URL for login page 1. Enter the username ‘wallace’
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 29
Writing Good Acceptance Tests
Login Test Possibilities 1. Direct browser to URL for login page 1. Enter the username ‘wallace’
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 29
Writing Good Acceptance Tests
Login Test Possibilities 1. Direct browser to URL for login page 1. Enter the username ‘wallace’
Build a Testable Environment First copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 29
Writing Good Acceptance Tests
BOC
Build Operate Check copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 30
Writing Good Acceptance Tests
Login Test Possibilities 1. Add some users to the system
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 31
Writing Good Acceptance Tests
Login Test Possibilities 1. Add some users to the system
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 31
Writing Good Acceptance Tests
Login Test Possibilities 1. Add some users to the system 3. Enter a value into the username field
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 31
Writing Good Acceptance Tests
Login Test Possibilities 1. Add some users to the system 3. Enter a value into the username field
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 31
Writing Good Acceptance Tests
Login Test Possibilities 1. Add some users to the system 3. Enter a value into the username field
Be Specific copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 31
Writing Good Acceptance Tests
Tests are Examples • Use concrete examples • Specify concrete behavior • No ambiguity allowed copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 32
Writing Good Acceptance Tests
Login Test Possibilities 1. Insert into User table values (’wallace’, ‘ilikecheeze’)
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 33
Writing Good Acceptance Tests
Login Test Possibilities 1. Insert into User table values (’wallace’, ‘ilikecheeze’)
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 33
Writing Good Acceptance Tests
Login Test Possibilities 1. Insert into User table values (’wallace’, ‘ilikecheeze’) 2. Open a browser to the URL http:// localhost/myapp
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 33
Writing Good Acceptance Tests
Login Test Possibilities 1. Insert into User table values (’wallace’, ‘ilikecheeze’) 2. Open a browser to the URL http:// localhost/myapp
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 33
Writing Good Acceptance Tests
Login Test Possibilities 1. Insert into User table values (’wallace’, ‘ilikecheeze’) 2. Open a browser to the URL http:// localhost/myapp
Avoid Implementation Details copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 33
Writing Good Acceptance Tests
Avoid Implementation Details Tests
UI
System Database copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 34
Writing Good Acceptance Tests
Login Test: Possible Solution • Add user to system: (’wallace’, ‘ilikecheeze’) • Process login with username ‘wallace’ and password ‘blah’
• Check login failed • Process login with username ‘wallace’ and password ‘ilikecheeze’
• Check login succeeded copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 35
Tools
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 36
Tools
Commercial Tools • WinRunner • Silk • RFT
TestPartner
EggPlant
QTP
TestComplete
Squish
WindowTester
Are not suitable for Acceptance Testing in an Agile environment copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 37
Tools
Open Source Options • FIT • FitNesse • Selenium
Sahi
Frankenstein
Watir
SharpRobo
Abbot
WET
Among the few tools that support Test Driven Development copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 38
Tools
FIT • Framework for Integrated Tests • Created by Ward Cunningham • Open Source • The most accepted solution for agile acceptance testing
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 39
Tools
FitNesse • Environment build around FIT • Makes everything easier • Created by Object Mentor, Inc. • Open Source copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 40
FIT
FitNesse
- Tests written in HTML - Tests are executed on the command line - Tables are executed - Non-table markup is ignored - Tables map to Fixtures - Fixtures are code that is aware of the system - Supplies foundational Fixtures - Implementations ported to many languages
- Stand alone web server - Is a wiki - Tests written in wiki text - Tests are executed from within the wiki - Translates tests into HTML - Uses FIT to execute tests - Supports test suites - Supports variables in tests - Supports test refactoring - Written in Java - Supports FIT implementations in any language
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 41
Thinking in Tables
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 42
Thinking in Tables
Only Tables Execute Ignored
Executed
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 43
Thinking in Tables
Foundational Table Structure Name of Fixture Interaction with Application
• Table structure depends on type of Fixture copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 44
Thinking in Tables
3 Foundation Fixtures • Column Fixture • Row Fixture • Action Fixture copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 45
Thinking in Tables
Column Fixture
package eg; // Copyright (c) 2002 Cunningham & Cunningham // Released under the terms of the GNU Genera Public … import fit.ColumnFixture;
public class Division extends ColumnFixture { public float numerator; public float denominator; public float quotient() { return numerator / denominator; } }
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 46
Thinking in Tables
Row Fixture Analogous to comparing against rows in a database table package fitnesse.fixtures; import fit.RowFixture; public class EmployeePayRecordsRowFixture extends RowFixture { public Object[] query() throws Exception { EmployeePayRecord[] records = new EmployeePayRecord[2]; records[0] = new EmployeePayRecord(1, 1000); records[1] = new EmployeePayRecord(2,2000); return records; } public Class getTargetClass() { return EmployeePayRecord.class; } }
public class EmployeePayRecord { public int id; private double salary; public EmployeePayRecord(int id, double salary){ this.id = id; this.salary = salary; } public double pay() { return salary; } }
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 47
Thinking in Tables
Action Fixture •
Think GUI window
Counter Window Counter:
public class CountFixture extends Fixture { private int counter = 0;
Counter: 6
public void count() { counter++; }
Count
public int counter() { return counter; } }
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 48
Exercise #2
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 49
Exercise #2
Login With Tables • Write the Login test using tables • Keep in mind what we learned from the previous exercise
• BOC • Examples • Avoid Implementation Details copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 50
Exercise #2
Possible Solution
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 51
FitLibrary • Extension to FIT • Written by Rick Mugridge • Adds some handy Fixtures copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 52
FitLibrary
FitLibrary Fixtures • ArrayFixture for ordered lists • SetFixture for unordered lists • SetUpFixture • Supports • Graphics • Tree structures • Nested Tables copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 53
FitLibrary
DoFixture • Broken tables • Highly readable • Flexibility
Text
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 54
Wiki
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 55
Wiki
What is it? • A collaborative web site • Editable by any • Created by Ward Cunningham • Every project should have one • http://c2.com/wiki • http://en.wikipedia.com copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 56
Wiki
Creating Tests • Use Wiki syntax to create a page with test tables • Label the page as a Test Page • Use a page name of the form Test… • Turn on the Test property • Make sure your Fixtures are in the classpath • Use !path widget • Mechanics • !path values are concatenated • Java command to start FitServer is executed • Testable HTML is passed to FitServer • FitServer runs the tests • Results are passed back to FitNesse copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 57
Wiki
Creating Suites There are 2 ways to make Suites
• •
Set the Suite property
• Create a page with the Suite property • Created test pages inside this page • When the suite is executed, all child test pages will be included in the suite execution
Use the !see widget
• !see
• All “included” tests pages will be included in the suite execution
Run a Suite by clicking the Suite button copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 58
Hands-on Session • Conference Proposal Submission Portal • Some sample Stories • • • • • • • •
Should be able to submit new proposal Should be able to list all submitted proposal Submitting proposal with same title should display appropriate error message Should be able to delete submitted proposal based on the title Should be able to delete submitted proposal based on the title Should be able to search proposals by title Should be able to search proposals by ID Should be able to find all proposal by an author's name
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 59
Break
http://www.fitnesse.org
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 60
Patterns
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 61
Patterns
Organizing Tests • Allowing customers to add new tests without breaking the build
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 62
Patterns
Version Control • Keeping the acceptance test in version control with the code.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 63
Patterns
Cross-Functional Pairing • Using FitNesse based acceptance tests for
collaboration between cross-functional team members.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 64
Patterns
ATDD
• Acceptance Test Driven Development
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 65
Patterns
CSTT
• Cleanup, Setup, Test, Teardown
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 66
Patterns
Independent Tests • Tests shouldn’t depend on each other. • Tests leave the system in the same state it started in.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 67
Patterns
Dynamic Stubbing
• Avoiding complications of external systems.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 68
Patterns
Non-Production Setup/Teardown
• Using non-production light weigh code for setup and teardown.
• Helps test only what you want to test.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 69
Patterns
Suite Levels • Creating different levels of suites depending on the depth/level of feedback desired.
• Smoke, Current Iteration/Sprint, Regression
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 70
Patterns
DRY
• Using !include to avoid repeating yourself.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 71
Patterns
Make it Real • Write ATs as close as possible to the real environment.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 72
Patterns
Fixture Evolution • Allow Fixture implementation to evolve over time.
• Treat fixtures as first class citizens.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 73
Patterns
At Least One Test/Story • Every story should have at least one acceptance test
• Avoid long/multipurpose tests.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 74
Anti-Patterns
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 75
Anti-Patterns
Developer ATs • Developers writing acceptance tests by themselves, for themselves.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 76
Anti-Patterns
Unit Testing • Don’t write ATs at the unit testing level • Unit tests are implementation specific • ATs are NOT implementation specific copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 77
Anti-Patterns
QA Testing Tool • Hard to write tests up front. • Perhaps only on large projects.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 78
Anti-Patterns
Silver Bullet • Trying to use FitNesse for all types of Acceptance Tests
• UI testing • XML testing copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 79
Anti-Patterns
Test After • Writing tests after the code is already written.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 80
Anti-Patterns
Hidden Test Data
• Hiding test data in the fixtures.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 81
Anti-Patterns
Implementation Dependant ATs
• Making test pages (tables) dependent on
implementation details and data structures.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 82
Anti-Patterns
Logging in Your Fixtures • Putting log statements or print statements in the fixture code.
• Fixtures are probably too complicated.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 83
The End
http://www.fitnesse.org
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 84