Testing Co-ordination -
new activity currently moving in 3 directions directions have commonalities .... ... but also each has its individual strengths
Concerns: - provide user guidance and infrastructure - keep an eye on the clock (Rome) (short term) - work towards a coherent flexible system (longer term)
P. Sherwood
Testing Co-ordination Software week 10 Dec. 2004
1
How to find out more Wiki (web pages): https://uimon.cern.ch/twiki/bin/viewAtlasOfflineTestingInfrastructure
-
condensed overview links to framework documentation testing co-ordination interactive discussions available via Atlas computing web pages
Meetings - mostly attended by testing developers - agendas and minutes available P. Sherwood
Testing Co-ordination Software week 10 Dec. 2004
2
Testing Frameworks Split the task of testing code into two part: specific to code not specific to the code (reusable)
‘test’ ‘testing framework’
The division is not unique.
P. Sherwood
Testing Co-ordination Software week 10 Dec. 2004
3
Existing Frameworks ATNight (ATN) User supplies short tests - run as part of the nightly build
Kit Validation (KV) Shell based test framework initially developed for testing distribution kits.
RunTimeTester (RTT) Python based testing framework. Currently running ~200 jobs/night at UCL.
P. Sherwood
Testing Co-ordination Software week 10 Dec. 2004
4
Automatic vs. Local running Automatic running Each framework can be run automatically: ATN - as part of the nightly build RTT - is run nightly as a cron job KV - can be run as a cron job Local Running Running framework under user control: KV - intrinsic ATN - included in design. Not much experience due to low demand. RTT - supported P. Sherwood
Testing Co-ordination Software week 10 Dec. 2004
5
User’s viewpoint Each test frameworks run jobs, and process results. Which to use? depends on: - Resources required to run tests - Results reporting mechanism - Test libraries - Local or automatic (nightly) testing - Language (shell or python) - Facilities (e.g. regression test support) - Ease of adding tests puzzling for the user: - boundaries are not so sharp - Some diffs are due to resources, not framework - Each framework has its own native usage P. Sherwood
Testing Co-ordination Software week 10 Dec. 2004
6
How to move forward (1) Target date 31/01/05 0. Continue current approaches 1.
changes to be made in small increments
Make standard tests callable from anywhere. e.g. tests to be run under ATN could use RTT libraries.
2.
Standardise language. KV → python.
3.
Make test specification a developer responsibility.
4.
Improve user interaction:
Encourages code reuse
P. Sherwood
move test specification to packages - developers in control - specification remains framework dependent - use CMT technology
- KV: move ATHENA transform parameters to config file - RTT: allow directory search to pick up users tests - RTT: allow users to specify library tests and parameters through configuration files Testing Co-ordination Software week 10 Dec. 2004
7
How to move forward (2) 4.
Identify information needed by the frameworks In config files, but also hard coded, or runtime environment
5.
6.
Create a common configuration standard Commonalities passed in the same manner Differences passed as framework specific (fat interface) Test configuration becomes finer grain. Meaningful queries become possible
Thin the interface
push parameter interpretation into frameworks
7.
8.
Move to a single Tester? both commonalities and specifics will have become clear single framework would have at least all current functionality diminish maintenance overhead
Handling test metadata
P. Sherwood
what tests where run when, what conditions, what results how to display needs thinking about Testing Co-ordination Software week 10 Dec. 2004
8