Radical Test Portability

Radical Test Portability Ian Dees - texagon.blogspot.com FOSCON 2007

Hi, I'm Ian Dees. I write software for a test equipment manufacturer in the Portland, Oregon area. This talk isn't really related to work, but our jumping-off point is sort of inspired by something that happened on the job once: I had the chance to evaluate some really crappy GUI testing toolkits.

Where there is a stink of [ there is a smell of being.

],

Antonin Artaud

But a funny thing about crap is that it can be fantastic fertilizer for growth. That's not what Artaud meant by this quote, by the way. But that's okay -- he was insane anyway.

Radical Test Portability

So, in keeping with FOSCON's "Really Radical Ruby" theme, let's talk about Radical Test Portability. It shouldn't be a revolutionary concept that tests can be portable. It should be old news.

But you wouldn't know that from reading the websites of the big-time test toolkit vendors. They say, "No programming." What that really means is, "No programming yet." These fire-and-forget tools typically generate a pile of code that you then have to go and customize.

MoveMouse(125,163); Delay(0.65); LeftButtonDown(); Delay(0.074); LeftButtonUp(); GetWindowText(hCtrl, buffer, bufferSize); if (0 != lstrcmp(buffer, L"Some text")) LOG_FAILURE("Text didn't match\n"); Delay(0.687); MoveMouse(204,78); //... ad nauseam And as often as not, you get a mess like this. Good luck trying to figure out where to insert your pass/fail tests, or maintaining this zoo a month from now, when the GUI designer moves a button.

def press_button(caption) send_msg 'PressButton ' + caption end press_button 'OK'

If the vendors would just implement clean protocols and document them well, you could write your test script in your choice of language instead of theirs. In this hypothetical case, it just so happens that the Ruby function name looks like the message we want to send. So the code can get even simpler....

def_msg :press_button, [:caption] press_button 'OK'

...by using Ruby's metaprogramming capabilities. And when the language gets clear, a funny thing starts to happen.

conversation

The tests become the centerpiece of a conversation between designers, developers, and testers. And in that spirit, I'd like to show you some sample code.

require 'spec_helper' describe 'A text editor' do it_should_behave_like 'a document-based app' it 'supports cut and paste' do @editor.text = 'chunky' @editor.select_all @editor.cut @editor.text.should == '' 2.times {@editor.paste} @editor.text.should == 'chunkychunky' end end Imagine that we're discussing a text editor's cut/paste feature. Using the RSpec test description language built on Ruby, we come up with something like this. The heart of the test is the third line from the end, where we express the intended behavior with "should."

require 'spec_helper' describe 'A text editor' do it_should_behave_like 'a document-based app' it 'can undo the most recent edit' do @editor.text = 'bacon' @editor.text = '' @editor.undo @editor.text.should == 'bacon' end end

Here's another example, testing the undo feature. See, you can show this stuff to anyone, no matter how much Ruby code they've seen in their lives. The setup code happens in that spec_helper file. We won't see the code for that -- it's on my blog. All we need to know for now is that it creates a TextEditor object.

require 'Win32API' class TextEditor @@keybd_event = Win32API.new 'user32', 'keybd_event', ['I', 'I', 'L', 'L'], 'V' KEYEVENTF_KEYDOWN KEYEVENTF_KEYUP VK_BACK

= 0x0000 = 0x0002 = 0x08

def keystroke(*keys) keys.each do |k| @@keybd_event.call k, 0, KEYEVENTF_KEYDOWN, 0 sleep 0.05 end keys.reverse.each do |k| @@keybd_event.call k, 0, KEYEVENTF_KEYUP, 0 sleep 0.05 end end end And here's a small piece of the TextEditor object's implementation for Windows Notepad. Here, we're teaching Ruby how to type a single character, possibly using modifier keys like Shift or Ctrl. We use the Win32API library to call the keybd_event function -- we press the keys down, then release them in reverse order.

class TextEditor def text=(string) unless string =~ /^[a-z ]*$/ raise 'Lower case and spaces only, sorry' end select_all keystroke VK_BACK string.upcase.each_byte {|b| keystroke b} end end

But the top-level test script never calls keybd_event directly. It calls text= instead, which breaks the message up into characters and types each one in turn. On Windows, a character is usually different than a keystroke (the "A" key might type an upper-case "A" or a lower-case "a"). So we're just going to deal with the easiest subset of messages where the character and keystroke are the same.

And here's what it looks like when you run it. I promise that's Ruby doing the typing! Now, on to portability. Since we didn't put any Windows-specific calls into our top-level test script, we can port it to other platforms, just by supplying a new implementation of the TextEditor class.

class TextEditor def initialize @window = JFrameOperator.new 'FauxPad' @edit = JTextAreaOperator.new @window @undo = JButtonOperator.new @window, 'Undo' end def text=(string) @edit.set_text '' @edit.type_text string end %w(text select_all cut paste).each do |m| define_method(m) {@edit.send m} end def undo; @undo.do_click end end Here's a test for a trivial editor for the Java runtime called FauxPad. You can run our same Ruby test for it using JRuby. I was going to show you just a piece of the TextEditor class for JRuby, but it turns out the whole thing fits on one slide. The text= method, which took up an entire slide in Windows, is just four lines here, thanks to the Jemmy GUI test library from NetBeans.

Now on to OS X. Of course there's AppleScript, but not every app exposes an AppleScript interface. Fortunately, you can turn on a setting in your preferences that will let control any app through its menus and buttons through the System Events interface.

tell application "System Events" tell process "TextEdit" keystroke "b" end tell end tell

script. application("System Events"). process("TextEdit"). keystroke!("b")

We can send AppleScript to the OS X command line pretty easily, but building the script up first can be cumbersome. All the commands except the last one start with "tell," and afterward there's an entire chain of "end tell" lines. But with a little Ruby magic, we can generate that long script from a little more terse starting point.

class AppleScript def initialize @commands = [] end def method_missing(name, *args) arg = args[0] arg = '"' + arg + '"' if String === arg command = name.to_s.chomp('!').gsub('_', ' ') @commands << "#{command} #{arg}" return name.to_s.include?('!') ? run! : self end end

The technique involves chaining a bunch of method calls together, and keeping them around in an array. When we encounter a method with a bang in it, we scoop up all those commands and pass them off to OS X.

class AppleScript def to_s inner = @commands[-1] rest = @commands[0..-2] rest.collect! {|c| "tell #{c}"} rest.join("\n") + "\n#{inner}" + "\nend tell" * rest.length end def run! clauses = to_s.collect do |line| '-e "' + line.strip.gsub('"', '\"') + '"' end.join ' ' `osascript #{clauses}`.chomp end end

The OS X command to run an arbitrary chunk of AppleScript is "osascript." It doesn't like long lines, so we break long commands up into pieces.

class TextEditor def initialize script.application("TextEdit").activate! end def text=(string) select_all delete string.split(//).each do |c| script. application("System Events"). process("TextEdit"). keystroke!(c) end end end Now that we can control Mac programs, we can write the TextEditor class yet again for a new application, Apple's TextEdit. Here's that same text= method, written for simplicity and not speed. We're spawning an osascript instance for every single keystroke -- definitely not something you'd want to do in a real project.

portability across apps

So far, we've seen one test script that can test three different apps: Notepad, FauxPad, and TextEdit.

portability across platforms

And that same test script runs on Windows, Mac, and other platforms. So what's next?

portability across languages

How about portability across languages? Our top-level test is in RSpec, but the underlying TextEditor object is just as at home in RBehave, another test language built with Ruby.

Story "exercise basic editor functions", %(As an enthusiast for editing documents I want to be able to manipulate chunks of text So that I can do my job more smoothly) do # one or more Scenarios here... end

While RSpec is geared toward small, self-contained examples, RBehave's basic unit of testing is the story. Stories are typically a little longer and more representative of how a real user would interact with the program. For now, we're just going to do a straight port from our RSpec test to RBehave.

Scenario "a chunky document" do Given "I start with the text", "chunky" do |text| @editor = TextEditor.new @editor.text = text end When "I cut once and paste twice" do @editor.select_all @editor.cut 2.times {@editor.paste} end Then "the document's text should be", "chunkychunky" do |text| @editor.text.should == text end end

Each Story consists of multiple Scenarios, which are phrased as Given... When... Then.... It's a little more verbose, but at least RBehave remembers its scenarios. So I can reuse the phrase "I start with the text" or "the document's text should be" in other scenarios without repeating the code that implements it.

Scenario "a bacon document" do Given "I start with the text", "bacon" When "I cut and then undo the cut" do @editor.select_all @editor.cut @editor.undo end Then "the document's text should be", "bacon" end

Like this. The only new behavior we've had to define for this scenario is "I cut and then undo the cut." The other two were already defined on the previous slide.

For more explorations like these, keep your eyes peeled for my book coming out in early 2008 from the Pragmatic Programmers.

thank you Ian Dees - texagon.blogspot.com

All of the source code for this exercise is available on my blog. Thanks for your time.