Automated Scenario Test

  • June 2020
  • PDF

This document was uploaded by user and they confirmed that they have the permission to share it. If you are author or own the copyright of this book, please report to us by using this DMCA report form. Report DMCA


Overview

Download & View Automated Scenario Test as PDF for free.

More details

  • Words: 7,220
  • Pages: 27
Purpose This document provides guidance and information that is necessary to be able to effectively implementing the automations of the test-cases.

Audience The audience of this documentation is the software SDETs1 responsible for the implementation of the automations of the test-cases. Implementing the automations involves translating the test-cases into XML documents where the steps in the test case are spelled out, and implementing custom actions (in the java programming language) that actually carry out the tasks to in each step. Therefore, knowledge of XML and programming in java are required on the side of the SDET.

Scope Each XML mentioned above captures a test-case (the intention and the steps) in a program-readable format. Throughout this documentation we will refer to that XML as scenario xml. This documentation explains all the elements and attributes expected / allowed in the scenario XML.

1 Software Development Engineer in Test

Scenario XML The scenario XML will be explained using several examples, starting with the one below. <scenario max-thread="5" repetition="1"> ➊ <desc> <wait timeout="2">eq(mediator("isStarted()"), true)



<args>

"lee" <steps>





<step name="step 1" mult="2" concurrent="true"> <desc>step 1 desc <args> "fu"





<desc>action 1.1 desc <args> list("a","b") range(7,8,1) 9 <desc>action 1.2 desc



<script> <args> list(10000, null)
<step name="step 2" mult="4" concurrent="false"> <wait>2 <desc>action 2.1 desc <args> list(3, true, "test", null) switch(ref(x), {true:concat("It's true! Look:", ref(x)), "zeppelin":"Wow that's a monster.", any:"Well..."})

Insert 1. Scenario XML #1

First pass through the scenario XML The <scenario> node ➊ is the root of a scenario XML. It has an optional attribute repetition, whose value can be any integer equals to or greater than 1 (by default it is set to 1). It is used to specify how many times the scenario will be ran, sequentially. If we set it to 0, the scenario will be repeated over and over, until we kill the process. Another optional attribute of <scenario> is max-thread, that specifies the maximum number of threads the scenario runner can use to run the actions. The number specified here is not a hard limit; there can be a little more threads than the number specified. As a general guideline, in order to avoid resorting into sequential execution of actions, pick the highest multiplicity in the scenario, add a constant to it, and assign the result to the max-thread attribute. In the above example it's 4 (the multiplicity of step 2), and the constant used is 1 (because step 2 is one level below the scenario). Therefore the chosen value for max-thread is 4 + 1 = 5. Inside the <scenario> node we can put a <desc> node, in which we write the (high-level) description of the scenario. We must enclose the description in a CDATA section if the description contains special characters which otherwise would render the scenario XML unprocessable. A scenario, as well as step and action, is a configurable node; it can take arguments. Those arguments are contained in the <args> node ➋. Each of them is represented by an <arg> node ➌, whose value can be a string literal, integer literal, bigdecimal literal (e.g.: 1.234), boolean literal (true / false), null, XML representation of a java object, XML tree, or function. The following table contains shows some examples of how to define an argument.

Type

Example

Value received in the java program (action class)

String literal

”foobar”

java.lang.String “foobar”

Int literal

78

java.lang.Integer 78

Boolean literal

false

java.lang.Boolean false

BigDecimal literal

7.5

java.math.BigDecimal 7.5

Null

null

An instance of com.baktun.bstrd.director.Null

Any

any

An instance of com.baktun.bstrd.director.Any

XML tree

<node key=”aVar”> Some beans

An instance of org.w3c.dom.Node

2

XML representation of Some beans

An instance of com.eshop.PurchaseOrder3

Function4

An instance of BigDecimal 6

mult(3,2)

Table 1. Examples of argument The arguments defined in a configurable node are visible to the child configurable-nodes. The usual scoping rules apply (i.e.: an argument defined in a configurable will override the argument with the same key defined in the parent node(s)).

2 Assigning null to an argument is not the same as not defining the argument. 3 The class must contain special (JAXB) annotations that would allow automatic mapping to XML . 4 All the available functions will be explained in separate section.

A scenario is composed of one or more steps, which will be executed one after another. Those steps must be contained within the <steps> node ➍. Each of them is represented by a <step> node ➎ that has the following attributes: • name (mandatory): the name of the step. Each step within the same scope5 must have a unique name. • offset (mandatory): the duration, in seconds, for the execution of the step to be delayed. The value must be an integer equal to or greater than 1. • mult (mandatory): the number of instances of the step to be created and executed. The value must be an integer equal to or greater than 1. • concurrent (optional): specifies whether the instances of the step will be executed sequentially or concurrently. The valid value is true (for concurrent) or false (for sequential). The default value is false. Inside the a step we define the actions that belong to the step. Those actions must be contained within the node ➏. Each of them is represented by an node ➐ that has the following attributes: • name (mandatory): the name that uniquely identifies the action within the step. • class (optional): the name of the java class that actually performs the task intended by the action. If we don't specify this, we must write the code directly inside the script node ➑, as exemplified by action 1.2. • asynch (optional): specifies whether the action is to be executed in the same thread as the containing step or in separate thread.

Running a scenario To see a scenario XML in action, a separate program has to be written. This program, which we will refer to as director throughout this documentation, in general performs the following tasks: Load the scenario XML (into a java object of type ScenarioDefinition). Create an instance of mediator. Create an instance of ScenarioRunner. Start the scenario runner by calling its run(...) method6, feeding in the scenario definition created in step #1. ➎ Start the mediator. ➊ ➋ ➌ ➍

The mediator is a generic java object; it can be anything. It is the entity we want to control, manipulate, or test throughout the execution of scenario, by means of actions. The following diagram depicts the relationship between scenario, mediator, and action.

5 The meaning of it will be clarified in the section where “action-nesting” is explained. 6 It has to run in a thread different from the one that runs the mediator.

Figure 1. The relationship between scenario, mediator, and actions. The following insert contains an example of such director. In this example, the mediator is an instance of com.baktun.bstrd.unittest.Frijol7. We have something quite similar for the spectel bridge simulator; the director is a java application named com.baktun.bridgesimulator.BridgeDirector, and the mediator is an instance of com.baktun.bridgesimulator.Bridge, wrapped inside an instance of com.baktun.bridgesimulator.BridgeServer. package com.baktun.bstrd.unittest; import import import import import

java.io.BufferedReader; java.io.FileReader; org.apache.log4j.Logger; com.baktun.bstrd.director.ScenarioDefinition; com.baktun.bstrd.director.ScenarioRunner;

public class FrijolDirector { private static final Logger logger = Logger.getLogger(FrijolDirector.class); private FrijolDirector() {} public static void main(String[] args) throws Exception { if (args == null || !(args.length == 2) ) { System.out.println("Usage: java FrijolDirector <scenario.xml> [keep-frijol-up]"); System.exit(-1); } boolean keepFrijolUp = true; if (args.length == 2) { keepFrijolUp = Boolean.parseBoolean(args[1]); } String scenarioDefXml = null; { BufferedReader br = new BufferedReader(new FileReader(args[0])); StringBuffer sb = new StringBuffer(); try { String line = br.readLine(); while (line != null) { sb.append(line).append("\n"); line = br.readLine();

7 Spanish word for bean.

} } finally { if (br != null) { br.close(); } } scenarioDefXml = sb.toString(); } final ScenarioDefinition scenarioDef = ScenarioRunner

➊ final Frijol frijol = new Frijol(); ➋ .loadScenarioDef(scenarioDefXml);

final ScenarioRunner scenarioRunner = new ScenarioRunner();



Thread scenarioRunningThread = new Thread(new Runnable() { public void run() { try { scenarioRunner.run(scenarioDef, frijol); } catch (Exception e) { e.printStackTrace(); } } }); scenarioRunningThread.setPriority(Thread.MIN_PRIORITY); scenarioRunningThread.start(); frijol.run();





} }

Insert 2. An example director The output of the execution of the director above, in the following insert, will help clarifying a couple of important points about the execution model of the scenario runner. 2009-09-23 15:42:24,564 [INFO] Thread-2 com.baktun.bstrd.unittest.Frijol - Frijol is going to sleep for 5 seconds.... 2009-09-23 15:42:24,583 [INFO] Thread-1 ➋ com.baktun.bstrd.director.ScenarioRunner Running scenario for the 1 times 2009-09-23 15:42:30,640 [INFO] Thread-1 com.baktun.bstrd.director.ScenarioRunner - About to execute step step 1

➊,5,main]:{jet=lee, kung=fu, a=b ➌, c=9, b=8 ➍} Thread[pool-1-thread-1 ➊,5,main]:{jet=lee, kung=fu, a=a ➌, c=9, b=7 ➍} Thread[pool-1-thread-2

----2009-09-23 15:42:30,661 [INFO] Thread-1 com.baktun.bstrd.director.ScenarioRunner - About to execute step step 2

➎,5,main]: Done! : 1913434059 Thread[pool-1-thread-4 ➎,5,main]: Done! : -498702880 Thread[Thread-1 ➋,1,main]:{jet=lee, y=Well..., x=3} Thread[pool-1-thread-3

--Thread[Thread-1,1,main]:{jet=lee, y=It's true! Look:true, x=true} --Thread[Thread-1,1,main]:{jet=lee, y=Well..., x=test} --Thread[Thread-1,1,main]:{jet=lee, y=Well..., x=null} ---

Insert 3. Output of the execution of example director

Threading in the scenario Since step 1 is declared as concurrent, there might be new thread(s) allocated for the execution of the instance(s) of step 1. In the above case, we also declared step 1 to have multiplicity 2, which might cause two instances of step 1 to be created, each of them will be running in separate threads8 ➊. In contrast, when a step is not declared as concurrent, as is the case of step 2, it will be executed in the same thread as the one in which the containing node is running ➋. In the above case, the containing node is the scenario, which is running in the main thread. The scenario runner will go through all the steps sequentially, in the document order. That means, step 2 will be executed only after all the instances of step 1 have finished running. Action will be running in the same thread as the one in which the containing step is running ➊, except when it is declared as asynchronous (the case of action 1.2). Asynchronous action will be running in yet separate thread ➎9. Implementions of action are not expected to either create or manage any threads (although currently there's nothing that prevents it from doing so). If an action is intended to run in the background (e.g.: as a listener), we can simply declare it as asynchronous in the scenario XML. The declarative style of doing things here is aimed at improving the communication among the teammembers, by making the intention spelled out clearly in the scenario XML. Above all, this test framework is a communication tool that fills the gap between the test-case document, and the executable that realizes the test-case. The test-framework takes common, repeated concerns (such as threading) away from us, while still making it accessible in another way (declarations in the scenario XML). This allows us to have a lean and compact implementation of actions, which are expected to consist of only a few lines of code that basically take the arguments passed-in to it, process them a little bit, and use them during the interaction with the mediator.

8 There is no guarantee that instances of a step declared as concurrent will be running concurrently. The scenario-runner makes its best effort to fulfill it, though, but if the maximum number of threads is already reached, the scenario-runner will resort to executing those steps in the same thread as the one in which the parent node is executing, resulting in sequential execution of those instances of step. 9 Care has been taken about creation of new threads. The engine utilizes Java's built-in thread-pooling mechanism. This has consequences such as: action that uses ThreadLocal as storage will likely break.

Argument's-value resolution Initially there was a simple mechanism: you declare the argument by simply specifying the key and the value, what you see (in the XML) is what you get (in the action). Things changed when we introduced dynamism into the scenario XML, by means of a domain-specific expression language. Some of the functions available in the expression language allows the value of an argument to be based on the value of other argument, for example: concat(ref(segment_id), "-", ref(participant_id)). Now we can make the value of an argument to vary according to the index in the iteration: list("white", "red", "black"). We can also make use of a construct similar to the switch-case-default statement in programming language like java: switch(ref(hobby), {"music":"give him a guitar", "literature":"buy him a kindle", null:"give him some cash"}). By combining that dynamic argument-value and multiplicity of steps, a whole set of combinations of inputs for the test-case can be packed into a single scenario XML. There are two functions that particularly deserves a couple of paragraphs (of clarification) here: • The list function, with syntax: list(element_1, element_2, ..., element_n) • The range function, with syntax: range(start_val, end_val, step) Assigning a list to an argument in the scenario XML does not mean that an instance of java.util.List will be passed-in to the action. Instead, the action will only see a single element bound to that argument, taken from the list. Example: <scenario repetition="1"> <steps> <step name="step 1" offset="0" mult="2" concurrent="true"> <args> list("green", "orange") <args> list(1980, 1990)

Insert 3. An example scenario definition with list(...) In the example presented above, there will be two threads. The value of the arguments accessible from the action running in the first thread: {var_a: green, var_b: 1980} . The second thread, on the other hand, will see the following values: {var_a: orange, var_b: 1990}.

The function range behaves in similar fashion. In fact, we can see a range as just another list, only that we don't have to specify all the elements manually. That is to say: • range(10, 15, 1) is equivalent to list(10, 11, 12, 13, 14, 15) • range(10, 15, 2)10 is equivalent to list(10, 12, 14) The number of elements in the list must be at least equal to multiplicity of the step to which it directly belongs. In the above example, list var_a and var_b both belong to the step step 1 whose multiplicity is set to 2. Therefore the length of those list be at least equal to 2. Extra elements at the end will simply be ignored / unused. The following insert contains another example that uses list, this time involves nesting of actions. In that case, the length of list var_c and var_d must be at least 1, which is the multiplicity of step 2 to which those lists directly belong. <scenario repetition="1"> <steps> <step name="step 1" offset="0" mult="2" concurrent="true"> <args> list("green", "orange") <args> list(1980, 1990) <step name="step 2" offset="0" mult="1" concurrent="true"> <args> list(9) <args> list("bar")

Insert 4. Another example scenario definition with list

10 Probably mod-check is necessary to eliminate misunderstanding / mis-expectations.

Available functions •





• •



• • •



sum: performs addition of all its arguments, returns an instance of java.math.BigDecimal. Examples: ◦ sum(4, 5) returns BigDecimal(9) ◦ sum(4, 7.15) returns BigDecimal(11.15) ◦ sum(4, 7, ref(x)) returns BigDecimal(16.5), assuming x resolves to 5.5 mult: multiplies all of its (two) arguments, returns an instance of java.math.BigDecimal. Examples: ◦ mult(4, 2.5) returns BigDecimal(10) ◦ mult(ref(x), ref(y)) returns BigDecimal(16), assuming x resolves to 5 and y resolves to 3.2. div: divides its first arguments by its second argument, returns an instance of java.math.BigDecimal. Examples: ◦ div(1, 2) returns BigDecimal(0.5) ◦ div(sum(ref(x), 4), mult(ref(y), 5)) returns BigDecimal(0.5), assuming x resolves to 1 and y resolves to 2. int: cast its argument to an integer, returns an instance of java.lang.Integer. Example: ◦ int(ref(x)) returns Integer(1), assuming x resolves to 1.2 mod: returns the remainder of division of its first argument by its second argument, as an instance of java.lang.Integer. Example: ◦ mod(ref(x), 5) returns Integer(2), assuming x resolves to 7. concat: concatenate all of its arguments, returns an instance of java.lang.String. Example: ◦ concat(ref(x), "_", mult(4, ref(y)), "_", 5) returns the string 7_12_5, assuming x resolves to 7 and y resolves to 3. list: returns a single element, as an instance of java.lang.Object, taken from the list, successively at each iteration. wheel: similar to the list function. However, instead of giving you an error, it quietly returns the first element in the list whenever the end of the list is passed. range: returns a single element, as an instance of java.lang.Object, taken from the range, successively at each iteration. The first input is the beginning of the range, the second input is the stop of the range, and the third input is the step (the difference between two adjacent elements in the range). The first and second input can be any integer (positive, zero, or negative). The first and the second input can't have the same value (otherwise an error will be thrown). If the first input is smaller than the second input, the range will be an “ascending” one. Otherwise it will be a “descending” one. The step can not be less than 1. Examples: ◦ range(2, 4, 1) is equivalent to list(2, 3, 4) ◦ range(4, 2, 1) is equivalent to list(4, 3, 2) ◦ range(4, 2, -1) throws an error. ◦ range(2, 2, 1) throws an error. ◦ range(2, ref(x), 1) is equivalent to list(2, 3, 4), assuming x resolves to 4. ref: takes the name of another argument as input, returns the value that is bound to that other argument. Examples: ◦ ref(x) returns Integer(5), assuming x is set to 5. ◦ ref(y) returns BigDecimal(3.2), assuming y resolves to 3.2 (y might be defined as

mult(1.6, 2), for example).











switch: takes the form switch(selector, map_of_cases), returns the value of the key-value pair whose key matches the value of the selector. Examples: ◦ switch(ref(x), {"boy": "Rama", "girl": "Shinta", any: ref(generic_name)}) returns: ▪ the string "Rama" if x resolves to "boy". ▪ the string "Shinta" if x resolves to "girl". ▪ the value the variable generic_name resolves to if x resolves to any value other than "boy" or "girl". neg: accepts either integer / bigdecimal / boolean, and returns the negated value of its input. Examples: ◦ neg(1) returns Integer(-1) ◦ neg(-2.5) returns BigDecimal(2.5) ◦ neg(ref(x)) returns Boolean(true), assuming x resolves to false. eq: compares its two inputs, returns true if they're equal. Except for some special cases, the equality is based on the value of the two inputs (i.e.: the result of calling equals method on one of the input, passing in the other input as the argument). Examples: ◦ eq(1, 1) returns Boolean(true). ◦ eq(1, “1”) returns Boolean(false). ◦ eq(1, ref(x)) returns Boolean(true), assuming x resolves to Integer(1). ◦ eq(ref(x), any) returns Boolean(true), assuming x resolves to any non-null value. ◦ eq(mediator(“getWatcher()”), null) returns Boolean(true), assuming the invocation of method getWatcher() on the mediator returns null. ◦ eq(mediator(“getWatcher()”), any) returns Boolean(true), assuming the invocation of method getWatcher() returns any non-null value. gt: compares its two inputs, returns Boolean(true) if the first input is greater than the second input. Will return error if any of the input is neither an instance of Integer nor BigDecimal. Examples: ◦ gt(2, 1) returns Boolean(true). ◦ gt(2.1, 1) returns Boolean(true). ◦ gt(1, 2) returns Boolean(false). ◦ gt(ref(x), 2) returns Boolean(true), assuming x resolves to any Integer or BigDecimal greater than 2. ◦ gt(“2”, 1) throws an error. lt: compares its two inputs, returns Boolean(true) if the first input is smaller than the second input. Will return error if any of the input is neither an instance of Integer nor BigDecimal. Examples: ◦ lt(1, 2) returns Boolean(true). ◦ lt(1, 2.1) returns Boolean(true). ◦ lt(2, 1) returns Boolean(false). ◦ lt(2, ref(x)) returns Boolean(true), assuming x resolves to any Integer or BigDecimal greater than 2. ◦ gt(1, “2”) throws an error.



and: performs logical AND operation on all11 of its inputs. All of the inputs must return an instance of Boolean. Examples: ◦ and(true, true) returns Boolean(true). ◦ and(true, false) returns Boolean(false). ◦ and(true, true, ref(x)) returns Boolean(false), assuming x resolves to Boolean(false). ◦ and(mediator(“isStarted()”), mediator(“isOkToTest()”)) returns Boolean(true) assuming the invocation of both isStarted() and isOkToTest()

method on the mediator returns true.

• •



◦ and(true, mediator(“getNumberOfLines()”)) throws an error, assuming the invocation of getNumberOfLines() method on the mediator returns anything other than Boolean (object / primitive). or: similar to the and function, only that the logical operation performed on the inputs is OR. rand: takes two inputs, both of them must resolve to integers; will generate a random integer between the range specified by those two inputs. Examples: ◦ rand(3, 7) can return any of the following values: 3, 4, 5, 6, or 7. ◦ rand(2, ref(x)) can return either 2, 3, 4 assuming x resolves to 4. mediator: takes the OGNL12 expression that will be applied on the mediator. The simplest way to see it: it allows us to specify the chain of method invocations (that starts on the mediator). This function returns the value returned from the invocation(s). Example: ◦ mediator(“isStarted()”), invokes the isStarted() method on the mediator, returns the value returned from that invocation. ◦ mediator(“getParticipant(#participant_id)”), invokes the getParticipant(...) method on the mediator, passing in the value of the argument whose key is “participant_id”. This requires an argument to be defined in the same node (action / step / scenario) or above as the node that in which the mediator function is used. Example: mediator(“getParticipant(#participant_id)”) ”part_2009_xx_00021”



◦ mediator(“getParticipant(#participant_id).getName()”), invokes the getName() method on the object returned by the invocation of getParticipant(...) method on the mediator. ◦ mediator(“launchRocket()”) throws an error if the method launchRocket() is neither defined nor accessible in the mediator object. context: takes the form context(path, context_var_key) . The path specifies the node in the scenario where the object stored under the key context_var_key13 is to be found. The syntax of the path is as follows: ◦ . (single dot): current node. ◦ ... (triple dot): root node (scenario) ◦ .. (double dot): parent ◦ ./action_1

11 There is no short-circuiting here; each and every arguments will be evaluated. 12 Object-Graph Navigation Language: http://en.wikipedia.org/wiki/OGNL 13 Storing object in the scenario-execution context will be explained in separate section.



◦ ../.. ◦ ../action_1 ◦ ../../../action_1 error: it is meant to be used only inside the <expectation> node, to specify the error that is expected to occur inside an execution-node. This function has two possible syntaxes: ◦ with one argument: error(exception_class_name), which will return true if any instance of error of the specified type occured. Examples: ▪ error(“java.lang.Exception”) ▪ error(“java.lang.IOException”) ◦ with two arguments: error(exception_class_name, ognl_expression), which

goes further than the first version, by doing a chain of method invocation starting at the instance of exception (using the specified ognl_expression). This functions returns the value returned from the method invocation(s). Examples: ▪ error(“java.io.FileNotFoundException”, “getMessage()”); returns the message of the instance of the FileNotFoundException. This version of error function is meant to be used together with the eq function. Example: ▪ eq(“c:\\theimportantfile.txt can not be found”, error(“java.io.FileNotFoundException”, “getMessage()”))



stepidx: this function returns the index of the instance of the step. It takes as input the path to the execution-node where the calculation of the index is based on14. For example, when we have a step declaring the following, the second instance of the step will see the integer 2 assigned to the argument the_index. ... <step name="step 1" mult="3" concurrent="true"> <desc>step 1 desc <args> stepidx(".") ... ...

Caution about the context_path that is used as input for this function: it must resolve to a node that is a step. Otherwise an error will be thrown. Example: ... <step name="step 1" mult="3" concurrent="true"> <desc>step 1 desc <args> stepidx(".") <args> ... ... ... ...

14 See the explanation of function context for some examples of path.

The following definition of argument the_index, however, will not throw an error. Instead, the action that is part of the second instance of the step will see the integer 2 assigned to the argument the_index. ... <step name="step 1" mult="3" concurrent="true"> <desc>step 1 desc <args> stepidx("..") <args> ... ... ... ...



stepmult: just like the stepidx function, it takes as input the path to the execution-node where the calculation of the multiplicity is based on. The same restriction as in the stepidx applies; the path must resolve to a step. It returns an integer, the multiplicity of the step. Examples: ... <step name="step 1" mult="3" concurrent="true"> <desc>step 1 desc <args> stepmult(".") ... ... ... <step name="step 1" mult="3" concurrent="true"> <desc>step 1 desc <args> stepmult(".") <args> ... ... ... ...

... <step name="step 1" mult="3" concurrent="true"> <desc>step 1 desc <args> stepmult("..") <args> ... ... ... ...

As you can see from the examples above, we can pass a function as argument of another function. This capability allows to build plethora of interesting combinations. The following grammar gives a complete picture of the expression language of this test framework.

stat: expr; expr: STR_LITERAL | INT_LITERAL | BIGDEC_LITERAL | NULL_EXPR | ANY_EXPR | func | bool_expr | mediator_expr | context_expr | rand_func | stepidx_func | stepmult_func | str_func; bool_expr: BOOL_LITERAL | eq_expr | gt_expr | lt_expr | and_expr | or_expr | error_expr; eq_expr: 'eq' '(' expr ',' expr ')' -> ^('eq' expr expr); gt_expr: 'gt' '(' expr ',' expr ')' -> ^('gt' expr expr); lt_expr: 'lt' '(' expr ',' expr ')' -> ^('lt' expr expr); and_expr: 'and' '(' expr ',' expr (',' expr)* ')' -> ^('and' expr expr+); or_expr: 'or' '(' expr ',' expr (',' expr)* ')' -> ^('or' expr expr+); mediator_expr: 'mediator' '(' STR_LITERAL ')' -> ^('mediator' STR_LITERAL); context_expr: 'context' '(' STR_LITERAL ',' STR_LITERAL ')' -> ^('context' STR_LITERAL STR_LITERAL); error_expr: 'error' '(' STR_LITERAL ')' -> ^('error' STR_LITERAL) | 'error' '(' STR_LITERAL ',' STR_LITERAL ')' -> ^('error' STR_LITERAL STR_LITERAL); func: ref_func | list_func | range_func | concat_func | mult_func | sum_func | neg_func | mod_func | switch_func | div_func | int_func | wheel_func | short_func; str_func: 'str' '(' expr ')' -> ^('str' expr); range_func: 'range' '(' arith_func_param ',' arith_func_param ',' INT_LITERAL ')' -> ^('range' arith_func_param arith_func_param INT_LITERAL); ref_func: 'ref' '(' VAR_NAME ')' -> ^('ref' VAR_NAME); mult_func: 'mult' '(' arith_func_param ',' arith_func_param ')' -> ^('mult' arith_func_param arith_func_param); sum_func: 'sum' '(' arith_func_param (',' arith_func_param)+ ')' -> ^('sum' arith_func_param arith_func_param+); div_func: 'div' '(' arith_func_param ',' arith_func_param ',' INT_LITERAL ')' -> ^('div' arith_func_param arith_func_param INT_LITERAL); mod_func: 'mod' '(' arith_func_param ',' arith_func_param ')' -> ^('mod' arith_func_param arith_func_param); neg_func: 'neg' '(' neg_func_param ')' -> ^('neg' neg_func_param); neg_func_param: arith_func_param | bool_expr; int_func: 'int' '(' short_int_func_param ')' -> ^('int' short_int_func_param); short_func: 'short' '(' short_int_func_param ')' -> ^('short' short_int_func_param); short_int_func_param: arith_func_param | bool_expr; rand_func: 'rand' '(' arith_func_param ',' arith_func_param ')' -> ^('rand' arith_func_param arith_func_param); stepidx_func: 'stepidx' '(' STR_LITERAL ')' -> ^('stepidx' STR_LITERAL); stepmult_func: 'stepmult' '(' STR_LITERAL ')' -> ^('stepmult' STR_LITERAL); arith_func_param: INT_LITERAL | BIGDEC_LITERAL | ref_func | mult_func | sum_func | neg_func | list_func | range_func | div_func | int_func | mod_func | switch_func | wheel_func | rand_func | stepidx_func | stepmult_func; list_func: 'list' '(' map_entry_key_value (',' map_entry_key_value)+ ')' -> ^('list' map_entry_key_value map_entry_key_value+); wheel_func: 'wheel' '(' map_entry_key_value (',' map_entry_key_value)+ ')' -> ^('wheel' map_entry_key_value map_entry_key_value+); switch_func: 'switch' '(' switch_func_selector ',' '{' switch_func_case (',' switch_func_case)* '}' ')' -> ^('switch' switch_func_case+ switch_func_selector); switch_func_selector: arith_func_param | concat_func; switch_func_case: map_entry_key_value ':' map_entry_key_value -> ^(SWITCH_CASE map_entry_key_value map_entry_key_value); map_entry_key_value: STR_LITERAL | NULL_EXPR | bool_expr | ANY_EXPR | concat_func | arith_func_param; concat_func: 'concat' '(' expr (',' expr)+ ')' -> ^('concat' expr expr+); STR_LITERAL: '"' ( options {greedy=false;} : . )* '"'; INT_LITERAL: '-'? ('0'..'9')+; BIGDEC_LITERAL: '-'? ('0'..'9')+ '.' ('0'..'9')+; BOOL_LITERAL: 'false' | 'true'; NULL_EXPR: 'null'; ANY_EXPR: 'any'; VAR_NAME: ('A'..'Z' | 'a'..'z' | '_' | '$') ('A'..'Z' | 'a'..'z' | '0'..'9' | '_' | '$')*; WS: (' ' | '\t' | '\n' | '\r')+ {skip();};

Insert 5. The grammar of the expression language of the test framework

Implementing action Once the scenario is completely laid-out and received an approval, we are for the final step that is implementing the actions that will actually carry out the tasks. There two ways of implementing action: (a) create the action class, or (b) script it directly inside the scenario XML. Creating an action class An action class is a (java) class that extends com.baktun.bstrd.director.Action. We only have to implement one method: callExecute(...). The following insert contains the source code of an example action class, PrintArgsAction that simply prints out all the arguments visible from it. package com.baktun.bstrd.unittest; import import import import import

java.util.Map; com.baktun.bstrd.director.Action; com.baktun.bstrd.director.ActionDefinition; com.baktun.bstrd.director.Step; com.baktun.bstrd.director.ValueProvider;

public class PrintArgsAction extends Action { public PrintArgsAction(ActionDefinition definition, Step step, Map<String, ValueProvider> valPros, ValueProvider waitValPro, ValueProvider condValPro, ValueProvider expectValPro, Object mediator) { super(definition, step, valPros, waitValPro, condValPro, expectValPro, mediator); } public Object execute(Object resultFromPreviousAction) throws Exception { System.out.println(Thread.currentThread() + ":" + getCopyOfArguments()); System.out.println("---"); return null; } }

Insert 6. The source code of an example action class The execute(...) method takes the result of the execution of the previous action, that is the action defined right before this action in the scenario XML (it's a way of sharing objects between actions, beside sharing through context which will be explained in separate section). Consequently, the execute(...) method returns an object (that will be passed on to the next action by the scenario runner). If there's nothing to share, simply returns null. From within the execute method, we can access the containing step, the mediator, and the arguments. We can also store / retrieve values stored in the context. The following methods let you do the aforementioned things: • • • • • • •

Step getStep() T getMediator() Object getArgument(String key) Object getArgument(String key, boolean lookUp) boolean hasArgument(String key) boolean hasArgument(String key, boolean lookUp) LinkedHashMap<String, Object> getCopyOfArguments()

• • • • • • • • • •

LinkedHashMap<String, Object> getCopyOfArguments(boolean lookUp) boolean hasValue(String key) boolean hasValue(String key, boolean lookUp) void setValue(String key, Object value) Map<String, Object> getCopyOfValues() Map<String, Object> getCopyOfValues(boolean lookUp) void unsetValue(String key) void unsetAllValues() ExecutionNode getParent() V getDefinition()

One thing worth mentioned here is the behavior of the method getArgument(...) and getValue(...): they will throw an exception if the corresponding argument can not be found (in the scenario XML) or there's no object associated with the specified key can be found (in the context), respectively. This is by design to force the SDET to state its intention in the scenario XML (in the case of getArgument(...)); if the value of a variable is intended to be null, don't leave it out in the XML. Instead specify its value to null, like in the example below. <scenario repetition="1"> <steps> <step name="step 1" offset="0" mult="2" concurrent="true"> <args> null

Insert 7. Specifying null in the scenario XML Inside the action class, we handle it the following way: ... public Object execute(Object resultFromPreviousAction) throws Exception { ... Object theVarA = null; if (getArgument("var_a") instanceof com.baktun.bstrd.director.Null) { theVarA = calculateDefaultValueOfVarA(...) } else { theVarA = calculateValueOfVarABasedOn(getArgument("var_a")); } ... } ...

Insert 8. Handling null in the action class Similarly if we want to state that var_a can take any value, set it to any in the scenario XML, as in the example below. <scenario repetition="1"> <steps>

<step name="step 1" offset="0" mult="2" concurrent="true"> <args> any

Insert 8. Specifying any in the scenario XML Inside the action class, we handle it the following way: ... public Object execute(Object resultFromPreviousAction) throws Exception { ... Object theVarA = null; if (getArgument("var_a") instanceof com.baktun.bstrd.director.Null) { theVarA = calculateDefaultValueOfVarA(...) } else if (getArgument("var_a") instance of com.baktun.bstrd.director.Any) { theVarA = calculateRandomValueOfVarA(...) } else { theVarA = calculateValueOfVarABasedOn(getArgument("var_a")); } ... } ...

Insert 8. Handling any in the action class To use that action class in the scenario XML, simply specify its fully-qualified name as the value of class attribute of the appropriate node. Example: <scenario repetition="1"> <steps> <step name="step 1" offset="0" mult="2" concurrent="true"> <args> list("green", "orange") <args> list(1980, 1990)

Insert 8. Using action class in the scenario XML

Scripting the action Alternatively, when the action is so simple15, we can directly script it inside the scenario XML. In that case, there is no need to specify a value for class attribute16 in the node. Instead, we have to write directly those lines of code inside the <script> node (and don't forget to wrap guard it inside a CDATA section). The scripting language is still java. The following insert contains an example of scenario XML with scripted action. <scenario repetition="1"> <desc> <steps> <step name="step 1" offset="0" mult="2" concurrent="true"> <desc>action 1.2 desc <script> <args> list(10000, null)

Insert 9. Example of scenario XML with scripted action From inside the script we can use the methods available in the Action class (page 19-20). However, we have to prepend the invocation with action..

15 No exact definition of simple provided. It can be taken as: "no more than 5 lines of code", for example. 16 In fact, we must omit the class attribute.

Passing objects through context Other than passing the return of the execute method from one action to the next one (in the same step), the framework allows actions to publish objects into the context, such that it can be later retrived and used by subsequent actions. The following code-snippet shows an example of how to use the setValue and getValue methods for storing object in the context and retrieving object from the context, respectively, ... public Object execute(Object resultFromPreviousAction) throws ActionExecutionException { ... RedBean redBean = processIt((GreenBean) getValue("theBeanThatIsShared")); setValue("theBeanToBeShared", redBean); ... } ...

Insert 10. Using getValue and setValue The above example, however, is not really useful because it tries to retrieve / store from the same execution-node; which translates to “sharing object(s) only with itself”, which is pretty much meaningless. Normally we would retrieve / store from the execution node at least one level above. The following snippet shows how an example of storing an object in the step that the action belongs to, so that subsequent actions within the same step can retrieve it. The snippet also shows an example of retrieving an object from the step. ... public Object execute(Object resultFromPreviousAction) throws ActionExecutionException { ... RedBean redBean = processIt((GreenBean) getStep().getValue("theBeanThatIsShared")); getStep().setValue("theBeanToBeShared", redBean); ... } ...

Insert 11. Store and retrive in / from a step Finally, the following snippet shows how to store in and retrieve from the scenario.

... public Object execute(Object resultFromPreviousAction) throws ActionExecutionException { ... RedBean redBean = processIt((GreenBean) getStep().getScenario().getValue("theBeanThatIsShared")); getStep().getScenario().setValue("theBeanToBeShared", redBean); ... } ...

Insert 12. Store and retrive in / from a scenario

Nesting actions Some scenarios requires nesting action(s) inside other action. We achieve nesting by, first, adding a <step> inside the . Further, inside the . Please note that unlike scenario, action can only have one child step. The structure of a scenario XML (with nesting) can be depicted the following way: •

Scenario ◦ Steps ▪ Step • Actions ◦ Action ◦ Action ▪ Step • Actions ◦ Action ◦ Action ◦ ... ◦ Action ◦ … ▪ Step

Insert 13. Tree-structure of a scenario XML Consider the following example which is taken from the real, current, project (LaCross). In the test scenario we needed to setup 20 conferences. Each conference in turn has to have 5 segments, and each segment has to be filled with 100 participants. In order to realize it, we have a step with multiplicity 20 – step “Create conference” ➊ – where we define the BILLING_CONFIGID for each conference that will be created ➋. Inside that step we have an action, a special kind of action that does nothing, com.baktun.bstrd.director.TransitAction ➌. It is a trick that allows us to go directly from a step to a (child)step without doing anything in between (because in this case we don't really create the instance of conference before creating the segments. The conference is only an abstract concept that is created at the time the segment is created). We already saw a nesting (of step “Create segment”) inside an action (action “Transit to segment creation (and population)” ➍). Further down, we nest another step – step “Create participants” ➏ – inside the “Create segment” action ➎. Another practical thing that can be observed in the example – which has been described in the argument's value resolution section – is that we can use the argument defined in the parent node from a child node, typically to construct a more complex pattern. For example: the value of PARTICIPANT_ID ➐ defined in the “Create participant” action is based on the value of SEGMENT_ID defined in the “Create segment” step. Similarly, the value of PHONE_NUM ➑ defined in the “Create participant” action is constructed by concatenating the value of BILLING_CONFID defined at the scenario level and the value of SEGMENT_ID defined in the “Create segment” step.

<scenario max-thread="20" repetition="0"> <desc>Scenario QA <wait timeout="10">eq(mediator("isStarted()"), true) <steps>



<step mult="20" concurrent="true" name="Create conference"> <args>



str(range(10101, 10120, 1))concat("SpectelEmu_", ref(BILLING_CONFID))



<step mult="5" concurrent="true" name="Create segment"> ➍ <args> short(sum(mult(stepidx("../.."), stepmult(".")), stepidx("."), 1)) ➎ <args> "OPERATOR_ASSISTED" false null null null <script> <step mult="100" concurrent="true" name="Create participants"> <wait>rand(5, 10) <args>



short(sum(mult(sum(ref(SEGMENT_ID), -1), stepmult("..")), stepidx(".."), 1))false false concat("Participant_", ref(CONF_NAME), "_", ref(SEGMENT_ID), "_", sum(stepidx(".."), 1)) "Baktun" concat(ref(BILLING_CONFID), "-", ref(SEGMENT_ID), "-", sum(stepidx(".."), 1)) ➑ <script>
<script> <script>
import com.baktun.bstrd.director.*; import com.baktun.bridgesimulator.*; Segment segment = mediator.getBridge().segments.get(action.getArgument("SEGMENT_ID")); for(Line lineItem : segment.getLines()) { //try { // Thread.sleep((long) (Math.random() * 5 * 1000)); //} catch (Exception ex) {} mediator.getBridge().addtoQAQueue(lineItem.getParticipantID()); } ]]>
<wait>int(mult(rand(1, 3), 60)) ➓ <script> <script> <script>


Specifiying wait In the above example we can see <wait> in action. A <wait> node can be placed in any execution node (scenario, step, or action). It will cause the execution of the node to be held on, until a specific condition is met. The condition is the expression that we type inside the <wait> node, which can be either any function that returns integer, or any function that returns boolean. In case the wait expression produces an integer (n), the scenario-runner will wait for (n) seconds before executing the node containing that wait. An example of this is the wait for action “Stop QA section” ➓. In case the wait expression is a boolean evaluation, as is the wait of scenario in the above example ➒, the scenario-runner will first evaluate the wait expression. The execution moves on normally if the wait expression evaluates to true. Otherwise, the thread in which the evaluation takes place will move suspended state (i.e.: calling the wait() method), and only wakes up when either the specified number of seconds (timeout attribute) has passed. When it wakes up, the wait expression will be re-evaluated. Wash. Rinse. Repeat. There's an advanced mechanism that causes the thread to wake up upon notification (thus avoiding the polling). For that, we need to have an object – an approriate type of object – that is registered as a listener for the events that take place inside the mediator. Upon receiving the notification from the mediator, that listener will have to invoke the notifyAll() method, such that the scenario-runner gets back to live from the suspended state, and re-evaluates the wait expression. Additionaly, the scenariorunner must be made aware of the existence of such listener. It is achieved by invoking the setMediatorListener method from inside the very first action that the scenario executes. The other important, but optional, property of <wait> is timeout. When it's set to true, the scenariorunner will evaluate the <wait> before it evaluates the (see specifying condition below). By default it is set to true.

Specifying condition An execution-node can have a node, where we specify the condition under which the execution-node will be executed. The node accepts boolean expression as its value. Normally we check the value stored in the context or a property of the mediator against a specific value. See the description of the context or mediator function to understand how to check the object stored in the context or mediator.

Specifying expectation Finally, an execution-node can have a <expect> node, where we specify the checkings to be performed at the end of the execution of the execution-node. Just like the , it accepts any boolean expression as its value, and normally we would make the context or mediator function together with either the eq, lt, or gt function. Additionaly, only for the <expect>, we can also use the error function, if we expect that the execution would throw an error.

Related Documents

Test Scenario
November 2019 1
04. Test Scenario
May 2020 19
Scenario
October 2019 65
Scenario
October 2019 70
Scenario
May 2020 42