Automated Software Validation And Verification Process

Automated Validation and Verification of Software Process Models

Abstract Using the process programming concept, processes are modeled as pieces of software. A process programming language is used to formally specify the process. Such a language resembles a conventional programming language, typically providing constructs such as iteration and selection. One advantage of this approach is that a process model once coded can be simulated easily. However, this approach also suffers from the same problems that plague traditional programming tasks, such as the question of whether the program itself is meaningful (i.e., semantically correct) or contains errors. Such program errors are the result of a miscoding of the model or of an error in the model itself. We present an automated approach for detecting errors in such process models. Our approach is based on static code analysis techniques such as dead code detection and register allocation used by optimizing compilers. By slightly adapting these well-understood techniques, errors can be found quickly before simulation. We have developed a tool to perform such semantic checks on processes modeled using the PML process modeling language.

1 Introduction In 1991 Lee Osterweil asserted that “software processes are software too” [1], and thus could (and should) be developed, analyzed, and managed using the same software engineering methods and techniques that are applied to software. This idea implies there is a software process life-cycle that resembles the software life-cycle, involving analysis, design, implementation, and maintenance of software processes [2]. One of the outgrowths of this line of research is the notion of process programming: the specification of pro-

1

cess models using process programming languages that resemble, and in some cases are derived from, conventional programming languages [3]. There is a large body of knowledge comprising techniques for analyzing conventional programming languages. These techniques enable programmers to assess the correctness of their programs, identify potential faults, and, as in the case of optimizing compilers, automatically re-design the implementation of a program to improve execution performance. Can these techniques be applied to process programs? Specifically, can program analysis techniques be applied to process programs, to Validate the correctness of process programs as models of real-world processes, and Aid in process redesign? In this paper we present a technique for analyzing the flow of resources through a process, as specified by a process program. This technique, derived from research into data-flow analysis of conventional programming languages, enables a process designer to answer important questions about a process model, including: Does a process actually produce the product it is supposed to? Are intermediate products consumed by later steps in a process actually produced by earlier steps? Does the flow of resources through a process match the flow of control? Does the flow of resources suggest opportunities for improving the process, for example by increasing parallelism or assigning tasks to different roles? The answers to these questions can result from errors in the specification, indicating a need for further capture and modeling activities; or, they may highlight flaws in the underlying process, indicating a potential for process improvement. We have developed a tool that implements this technique to analyze specifications written in the PML process programming language. Based on well understood data-flow analysis techniques, the analysis tool allows a process designer to quickly and automatically analyze a PML process model to aid in assessing its correctness, and for use in redesign. 2

We begin with a brief overview of PML, in order to provide a context for discussing our technique. Then, we present our analysis technique, and the implications for process engineering and re-engineering of the data it provides. Following that, we discuss the implementation of the analysis tool, and the results of its application to actual PML process specifications. We conclude with our assessment of the technique and its implementation, and potential directions for future work.

2 The PML Process Programming Language PML1 is a simple process programming language intended to model organizational processes at varying levels of detail [4]. PML was designed specifically for rapid, incremental process capture, to support both process modeling and analysis, and process enactment [5]. PML’s design emphasizes simplicity, of both the language itself, and the models it specifies. As such, it is a true modeling language intended to capture the essential aspects of a process while abstracting away other details. Using PML, a process can be specified initially at a very high level that contains only major process steps and control flow. Deployment can be incremental as well: high level specifications can be syntactically correct and thus enactable, yet may be missing significant detail. As understanding of process details emerges, the specification can be elaborated to capture them.

2.1 Overview PML reflects the conceptual model of process enactment developed by Mi and Scacchi [6]. This model views a process as a situation in which agents use tools to perform tasks that require and produce resources (Figure 1). PML models processes as collections of actions that represent atomic process tasks. An action declaration contains an identifier and a (possibly empty) set of additional fields, shown in Table 1. Note that only the name field is required; all others are optional. This supports the goal of incremental specification: a process can be quickly described as a set of named actions, validated by process performers, then elaborated over time as the details are understood. 1 The name “PML” is an acronym, originally for “Process Markup Language.” The language quickly evolved to resemble a conventional programming language more than a markup language, but the PML name was retained for historical reasons.

3

Control Structures PML specifies the order in which actions should be performed using conventional programming language control flow constructs such as sequencing, iteration, and selection, as well as concurrent branching of process flows. sequence A series of tasks to be performed in order.

sequence { action first {} action second {} action third {} }

iteration A series of tasks to be performed repeatedly.

iteration { action first {} action second {} action third {} } action go_on {}

Note that iterations have no explicit termination condition. Instead, the decision of when to exit the loop is left to the actor performing the process. selection A set of tasks from which the actor should choose one to perform.

selection { action choice_1 {} action choice_2 {} action choice_3 {} }

branch A set of tasks which can be performed concurrently; all of the tasks in a branch must be performed before the process can continue execution.

4

branch { action path_1 {} action path_2 {} action path_3 {} }

2.2 Resource Predicates and Resource Flow The provides and requires fields of an action specify how resources are transformed as they flow through a process. As such, they capture several important facts about a process, namely: What conditions must exist before an action can begin. What conditions exist after an action is completed. As a result, the requires and provides predicates specify the purpose of an action, in terms of how the action affects the product(s) under development, and the goal of a process as a whole, in terms of the products it produces. The simplest form of a resource predicate just names the resource provide or required:

provides { resourceName }

This predicate states that the output of an action is a resource bound to the variable resourceName. Resource specifications may also be predicates that constrain the state of the resource:

requires { resourceName.attributeName op value }

where op is a comparison operator (==, !=, <, <=, >, >=). Clauses may be joined by Boolean operators:

provides { resource1.attribute1 op value && resource2.attribute2 op value2 ...}

Resource predicates allow process designers to specify in some detail how a product evolves as a process progresses, as well as what resources are required to produce a product, and the state those resources must have before the process can proceed. 5

3 Analyzing Resource Flow What can we learn from analysis of syntactically correct process programs? Analysis helps in two phases of the process life-cycle. First, by analyzing the flow of resources through a process specification, we can identify situations where provided and required resources do not match. This information is useful for validating process specifications against reality; such inconsistencies may indicate gaps in process capture and understanding. Second, resource analysis can also point out potential areas of improvement in the process being modeled. Inconsistencies between provided and required resources signal a potential for re-engineering to make the process more effective. For example, if a sequence of actions does not have a resource flowing from one action to the next, it may be possible to perform those actions concurrently. At the extreme, an action that does not consume or produce any resources may be redundant and therefore a candidate for elimination. In the following sections, we examine in detail the kinds of inconsistencies that can exist in a specification, and their potential impact on a process. Then, we discuss the design of a tool for detecting these inconsistencies in PML specifications.

3.1 Categories of Inconsistencies Resource flow inconsistencies can be classified into several situations: 1. Required resource not provided. In this situation, an action requires a resource that is not provided by any preceding action:

action a { provides { r } } action b { requires { s } }

2. A provided resource is never used. An action might provide a resource that is never required by a subsequent resource. 6

action a { provides { r && s } } action b { requires { r } }

Inconsistencies due to unprovided or unrequired resources are not necessarily errors: an unrequired resource could indicate an action that represents an output of a process; an unprovided resource could indicate a point where the process receives input from another process. Depending on the context surrounding a particular process, unprovided or unrequired resources represent points where a process consumes some input from the environment, or produces some output to the environment. 3. A provided resource does not match a subsequent resource requirement. Here, the resource is not missing, but rather in the wrong state.

action a { provides { r.status == 1 } } action b { requires { r.status == 2 } }

4. A resource is provided by an action that does not require any resources. This situation (termed a “miracle” [2]) occurs when an action produces something without consuming any resources.

action miracle { provides { r } }

It could represent a modeling error where the modeler failed to capture an action’s inputs; or, it could represent a real situation where the actor generates something like a document from (intangible) ideas.

7

action describe_problem { /* requires inspiration from experience, etc. */ provides { problem_description } }

5. A resource is required by an action that does not provide any resources. This situation (termed a “black hole” [2]) represents a process step that consumes resources, but does not produce any product.

action black_hole { requires { r } }

This could represent a legitimate activity, such as a task that requires the actor to read certain documents and develop an “understanding” of their contents; the action produces no tangible results, but is worthwhile nevertheless:

action understand_problem { requires { problem_description } /* Provides nothing tangible, but the actor presumably understands the problem issues now. */ }

6. A resource is required, and a different resource is provided.

action transform { requires { r } provides { s } }

Occasionally, this situation represents a modeling error, but is more often the desired result: an action consumes some resources in the production of another. A simple example happens when a document is assembled from different sections: the action requires each section, and provides the completed document.

8

action submit_design_report { requires { use_cases && architecture && design_rationale } provides { design_report } }

3.2 Analysis Tool Design Our analysis tool, called pmlcheck, is intended to complement the PML compiler. The compiler generates executable models, useful for simulation and enactment. The analyzer tells us interesting things about these models. Pmlcheck is a command-line tool that can be invoked on a set of PML models. The user can specify which inconsistencies to look for using command-line options. These are summarized in Table 2. Pmlcheck reports the name of the action that is the source of an inconsistency, the location in the PML file, and the kind of inconsistency detected:

% pmlcheck -pr example.pml example.pml:5: s in action ’b’ is unprovided example.pml:8: t in action ’c’ is never required

Pmlcheck analyzes a process model as follows: 1. Parse the PML source code and construct a graph of the control-flow. 2. For each action in the control-flow graph (a) Determine whether the action is empty, is a miracle or black hole, or transforms resources. (b) Search backward for actions that provide the current action’s required resources. (c) Search forward for actions that require the current action’s provided resources. (d) Record findings. 3. Report specified inconsistencies.

9

3.2.1 Issues Parallel paths. PML’s select construct introduces the possibility that a resource may be provided along one path, but not another:

selection { action a { provides { r } } action b { provides { s } } } action c { requires { r } }

In the fragment above, if action a is selected, c’s requirements are satisfied; if b is selected, c’s requirements are not satisfied. Pmlcheck can be directed to treat this situation in one of two ways:

1. Lax analysis determines that at least one path through a process has a consistent resource flow. This indicates that the process may work, if the right path is chosen. If this option is chosen, no inconsistency will be reported. This is useful from a validation perspective: if no consistent paths can be found, yet the process is known to work in practice, the process specification is likely in error. This is the default analysis mode. 2. Strict analysis requires that all paths through a process have consistent resource flows. This means that if the process steps are performed correctly, the process will always work in the sense that it will produce the intended products. Pmlcheck will report an inconsistency unless this condition is met. This is useful for process redesign: if analysis shows that some inconsistent paths exist, it may identify why a process occasionally breaks down in practice. Strict analysis of the example above produces the following result:

% pmlcheck -s select.pml

10

select.pml:10: r in action ’c’ is possibly unprovided %

Iteration

An iteration over a selection presents a situation similar to the selection problem above. It is possible for

one path of a selection to provide a resource, and another to require it:

iteration { selection { action a { provides { r } } action b { requires { r } } } }

If action a is selected the first time through the loop, and b the second, b’s resource requirements will be satisfied; this is not the case if the order is reversed. By default, pmlcheck performs lax analysis on these situations: if any ordering exists that satisfies an action’s resource requirements, no inconsistency is reported. Pmlcheck can be directed to flag potential inconsistencies if some ordering would fail to satisfy an action’s resource requirements:

% pmlcheck -I iterate-select.pml iterate-select.pml:7: r in action ’b’ is possibly unprovided %

Separate Analyzer or Stricter Compilation? Why not force global consistency at compile time, as modern programming languages do, rather than using a separate analysis tool? First, this is not necessary: useful process analysis and enactment are possible without global consistency. Second, it’s not always possible. Process capture is an iterative process that uncovers hidden activities over time, as process understanding emerges. Thus it is desirable to have specifications that are incomplete or inconsistent, but that still provide useful input to downstream tools.

11

Finally, valid models can be inconsistent, because the underlying process being modeled is inconsistent. An organization’s processes may contain useless steps, missing steps, or sequences of activities that do not produce desired results. Nevertheless, it is important to document these processes accurately, to establish a baseline for process redesign. Therefore, the process engineering environment must be tolerant of inconsistencies that exist in the real world. In the following section, we examine the use of these techniques, as realized in pmlcheck, to analyze a set of software development processes.

4 Examples and Results To assess the effectiveness of pmlcheck, we analyzed two software development processes: the development process used to conduct Computer Engineering Senior Design projects at Santa Clara University, and a graduate Software Engineering course software development process.

4.1 SCU Senior Design Process Our first experiment employed pmlcheck to aid in the creation of a model of the Santa Clara University Computer Engineering department’s senior design project process. The process spells out a set of milestones and associated deliverables roughly based on Boehm’s Anchoring Milestones [7], as shown in Table 3. The experiment was performed in two iterations to capture, model, analyze, and refine the process specification in PML, as follows: 1. Initial capture, in which we simply translated the narrative specification into PML; 2. Refinement, in which we used the analysis provided by pmlcheck to improve the accuracy of the model by correcting specification errors and elaborating resource specifications. The first version of the model was a simple translation of the narrative specification into a PML specification. We modeled each milestone as a sequence of actions, each action producing a single deliverable. The initial model is listed in Appendix A. Pmlcheck reported sixty-three potential inconsistencies in this initial model (see Table 5). How many of these were actual errors? To determine the answer, we analyzed the reported inconsistencies in detail, categorizing them as 12

follows: Specification error. The modeler made a mistake in the specification, such as misspelling a resource name. Modeling error. The model did not match the underlying process. For example, an action was out of order or missing. Process error. The model was correct, but the underlying process contained an inconsistency. Spurious error. Pmlcheck correctly identified an error, but the error was triggered by another error. For example, if the model misspells the name of a provided resource, an action later in the model that requires the resource (with the correct spelling) will trigger an “unprovided resource” error. No error. Pmlcheck incorrectly reported an inconsistency. Of sixty-three reported inconsistencies, three were specification errors where a resource name was misspelled, and two were spurious errors, caused by the specification errors. An additional two were not errors; these represented output (the “system prototype” and “problem statement document” resources). The remaining fifty-six errors were the result of incorrectly modeling some aspect of the process, such as omitting a required or provided resource from an action (forty-two inconsistencies). These conclusions are summarized in Table 4. Perhaps most interesting from a process engineering viewpoint, thirteen errors were the result of omitting actions to capture and deliver document components as a single document; in the fragment shown in Figure 2, the sequence is missing a “submit design report” action to assemble the document parts and deliver them as a completed “design report” resource. We used this analysis of the initial version of the model to correct the modeling errors uncovered by pmlcheck. The revised specification is listed in Appendix B; the output from pmlcheck is shown in Figure 3. We again analyzed this model with pmlcheck; twelve inconsistencies were reported, none of which were errors, as they represented process input or output.

13

4.2 Graduate Software Processes We also analyzed twenty-four process models developed by graduate software engineering students to describe the class project development process. The intent was to develop formal models of the processes specified by the instructor as narrative text in class assignment documents and class lectures, augmented by the students’ personal experience in performing these processes. The analysis results are shown in Table 5.

4.3 Discussion It appears from these experiments that the majority of inconsistencies reported by pmlcheck are unprovided or unrequired resources. This is the chief limitation of pmlcheck: because PML does not distinguish between provided and required resources and process inputs and outputs, pmlcheck takes a conservative approach and reports process inputs as unprovided resources, and process outputs as unrequired resources. Curiously, the Graduate Software Development processes contained only two miracles and no black holes among ninety-five actions; in contrast, the initial Senior Design model had twenty-eight miracles and fifteen black holes. This appears to be the due to careful attention to detail on the part of the three modelers who wrote these specifications. Pmlcheck reported sixty-seven transformations in the Graduate Software Development processes; thirty-one of these proved to be specification errors that caused the provided resource to appear to be a new resource rather than a modification of the required resource. This was a surprise: we had anticipated that most actions identified as transformations would actually transform resources into new resources. Thus, it appears to be useful to optionally flag actions that transform resources for closer examination.

5 Related Work 5.1 Program Analysis Many of the checks performed by our tool are analogous to those performed by optimizing compilers such as gcc [8] and static checkers such as lint [9]. Optimizing compilers typically warn the user regarding possibly uninitialized variables (variables whose values are used before being assigned). Our analysis tool informs the user regarding resources that are required without possibly being provided. As another example, register allocation [10], the process of 14

effectively assigning registers to variables to increase execution speed, requires knowledge of the lifetimes of variables in a program. Such knowledge is obtained by computing when a variable is first and last possibly referenced, which is analogous to determining when a resource is first provided and last required. Other common optimizations that follow the flow of variables through a program include common-subexpression elimination and code motion (moving loop-invariant code out of a loop) [11]. Algorithms for understanding and analyzing programs described as graphs are well-known [11, 12, 13], as are algorithms for computing properties of the graphs themselves [14]. Finally, other tools to aid the programmer in finding errors in programs include program slicing tools [15, 16, 17] and assertion checkers [18].

5.2 Process Validation Cook and Wolf [19] discuss a method for validating software process models by comparing specifications to actual enactment histories. This technique is applicable to downstream phases of the software life-cycle, as it depends on the capture of actual enactment traces for validation. As such, it complements our technique, which is an upstream approach. Similarly, Johnson and Brockman [20] use execution histories to validate models for predicting process cycle times. The focus of their work is on estimation rather than validation, and is thus concerned with control flow rather than resource flow. Woflan [21] is a tool for process specification verification based on a set of process correctness measures derived from properties of Petri-nets. This approach first translates the process description into an equivalent Petri-net, which is then analyzed for properties Petri-nets that imply process properties such as absence of deadlocks. While these properties ensure that the models are executable, it is not clear how other Petri-net properties relate to real world processes. In contrast, the inconsistencies that pmlcheck reports are derived from actual experience with industrial process models [22, 5, 23]. Scacchi’s research employs a knowledge-based approach to analyzing process models. Starting with a set of rules that describe a process setting and models, processes are diagnosed for problems related to consistency, completeness, and traceability [2]. Conceptually, this work is most closely related to ours; many of the inconsistencies uncovered by pmlcheck are also revealed by Scacchi and Mi’s Articulator [24]. As mentioned in Section 2, PML and the Articulator

15

share the same conceptual model of process activity. However, there are some important differences. First, their approach is based on knowledge-based techniques, with rule-based process representations and strong use of heuristics. This is a different approach to process modeling than PML’s, which closely resembles conventional programming. Thus, our analysis technique is derived from programming language research.

6 Conclusion What can we conclude about data-flow analysis of process programs? Data-flow analysis can uncover specification errors, such as misspelled resource names, that can exist in otherwise syntactically correct process specifications. Without analysis, these errors would not be detectable until the process is executed. Also, resource flow analysis can identify inconsistencies between a specification and the process it models. This was shown in Section 4, where our initial Senior Design process model was missing several resource dependencies that were important to the process. Further, data-flow analysis can validate that a process produces the products it was intended to produce. By verifying that the resource flow specified by the process program proceeds correctly from beginning to end, the process designer can validate that the process does in fact transform its inputs into the desired outputs. Finally in addition to identifying potential errors in a process specification, resource flow analysis can suggest opportunities for redesign of a valid process. For example, in Section 3, our revised Senior Design model contains six actions that require the “problem statement” resource (Appendix B, lines 5–24). Where does this resource come from? At present, the process assumes that the problem statement exists prior to the beginning of the process. But the intent of the process is for professors to provide problems for student teams to solve; so the process should include a phase where students and professors negotiate the problem statement, which then serves as the input to the Conception phase.

6.1 Future Work The use of data-flow analysis for process re-engineering suggests several opportunities for future work.

16

6.1.1 Automatic Re-engineering A sequence specifies a temporal dependency between actions: a predecessor must be completed before the successor can begin. This implies that the predecessor does something that the successor needs; in other words, the predecessor provides something that the successor requires. If the resources analysis shows that no resource flows between sequential actions, however, it may indicate an opportunity for concurrency. In this case, the process specification indicates a dependency among actions that does not exist. The opposite situation occurs when the control-flow specification indicates that actions can be performed concurrently, but the resource flow among them requires that they performed in a certain order. This situation may indicate either an error in process capture, or a problem with the process itself. This suggests a tool for automatically transforming a specification into an equivalent specification based on the resource flow graph. Such a tool would analyze the resource dependencies among actions, then re-arrange their ordering so that the flow of control matches the resource flow.

6.1.2 Process Decomposition In certain situations, it is useful to decompose a process into independent fragments, that can be assigned to separate actors that may be part of independent, autonomous organizations [25]. In order to do this, we need to know where a process transitions from one agent to another, and what resources flow across this transition. Resource flow analysis identifies resource dependencies among actions. A similar analysis approach that examines the agent field of actions could reveal actions that are performed by various actors, producing a set of sub-graphs, each of which is bound to a single actor. Resource flow analysis would then determine how these sub-graphs are connected, through shared resources.

6.1.3 PML Language Enhancements The analysis of actual PML programs discussed in Section 4 revealed certain deficiencies in PML. Specifically, because PML makes no distinction between resources provided by or required from actions within the process and resources provided by or to the external environment, pmlcheck cannot distinguish between an unprovided resource and a process input, and likewise between an unrequired resource and a process output. This suggests the need for an enhancement

17

to PML to allow the process modeler to specify the process inputs and outputs.

References [1] L. Osterweil, “Software processes are software too,” in Proceedings of the 9th International Conference on Software Engineering, (Monterey, CA USA), Mar. 1987. [2] W. Scacchi, “Understanding software process redesign using modeling, analysis and simulation,” Software Process–Improvement and Practice, 2000. [3] S. M. Sutton, Jr., L. J. Osterweil, and D. Heimbigner, “APPL/A: A language for software process programming,” ACM Transactions on Software Engineering and Methodology, vol. 4, pp. 221–286, July 1995. [4] J. Noll and W. Scacchi, “Supporting software development in virtual enterprises,” Journal of Digital Information, vol. 1, Jan. 1999. http://jodi.ecs.soton.ac.uk/Articles/v01/i04/Noll/. [5] W. Scacchi and J. Noll, “Process-driven intranets: Life-cycle support for process engineering,” IEEE Internet Computing, Sept./Oct. 1997. http://www.usc.edu/dept/ATRIUM/Papers/PDI.pdf. [6] P.-W. Mi and W. Scacchi, “A knowledge-based meta-model for formulating models of software development processes,” Decision Support Systems, vol. 17, no. 3, 1996. [7] B. W. Boehm, “Anchoring the software process,” IEEE Software, vol. 13, pp. 73–82, July 1996. [8] R. M. Stallman, GCC Reference Manual. Cambridge, MA: Free Software Foundation, 1991. [9] S. C. Johnson, “Lint: A C program checker,” computing science technical report, AT&T Bell Laboratories, Murray Hill, NJ, July 1978. [10] F. C. Chow and J. L. Hennessy, “A priority-based coloring approach to register allocation,” ACM Trans. Programming Languages and Systems, vol. 12, pp. 501–536, Oct. 1990. [11] A. V. Aho, R. Sethi, and J. D. Ullman, Compilers: Principles, Techniques, and Tools. Reading, MA: AddisonWesley, 1986. [12] R. Cytron, J. Ferrante, B. K. Rosen, M. N. Wegman, and F. K. Zadeck, “Efficiently computing static single assignment form and the control dependence graph,” ACM Trans. Programming Languages and Systems, vol. 13, pp. 451–490, Oct. 1991. [13] J. Ferrante, K. J. Ottenstein, and J. D. Warren, “The program dependence graph and its use in optimization,” ACM Trans. Programming Languages and Systems, vol. 9, pp. 319–349, July 1987. [14] T. Lengauer and R. E. Tarjan, “A fast algorithm for finding dominators in a flowgraph,” ACM Trans. Programming Languages and Systems, vol. 1, pp. 121–141, July 1979. [15] M. Weiser, “Program slicing,” IEEE Trans. Software Engineering, vol. SE-10, pp. 352–357, July 1984. [16] M. Mock, D. C. Atkinson, C. Chambers, and S. J. Eggers, “Improving program slicing with dynamic points-to data,” in Proceedings of the 10th ACM International Symposium on the Foundations of Software Engineering, (Charleston, SC), Nov. 2002. (To appear). [17] F. Tip, “A survey of program slicing techniques,” Journal of Programming Languages, vol. 3, pp. 121–189, Sept. 1995. [18] D. Jackson, “ASPECT: An economical bug-detector,” in Proceedings of the 13th International Conference on Software Engineering, (Austin, TX), pp. 13–22, May 1991.

18

[19] J. E. Cook and A. L. Wolf, “Software process validation: Quantitatively measuring the correspondence of a process to a model,” ACM Transactions on Software Engineering and Methodology, vol. 8, pp. 147–176, April 1999. [20] E. W. Johnson and J. B. Brockman, “Measurement and analysis of sequential design processes,” ACM Transactions on Design Automation of Electronic Systems (TODAES), vol. 3, no. 1, pp. 1–20, 1998. [21] W. M. P. van der Aalst, “Woflan: A petri-net-based workflow analyzer,” Systems Analysis - Modelling - Simulation, vol. 34, no. 3, pp. 345–357, 1999. [22] J. Noll and W. Scacchi, “Specifying process-oriented hypertext for organizational computing.,” Journal of Networking and Computer Applications, vol. 24, Jan. 2001. http://www.academicpress.com/jnca. [23] W. Scacchi, “Experience with software process simulation and modeling,” The Journal of Systems and Software, vol. 46, pp. 183–192, April 1999. [24] W. Scacchi and P. Mi, “Process life cycle engineering: A knowledge-based approach and environment,” Intelligent Systems in Accounting, Finance, and Management, vol. 6, pp. 83–107, 1997. [25] J. Noll and B. Billinger, “Modeling coordination as resource flow: An object-based approach,” in Proceedings of the 2002 IASTED Conference on Software Engineering Applications, (Cambridge, MA, USA), Nov. 2002. http://www.cse.scu.edu/˜jnoll/sea-02.pdf.

A

Initial Senior Design Model 1 /* COEN Senior Design Process */ 2 process senior_design { 3 sequence problem_statement { 4 action understand_problem { 5 provides { problem_statement } 6 } 7 action create_project_motivation { 8 requires { problem_statement } 9 provides { motiviation } 10 } 11 action create_background { 12 requires { problem_statement } 13 provides { background } 14 } 15 action create_business_case { 16 requires { problem_statement } 17 provides { business_case } 18 } 19 action create_user_scenarios { 20 requires { problem_statement } 21 provides { scenario } 22 } 23 action create_evaluation_criteria { 24 requires { problem_statement } 25 provides { evaluation_criteria } 26 } 27 action submit_problem_statement { 28 requires { problem_statement && motivation && background && business_case && scenario 29 && evaluation_criteria } 30 provides { problem_statement_document } 31 } 32 } 33 sequence design_report {

19

34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91

action create_project_plan { provides { project_plan } } action create_risk_analysis { provides { risk_analysis } } action create_use_cases { provides { use_cases } } action create_conceptual_model { provides { conceptual_model } } action create_requirements_list { provides { requirements } } action create_sequence_diagram { provides { sequence_diagram } } action create_architecture { provides { architecture } } action create_design_rationale { provides { design_ratinale } } action create_technologies_used { provides { technologies_used } } action create_component_state_charts { provides { state_chart } } action create_test_plan { provides { test_plan } } action create_user_manual { provides { user_manual } } action capture_test_results { provides { test_results } } } sequence design_review { action create_system_prototype { provides { system_prototype } } action create_collaboration_diagrams { provides { collaboration_diagram } } action create_class_diagram { provides { class_diagram } } action create_design_presentation { provides { presentation } } action perform_design_review { } } sequence revised_design_report {

20

92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149

action revise_risk_analysis { requires { risk_analysis } } action revise_use_cases { requires { use_cases } } action revise_conceptual_model { requires { conceptual_model } } action revise_requirements_list { requires { requirements } } action revise_sequence_diagram { requires { sequence_diagram } } action revise_architecture { requires { architecture } } action revise_design_rationale { requires { design_rationale } } action revise_technologies_used { requires { technologies_used } } action revise_component_state_charts { requires { state_chart } } action revise_test_plan { requires { test_plan } } action revise_user_manual { requires { user_manual } } action revise_collaboration_diagrams { requires { collaboration_diagrams } } action revise_class_diagram { requires { class_diagram } } action revise_project_plan { requires { project_plan } } } sequence operational_system { action create_implementation { provides { code } } action create_build_scripts { provides { build_script } } action create_deployment_instructions { provides { deployment_instructions } } action create_test_cases { provides { test_cases } } action capture_test_results { provides { test_results }

21

150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 }

} } sequence design_conference { action revise_presentation { requires { presentation } provides { presentation } } action create_system_demo { provides { system_demo } } action perform_presentation { requires { presentation && system_demo } } } sequence project_report { action create_API_documentation { provides { api_doc } } action create_maintenance_guide { provides { maintenance_guide } } action create_suggested_changes { provides { suggested_changes } } action create_lessons_learned { provides { lessons_learned } } action capture_test_results { requires { test_cases } provides { test_results } } } sequence final_implementation { action revise_implementation { requires { code } provides { code } } action revise_build_scripts { requires { build_script } provides { build_script } } action revise_deployment_instructions { requires { deployment_instructions } provides { deployment_instructions } } action enhance_test_cases { requires { test_cases } provides { test_cases } } action capture_test_results { requires { test_cases } provides { test_results } } }

22

B Revised Senior Design Model 1 /* COEN Senior Design Process */ 2 process senior_design { 3 sequence problem_statement { 4 branch { 5 action create_project_motivation { 6 requires { problem_statement } 7 provides { motivation } 8 } 9 action create_background { 10 requires { problem_statement } 11 provides { background } 12 } 13 action create_business_case { 14 requires { problem_statement } 15 provides { business_case } 16 } 17 action create_user_scenarios { 18 requires { problem_statement } 19 provides { scenario } 20 } 21 action create_evaluation_criteria { 22 requires { problem_statement } 23 provides { evaluation_criteria } 24 } 25 } 26 action submit_problem_statement { 27 requires { problem_statement && motivation && background && business_case && scenario 28 && evaluation_criteria } 29 provides { problem_statement_document } 30 } 31 } 32 sequence design_report { 33 branch { 34 action create_project_plan { 35 requires { problem_statement_document } 36 provides { project_plan } 37 } 38 action create_risk_analysis { 39 requires { problem_statement_document } 40 provides { risk_analysis } 41 } 42 action create_use_cases { 43 requires { problem_statement_document } 44 provides { use_cases } 45 } 46 action create_conceptual_model { 47 requires { problem_statement_document } 48 provides { conceptual_model } 49 } 50 action create_requirements_list { 51 requires { problem_statement_document } 52 provides { requirements } 53 } 54 action create_sequence_diagram { 55 requires { problem_statement_document } 56 provides { sequence_diagram }

23

57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114

} action create_architecture { requires { problem_statement_document provides { architecture } } action create_design_rationale { requires { problem_statement_document provides { design_rationale } } action create_technologies_used { requires { problem_statement_document provides { technologies_used } } action create_component_state_charts { requires { problem_statement_document provides { state_chart } } action create_test_plan { requires { problem_statement_document provides { test_plan } } action create_user_manual { requires { problem_statement_document provides { user_manual } } action capture_test_results { requires { test_cases } provides { test_results } }

}

}

}

}

}

}

} action submit_design_report { requires { project_plan && risk_analysis && use_cases && conceptual_model && requirements && sequence_diagram && architecture && design_rationale && technologies_used && state_chart && test_plan && user_manual && test_results } provides { design_report } } } sequence design_review { branch { action create_system_prototype { requires { use_cases && requirements } provides { system_prototype } } action create_collaboration_diagrams { requires { architecture } provides { collaboration_diagram } } action create_class_diagram { requires { requirements && use_cases && scenario } provides { class_diagram } } } action create_design_presentation { requires { system_prototype && collaboration_diagram && class_diagram } provides { presentation } } action perform_design_review { requires { presentation }

24

115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172

provides { design_review } } } sequence revised_design_report { branch { action revise_project_plan { requires { project_plan } provides { project_plan } } action revise_risk_analysis { requires { risk_analysis } provides { risk_analysis } } action revise_use_cases { requires { use_cases } provides { use_cases } } action revise_conceptual_model { requires { conceptual_model } provides { conceptual_model } } action revise_requirements_list { requires { requirements } provides { requirements } } action revise_sequence_diagram { requires { sequence_diagram } provides { sequence_diagram } } action revise_architecture { requires { architecture } provides { architecture } } action revise_design_rationale { requires { design_rationale } provides { design_rationale } } action revise_technologies_used { requires { technologies_used } provides { technologies_used } } action revise_component_state_charts requires { state_chart } provides { state_chart } } action revise_test_plan { requires { test_plan } provides { test_plan } } action revise_user_manual { requires { user_manual } provides { user_manual } } action revise_collaboration_diagrams requires { collaboration_diagram provides { collaboration_diagram } action revise_class_diagram {

25

{

{ } }

173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230

requires { class_diagram } provides { class_diagram } } } action submit_revised_design_report { requires { project_plan && risk_analysis && use_cases && conceptual_model && requirements && sequence_diagram && architecture && design_rationale && technologies_used && state_chart && test_plan && user_manual && test_results && collaboration_diagram && class_diagram } provides { design_report } } } sequence operational_system { branch { action create_implementation { requires { design_report } provides { code } } action create_build_scripts { requires { code } provides { build_script } } action create_deployment_instructions { requires { code } provides { deployment_instructions } } action create_test_cases { requires { requirements && use_cases && user_manual } provides { test_cases } } action capture_test_results { requires { test_cases } provides { test_results } } } /* Following added after pmlcheck shows previous provides statements never required. */ action demo_operational_system { requires { code && build_script && deployment_instructions && test_cases && test_results } provides { operational_system } } } sequence design_conference { branch { action revise_presentation { requires { presentation } provides { presentation } } action create_system_demo { requires { code && build_script } provides { system_demo } } } action perform_presentation { requires { presentation && system_demo } provides { conference_evaluation } } }

26

231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288

sequence project_report { branch { action create_API_documentation { requires { architecture && code } provides { api_doc } } action create_maintenance_guide { requires { code } provides { maintenance_guide } } action create_suggested_changes { requires { code } provides { suggested_changes } } action create_lessons_learned { requires { code } provides { lessons_learned } } action capture_test_results { requires { test_cases } provides { test_results } } } action submit_project_report { requires { api_doc && maintenance_guide && suggested_changes && lessons_learned && test_results } provides { project_report } } } sequence final_implementation { branch { action revise_implementation { requires { code } provides { code } } action revise_build_scripts { requires { build_script } provides { build_script } } action revise_deployment_instructions { requires { deployment_instructions } provides { deployment_instructions } } action enhance_test_cases { requires { test_cases } provides { test_cases } } action capture_test_results { requires { test_cases } provides { test_results } } } /* Following added after pmlcheck shows previous provides statements never required. */ action demo_final_system { requires { code && build_script && deployment_instructions && test_cases && test_results } provides { final_system } }

27

289 290 }

}

28

List of Figures 1 2 3

The PML Conceptual Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fragment of Senior Design Process with Missing Action . . . . . . . . . . . . . . . . . . . . . . . . Pmlcheck Analysis of Revised Senior Design Process. . . . . . . . . . . . . . . . . . . . . . . . . . .

29

30 31 32

Agent Use

Perform

Task

Tool

Provide/Require

Modify Resource

Figure 1: The PML Conceptual Model

30

sequence design_report { action create_project_plan { provides { project_plan } } action create_risk_analysis { provides { risk_analysis } } action create_use_cases { provides { use_cases } } action create_conceptual_model { provides { conceptual_model } } action create_requirements_list { provides { requirements } } action create_sequence_diagram { provides { sequence_diagram } } action create_architecture { provides { architecture } } action create_design_rationale { provides { design_ratinale } } action create_technologies_used { provides { technologies_used } } action create_component_state_charts { provides { state_chart } } action create_test_plan { provides { test_plan } } action create_user_manual { provides { user_manual } } action capture_test_results { provides { test_results } } }

Figure 2: Fragment of Senior Design Process with Missing Action

31

pmlcheck -oiemb senior_design_2.pml senior_design_2.pml:5: problem_statement in action ’create_project_motivation’ is unprovided senior_design_2.pml:9: problem_statement in action ’create_background’ is unprovided senior_design_2.pml:13: problem_statement in action ’create_business_case’ is unprovided senior_design_2.pml:17: problem_statement in action ’create_user_scenarios’ is unprovided senior_design_2.pml:21: problem_statement in action ’create_evaluation_criteria’ is unprovided senior_design_2.pml:26: problem_statement in action ’submit_problem_statement’ is unprovided senior_design_2.pml:87: design_report in action ’submit_design_report’ is never required senior_design_2.pml:113: design_review in action ’perform_design_review’ is never required senior_design_2.pml:210: operational_system in action ’demo_operational_system’ is never required senior_design_2.pml:226: conference_evaluation in action ’perform_presentation’ is never required senior_design_2.pml:254: project_report in action ’submit_project_report’ is never required senior_design_2.pml:284: final_system in action ’demo_final_system’ is never required

Figure 3: Pmlcheck Analysis of Revised Senior Design Process.

32

List of Tables 1 2 3 4 5

Action Fields . . . . . . . . . Pmlcheck Analysis Options. . Senior Design Process . . . . Summary of Analysis Results . Errors Reported by pmlcheck .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

33

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

34 35 36 37 38

type

agent script

tool requires provides

Either manual or executable (optional). If an action is type executable, the run-time system will pass the action’s script attribute to the underlying platform for execution; these actions, in effect, are performed automatically without human intervention. Manual actions are tasks to be performed by human actors. The role of the actor that should perform the action (optional). For executable actions, the script is an executable string to be run by the operating system; for manual actions, the script contains documentation about how to perform the task represented by the action (optional). An application used by the actor to perform the task associated with the action (optional). Resources required to perform the action. These can be considered the action’s input (optional). Resources produced as a results of performing the action; the action’s output (optional). Table 1: Action Fields

34

Option p r e m b t s I

Inconsistencies Reported Provided resource never required. Required resource never provided. Action neither requires nor provides resources. Action provides, but does not require resources (“miracle”). Action requires, but does not provide resources (“black hole”). Action provides different resources than required (“transformation”). Perform “strict” analysis on selections. Perform “strict” analysis of iterations. Table 2: Pmlcheck Analysis Options.

35

Milestone Problem Statement

Design

Design Review

Revised Design Operational System

Final System

Deliverables Conception Phase Motivation Background Business Case User Scenarios Evaluation Criteria Elaboration Phase Project plan Project Risks Use Cases Conceptual Model List of Requirements (functional and non-functional) System Sequence Diagram/Flow Chart Architectural Diagram/Circuit Diagram/Hardware Block Diagram Design Rationale Technologies Used Component State Chart(s) Test Plan, including test cases User Manual Test or Experimental Results System Prototype Collaboration Diagrams Class Diagram or Schematic Risk resolution/mitigation Revised Design items Construction Phase Source code Build and install scripts Deployment instructions Test cases Test results Delivery Phase Final source code Final test cases Build and install scripts Deployment instructions User Manual API documentation Maintenance Guide Suggested Changes Experiences/lessons learned Complete test results

Table 3: Senior Design Process

36

Model senior design.pml senior design 2.pml

Number 63 12

Specification 3 0

Modeling 56 0

Process 0 0

Table 4: Summary of Analysis Results

37

Spurious 2 0

Not an Error 2 12

Model Senior Design senior design.pml senior design 2.pml TOTAL Graduate S/W Development Architecture.pml Checkout.pml Commit.pml Edit.pml PostMortem.pml Update.pml checkin.pml checkout.pml make.pml milestone1.pml milestone5.pml updateANDresolve.pml Analysis.pml FunctionalRequirements.pml Milestone2.pml Milestone3.pml NonFunctionalRequirements.pml OperationalConcept.pml ProjectLog.pml RepositoryCheckIn.pml RepositoryCheckOut.pml RepositorySynchronize.pml RiskIdentification.pml SourceCodeEdit.pml TOTAL TOTAL

Size

Actions

# Res.

Empty

Unprovided

Unrequired

Miracles

Blk. Holes

Xforms

204 290 494

35 37 72

69 122 191

1 0 1

3 7 10

16 6 22

28 0 28

15 0 15

36 42 78

130 11 12 27 44 18 13 16 13 172 141 22 10 10 59 45 10 10 10 10 20 20 10 33 866 1360

16 1 1 3 7 2 1 1 1 18 15 2 1 1 7 5 1 1 1 1 2 2 1 4 95 167

26 2 2 6 4 4 2 2 2 36 30 4 3 3 21 15 3 3 3 3 5 5 3 6 193 384

3 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 8 9

5 1 1 2 0 2 1 1 1 13 10 2 2 2 7 5 2 2 2 2 3 3 2 3 74 84

5 1 1 2 2 2 1 1 1 13 10 2 1 1 7 5 1 1 1 1 2 2 1 3 67 89

0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 30

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15

13 1 1 3 3 2 1 1 1 8 6 1 1 1 7 5 1 1 1 1 2 2 1 3 67 145

Table 5: Errors Reported by pmlcheck

38