The Effectiveness Of Multi-criteria Intelligence Matrices In Intelligence Analysis

THE EFFECTIVENESS OF MULTI-CRITERIA INTELLIGENCE MATRICES IN INTELLIGENCE ANALYSIS

LINDSEY N. JAKUBCHAK

A Thesis Submitted to the Faculty of Mercyhurst College In Partial Fulfillment of the Requirements for The Degree of MASTER OF SCIENCE IN APPLIED INTELLIGENCE

DEPARTMENT OF INTELLIGENCE STUDIES MERCYHURST COLLEGE ERIE, PENNSYLVANIA MAY 2009 i

ii

DEPARTMENT OF INTELLIGENCE STUDIES MERCYHURST COLLEGE ERIE, PENNSYLVANIA THE EFFECTIVENESS OF MULTI-CRITERIA INTELLIGENCE MATRICES IN INTELLIGENCE ANALYSIS A Thesis Submitted to the Faculty of Mercyhurst College In Partial Fulfillment of the Requirements for The Degree of MASTER OF SCIENCE IN APPLIED INTELLIGENCE

Submitted By: LINDSEY N. JAKUBCHAK

Certificate of Approval: _________________________________ Kristan J. Wheaton Assistant Professor Department of Intelligence Studies _________________________________ James Breckenridge Chairman/Assistant Professor Department of Intelligence Studies _________________________________ Phillip J. Belfiore Vice President Office of Academic Affairs MAY 2009 iii

iv

Copyright © 2009 by Lindsey N. Jakubchak All rights reserved.

v

“The work of managers, of scientists, or engineers, of lawyers-the work that steers the course of society and its economic and government organizations-is largely work of making decisions and solving problems. It is work of choosing issues that require attention, setting goals, finding or designing suitable courses of action, and evaluating and choosing among alternative actions.” -Nobel Laureate Herbert Simon “Life is the sum of all your choices. History equals the accumulated choices of all mankind.” -Albert Camus

vi

DEDICATION

This work is dedicated to: -My parents for their continued support and confidence in me and everything I have done throughout my life. -My friends who have greatly impacted the person that I am today. -My classmates who are constantly pushing me to achieve new levels of academic and professional excellence.

vii

ACKNOWLEDGEMENTS I would like to acknowledge my thesis advisor and primary reader, Kristan Wheaton, for his guidance throughout the thesis process and throughout my graduate studies at Mercyhurst College.

I would like to thank Professor Hemangini Deshmukh for her assistance with the statistical analysis on this work.

I would also like to thank Justin Hiskey for his assistance with editing my thesis, and for his overall support throughout my graduate studies

Finally, I would like to acknowledge Joshua Peterson for making his work available to me, and for his advice throughout the process.

viii

ABSTRACT OF THE THESIS The Effectiveness of Multi-Criteria Intelligence Matrices in Intelligence Analysis By Lindsey N. Jakubchak Master of Science in Applied Intelligence Mercyhurst College, 2009 Professor Kristan J. Wheaton, Chair

While there is a substantial body of literature related to the use and effectiveness of Multi-Criteria Decision Making (MCDM) in the general sense, there is none regarding the use of this method in intelligence analysis. This study discusses relevant literature on the conventional form of MCDM and addresses the differences necessary to convert MCDM to an intelligence methodology (tentatively titled Multi-Criteria Intelligence Matrices-MCIM).

Additionally, a controlled experiment was conducted to test the

hypothesis that MCIM is a valuable method to use in intelligence analysis. Findings suggest that MCIM is likely a valuable method to use when conducting analysis, though more research is recommended to confirm these results.

ix

TABLE OF CONTENTS Page COPYRIGHT PAGE …………………………………………………………...

iii

QUOTES………………………………………………………………………...

iv

DEDICATION………………………………………………………………….

v

ACKNOWLEDGMENTS………………………………………………………

vi

ABSTRACT…………………………………………………………………….

vii

TABLE OF CONTENTS……………………………………………………….

viii

LIST OF FIGURES……………………………………………………………..

xi

CHAPTER 1

INTRODUCTION………………………………………………

1

2

LITERATURE REVIEW……………………………………….

4

3

Intuition vs. Structured Methodology……..……………. Benefits of Structured Methodology in Intelligence……. Testing Structured Methodology in Intelligence…….….. MCDM………………………...………………………… The MCDM Process ……………………….……….…... Validity of MCDM……………………………..…….….. Converting MCDM to MCIM…………………………… Hypotheses ………………………………………………

4 6 8 9 12 18 24 25

METHODOLOGY………………………………………………

27

Research Design……………………………………….. Selection of Subjects…………………………………... Recruitment of Subjects……………………………….. Process………………………………………………… Control Group…………………………………………. Experiment Group…………………………….............. Data Analysis Procedures……………………………..

x

27 28 28 30 30 31 33

4

RESULTS……………………………………………………….. Pre-Test Findings……..………………………………..... Post-Test Findings……………………………….…….... Product…………………………………………...……… Process…………………………………………………… Timeliness………………………………………………... Analytic Confidence…………………………………….. Objectivity and the Incorporation of Alternative Analysis………….. Logical Argumentation…………………………………. Bullet Point Analysis……………………………………. Potential of Future Use…………………………………..

5

36 36 38 40 42 43 45 46 46 47 48

CONCLUSIONS………………………………………………...

50

Pre-Test and Post-Test Conclusions……………………... Process and Product Conclusions……………………….. A Step Forward…………………………………………..

50 52 53

BIBLIOGRAPHY……………………………………………………………….

56

APPENDICES…………………………………………………………………..

59

APPENDIX 1: Analytic Methods Experiment Sign-Up Form…..

60

APPENDIX 2: Institutional Review Board Proposal Form……..

61

APPENDIX 3: Participation Consent Form. ……………………

66

APPENDIX 4: Experiment Section 1 Handout (Control Group)..

67

APPENDIX 5: Analytic Methods Experiment Links……………

69

APPENDIX 6: Pre-Test Questionnaire (Control and Experiment Groups) …………………………………………

70

APPENDIX 7: Post-Test Questionnaire (Control Group)…………………….…………………….

72

APPENDIX 8: MCDM Participation Debriefing……………….

74

APPENDIX 9: MCDM Lecture Notes………………….………

75

xi

APPENDIX 10: Experiment Section 2 Handout (Experimental Group)…………………………………………

77

APPENDIX 11: Post-Test Questionnaire (Experimental Group)…………………………………………

80

APPENDIX 12: Statistical Data………………………………..

82

xii

LIST OF FIGURES Page Figure 1.1 Figure 1.2 Figure 1.3 Figure 1.4 Figure 1.5 Figure 3.1 Figure 3.2 Figure 4.1 Figure 4.2 Figure 4.3 Figure 4.4 Figure 4.5

General Example Matrix Used in MCIM Step Two in the MCIM Process Steps Three and Four in the MCIM Process Steps Five and Six in the MCIM Process Completed Example Matrix Subjects by Class Year Demographic Breakdown by Experiment Group Control Group Results Experiment Group Results Length of Analysis Completion Time Levels of Analytic Confidence Bullet Point Analysis

xiii

13 15 16 17 18 29 33 41 41 44 45 48

1

CHAPTER 1: INTRODUCTION

We all make decisions on a daily basis. In fact, we often make so many decisions that we may fail to recognize we are even doing so; it is an act that has become second nature to us. Some decisions, such as what we choose to eat for breakfast in the morning, or which road we use to travel to work, are typically made with our “gut feeling” or intuition. While these decisions may require the balancing of multiple factors (how hungry we are or how much time we have), they require little forethought or analysis. These decisions are not life altering, and they do not generally have a long-term affect on us. On the other hand, decisions such as what kind of car we buy, or what career path we embark on, are decisions that are affected by multiple criteria and may alter our life in some way. In the field of intelligence, decisions made by others can affect matters of national security, law enforcement and business operations. These decisions are directly related to the safety and survival of both nations and organizations; therefore, they mandate appropriate analysis. Since the ability to make the “right” decision or predict an adversary’s most likely course of action (COA) can have heavily influence the society in which we live, it is imperative that an analyst makes sure that he or she fully grasps the relevant information to provide effective analysis to a decision maker. One method of ensuring appropriate evaluation of significant criteria, as well as the assessment of suitable alternatives, is through the use of a structured methodology, such as MCDM.

2

MCDM is a generic term used to encompass a broad range of analytical methodologies that use matrices as the basis for their conclusions. More specifically, MCDM is an internally focused decision support method that assesses possible COAs based on the evaluation of multiple criteria, goals, and objectives of conflicting nature. It provides substance to a decision maker, as it allows for selecting an option based on the appropriateness of those alternatives weighted against one another. Given recent intelligence failures, substantial emphasis has been placed on improving intelligence analysis and capabilities.

In the Vision 2015 document, the

Director of National Intelligence specifically addresses the need to improve the Intelligence Community (IC) and “create decision advantage.”1 One way to create decision advantage, as well as improve competitive analysis, and counterintelligence capabilities, is by gaining insights into the thoughts of one’s competitor or adversary through the use of Multi-Criteria Intelligence Matrices (MCIM).2 MCIM is a twist to the conventional form of MCDM, as it focuses not on an organization’s own COA but rather on the likely COA of others outside the organization. This method, once validated in an intelligence environment, might promote a more efficient way of structuring intelligence data and may increase an analyst’s ability to learn what is happening with his or her adversary or the environment in which they are working in. Additionally, it may ensure that an analyst examines all relevant COAs allowing for more thorough analysis of a situation. Finally, it may help an analyst escape analytic pitfalls, and provide more credibility to an analyst’s work by providing a logical, transparent process to a decision maker in the form of an easy to understand matrix. 1

United States Government, Director of National Intelligence, Vision 2015, Available at http://www.dni.gov/Vision_2015.pdf; Internet (Accessed 20 March 2009). 2 MCIM is used to make the distinction between externally focused matrix analysis and MCDM, or internally focused matrix.

3

While there is a substantial body of literature related to the use and effectiveness of MCDM in the general sense, there is none regarding the use of this method in intelligence analysis. The purpose of this experimental research is to determine the effectiveness of using MCIM as an analytical methodology in the field of intelligence. Currently no research has been conducted on the topic, and this study hopes to take the first steps of many in increasing an analyst’s ability to thoroughly examine all possible COAs in a timely manner as well as accurately forecast what COA one’s adversary is likely to choose.

4

CHAPTER 2: LITERATURE REVIEW

Some individuals will always argue that nothing can compete with the use of an analyst’s “gut feeling” or intuition to make decisions, or that on occasion, there just isn’t time to apply a structured methodology to an intelligence problem. After all, most analysts are working under a deadline and cannot afford to incorporate time-consuming methodologies that are difficult to use. However, in failing to implement a structured method of problem solving, an analyst may be sacrificing quality analysis.

Intuition vs. Structured Methodology In Michael R. LeGault’s book, Think: Why Crucial Decision Can’t Be Made in the Blink of an Eye, he suggests that individuals should use their knowledge and abilities to make informed decisions, rather than base decisions merely on intuition or impulse. He further stresses the importance of critical or complex thinking by stating, “It can be used simply to dig into things to enhance one’s awareness and discernment.”3 This quote can be directly related to the analytic process in intelligence. When an analyst draws conclusions based on their own experiences or intuition, he or she may fail to account for conditions or factors in their adversary’s environment that may lead to a different conclusion. By systematically reviewing all pertinent information, through the use of a structured analytic method, an analyst may increase his or her chance of creating accurate analytic judgments.

3

LeGault, Michael R, Think: Why Crucial Decision Can’t Be Made in the Blink of an Eye, New York, Threshold Editors, 2006, Page 305.

5

Ernest Forman, Professor of Management Science at George Washington University, also highlights the benefit of structured thinking over intuitive thinking. He states, “Although the vast majority of everyday decisions made intuitively are adequate, intuition alone is not sufficient for making complex, crucial decisions.”4 He argues that “organizations that use modern decision support methods can gain and maintain a competitive edge in leading and managing global business relationships that are influenced by fast changing technologies and complicated by complex interrelationship between businesses and governments.”5

Forman’s position highlights the need for

teaching and implementing competitive decision making techniques, or structured analytic methods. Like Forman, an additional argument by LeGault demonstrates the importance of teaching structured methodology early on in an analyst’s career.

He states, “It [critical

thinking] is a skill that every individual is born with; however, it requires development upon laying a foundation.”6

Therefore, if an analyst learns how to effectively use

analytical methodologies in the early stages of his or her career, they may expand and develop the critical thinking skills necessary to make better decisions and more accurate forecasts throughout their career. It is also important to recognize here that intuitive decision making is not always available or beneficial to entry-level intelligence professionals. These individuals often lack experience in their field, thus suggesting the need for a more organized approach to deal with complex problems, in order to produce better analysis. Benefits of Structured Methodology in Intelligence 4

Forman, Ernest, Decision by Objectives (How to Convince Others That You Are Right, Available at http://mdm.gwu.edu/forman/DBO.pdf; Internet. 5 Ibid. 6 LeGault, Page 32.

6

Numerous experts in the field of intelligence have discussed the potential benefits of using structured methodology in analysis. According to Stephen Marrin, “the primary value of analytic techniques or structured methods is that they provide a way to account for the analytic judgment; an analytic ‘audit trail’ as it were.”7

Thus, thru the

implementation of a structured methodology, the analyst’s thought process becomes more transparent to others. Marrin also states, “With an analytic audit trail, analysts and their colleagues can discover the sources of analytic mistakes when they occur and evaluate new methods or new applications of old methods. In this way, structured methods make it possible to advance the analytic tradecraft.”8 In other words, if the desired outcome was not achieved, an analyst or decision maker can review what went wrong, or learn how to approach and evaluate problems better in the future. According to Richards Heuer, the use of structured methodology in intelligence analysis may also help to prevent analytic pitfalls, such as the satisficing strategy or groupthink.9

One structured methodology in particular, the Analysis of Competing

Hypothesis (ACH), requires analysts to weigh alternatives, in order to reach some type of conclusion. Additionally, ACH provides a structured outline as to how an analyst reached a specific conclusion, thus providing substance to a decision maker regarding his or her estimates.

In regards to ACH, Heuer that stated the following benefits the methodology: 7 Marrin, Stephen, “Intelligence Analysis: Structured Methods or Intuition,” American Intelligence Journal, Page 7, Summer 2007. 8 Ibid, Page 7. 9 Heuer, Richards, Psychology of Intelligence Analysis [book on-line] (Langley, Virginia: Central Intelligence Agency Center for the Study of Intelligence, 1999), Available at https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-andmonographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf; Internet, Page 109.

7

This procedure [ACH] leads you through a rational, systematic process that avoids some common analytical pitfalls. It increases the odds of getting the right answer, and it leaves an audit showing the evidence used in your analysis and how this evidence was interpreted. If others disagree with your judgment, the matrix can be used to highlight the precise areas of disagreement. Subsequent discussion can then focus productively on the ultimate source of the differences.10 Thus, structured methods such as ACH provide analysts with a more efficient way to approach and organize complex problems Structured methodology may also help an analyst record all relevant pieces of information. Sometimes, ideas and thoughts that analysts do not write down may get lost or forgotten about. In fact, George A. Miller stated that “the number of things most people can keep in working memory at one time is seven, plus or minus two.”11 Thus, by implementing a structured method, an analyst would have an organized way to keep track of information so that it is not lost or forgotten. Structured methodology in intelligence may also facilitate an analyst’s understanding of how pieces of a problem are related. Heuer stated the following regarding coping with complexity in analysis: “put all the parts down on paper or a computer screen in some organized manner such as a list, matrix, map, or trees so that we and others cans see how they interrelate as we work with them.” 12 Thus, Heuer is

10

Heuer, Richards J., Psychology of Intelligence Analysis [book on-line] (Langley, Virginia: Central Intelligence Agency for the Study of Intelligence, 1999), Available at https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-andmonographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf; Internet, Page 109. 11 Miller, George. A., “The Magic Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information,” The Psychological Review, Vol. 63, No. 2 (March 1996). As cited by Heuer, Richards Jr. in “Taxonomy of Structured Analytic Techniques,” Paper prepared for the International Studies Association 2008 Annual Convention, March 26-29, 2008, San Francisco, CA. 12 Heuer, Richards, Jr., “Taxonomy of Structured Analytic Techniques,” Paper prepared for the International Studies Association 2008 Annual Convention, March 26-29, 2008, San Francisco, CA, Available at http://www.allacademic.com//meta/p_mla_apa_research_citation/2/5/4/1/2/pages254125/p254125-1.php; Internet; (Accessed 20 November 2008).

8

affirming the idea that structured methodology may provide some organized way for analyst to visualize and evaluate data and see how pieces of criteria relate to each other.

Testing Structured Methodology in Intelligence An experiment conducted in 2000 by MSgt Robert D. Folker, Jr. of the United States Air Force (USAF) specifically addressed the use of structured methodology in intelligence. The base of Folker’s research revolved around an experiment designed to test the effectiveness of structured methodology in providing “correct” intelligence analysis, by comparing it to intuitive analysis.13 Folker's subjects were various students at the Joint Military Intelligence College (JMIC), which he divided into two groups: a control group and an experimental group. His experimental group was trained on a structured methodology and asked to incorporate that methodology in their analysis, while the control group was not asked to use a specific methodology. Folker gave both groups the same two hypothetical intelligence scenarios based off of facts from a real world problem. By forming the basis of his experiment around an event that actually took place, Folker was able to assess an analyst’s ability to pick the “correct” COA. Upon review of the results, Folker’s found that the group of analysts, who used structured methodology in their analysis, provided significantly better analysis than the group who used no formal structured methodology. 14 He also concluded that these results were not linked to “rank, experience, education or

13

Folker, MSgt Robert D, Jr., “Intelligence Analysis in Theater Joint Intelligence Centers: An Experiment in Applying Structured Methods,” Joint Military Intelligence College, Washington DC, NDIC Press 2000. Available at http://www.fas.org/irp/eprint/folker.pdf; Internet, Page 15 14 Ibid, Page 32

9

branch of service.”15 Folker further stated, in order to be effective, intelligence analysts must be properly trained on the specific type of methodology.16 Folker’s research was important as it reinforced the idea that the implementation of structured methodology can assist analysts in breaking down problems and producing better analysis. MCDM In the Psychology of Intelligence Analysis, Richards Heuer, also distinguishes between merely “sitting down and thinking about a problem and really analyzing a problem.”17 Like many others, he suggests that in order to fully analyze a complex problem, an individual must break it down into its’ smaller components, evaluate each portion, and then piece the subcomponents back together to make a decision.18 One method of doing just that is through the use of decision support systems such as MCDM. As noted earlier, MCDM is a generic term used to encompass a broad range of analytical methodologies that use matrices as the basis for their conclusions. More specifically, MCDM is an internally focused decision support method that assesses COAs based on the evaluation of multiple criteria, goals, and objectives of conflicting nature. Prior to demonstrating the validity of MCDM, as well as judging the effectiveness of MCIM, it is necessary to begin by providing a brief overview of the traditional process. As noted earlier, the conventional process of MCDM has numerous variations.

15

Folker, Page 32 Folker, Page 32 17 Heuer, Richards, Psychology of Intelligence Analysis, [book on-line] (Langley, Virginia: Central Intelligence Agency Center for the Study of Intelligence, 1999), Available from https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-andmonographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf: Internet, Page 94. 18 Ibid, Page 94. 16

10

Dr. Thomas Saaty developed the Analytic Hierarchy Process (AHP), one variation of MCDM.

AHP allows decision makers to structure decisions hierarchically to

determine which one most suits their needs. Typically, the overall goal or decision to be made, is stated at the top of the hierarchy, criteria used evaluate the alternatives are in the middle, and the various COA are listed at the bottom. In regards to the benefits of his methodology Saaty states: The AHP is about breaking a problem down and then aggregating the solutions of all the sub problems into a conclusion. It facilitates decision making by organizing perceptions, feelings, judgments, and memories into a framework that exhibits the forces that influence a decision. In the simple and most common case, the forces are arranged from the more general and less controllable to the more specific and controllable.19 Another variation of MCDM is the WSM. According to research conducted by Evangelos Triantaphyllou, the WSM is “probably the most commonly used approach, especially in single dimensional problems.”20 Additionally, Triantaphyllou states that this method works best “in single dimension problems where all the units are the same, (e.g. dollars, feet, seconds)” and “difficulty arises when it is applied to multi-dimensional decision making problems.”21 The WPM is another popular MCDM variation that is similar to the WSM model. In fact, the main factor that distinguishes the WPM from the WSM is that the WPM applies multiplication to criterion, in order to compare alternatives, where as the WSM, uses addition.22 19

Saaty, Thomas L, “How To Make A Decision: The Analytic Hierarchy Process,” Available at http://sigma.poligran.edu.co/politecnico/apoyo/Decisiones/curso/Interfaces.pdf; Internet (Accessed 24 March 2009). 20 Ibid, Page 6. 21 Triantaphyllou, Evangelos and Stuart H. Mann, “An Examination of the Effectiveness of MultiCriteria Decision Making Methods: A Decision Making Paradox,” Available At http://www.csc.lsu.edu/trianta/index.html?http://www.csc.lsu.edu/trianta/Books/DecisionMaking1/Book1.ht m; Internet (Accessed 20 March 2009). 22 Triantaphyllou, Evangelos and Panos M. Parlos (ed.), Multi-Criteria Decision Making Methods: A Comparative Study, The Netherlands: Kluwer Academic Publishers, 2000, Page 8.

11

The staff study is also a MCDM variant that is commonly studied and utilized in the military community. As outlined in the United States Army Field Manual (FM) 1015, Staff Organizations and Operations, there are several types of military briefs including information, decision, mission and staff.23

FM 101-5 states that the decision brief,

designed to arrive at the solution to a problem/decision, “requires that an analyst be prepared to present his assumptions, facts, alternative solutions, reason for choosing the recommended solution, and the coordination involved.” The staff study format, outlined in Appendix D of FM 101-5,24 provides the means to do this by outlining multiple COAs, reviewing pertinent criteria, and evaluating the most appropriate solution.

23

The United States Army. Field Manual 101-5. Staff Organization and Operations. 31 May 1997. Page 115. Available at http://www.fs.fed.us/fire/doctrine/genesis_and_evolution/source_materials/FM-1015_staff_organization_and_operations.pdf; Internet. 24 Ibid, Page 105.

12

The MCDM Process Although MCDM is an umbrella term used for a collection of methodology types, they all involve the idea of decision alternatives, or COAs, determined by numerous criteria weighted against each other.

In MCDM, emphasis is on placing value, or

judgment of an item’s worth and desirability, on pieces of criterion, a standard on which a judgment or decision may be based.25 By providing a breakdown of the conventional MCDM process, I hope to demonstrate that this methodology is relatively easy to understand, and when used properly, it is an effective way to approach complex problems. For this research, I have chosen to use a MCDM model derived from the Army’s staff study. This model seemed like an appropriate choice given its simplicity and ability to accommodate a large amount of unstructured data, which is common to intelligence analysis. In general, the basic parts of the MCDM process can be broken down into the following eight steps: 1. Requirement/Question and Collection 2. Establish Possible COAs 3. Establish Screening Criteria 4. Screen COAs 5. Establish Evaluation Criteria 6. Weight Evaluation Criteria 7. Evaluate COAs 8. Make Recommendations

25

Internet.

Webster’s Dictionary Online, Available at http://www.merriam-webster.com/dictionary/criteria;

13

Requirement/Question and Collection The first step in MCDM, establishing a clearly defined question, is essential. An analyst simply cannot have a good analysis without a clearly defined question.

In this

step, it is equally important that the analyst or decision maker form the question in a way, which will elicit possible COAs. For the purpose of working through the conventional process of MCDM, I have chosen to explain each step by addressing the question of which vehicle to buy. MCDM normally uses a matrix to display the results of the analysis. The general matrix used in MCDM normally looks like the one below, where COA represents the various options realistically open to the decision maker and criteria represents the various relevant ways in which a decision maker will judge each COA.

Figure 1.1 Example of a General Matrix Used in MCDM Question: Which kind Criteria 1: of vehicle should I buy? COA 1: COA 2: COA 3: COA 4: COA 5: COA 6:

Criteria 2:

Criteria 3:

Criteria 4:

Total/ (Rank Order):

14

Establish Possible COA The second step in the MCDM process is establishing relevant possible COAs. According to numerous experts, there are two categories for further classifying MCDM problems based on the possible COAs: Multi-Objective Decision Making (MODM) and Multi-Attribute Decision Making (MADM).26 The primary difference between these two categories is that with MODM problems, the decision alternatives are endless; whereas, MADM problems focus on distinct decision spaces, or predetermined COAs.27 In the case of deciding what car to buy, the example question falls in line with the idea of MODM, as there are endless possibilities of cars for an individual to choose from, and an individual is not required to choose from pre-determined COAs. In this step, it is important for an analyst to identify as many COAs as possible, since there is no “correct” number of alternatives. The range of possible COAs merely derives from the extent of the problem and the creativity of the analysts in solving the problem. The following example matrix outlines six types of vehicles that an individual may purchase. It is important to note that this is not an all inclusive list, and is merely an abridged list for the purpose of demonstrating the use of matrices.

26

Triantaphyllou, Evangelos and Panos M. Parlos (ed.), Multi-Criteria Decision Making Methods: A Comparative Study, The Netherlands: Kluwer Academic Publishers, 2000, Page 1. 27 Ibid, Page 1.

15

Figure 1.2 Step Two in the MCDM Process Question: Which kind Criteria 1: of vehicle should I buy? COA 1: Family compact car COA 2: Mini Van COA 3: Sports car COA 4: Station wagon COA 5: SUV COA 6: Motorcycle

Criteria 2:

Criteria 3:

Criteria 4:

Total/ (Rank Order):

Establish Screening Criteria and Screen Criteria After completion of the second step in the MCDM process, an individual might be overwhelmed at the plethora of COAs available; therefore, he or she must begin to eliminate certain COAs by establishing screening criteria. It is necessary to distinguish this step separate from the previous step, which involved establishing possible COAs. On occasion, individuals subconsciously rule out possible COAs, without even recognizing they are going through a screening process. For example, when choosing which car to buy, a student may not even consider options that may be out of range for either financial reasons (too costly) or due to appearance, (it is not a dream car), thus,ruling out possible alternatives before having the opportunity to compare it against other COAs. In the example case, screening criteria for our aforementioned problem may be “must be vehicle two wheeled vehicle.” By establishing “must have/be” guidelines for COAs an individual can quickly eliminate unlikely possible COAs.

Based on the

16

previous evaluation criteria, the possible COA, “motorcycle” may be eliminated as it is not a four wheel vehicle. The following matrix shows the elimination of one possible COA (a motorcycle), after implementing the screening criteria. Figure 1.3 Steps Three and Four in the MCDM Process Question: Which kind Criteria 1: of vehicle should I buy? COA 1: Family Compact Car COA 2: Mini Van COA 3: Sports Car COA 4: Station Wagon COA 5: SUV COA 6: X Motorcycle

Criteria 2:

Criteria 3:

Criteria 4:

Total/ (Rank Order):

X

X

X

X

Establish Evaluation Criteria and Weight Evaluation Criteria The next steps in the process, establishing evaluation criteria and weighting the evaluation criteria, allows the analyst to provide rank to the remaining COAs. At this point, the analyst uses evaluation criteria to establish that “all things being equal, this is the best COA for me." The analyst should develop a system of rank that stresses the significance of one criterion over another. The analyst can provide rank as either a tangible means, things that he or she can measure, or by intangible means, things that he or she cannot easily measure. Evaluation criteria for the stated problem may be specific, “must cost less than $25,000,” or it may be generalized, “with all other things being equal, the most inexpensive car is the best.” This step involves evaluating what an analyst or decision

17

maker really cares about, or what is most important in achieving the desired outcome. Additionally, this step allows an analyst to provide a system to weight the most important criteria to them. For example, if the individual was a student of limited financial means, price may be the most important criteria when looking to buy a vehicle. Therefore, the price category will be multiplied by two, in order to demonstrate its significance. The following matrix outlines the criteria, while applying weight and rank order:

Figure 1.4 Steps Five and Six in the MCDM Process Question: Which kind Criteria 1: of vehicle should I buy? Price Cheaper is Better (x2) COA 1: 1 (2) Family Compact Car COA 2: 2 (4) Mini Van COA 3: 5 (10) Sports Car COA 4: 3 (6) Station Wagon COA 5: 4 (8) SUV

Criteria 2: Size Room for family

Criteria 3: Safety For Family

Criteria 4: Appeal Attractive Look

Total/ (Rank Order):

4

1

2

9

1

2

4

11

5

5

1

21

3

3

5

17

2

4

3

17

Evaluate COA and Make Recommendations After establishing evaluation criteria, it is necessary to provide a system of rank order to the possible COAs by establishing a scale. When addressing our example question, this task could be as simple as ranking the cars most likely to purchase, to least likely to purchase, by utilizing a scale from 1 to 3, 1 to 5, or 1 to 10. The following completed matrix highlights the rank order of the possible COAs:

18

Figure 1.5 Completed Example Matrix Question: Which kind Criteria 1: of vehicle should I buy? Price Cheaper is Better (x2) COA 1: 1 (2) Family Compact Car COA 2: 2 (4) Mini Van COA 3: 5 (10) Sports Car COA 4: 3 (6) Station Wagon COA 5: 4 (8) SUV

Criteria 2: Size Room for family

Criteria 3: Safety For Family

Criteria 4: Appeal Attractive Look

Total/ (Rank Order):

4

1

2

9 (1)

1

2

4

11 (2)

5

5

1

21 (5)

3

3

5

17 (3)

2

4

3

17 (3)

After ranking the possible alternatives, the individual now has a structured format, which provides substance to their decision. The family compact car appears to be the most idea vehicle, followed by the minivan, station wagon and SUV, and sports car. Upon completion of the first MCDM analysis, one COA that may be chosen is a sports car. It is important to note, that upon determining the first ideal COA, a second matrix analysis may be completed in order to determine the more specific type of sports car.

Validity of MCDM In his book, Multi-Criteria Decision Making Models: A Comparative Study (2000), Evangelos Triantaphyllou explores some of the variations on MCDM to determine what the most “ideal” variation is to use. Since it was not possible to include every possible MCDM variation, Triantaphyllou focused on the more popular methods,

19

such as AHP, WSM, and the WPM, noted above. Although unrealistic to determine the most “ideal” method to use in a given situation, Triantaphyllou’s extensive research of the variations of MCDM provides strong evidence that there is a plethora of methods available which allow an efficient and organized approach to effective decision making. Triantaphyllou also points out the irony in his research commenting, “Deciding on which one is the best method can be viewed as an MCDM problem itself whose solution requires the use of the best MCDM.”28 Valerie Belton and Theodor Stewart also provide an excellent overview of the various types of MCDM models and theories that have developed over the past 25 years, in their book Multiple Criteria Decision Analysis: An Integrated Approach (2002). The authors reinforce the idea that although MCDM cannot guarantee a “correct” answer, the methodology should “facilitate decision makers’ learning about the understanding of the problem faced, about their own, other parties’ and organizational priorities, values and objectives and through exploring these in the context of the problem to guide them in identifying a preferred COA.”29 The book concludes with the authors’ opinion regarding the future of MCDM, specifically stating that integration of various perspectives and identifying common strengths and weaknesses, is essential to the methodology’s “growth and success.”30 Aside from the research surrounding MCDM as outlined in the aforementioned academic books, MCDM has multiple applications in the real world, contributing to such

28

Triantaphyllou, Evangelos, “Can We Always Determine the Right Alternatives in Business Problems,” 18 August 2002. 29 Belton, Valerie and Theodor J. Stewart, Multiple Criteria Decision Analysis: An Integrated Approach, The Netherlands: Kluwer Academic Publishers, 2002, Pages 2-3. 30 Belton and Stewart, Pages 333-343.

20

fields as finance and economics, environmental management and marketing.31 In his article “Decision making with the analytic hierarchy process,” Saaty justifies the application of AHP by citing examples where the methodology has been used in the real world. The following examples are just a few of the ones outlined by Saaty, and are taken directly from his publication: • •

• • •

British Airways used the AHP in 1998 to choose the entertainment system vendor for its entire fleet of airplanes. In 2001, the methodology was used to determine the best relocation site for the earthquake devastated Turkish city Adapzari. Xerox Corporation has used AHP to allocate close to a billion dollars to its research projects. In sports, it was used in 1995 to predict which football team would to go to the Superbowl and win. The AHP was also applied in baseball to analyze which players should be retained on a team. In 1999, the Ford Motor Company used the AHP to establish priorities for criteria that improve customer satisfaction. Ford gave Expert Choice Inc, an Award for Excellence for helping them achieve greater success with its clients.32

The application of MCDM problems has also surfaced in financial planning journals. In the March 2008 Journal of Financial Planning, William Z. Suplee and Steven R. Dzubow discuss the importance of using structured methodology in helping their clients make critical financial decisions. In their article, “Using Multiple-Criteria Decision Analysis to Simply the Financial Planning Process,” they demonstrate the value of using matrix analysis by a parent and child deciding on the best college for the child to

31

Zopounidis, Constantin and Doumpos, Michael, “Multiple-Criteria Decision Making,” Thomson Corporation, 2006. 32 Satty, Thomas , “Decision making with the analytic hierarchy process,” Int. J. Services Sciences, Vol. 1, No. 1, 2008, Available at http://inderscience.metapress.com/media/pgwf2qtuyg3xnvnhxnby/contributions/0/2/t/6/02t637305v6g65n8. pdf; Internet (Accessed 26 March 2009).

21

attend through the process of evaluating a plethora of criteria including tuition costs, graduation rate, campus life, student/teacher ratio and post-graduation placement.33 In addition to financial journals, the relevance of MCDM has also appeared in environmental management journals. In their article, the “Application of Multicriteria Decision Analysis in Environmental Decision Making,” in The Integrated Environmental Assessment and Management Journal, Gregory A. Kiker, Todd S. Bridges, Arun Varghese, Thomas P. Seager and Igor Linkov discuss the application of MCDM in various environmental projects. The authors highlight numerous environmental case studies, by outlining the specific MCDM method used, the decision context (a range of topics in the areas of environmental management, stakeholder involvement and contaminated sites) and the funding agency (a range of universities, as well as US and international government agencies).34 This article is mentioned as it demonstrates the flexibility and use of MCDM in a multiplicity of environmental topics conducted by an array of organizations. A surplus of research and studies has also been conducted as to the application and relevance of MCDM methods in engineering.

In the fields of civil and

environmental engineering, the Elimination and Choice Translating Reality (ELECTRE) II and III, are cited as two MCDM variations that are widely used.35 While conducting 33

Dzubow, PhD and William Z. Suplee IV, CFA, CFP, ChFC, CAS, “Using Multiple-Criteria Decision Analysis to Simplify the Financial Planning Process,” Journal of Financial Planning. March 2008, Available at http://www.library.idsc.gov.eg/GUI/Globals/Upload/BULLETIN_ATTACHMENT/92/efiles/manegement%20and%20economics/using%20multiple.pdf; Internet ( Accessed 5 December 2008). 34 Kiker, Gregory A., Todd S. Bridges, Arun Varghese, Thomas P. Seager and Igor Linkov. “Application of Multicriteria Decision Analysis in Environmental Decision Making.” Integrated Environmental Assessment and Management. Volume 1, Number 2, pages. 95-108. 2005, Available athttp://www.allenpress.com/pdf/ieam-01-02_95_108.pdf; Internet. (Accessed 20 November 2008). 35 Wang, Xiaoting, “Study of Ranking Irregularities When Evaluation Alternatives By Using Some ELECTRE Methods and a Proposed New MCDM Method Based on Regret and Rejoicing,” A Thesis Submitted to the Graduate Faculty of the Louisiana State University and Mechanical College. August 2007. Available online at http://etd.lsu.edu/docs/available/etd-07112007012708/unrestricted/Wang_thesis.pdf; Internet (Accessed 24 March 2009), Page iv.

22

his research in regards to MCDM in engineering, Xiaoting Wang mentions several past case studies in which MCDM methods were applied. Once again, the variety of topics covered highlights the flexibility in the application of MCDM problems. Some of the real life case studies utilized in Wang’s research included “choosing a solid waste management system [Hokkanen, J., and P. Salminen (1997)], choosing an alternative fuel system for land transportation [Poh, K.L., and B.W. Ang, (1999), and selection of an alternative electricity power plant [Leyva-Lopes, J.C., and E. Fernandez-Gonzalez (2003)].”36 In their article “Multi-Criteria Decision Making Process for Buildings,” J.D. Balcomb and A. Curtner demonstrate MCDM’s value in the building design planning process.

The authors suggest that by evaluating criteria such as life cycle cost,

architectural quality, functionality, air quality, and maintainability, various building designs can be evaluated to ensure maximum sustainability. The authors indicate that as fundamental decisions are being made, the methodology has the ability “to assist design teams in prioritizing their goals, setting performance targets and evaluating design options to ensure that the most important issues affecting building sustainability are considered.”37

Additionally, the paper highlights the various strengths of the

methodology including ease of use, its inexpensive nature, its ability to document how the team arrived at conclusions, as well as its ability to provide visual representation of the team’s thought process which can be interpreted by almost anyone.38

36

Ibid, Page 25. Balcomb, J.D. and A. Curtner, “Multi-Criteria Decision-Making Process for Building,” paper to be presented at the American Institute of Aeronautics and Astronautics Conference, Las Vegas, Nevada, July 24-28. 2000. Available online at http://www.nrel.gov/docs/fy00osti/28533.pdf; Internet (Accessed 23 March 2009), Page 1. 38 Balcomb, J.D. and A. Curtner, Pages 2 and 8. 37

23

Research conducted by Stacy Gilchrist, Master of Science in Applied Intelligence from Mercyhurst College, examined the use of the military staff study in intelligence by transforming the conventional problem solving process into an intelligence-focused methodology.39 For the purpose of his research, Gilchrist applied the methodology to the following question regarding the Mara Salvatrucha gang (MS-13): “How will MS-13 respond or react to the current law enforcement efforts led by the Immigration and Customs Enforcement (ICE) branch of the Department of Homeland Security (DHS), specifically Operation Community Shield?”40 Gilchrist noted that first and foremost, the intelligence-focused staff study was decision maker focused and provided a means to track the analyst’s thought process. 41 A second strength noted of the intelligence-focused staff study by Gilchrist is that the process is structured and organized.42 A final strength that Gilchrist noted was that the intelligence-focused staff study produced objective analysis, stating that cognitive biases are likely to occur less frequently, due to the individual evaluation of each criterion in regards the overall objective.43 Gilchrist also noted possible weaknesses including the potential for the methodology to be both time-consuming and contain flawed results.44 Gilchrist’s findings are important as they address the issues that many analysts face when working with structured methodology.

Additionally, Gilchrist’s research

provides valuable insight to the use of the intelligence-focused MCIM matrix.

39

Gilchrist, Stacy. “Staff Study,” (Research Paper for Advanced Analytic Techniques, Mercyhurst

College). 40

Ibid, Page 3. Ibid, Page 1. 42 Ibid, Page 3 43 Ibid, Page 1. 44 Ibid, Page 1. 41

24

Converting MCDM to MCIM Since the conventional form of MCDM has proven to be valuable, why not replicate the process, while making slight modifications, in order to create a valuable structured methodology that may lead to the production of better intelligence analysis? The only difference with using the methodology in intelligence is that the analyst would apply the method to an external entity-to best predict the likely COA an enemy or competitor will choose. MCIM would likely have several benefits for intelligence analysts. Just as Heuer states in regards to ACH, although the use of MCIM can never guarantee a “correct” answer, it may increase the odds of choosing the COA one’s adversary or competitor is most likely to pursue, based on a thorough breakdown of all relative pieces of information.45 Additionally, the analyst can provide the decision maker with a supporting matrix that outlines their thought process, which may give more credibility to their work. The matrices can also highlight the areas of disagreement or conflict between two individuals, by identifying the weight of importance that an analyst places on each individual criterion.46 Finally, a decision maker or analyst can adjust the matrix to accommodate new data and “what-if” scenarios. Assuming the role of an external entity would also benefit intelligence analysis. In fact, this is something the army has already been doing for years through the Red Team

45

Heuer, Richards J., Psychology of Intelligence Analysis [book on-line] (Langley, Virginia: Central Intelligence Agency for the Study of Intelligence, 1999), Available at https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-andmonographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf; Internet (Accessed 11 November 2008), Page 109. 46 Heuer, Richards J., Psychology of Intelligence Analysis [book on-line] (Langley, Virginia: Central Intelligence Agency for the Study of Intelligence, 1999), Available at https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-andmonographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf; Internet, Page 109.

25

methodology, a tactic frequently used to increase the odds of predicting how one’s opponent is likely to react in a specific situation.47 Additionally, in his book, StrategySpecific Decision Making: A Guide for Executing Competitive Strategy, William Forgang states that “external analysis has the ability to deter an organization from pursuing an option they might have otherwise chosen to accept.”48 Thus, by using structured methods with an external focus such as MCIM, an analyst might gain insight to factors outside of a decision maker’s immediate surroundings, and possible deter them from costly or poor decisions they would otherwise have chosen to pursue.

Hypotheses The following experiment seeks to demonstrate that MCIM can prove itself as a useful intelligence analysis methodology. Specifically, I hypothesized the following: Hypothesis 1: The experimental group using MCIM to conduct analysis will have a more accurate product than the students in the control group. Hypothesis 2: The analytical process for the experimental group using MCIM will be more transparent than the control group. Hypothesis 3: The experimental group using MCIM will complete their analysis faster than the control group. Hypothesis 4: The experimental group using MCIM will have a higher level of analytic confidence in their product than the control group.

47

2008 U.S. Army Posture Statement. Information Papers: Red Team Education and Training, Available at http://www.army.mil/aps/08/information_papers/prepare/Red_Team_Education_and_Training.html; Internet (Accessed 26 September 2008). 48 Forgang, William G., Strategy-Specific Decision Making: A Guide for Executing Competitive Strategy. M.E. Sharpe, Inc., Armonk, New York, 2004.

26

Hypothesis 5: The experimental group using MCIM will be more objective during the analytic process and examine more alternative possibilities. Hypothesis 6: The experimental group using MCIM will provide better logical argumentation than the control group.

27

CHAPTER 3: METHODOLOGY

In order to test the given hypotheses, I designed an experiment to assess whether or not MCIM methodology improves intelligence analysis. When the experiment was completed, I evaluated the analyst’s process and product, including the length of time it took the student-analysts to complete analysis and their level of analytic confidence. This methodology section will detail the research design of this experiment.

Research Design The experiment I conducted broke the subjects into two groups. One of the groups was an experimental group (comprised of 24 student-analysts), and one was a control group (comprised of 21 students-analysts). I gave the control group a 10 minute instruction/information period regarding the experiment tasking, and then asked them to complete an analysis of a real life intelligence scenario. I taught the experimental group the method of MCDM/MCIM over the course of approximately 35 minutes, gave them a 10 minute instruction/information period regarding the experiment tasking, and then asked them to provide an analysis regarding the same real life intelligence scenario. The only difference between the two groups was that the experimental group received instruction regarding MCDM/MCIM and then was asked to complete the analysis using that methodology. I gave the two groups access to the same resources and provided them the same amount of time to complete their analysis.

28

Selection of Subjects I used students at the Mercyhurst College Institute for Intelligence Studies (MCIIS) as my research subjects. Mercyhurst College, located in Erie, Pennsylvania, is the originator of a four-year program specifically designed to prepare students for entrylevel careers in business, law enforcement and national security intelligence analysis Additionally, Mercyhurst College offers a master’s program, with a thesis requirement. Since the students of Mercyhurst College are training to be efficient analysts, they could arguable serve as a proxy for entry-level analysts in the intelligence community, who would have been difficult, (if not impossible), to recruit due to security clearances and access problems.

Recruitment of Subjects In order to recruit participants for my experiment, I approached several professors in the intelligence department asking for permission to recruit student-analysts during their designated class time. Upon receiving their approval, and approximately one and a half weeks prior to the start of my experiment, I gave a brief oral presentation at the beginning of several classes that included information regarding the time, place and subject matter of the experiment. The students were informed that the experiment would be conducted during the second week of winter term. The presentation also indicated that several professors in the department would be offering extra credit for their class if students chose to participate in the experiment. Offering extra credit may have persuaded several students to participate when they otherwise might not have. Upon agreeing to participate, each student-analyst was handed a sign-up sheet (see Appendix 1) and asked

29

to choose the appropriate date and time that they were available to participate. The students could choose from either a Wednesday or Thursday night time slot based on their availability or preference. The time frame on both days was kept the same so that the student-analysts were unable to determine which group was the control group and which group was the experimental group. If students did not have a preference for availability, they selected a third choice, in which they were randomly assigned a day to complete their experiment. The experiment was open to all students in the intelligence department, including both undergraduate and graduate students (See Figure 3.1 for breakdown by of the subjects by class year). A total of 45 students participated in experiment. Although the experiment was open to all class years, no freshmen participated in the experiment.

Figure 3.1 Subjects by Class Year

30

Process Prior to conducting my experiment, I needed approval from the Mercyhurst College Institutional Review Board (IRB). The IRB requires any student anticipating conducting an experiment involving human subjects to submit a proposal for review (See Appendix 2). The purpose of the experiment proposal is to outline the experiment and identify any possible risks, if any, which participants may come into contact with as a result of participating in the experiment. After receiving approval from the IRB, I also had to secure consent from participants in the experiment. At the beginning of each experiment session, I outlined what the experiment entailed. I then provided each student-analyst with a participation consent form (See Appendix 3). The consent form asked for the student’s name, class year, and name of their professor/professors, in order for me to pass along the appropriate information to award them extra credit.

Before starting each group’s section of the

experiment, I answered any questions that the subjects may have had regarding the experiment. The process of each group was carried out the same way, only adjusting those items necessary to test the effectiveness of the methodology.

Control Group The first group of participants that I used to test my hypotheses on was the control group. When the sessions started, I provided the group with a brief set of instructions and provided them with a handout regarding a real life intelligence scenario. The handout provided the subjects with room to write their analysis, as well as a section to identify their level of analytic confidence (See Appendix 4). I provided the subjects with a

31

handout containing a list of websites to use as a starting point for their research (See Appendix 5). I informed the students that these links were merely a starting point for their analysis, and it was their choice if they wanted to use the links for information. I asked the students to carefully read through the problem and gain an understanding of the requirements.

I then handled any questions they had regarding the process of the

experiment. After gaining an understanding of the expectations, and prior to completing their tasking, I asked the subjects to complete a pre-test questionnaire and provide brief demographic information, both of which I used to further compare the experimental and control groups (See Appendix 6). I then gave the group two hours to complete their analysis using whichever sources and method they preferred. Upon completing their analysis, I asked the control group to complete a post-test questionnaire (See Appendix 7), and then provided them with a debriefing form (See Appendix 8).

Experimental Group The second group of participants that I used to test my hypothesis was the experimental group. At the beginning of the session, the experimental group received a 35 minute long presentation on MCDM/MCIM methodology, where the subjects worked through examples of MCDM/MCIM problems. In addition, the group received a handout on MCDM/MCIM that they were able to refer back to later on in the experiment (See Appendix 9). I gave the experimental group the same 10 minute instruction period as the control group, regarding the tasking. In addition, I provided them with a handout (See Appendix 10) which asked for each subject to provide analysis for the same intelligence related problem within a period of two hours. The only difference between the two

32

groups was that I asked the experimental group to complete their analysis using MCIM. The handout provided room for the student-analysts to write their analysis and to indicate their level of analytic confidence; however, it also instructed each subject to create an MCIM matrix in Excel format. I provided the experimental group with an example of what the matrix should resemble when their product was completed. Additionally, they had the option of completing an outline, which asked for possible COAs, a listing of screening criteria, the eliminated possible COAs and the evaluation criteria.

After

receiving instruction, but prior to completing the requirement, I asked the experimental group to complete the same pre-test questionnaire as the control group. Upon completion of their analysis, I asked each member of the experimental group to fill out a post-test questionnaire (See Appendix 11), which differed slightly from the control group, and received the same debriefing form. In order to obtain minimal outside influence with the results, I intended to keep the control and experimental group as similar as possible as far as number of participants and class year (See Figure 3.2 for a breakdown demographic breakdown of subjects by group). Although the number of participants in each group was similar, (a total of 21 student-analysts participated in the control group and a total of 24 student-analysts participated in the experimental group section), there was an unpreventable discrepancy regarding the number of students in each class year. Specifically, there were a greater number of first year graduate students within the experimental group. This was due to uncontrollable scheduling conflicts regarding evening classes and meetings, as well as the availability of students who wanted to participate in the experiment. In order to identify if this discrepancy had any effect on the results, specific questions related to familiarity

33

and experience with analytic methods were addressed in both the pre-test and post-test questionnaires.

Figure 3.2 Demographic Breakdown by Experiment Group

Data Analysis Procedures Since the intent of the experiment was to test whether MCIM is a valuable methodology to use in the field of intelligence, I evaluated several factors upon completion of the experiment. Prior to the review of each analyst’s process and product, it was necessary for me to draw some conclusions from both the pre-test and post-test questionnaires in order to identify possible outside influence. A majority of the questions required the students to rate their answer on a 5 level Likert scale. I recorded the number of responses for each numeric category in a spreadsheet, found the percentage of subjects

34

who answered each number within each group, and figured out the average of both groups. By comparing the averages, I was able to pinpoint any differences between the two groups and test whether or not these differences were statistically significant at the 5% level. In addition to these questionnaires, I asked each student-analyst to indicate their level of analytic confidence. When determining his or her level of analytic confidence, each student-analyst was asked to consider the following seven factors:49 1. 2. 3. 4. 5. 6. 7.

Use of Structured Method (s) in Analysis Overall Source Reliability Source Corroboration/Agreement Level of Expertise on Subject/Topic & Experience Amount of Collaboration Task Complexity Time Pressure In order to measure analytic confidence, I asked the students to rate their level of

analytic confidence by marking their level of confidence on the modified scale shown below.50 Low |------------------------------------------------------| High

The scale was 10 centimeters in length, and they placed a slash mark where they felt their level of analytic confidence fell.

In order to compare levels of analytic

confidence between groups, I measured each analyst’s slash mark to the nearest half centimeter. I then determined the minimum and maximum level of analytic confidence in the experimental group and control group, as well as the average level within each group.

49

Peterson, Joshua. “Appropriate Factors to Consider When Assessing Analytic Confidence in Intelligence Analysis.” (Master's Thesis, Mercyhurst College, March 2008) 50 Ibid

35

Upon completion of their analysis, I also asked each student-analyst to record the amount of time it took them to complete their analysis (in minutes). In order to make comparisons regarding length of time to complete their analysis, I found the average amount of time, as well as the minimum and maximum amount of time within each group.

Finally, I reviewed each individual analysis, in order to make comparisons

regarding the thoroughness of their process, and the accuracy of their product.

36

CHAPTER 4: RESULTS

Statistical analysis was conducted on the results in order to determine if the findings were significant. If the results indicated a P-value < 0.05, then the results were considered significant, meaning it is unlikely to have occurred by chance. If the results indicated a P-value > 0.05, then the results were considered not significant, meaning it was likely to have occurred by chance. This section will discuss the results of both the pre-test and post-test questionnaires, as well as the results of the control group and experimental group viewed both individually and compared as a whole. A more complete review of the statistical data can be found in Appendix 12.

Additionally, the

consequences of some of these results are discussed further in Chapter 5.

Pre-Test Findings I distributed identical pre-test questionnaires to the control group and the experimental group prior to the analysts beginning their research. The subjects were asked to answer questions on a 5 level Likert scale. The purpose of the questions was to determine if there were any significant factors that would explain differences between the two group’s results that were not due to the experiment itself.

Specifically, these

questions were designed to provide insight as to the subject’s level of interest in participating in the experiment, as well as their level of familiarity and interest associated with the topic (Russia’s relationship with OPEC).

37

The first question was designed to determine the level of interest regarding participation in the experiment between the two groups. Both groups expressed roughly the same level of interest regarding participation in the experiment, with the control group expressing an average interest level of 3.48 (out of 5) and the experimental group expressing an average interest level of 3.75 (out of 5). Statistical analysis revealed a Pvalue of 0.586, meaning that this result was not significant. Extra credit appeared to be a driving force behind the majority of subject participation with the control group average at 3.9 (out of 5) and the experimental group average at 4.25 (out of 5). Specifically, extra credit appeared to be important to the undergraduate and first year graduate students. Only four students indicated that extra credit was “not at all important,” all of whom were second year graduate students. The next two questions were designed to see if one group had more knowledge in regards to the subject matter (Russia’s relationship with OPEC), or interest in the topic, prior to conducting research.

Results indicated that the control group was more

knowledgeable on the topic, with an average score of 2.5 (out of 5), compared to the average score of 1.7 (out of 5) expressed by the experimental group, a difference found to be statistically significant with a P-value of 0.012. The control group also indicated that they were slightly more interested in the topic, with an average control group score of 3.48 (out of 5) and an experimental group average of 2.83 (out of 5), a difference also found to be statistically significant with a P-value of 0.032.

38

Post-Test Findings As discussed in Chapter 3, the control group and experimental group were required to fill out a post-test questionnaire upon completion of their analysis. Each group received several identical questions. These questions were designed to provide insight as to the frequency that each subject uses analytic methods when providing analysis and their opinion regarding the ability of analytic methods to improve intelligence analysis.

Additionally, the questions were designed to determine the

subject’s opinion in regards to having adequate information and time to complete his or her analysis, and to assess their overall enjoyment of the process. When I asked the student-analysts how frequently they use structured analytic methods when solving intelligence related problems, the control group responded with an average of 3.14 (out of 5) and the experimental group responded with an average of 3.5 (out of 5). This result has a P-value of 0.150, thus this difference was not found to be statistically significant. In addition, I asked the two groups to rate their level of enjoyment during the process. The control group average was 3.7 (out of 5) and the experimental group’s average was 3.46 (out of 5). This result yielded a P-value of 0.101, thus the difference was not found to be statistically significant. These results suggest that both the control group and the experimental group felt that they had access to a reasonable amount of information that was necessary to evaluate the problem to its full extent, and that the use of the methodology by the experimental question did not seem to overly burden the studentanalysts. I asked the student-analysts to determine if they were given enough time to complete their analysis. The control group’s average response was 4.43 (out of 5) and the

39

experimental group responded with an average of 4.83 (out of 5), a difference that was statistically significant with a P-value of 0.025. This result suggests that the experimental group felt that they had more than enough time to complete the estimate, a result that is particularly interesting when compared to the actual time each group took to complete the estimate. (See the section on timeliness discussed later in this chapter). Aside from the questions previously discussed, I asked both groups several questions that were designed to gain feedback regarding the experiment set-up, as well as gauge the student’s prior exposure to analytic methods and desire to use structured methodology in future analysis.

These questions were not tested for statistical

significance as they did not likely have any influence regarding the actual process or product of the student’s analysis. These questions merely had to do with the experiment itself and therefore, there was no basis for comparison. In the post-test questionnaire, I asked the student-analysts in both groups how familiar they were with the process of MCIM. It was predictable that the experimental group would indicate that they were more familiar with the process of MCIM (after all they did just receive instruction on the methodology). The control group rated a level of familiarity as an average of 2.29 (out of 5) with six students claiming they had “no familiarity at all with the methodology.”

The experimental group rated a level of

familiarity of 3.29 (out of 5); with three students claiming they had “no familiarity at all with the methodology.” Based on the experimental group’s responses it is likely that the three students who indicated that they had “no familiarity at all with the methodology” interpreted the question to mean familiarity prior to the experiment session. After all, the

40

students had just received instruction on the methodology and completed their analysis using the methodology. Despite the difference in level of familiarity regarding MCIM, both groups did indicate very similar levels of familiarity with another structured analytic method, the ACH. The control group indicated an average of 4.29 (out of 5), with no subjects providing a response of 1 or 2 (responses that would have indicated no familiarity with the methodology, or minimal at best). The experimental group indicated an average of 4.42 (out of 5), with no subjects provided a response of 1, 2, or 3 (responses that would have indicated no familiarity with the methodology, or minimal at best). In spite of these statistics indicating that the students had some level of familiarity with analytic methods, no student-analyst in the control group actually used a structured method during their analysis.

Product I asked the student-analysts the following question: “How is Russia likely to seek to interact with OPEC after the December 17th, 2008 meeting in Algeria.” Figure 4.1 expresses the COAs derived from the 21 members of the control group. Figure 4.2 expresses the COAs derived from the 24 members of the experiment group. However, one student-analyst in the experiment group provided two answers; therefore the data in that chart reflects 25 responses instead of 24.

41

Figure 4.1 Control Group Results

Figure 4.2 Experimental Group Results The pie charts show that the experimental group arrived at a broader range of possible COAs. The control group came up with five possible COAs; whereas, the experimental group came up with seven possible COAs. Four of the COAs were roughly the same between the two groups (Russia will closely coordinate with OPEC regarding production, Russia will build a reserve to store oil, Russia will develop stronger ties with OPEC, and Russia will engage in future talks with OPEC). The majority of the studentanalysts in both groups predicted that Russia will closely coordinate with OPEC regarding oil production. This difference in number of suggested possible COAs may imply that the use of MCIM aided the student-analysts in generating more creative and

42

unique solutions to the problems, rather than choosing the first COA that seemed logical without reviewing alternatives. When I originally picked the real world scenario (Russia’s relationship with OPEC after the December 17th 2008 meeting) for the student-analysts to provide their analysis, I chose a situation that I felt would have a clear cut answer within a few weeks of the meeting’s termination. This has proved to not be the case up to this point. With no clear cut answer at this time, it is impossible to evaluate the method with regard to the accuracy of its product.

Process Despite the lack of a clear cut answer to the question, it is beneficial to examine the process by which both the control and experimental group derived their conclusions. The Intelligence Community Directive (ICD) Number 203 is a document that manages “the production and evaluation of national intelligence analysis.” 51 While the entire document does not apply to my research, several of the IC Analytic Standards outlined in section D4 are worthy of noting. These standards include timeliness, expression of analytic confidence, objectivity and the incorporation of alternative analysis, as well as the use of logical argumentation.

Timeliness

51

Intelligence Community Directive, Number 203. Analytic Standards (Effective June 21, 2007) Available at: http://docs.google.com/gview?a=v&attid=0.1&thid=11f1d65194a3fe1b&mt=application%2Fpdf&pli=1; Internet.

43

The ICD recognizes the importance of timeliness in intelligence analysis. The document emphasizes that analytic products have the obligation to meet the time standards set by the consumer, to ensure that they can be actionable products. This is why I choose to evaluate the length of time it took each analyst to complete their work, as there would be no purpose in utilizing a time consuming method; especially, if it is not likely to yield better results. Both groups were given two hours to complete their analysis. At the end of each experiment section, each subject also recorded the actual time it took them to complete their analysis, not including the instruction session. The results regarding time were interesting. In the control group, the amount of time to complete analysis ranged from a minimum of 45 minutes to a maximum of 105 minutes, with the average time at 70 minutes.

The experimental group completed their analysis slightly faster, with a

minimum completion time of 30 minutes, a maximum completion time of 90 minutes, and an average completion time of 58.96 minutes. The average length of completion time was a result found to be statistically significant, with a P-value of 0.036. This result suggests that the experimental group was able to structure and complete their analysis in a notably more time efficient way, almost 20% faster, than the control group. Figure 4.3 represents the completion time for 20 student-analysts in the control group (one member failed to provide their length of completion time) and for the 24 student-analysts in the experimental group.

Figure 4.3 Length of Analysis Completion Time

44

Upon completion, the student-analysts were also asked if this was enough time to complete their analysis. The control group indicated an average response of 4.43 (out of 5) and the experiment group indicated a response of 4.83 (out of 5), a difference that was found to be statistically significant with a P-value of 0.025. The results regarding time are important for two reasons. Not only were the student-analysts able to complete their analysis more quickly, but their perception for the process was that they had more than enough time.

Analytic Confidence The ICD also mandates that analytic products express the analyst’s level of confidence in their analytic judgment. After analysis was completed, each studentanalyst was asked to rate his or her level of analytic confidence. The control group had a minimum level of analytic confidence of 1, a maximum level of analytic confidence level of 9, and an average response of 5.93.

The

45

experimental group expressed a slightly higher level of analytic confidence, with a minimum level of 1.5, a maximum level of 9, and an average of 6.31. This difference was not found to be statistically significant, with a P-value of 0.2335. Figure 4.4 displays the levels of analytic confidence for both groups.

Figure 4.4 Level of Analytic Confidence

Objectivity and the Incorporation of Alternative Analysis The ICD also mandates that analysts remain objective in their work and that their analytic products incorporate alternative analysis when appropriate. This was one of the primary differences I saw between the control group and the experimental group. While a few students in the control group provided one or two alternative COAs, the majority of the student-analysts merely provided one COA with few comparisons to any alternatives, thus, not providing any insight to whether or not alternative solutions were considered. In the experimental group, the student-analysts, who used MCIM, provided a list of all possible COAs, and identified the importance of specific criterion or various factors to

46

those COAs. Their matrices highlighted strengths and weaknesses of each COA allowing a hypothetical decision maker (or in this case, researcher) to compare each COA against the alternatives. The matrices received from the experimental group provided a much clearer outline regarding the thought process of each student-analyst and allowed me to determine what alternative factors and COAs that the student-analysts considered. Additionally, by providing a list of alternatives, I can assume that the student-analysts took a more comprehensive look at different options, and did not merely find information to support the first COA they found fitting.

Logical Argumentation The ICD requires that key findings and analysis be supported by appropriate facts and information, as well as address any divergent ideas. As part of the research process, I asked both groups of student-analysts to use bullet points to explain their analysis. Although the bullets provided some indication as to the thought process of the studentanalyst, the matrices completed by the experimental group were much easy to understand and visibly conveyed the written analysis provided.

Bullet Point Analysis I also asked the students to provide bullet point analysis so that I could capture the data later on. One method I used to determine the thoroughness of analysis was by evaluating the number of bullet points that each student-analyst used. I found the average number of bullet points for each group to determine if there was a difference between groups. The control group had a total of 146 bullet points, used a minimum of three

47

bullet points, a maximum of 11 bullet points, and an average of 6.95 bullet points. The experimental group had a total of 139 bullet points, used a minimum of three bullet points, a maximum of 15 bullet points and an average of 5.79 bullet points. This difference was found to be statistically significant, with a P-value of 0.0415. I did not evaluate the quality of each bullet point in this step; however, I do recognize that each bullet point may not have been entirely relevant to the analysis. I am assuming that the amount of insignificant bullet points between the two groups was roughly the same; therefore, they would cancel each other out. Figure 4.5 provides a statistical breakdown of the number of bullet points used by each member of the control and experimental groups.

Figure 4.5 Bullet Point Analysis

48

Potential of Future Use Since MCIM is relatively new to the field of intelligence, I wanted some feedback as to the student’s opinion of future use of the methodology. Upon completion of their analysis, I asked the experimental group several post-test questions designed to determine if the student-analysts had enough instruction to complete the process, determine the ease of using MCIM, and assess the likelihood that each analyst would use MCIM in the future. When I asked the student-analysts if they felt like they had received enough instruction to complete the process using MCIM (the group received a 35 minute lecture on the process of MCIM in which I provided background information, and the group worked through two example problems), they responded with an average answer of 4.5 (out of 5). Most of the student-analysts indicated this method was relatively simple to use, rating it with an average score of 3.83 (out of 5). The majority of the studentanalysts also indicated that this was a method they would likely use again in the future, with an average score of 3.67 (out of 5). Thus, even with minimal instruction time, the student-analysts felt that the methodology was relatively easy to understand and use, and it is something they would consider using in future intelligence analysis.

49

CHAPTER 5: CONCLUSIONS

The purpose of this experiment was to test the value of using MCIM when solving intelligence related problems.

By controlling the method in which student-analysts

worked through an intelligence scenario, I was able to evaluate the significance of MCIM in intelligence analysis. These factors included the analyst’s thought process and product, their level of analytic confidence, as well as the length of time it took to complete their analysis.

Pre-Test and Post-Test Conclusions The pre-test and post-test questionnaire proved to be valuable, as they were an indicator as to some of the factors outside of the experiment that may have contributed to the results in some way. Additionally, they gave me a basis for which to compare the base level of knowledge and level of interest in participation between the two groups, prior to beginning research. One interesting conclusion drawn from the pre-test was that although the experimental group indicated a lower level of knowledge in regards to the topic (Russia’s relationship to OPEC) and expressed a lower level of interest with the topic, both of which were found to be statistically significant, they were able to arrive at a broader range of possible COAs. This result suggests that despite the control group having a jump start on the research, the experimental group was able to provide roughly the same possible COAs, as well as a few alternatives. Another interesting conclusion drawn after

50

reviewing the products of the two groups occurred in regards to time. The experimental group, who used MCIM, was able to complete their analysis in a more time efficient way (faster), while still perceiving that they had more than enough time (according to the pretest). Specifically, the average completion time for the control group was 70 minutes and the average time for experimental group was 58.6 minutes. Therefore, when looking at the big picture, although the experimental group seemed less knowledgeable and less interested, they were able to arrive at a more complete list of relevant possible COAs, and they completed their analysis in less time. Finally, the bullet point analysis indicated that the control group averaged approximately one more bullet point in the analysis part of their product. This difference was found to be statistically significance with a P-value of 0.0415. Although this result suggests that the control group was able to provide more analysis than the experimental group, it is important to remember that the worth of each bullet point was not assessed. It is also entirely possible that the student-analysts in the experimental group felt that the matrix was able to more accurately support their analysis, thus, they provided less bullet point reasoning. The results from the post-test suggest that all student-analysts participating in the experiment had some level of familiarity with structured analytic methodologies. In addition to familiarity with structured analytic methods, the results of the post-test indicated that all of the subjects that participated in the experiment have actually used these methods when conducting prior analysis. They also expressed that these methods improve intelligence analysis on some level. Despite these factors, no one in the control group utilized any sort of structured method when completing their analysis. There is no

51

apparent reason as to why this happened.

The student-analysts may have felt that

structured methodology wasn’t necessary for this type of problem, or perhaps they felt that using an informal process was both quicker and easier. Any future research or experiments conducted regarding the use of structured methodology in intelligence analysis, should include a post-test question asking why the student either chose to use, or chose not to use methodology. By asking this question, more information could be gathered as to what makes an analyst chose the method that they did.

Process and Product Conclusions As previously mentioned, when I originally picked the real world scenario (Russia’s relationship with OPEC after the December 17th, 2008 meeting) for which the student-analysts to provide their analysis, I chose a situation that I felt would have a clear cut answer within a few weeks. Although this has proved to not be the case, the use of a MCIM model has proven to be beneficial in a variety of ways. The matrix clearly explains how each analyst in the experimental group arrived at their conclusion. The student-analysts outlined the criteria that he or she felt to be important and explained how significant each of these factors was at arriving at their decision by the use of a weighting system. Additionally, if any of the criteria should change as time progresses, the matrix is easily modified by adding or eliminating criteria, or by merely adjusting the weights to arrive at an alternative COA. In other words, all the hard work has already been done on the problem, and only minor changes would need to be made to suggest an alternative COA needed due to changes in the adversary’s environment or priorities. Since no one in the control group used any form of structured methodology to arrive at their conclusion,

52

their decision maker (or in this case, researcher) would have a minimal idea of how each arrived at their conclusion. Although a majority of the control group’s bullet point analysis does support the COA that they chose to be most likely, there is minimal mention of any other possible COA, or of the significance of each factor they considered to support their predicted COA. If there was a change in the adversary’s environment that would suggest the need for an alternative COA, the student-analysts in the control group would need to go back and figure out what is the next most likely solution based on the importance of each factor that they considered. In general MCIM is beneficial to the field of intelligence due to its flexibility and versatile nature. The use of MCIM is not specific to a given intelligence field, therefore it is applicable to business, law enforcement and national security. Additionally, it is an appropriate method to use when working with both qualitative and quantitative data.

A Step Forward While there are a number of potential benefits with using this methodology, there are also several prospective weaknesses. Although the matrix seeks to identify the most important criteria involved in the decision making, it has the potential to be flawed as the analyst is doing the scoring of the criteria. Therefore, the process of assigning weight to pieces of criteria has the potential to be based on the analyst’s intuition and may contain bias. Additionally, an analyst must know how and when to make a distinction when weighing different criteria. For example, I would like to refer back to the example of buying a car that I discussed in Chapter 2. How would an analyst rank a difference in price for values that are close, such as $25,000 and $25,001? In other words, at what

53

point do they numbers quit being the same? Finally, this process seeks to forecast something in which the analyst has no actual control over. Individuals only have total control over what they do, but not competitors, enemies, or criminals. Thus, even with a detailed, structured approach that hopes to identify the COA that an opponent is most likely to chose, it is possible that the adversary will go with a COA that is completely random based on their intuition or “gut feeling.” Heuer draws attention to this idea when he said, “Caution is in order, however, whenever one thinks of predicting or even explaining another person’s decision, regardless of whether the person is American or foreign. People do not always act rationally in their own best interests. Their decisions are influenced by emotions, habits, what others might think, or values that others many not be aware of.”52 In all, the benefits of using a structured methodology in intelligence analysis, as well as the overall general positive responses received from the student-analysts regarding the use of MCIM, suggest that this topic is worthy of continued research. Larger sample sizes would be beneficial in any future experiments in order to confirm hypotheses surrounding use of the method. Additionally, in future experiments, analysis should be provided for a real world situation that would have a clear cut answer within a short time frame of the analysis in order to further evaluate MCIM’s ability to aid decision makers in efficient forecasting. Despite the possible flaws with the methodology, the results of this experimental research highlight the potential benefits of using structured analytic methods, such as MCIM, during the analytic process. Although the use of MCIM can never guarantee a “correct” answer, it may increase the odds of choosing the most likely answer, by 52

Heuer, Richards, J., Email. February 12, 2009.

54

providing a more efficient way of handling unstructured data in both qualitative and quantitative form. Additionally, matrix analysis provides a clear explanation to a decision maker regarding the thought process of each analyst. By identifying the weight that an analyst places on each individual criterion, MCIM can highlight the areas of disagreement between two individuals, and it is easily adjusted to accommodate new data or “what-if” scenarios. Additionally, as demonstrated in the experiment, using the MCIM methodology may allow an analyst to structure and complete their analysis in a more time efficient. While the jury is still out regarding the accuracy of the analysis, MCIM can help teach an analyst the importance of considering the significance of certain factors. This study hopes to take the first steps of many in increasing an analyst’s ability to thoroughly examine all possible COAs in a timely manner, as well as accurately forecast what COA one’s adversary is likely to choose.

55

BIBLIOGRAPHY Balcomb, J.D. and A. Curtner. “Multi-Criteria Decision-Making Process for Building.” Paper to be presented at the American Institute of Aeronautics and Astronautics Conference, Las Vegas, Nevada, July 24-28. 2000. Available online at http://www.nrel.gov/docs/fy00osti/28533.pdf; Internet; accessed 23 March 2009. Belton, Valerie and Theodor J. Stewart. Multiple Criteria Decision Analysis: An Integrated Approach. The Netherlands: Kluwer Academic Publishers, 2002. Dzubow, PhD and William Z. Suplee IV, CFA, CFP, ChFC, CASL. “Using MultipleCriteria Decision Analysis to Simplify the Financial Planning Process.” Journal of Financial Planning. March 2008. Available at http://www.library.idsc.gov.eg/GUI/Globals/Upload/BULLETIN_ATTACHMEN T/92/e-files/manegement%20and%20economics/using%20multiple.pdf; Internet; accessed 5 December 2008. Forgang, William G., Strategy-Specific Decsion Making: A Guide for Executing Competitive Strategy. M.E. Sharpe, Inc., Armonk, New York, 2004. Forman, Ernest, Decision by Objectives (How to Convince Others That You Are Right). Available at http://mdm.gwu.edu/forman/DBO.pdf; Internet. Gilchrist, Stacy. “Staff Study.” Research Paper for Advanced Analytic Techniques, Mercyhurst College. Heuer, Richards, Jr. Email. 12 February 2009. Heuer, Richards, Jr. Psychology of Intelligence Analysis [book on-line]. Langley, Virginia: Central Intelligence Agency Center for the Study of Intelligence, 1999. Available at https://www.cia.gov/library/center-for-the-study-of-intelligence/csipublications/books-and-monographs/psychology-of-intelligenceanalysis/PsychofIntelNew.pdf; Internet. Heuer, Richards, Jr. “Taxonomy of Structured Analytic Techniques.” Paper prepared for the International Studies Association 2008 Annual Convention, March 26-29, 2008, San Francisco, CA. Available at http://www.allacademic.com//meta/p_mla_apa_research_citation/2/5/4/1/2/pages2 54125/p254125-1.php; Internet; accessed 20 November 2008. Intelligence Community Directive, Number 203. Analytic Standards (Effective 21 June 2007). Available at http://docs.google.com/gview?a=v&attid=0.1&thid=11f1d65194a3fe1b&mt=appl ication%2Fpdf&pli=1; Internet.

56

Kiker, Gregory A., Todd S. Bridges, Arun Varghese, Thomas P. Seager and Igor Linkov. “Application of Multicriteria Decision Analysis in Environmental Decision Making.” Integrated Environmental Assessment and Management. Volume 1, Number 2, Pages 95-108. 2005. Available at http://www.allenpress.com/pdf/ieam01-02_95_108.pdf; Internet; accessed 20 November 2008. LeGault, Michael R. Think: Why Crucial Decision Can’t Be Made in the Blink of an Eye. New York, Threshold Editors, 2006. Page 305. Marrin, Stephen. “Intelligence Analysis: Structured Methods or Intuition.” American Intelligence Journal. Page 7. Summer 2007. Peterson, Joshua. “Appropriate Factors to Consider When Assessing Analytic Confidence in Intelligence Analysis.” (Master's Thesis, Mercyhurst College, March 2008). Satty, Thomas. “Decision making with the analytic hierarchy process,” Int. J. Services Sciences, Vol. 1, No. 1, 2008. Available at http://inderscience.metapress.com/media/pgwf2qtuyg3xnvnhxnby/contributions/0 /2/t/6/02t637305v6g65n8.pdf; Internet; accessed 26 March 2009. Saaty, Thomas L. “How To Make A Decision: The Analytic Hierarchy Process.” Available at http://sigma.poligran.edu.co/politecnico/apoyo/Decisiones/curso/Interfaces.pdf; Internet; accessed 24 March 2009. Simon, Herbert and Associates. “Decision Making and Problem Solving.” Reprinted with permission from Research Briefings 1986: Report of the Research Briefing Panel on Decision Making and Problem Solving. 1986 by the National Academy of Sciences. Published by National Academy Press, Washington, DC. Available at http://dieoff.org/page163.htm; Internet; accessed 20 November 2008. The United States Army. Field Manual 101-5. Staff Organization and Operations. 31 May 1997. Available at http://www.fs.fed.us/fire/doctrine/genesis_and_evolution/source_materials/FM101-5_staff_organization_and_operations.pdf; Internet. Page 115. The United States Army 2008 U.S. Army Posture Statement. Information Papers: Red Team Education and Training. Available at http://www.army.mil/aps/08/information_papers/prepare/Red_Team_Education_a nd_Training.html; Internet; accessed 26 September 2008. The United States Government. Director of National Intelligence, Vision 2015. Available at http://www.dni.gov/Vision 2015.pdf; Internet.

57

Triantaphyllou, Evangelos and Stuart H. Mann. “An Examination of the Effectiveness of Multi-Criteria Decision Making Methods: A Decision Making Paradox.” Available at http://www.csc.lsu.edu/trianta/index.html?http://www.csc.lsu.edu/trianta/Books/D ecisionMaking1/Book1.htm; Internet. Triantaphyllou, Evangelos. “Can We Always Determine the Right Alternatives in Business Problems.” 18 August 2002. Triantaphyllou, Evangelos and Panos M. Parlos (ed.). Multi-Criteria Decision Making Methods: A Comparative Study. The Netherlands: Kluwer Academic Publishers, 2000. Wang, Xiaoting. “Study of Ranking Irregularities When Evaluating Alternatives By Using Some ELECTRE Methods and a Proposed New MCDM Method Based on Regret and Rejoicing,” A Thesis Submitted to the Graduate Faculty of the Louisiana State University and Mechanical College. August 2007. Available online at http://etd.lsu.edu/docs/available/etd-07112007012708/unrestricted/Wang_thesis.pdf; Internet; accessed 24 March 2009. Webster’s Dictionary Online; available from http://www.merriamwebster.com/dictionary/criteria; Internet. Zopounidis, Constantin and Doumpos, Michael. “Multiple-Criteria Decision Making.” Thomson Corporation. 2006.

58

APPENDICES

59

Appendix 1: Analytic Methods Experiment Sign-Up Form

Name: Class Year: Phone Number: E-mail Address:

Instruction Session Dates/Times: (Please select the day you are able to participate) Wednesday, 10 December 2008 Thursday, 11 December 2008

5:00-8:00pm 5:00-8:00pm

If you are able to participate on either day please check here, and you will be randomly assigned a day

____ ____ ____

*** I will send you an e-mail confirming your experiment day and time

Upon completion, please return this form to Lindsey Jakubchak or Travis Senor in CIRAT. Contact Info: [email protected]

60

Appendix 2: Institutional Review Board Proposal Form Date Submitted: 10/10/2008

Advisor's Name (if applicable): Kristan Wheaton

Investigator(s): Lindsey Jakubchak

Advisor’s E-mail: [email protected]

Investigator Address:

Advisor's Signature of Approval: [X] Place X here if advisor has approved research

Investigator(s) E-mail: [email protected]

Investigator Telephone Number:

Title of Research Project: The Effectiveness of Multi-Criteria Decision Making (MCDM) in the Field of Intelligence Date of Initial Data Collection: TBD, anticipate October-December 2008

_______________________________________________________________________ _ Please describe the proposed research and its purpose, in narrative form: Multi-Criteria Decision Making (MCDM) is a generic term used to encompass a broad range of analytic methodologies that use matrices as the basis for their conclusions. More specifically, MCDM is an internally focused support system that confronts real world decisions based on the evaluation of multiple criteria, goals, and objectives of conflicting nature. Additionally, it provides substance to a decision maker as it allows for selecting an option based on the appropriateness of those alternatives weighted against the others. While there is a substantial body of literature related to the use and effectiveness of MCDM in the general sense, there has been little research done in the field of intelligence. The purpose of this proposed research and experiment is to take an alternative look at this methodology. In other words, if MCDM was switched to an external focus, and used in the field of intelligence, is it likely that this methodology

61

would be a valuable way to predict what course of action one’s adversary is likely to choose? I have developed an experiment which I feel will test this hypothesis, in both process and product, when solving intelligence related problems. I plan to test if MCDM truly is a viable methodology by using intelligence analysts (both undergraduate and graduate students in the intelligence program at Mercyhurst College) to solve a real world problem. Indicate the materials, techniques, and procedures to be used (submit copies of materials): Materials: Exercise Scenarios Writing Utensils Pre-Test and Post-Test Questionnaire Procedure: One month prior to the experiment, I will solicit both undergraduate and graduate students at Mercyhurst College to participate through department wide e-mails, fliers and through short information sessions at the beginning of classes (with permission of the designated professor). Students will be selected on a first come, first serve basis. One week prior to the study, I will send out reminders to those who have volunteered to participate. I will send out another reminder the day before the study. During the actual study, I will begin by going over the directions. I will then explain what is expected of the participants, and how they will be getting credit for their participation. After the introduction, I will pass out a pre-test questionnaire (attached at the end of this form), the study’s materials, and instruct the students to begin their analyses. I will then provide the students with one week in which they can complete their analysis and provide a time in which they can return their finished product to me. This process will be the same for both groups (the control group and the experimental group), with the only exception being that the experimental group will receive a one hour long instructional session on the how to use a variant of MCDM in an intelligence context (in order to complete their analysis), after the introduction. Following the completion of the exercise, I will ask the participants to fill out a questionnaire (attached at end of this form) and provide feedback regarding both the topic and the experiment. I plan to conduct my experiment two times, on two different nights. They will vary only in which handout I give them. There will be one experimental group and a control group.

62

All groups will be given the exact same scenario; I will just vary the methodology utilized to solve the problem. (Please see the forms below)

63

1. Do you have external funding for this research (money coming from outside the College)? Yes[ ] No[X] Funding Source (if applicable): 2. Will the participants in your study come from a population requiring special protection; in other words, are your subjects someone other than Mercyhurst College students (i.e., children 17-years-old or younger, elderly, criminals, welfare recipients, persons with disabilities, NCAA athletes)? Yes[ ] No[X] If your participants include a population requiring special protection, describe how you will obtain consent from their legal guardians and/or from them directly to insure their full and free consent to participate. N/A Indicate the approximate number of participants, the source of the participant pool, and recruitment procedures for your research: I plan to have approximately 75 participants. I plan to recruit undergraduate and graduate students in the intelligence studies department through a department-wide email, fliers hung in the building, and by information sessions held at the beginning of classes (with permission of the designated professor). I will select the students on a first come, first serve basis. Will participants receive any payment or compensation for their participation in your research (this includes money, gifts, extra credit, etc.)? Yes[X] No[ ] If yes, please explain: In the past, most of the intelligence professors have been willing to grant extra credit for participating in an experiment. 3. Will the participants in your study be at any physical or psychological risk (risk is defined as any procedure that is invasive to the body, such as injections or drawing blood; any procedure that may cause undue fatigue; any procedure that may be of a sensitive nature, such as asking questions about sexual behaviors or practices) such that participants could be emotionally or mentally upset? Yes[ ] No[X] Describe any harmful effects and/or risks to the participants' health, safety, and emotional or social well being, incurred as a result of participating in this research, and how you will insure that these risks will be mitigated: None.

64

4. Will the participants in your study be deceived in any way while participating in this research? Yes[ ] No[X] If your research makes use of any deception of the respondents, state what other alternative (e.g., non-deceptive) procedures were considered and why they weren't chosen: N/A 5. Will you have a written informed consent form for participants to sign, and will you have appropriate debriefing arrangements in place? Yes[X] No[ ] Describe how participants will be clearly and completely informed of the true nature and purpose of the research, whether deception is involved or not (submit informed consent form and debriefing statement): Prior to the training sessions, participants will be provided with a general overview of what will occur during the session as well as the consent form, which will also describe what is expected of them. Following the administrative questionnaire, participants will be provided with a debriefing statement that will explain how the results from the session will be used (please see forms at the end of this proposal). Please include the following statement at the bottom of your informed consent form: “Research at Mercyhurst College which involves human participants is overseen by the Institutional Review Board. Questions or problems regarding your rights as a participant should be addressed to Mr. Tim Harvey, Institutional Review Board Chair; Mercyhurst College; 501 East 38th Street; Erie, Pennsylvania 16546-0001; Telephone (814) 824-3372.” 6. Describe the nature of the data you will collect and your procedures for insuring that confidentiality is maintained, both in the record keeping and presentation of this data: Names are not required for my research and thus no names will be used in the recording of the results or the presentation of my data. Names will only be used to notify professors of participation in order for them to correctly assign extra credit. 7. Identify the potential benefits of this research on research participants and humankind in general. Potential benefits include: For participants: For some students, this experiment will provide an opportunity to learn a new analytical tool that they may use in intelligence related decision making. For other students, it will provide an opportunity to practice decision making skills that they have learned in the classroom or developed in the real world. Students are often asked to utilize various

65

analytical tools in order to gain a more thorough understanding of the problem at hand, and this may be an effective methodology for doing that. For the Intelligence Community (IC): This experiment hopes to prove that MCDM is an effective methodology to use when making intelligence related decisions. In doing so, the IC would benefit from a methodology that allows our nation’s decision makers to make more informed decisions. Please submit this file and accompanying materials to the IRB Chair, Tim Harvey, via electronic mail ([email protected]) for review.

Appendix 3: Multi-Criteria Decision Making Participation Consent Form The purpose of this research is to test the effectiveness of a particular method, in both process and product, when making intelligence related decisions. Your participation involves an instruction period, completion of a short intelligence analysis, and filling out a pre and post test questionnaire. This process should take no longer than 3 hours. Your name WILL NOT appear in any information disseminated by the researcher. Your name will only be used to notify professors of your participation in order for them to assign extra credit. There are no foreseeable risks or discomforts associated with your participation in this study. Participation is voluntary and you have the right to opt out of the study at any time for any reason without penalty. I, ____________________________, acknowledge that my involvement in this research is voluntary and agree to submit my data for the purpose of this research. _________________________________ __________________ Signature _________________________________ __________________ Printed Name

Date

Class

Name(s) of professors offering extra credit and class(es) you are enrolled in: (Professor Breckenridge, Professor Marrin, Professor Welch, and Professor Wheaton) _______________________________________________________________________ _ Researcher’s Signature: __________________________________________________________ If you have any further question about methodology used, or this research, you can contact me at [email protected] Research at Mercyhurst College which involves human participants is overseen by the Institutional Review Board. Questions or problems regarding your rights as a participant should be addressed to Tim Harvey; Institutional Review Board Chair; Mercyhurst College; 501 East 38th Street; Erie, Pennsylvania 16546-0001; Telephone (814) 824-3372. [email protected]

Lindsey Jakubchak, Applied Intelligence Master’s Student, Mercyhurst College Kristan Wheaton, Thesis Advisor, Mercyhurst College

Appendix 4: Experiment Section #1 (Control Group) You are a national security analyst specializing in Russian relations. Your director is interested in the outcome of the upcoming meeting of the Organization of Petroleum Exporting Countries (OPEC), taking place in Algeria on December 17th, 2008. Please provide an answer to the following question: How is Russia likely to seek to interact with OPEC after the December 17th, 2008 meeting in Algeria? STEP 1: Please complete the following analysis in the space provided below. *You may use any sources you would like It is likely that Russia will seek to_______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ _____ STEP 2: Please list your reason(s) for your analysis: *You do not need to provide your sources        

    STEP 3: Please rate your level of analytic confidence. Important Information: Analytic Confidence: Analytic Confidence reflects the level of confidence an analyst has in his or her estimates and analyses. It is not the same as using words of estimative probability, which indicate likelihood. It is possible for an analyst to suggest an event is virtually certain based on the available evidence, yet have a low amount of confidence in that forecast due to a variety of factors or vice versa. As you are considering your level of analytic confidence, please consider these seven factors: 1. Use of Structured Method(s) in Analysis 2. Overall Source Reliability 3. Source Corroboration/Agreement 4. Level of Expertise on Subject/Topic & Experience 5. Amount of Collaboration 6. Task Complexity 7. Time Pressure Mark your analytic confidence on the scale below. Low |-----------------------------------------------------------------------| High

Appendix 5: Analytic Methods Experiment Links

Current Situation: •

Although Russia is not currently an OPEC member, it does maintain a relationship with the organization.

Links to get you started… • You are not limited to these sources; they are merely starting points to familiarize yourself with the topic! • Feel free to use whatever sources you would like! Monthly Oil Market Report-November 2008: Source: Organization of the Petroleum Exporting Countries (OPEC) http://www.opec.org/home/Monthly%20Oil%20Market%20Reports/2008/pdf/M R112008.pdf

Non-OPEC Oil Production: Source: Council on Foreign Relations http://www.cfr.org/publication/14554/nonopec_oil_production.html

Organization of the Petroleum Exporting Countries (OPEC): Source: OPEC Homepage http://www.opec.org/home/

Outlook for Non‐OPEC Oil Supply Growth in 2008‐2009: Source: Energy Information Administration http://www.eia.doe.gov/emeu/steo/pub/special/2008-non-opec-oil-supply.pdf

Russia Energy Data, Statistics and Analysis - Oil, Gas, Electricity, Coal: Source: Energy Information Administration http://www.eia.doe.gov/emeu/cabs/Russia/pdf.pdf

Appendix 6: Pre-Test Questionnaire (Control Group and Experimental Group) Answer the following questions by circling the best response. Please answer all the questions honestly. On a scale from 1 to 5, where 1 represents not interested at all and 5 represents extremely interested, how interested are you in participating in this experiment? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not knowledgeable at all and 5 represents extremely knowledgeable, how knowledgeable are you with this topic? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not interested at all and 5 represents extremely interested, how interested are you in this topic? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not interested at all and 5 represents extremely interested, how interested are you in learning about analytic methodologies? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not enough time and 5 represents more than an adequate amount of time, do you think you will have enough time to complete this analysis? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not important at all and 5 represents extremely important, how important was offering extra credit in participating in this experiment? 1

2

What is your class rank? FR

SO

JR

SR

G1

G2

3

4

5

Please provide any additional comments or feedback in the space provided: _______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ _____

Appendix 7: Post-Test Questionnaire (Control Group) Answer the following questions by circling the best response. Please answer all the questions honestly. On a scale from 1 to 5, where 1 represents not at all familiar and 5 represents extremely familiar, how familiar are you with the process of Multi-Criteria Decision Making (MCDM)? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not at all familiar and 5 represents extremely familiar, how familiar are you with the process of Analysis of Competing Hypothesis (ACH)? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not at all and 5 represents extremely frequently, how frequently do you use structured analytic methods in solving intelligence related problems? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents no improvement at all and 5 represent a significant improvement, how much do you feel analytic methods improve intelligence analysis? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not enough at all, and 5 represents more than enough, do you feel you found enough adequate information to complete this analysis? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not enough time at all and 5 represents more than adequate amount of time, do you feel you had enough time to complete this process? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents no enjoyment at all and 5 represents thorough enjoyment, how much did you enjoy this process? 1

2

3

4

5

Approximately how much time did you spend completing this analysis? ___________________ What do you think the purpose of this experiment was? ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ What is your class rank? FR

SO

JR

SR

G1

G2

Please provide any additional comments or feedback in the space provided: _______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ ______

Appendix 8: Multi-Criteria Decision Making Participation Debriefing Thank you for participating in this research process. I appreciate your contribution and willingness to support the student research process. The purpose of this study was to determine the effectiveness of using MCDM as an analytical methodology in the field of intelligence. Currently there has been little research done on this topic, and this study hopes to take the first of many steps in increasing one’s ability to accurately predict both what course of action one’s adversary is likely to choose, as well as forecast what is happening in the environment in which an analyst works in. My experiments today were designed to demonstrate the significance of using this methodology in solving intelligence related problems. By shifting the process of MCDM from an internal focus to an external focus, it is keeping with the theme of intelligence. Additionally, utilizing methods that promote a more efficient method of structuring data may increase an analyst’s ability to learn what is happening with his/her adversary or the environment in which they are working in. I plan to use the results from this study to determine if the use of MCDM in solving intelligence related problems improves the analytic process and product. Please note, you will be able to review the results of this experiment. The results will be included in a paper that I am preparing to present at the International Studies Association (ISA) Conference, and will be accessible online. Additionally, results and conclusions will be accessible via my completed thesis. If you have any further question about MCDM or this research you can contact me at [email protected]

Appendix 9: Multi-Criteria Decision Making Lecture •

Everybody Makes Decisions On A Daily Basis o Some decisions can be made with our intuition or “gut feelings” as they are not life altering and require little to no analysis.  Examples: What will I eat for breakfast in the morning? What road will I use to travel to work? o Other decisions require the balancing of multiple factors or criteria, and do affect our life in some way.  Examples: Where will I go to college? What kind of car will I buy? What career will I choose to embark on? o In the field of intelligence, decision related to national security, law enforcement, and business directly relate to the safety of a nation, the rise and fall of a country, and the stability of an organization. These decisions mandate appropriate analysis.  Examples: What is likely to happen in Country X in the next two years? What will likely happen to the oil fields in Country Y over the next 12-24 months?

•

Multi-Criteria Decision Making (MCDM) o The process of evaluating possible courses of action (COA) in a systematic way against a set of fixed criteria. o Involves the breakdown of complex problems into smaller pieces in order for more thorough analysis. o Often called “matrix analysis.” o MCDM is a generic term used to encompass a broad range of analytical methodologies o In MCDM, emphasis is on placing value, or judgment of an item’s worth and desirability, on pieces of criterion, a standard by which a judgment or decision may be made.

•

What Constitutes a MCDM problem? o MCDM is used when a decision needs to be made that has two or more known COAs or two or more possible COAs. o Either a “need to find best solution” or “we need to find a solution.” o Must have multiple criteria.

•

Process Of MCDM (8 Steps) o Requirement/Question and Collection  Good Essential Question  Understanding INTENT of question, is essential to good analysis

o Establish Possible COA  Brainstorm  Want to identify as many COAs as possible o Establish “Screening Criteria”  Designed to eliminate COAs  “Must have/be” guidelines o Screen COA o Establish “Evaluation Criteria”  Designed to rank COAs  Can be same as screening criteria. o Weight Evaluation Criteria o Evaluate COA  What scale? • Best to worst, 1 to 3, 1 to 5, 1 to 10? • Based on? Intuition? Standards? Even distribution? o Make Recommendations/Estimate •

Conventional Use Of MCDM Vs. Use Of MCDM In Intelligence o Difference of perspective  The conventional MCDM process focuses on one’s own decisionmaking process, a situation where an individual has complete control.  The intelligence focused MCDM process focuses on the decisionmaking process of others, a situation in which an analyst has no control over. o Purpose  By shifting the process of MCDM from an internal focus to an external focus, it is keeping with the theme of intelligence  Therefore, instead of establishing the COAs about the “best” decision for an individual, an analyst seeks to determine which COA one’s adversary or competitor is likely to choose.

•

Likely Benefits Of The MCDM Process In Intelligence o Utilizing methods that promote a structured review of external information increases an analyst’s awareness of his/her adversary, and the environment in which they are working in.  By continually assessing the social, political, economical and technological aspects of an adversary’s environment, an analyst might gain insight to factors outside of a decision maker’s immediate control, deterring them from decisions they would otherwise likely make. o The use of an intelligence focused MCDM process may help to prevent analytic pitfalls, such as the satisficing strategy and groupthink.

o Should allow for a more complete analysis of the situation and account for possible COAs that would otherwise have not been considered.

Appendix 10: Experiment Section #2 (Experimental Group) You are a national security analyst specializing in Russian relations. Your director is interested in the outcome of the upcoming meeting of the Organization of Petroleum Exporting Countries (OPEC), taking place in Algeria on December 17th, 2008. Using Multi-Criteria Decision Analysis, please supply an answer to the following question: How is Russia likely to seek to interact with OPEC after the December 17th, 2008 meeting in Algeria? The following is designed to guide you through the process of MCDM. Please note, Steps 2, 3 and 4 are mandatory. Step 1 is optional, and is designed to assist you with completion of the matrix. *You may use any sources you would like. STEP 1: This step is optional

•

Possible Course of Action: What courses of actions are possible for Russia? You may have as many courses of action as you would like

COA 1: COA 2: COA 3: •

Screening Criteria: Designed to eliminate COA’s o o

•

Remaining Courses of Action: What courses of actions are possible for Russia? (After screening applied) You may have as many courses of action as you would like

COA 1: COA 2: •

Evaluation Criteria: Designed to rank COAs. Evaluation Criteria may be tangible items (things that can be measured) or intangible items (things things that cannot be measured). These criteria can be ranked by scale: best to worst 1 to 3, 1 to 5, etc. They can also be based on intuition, standards, even distribution, etc.

Eval 1: Eval 2: Eval 3:

Step 2: This step is mandatory Matrix: Please complete the Matrix in Excel (format is on the shared drive). Below is to demonstrate what the matrix should look like. Criteria 1:

Criteria 2:

Criteria 3:

Criteria 4:

Total/ Rank Order

COA 1: COA 2: COA 3: COA 4: COA 5:

It is likely that Russia will seek to ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ STEP 3: This step is mandatory Please list your reason(s) for your analysis: *You do not need to provide your sources       

      STEP 4: This step is mandatory Please rate your level of analytic confidence. Important Information: Analytic Confidence: Analytic Confidence reflects the level of confidence an analyst has in his or her estimates and analyses. It is not the same as using words of estimative probability, which indicate likelihood. It is possible for an analyst to suggest an event is virtually certain based on the available evidence, yet have a low amount of confidence in that forecast due to a variety of factors or vice versa. As you are considering your level of analytic confidence, please consider these seven factors: 1. Use of Structured Method(s) in Analysis 2. Overall Source Reliability 3. Source Corroboration/Agreement 4. Level of Expertise on Subject/Topic & Experience 5. Amount of Collaboration 6. Task Complexity 7. Time Pressure. Mark your analytic confidence on the scale below. Low |-----------------------------------------------------------------------| High

Appendix 11: Post-Test Questionnaire (Experimental Group) Answer the following questions by circling the best response. Please answer all the questions honestly. On a scale from 1 to 5, where 1 represents not at all familiar and 5 represents extremely familiar, how familiar are you with the process of Multi-Criteria Decision Making (MCDM)? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not at all familiar and 5 represents extremely familiar, how familiar are you with the process of Analysis of Competing Hypothesis (ACH)? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not at all and 5 represents extremely frequently, how frequently do you use analytic methods in solving intelligence related problems? 1

2

3

4

5

On a scale from 1 to 5, where 1 no improvement and 5 represents a significant improvement, how much did this using this methodology improve your intelligence analysis? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents no enjoyment at all and 5 represents thorough enjoyment, how much did you enjoy this process? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not enough at all, and 5 represents more than enough, do you feel you found enough adequate information to complete this analysis? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not enough time at all and 5 represents more than adequate amount of time, do you feel you had enough time to complete this process? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not enough instruction at all and 5 represents more than enough instruction, do you feel you had enough instruction to complete this process using MCDM? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents extremely difficult and 5 represents extremely easy, how was this methodology to use? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not at all interested and 5 represents extremely interested, how interested are you in learning about different analytic methods after completion of this exercise? 1

2

3

4

5

On a scale from 1 to 5, where 1 represents not at all likely and 5 represents extremely likely, how likely is it that you will use this methodology in the future? 1

2

3

4

5

Approximately how much time did you spend completing this analysis (Not including classroom instruction)? ___________________ What do you think the purpose of this experiment was? ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ What is your class rank? FR

SO

JR

SR

G1

G2

Please provide any additional comments or feedback in the space provided: _______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ ____________________________________________________________________________ __________________________________________________________________________________

Appendix 12: Statistical Data Length of Time to Complete Analysis (in minutes): Null: Group B does not take less time to finish. Alternative: Group B takes less time to finish than group A. – claim Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30. Te sts of Normality a

Group A (Control) Group B (Experiment)

Kolmogorov-Smirnov Statistic df Sig. .221 20 .012 .212 20 .019

Shapiro-Wilk Statistic df .924 20 .935 20

Sig. .116 .193

a. Lilliefors Significance Correction

Shapiro-Wilk test gives p-values > ( = 0.05), thus normality assumption is satisfied for both samples.

Normal Q-Q Plot of Group A (Control) Expected Normal

2

Most points are close to the line thus the assumption of normality is satisfied for the group A.

1 0 -1 -2 40

50

60

70

80

90

Observed Value

100

110

Normal Q-Q Plot of Group B (Experiment) Expected Normal

2

Most points are close to the line thus the assumption of normality is satisfied for the group B.

1

0

-1

30

40

50

60

70

80

90

Observed Value 110 100

Box plot shows no outliers for group A.

90 80 70 60 50 40 Group A (Control)

90

Box plot shows no outliers for group B.

80 70 60 50 40 30 Group B (Experiment)

Group Statistics Group Length of time in minutes Group A - Control Group B - Experiment

N 20 24

Mean 70.0000 58.9583

Std. Deviation 17.09109 16.61450

Std. Error Mean 3.82168 3.39142

Independent Sample s Test Levene's Test for Equality of Variances

F Length of time in minutes Equal variances assumed Equal variances not assumed

Sig.

.542

.466

t-test for Equality of Means

t

df

Sig. (2-tailed)

Mean Difference

Std. Error Difference

95% Confidence Interval of the Difference Lower Upper

2.167

42

.036

11.04167

5.09607

.75738

21.32596

2.161

40.143

.037

11.04167

5.10950

.71612

21.36721

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.466) > ( = 0.05), thus assumption of equal variances is satisfied. Independent Samples Test Levene's Test for Equality of Variances F Sig. Length of time in minutes Equal variances assumed Equal variances not assumed

.542

.466

t-test for Equality of Means t df Sig. (2-tailed) 2.167

42

.036

2.161

40.143

.037

According to above table, t-test value = 2.167, P-value = 0.036. Since (P-value = 0.036) < ( = 0.05), null hypothesis is rejected. Conclusion: At 5% level, Group B takes less time to finish than group A.

Level of Analytic Confidence: Null: Group B does not have higher level of analytic confidence. Alternative: Group B does have higher level of analytic confidence than group A. – claim Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30. Te sts of Normality Groups for Analytic Confidence Level of Analytic Group A (Control) Confidence Group B (Experiment)

a

Kolmogorov-Smirnov Statistic df Sig. .165 21 .137 .137 24 .200*

Statistic .945 .939

Shapiro-Wilk df 21 24

Sig. .268 .157

*. This is a lower bound of the true significance. a. Lilliefors Significance Correction

Kolmogorov-Smirvov test gives p-values > ( = 0.05), thus normality assumption is satisfied for both samples.

Normal Q-Q Plot of Level of Analytic Confidence Expected Normal

for groupAC= Group A (Control)

2

Most points are close to the line thus the assumption of normality is satisfied for the group A.

1 0 -1 -2 0

2

4

6

8

10

Observed Value

Normal Q-Q Plot of Level of Analytic Confidence Expected Normal

for groupAC= Group B (Experiment)

2

Most points are close to the line thus the assumption of normality is satisfied for the group B.

1 0 -1 -2 2

4

6

Observed Value

8

10

Level of Analytic Confidence

10.00

Box plot shows outliers for both groups. Even with the presence of outliers, normality is satisfied. Also these values are important for the analysis. Thus the decision is not to remove the outliers.

8.00 6.00 4.00 2.00

31

4

0.00 Group A (Control)

Group B (Experiment)

Groups for Analytic Confidence Group Statistics Groups for Analytic Confidence Level of Analytic Group A (Control) Confidence Group B (Experiment)

N

Mean 5.9286 6.3125

21 24

Std. Deviation 1.75560 1.74961

Std. Error Mean .38310 .35714

Inde pe nde nt Sample s Test Levene's Test for Equality of Variances

F Level of Analytic Equal variances Confidence assumed Equal variances not assumed

.002

Sig.

t-test for Equality of Means

t

.967

df

Sig. (2-tailed)

Mean Difference

Std. Error Difference

95% Confidence Interval of the Difference Lower Upper

-.733

43

.467

-.38393

.52363

-1.43993

.67207

-.733

42.171

.468

-.38393

.52375

-1.44078

.67292

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.967) > ( = 0.05), thus assumption of equal variances is satisfied. Indepe ndent Sample s Te st Levene's Test for Equality of Variances F Sig. Level of Analytic Equal variances Confidence assumed Equal variances not assumed

.002

.967

t

t-test for Equality of Means df Sig. (2-tailed)

-.733

43

.467

-.733

42.171

.468

According to above table, t-test value = -0.733, P-value = 0.467/2 = 0.2335 (need to divide the P-value by 2 as we have one-tailed test where as SPSS gives you 2-tailed test P-value. Since (P-value = 0.2335) > ( = 0.05), null hypothesis is not rejected. Conclusion: At 5% level, Group B does not have higher level of analytic confidence than group A.

Bullet Point Reasoning: Null: Group B does not have less bullet points than group A. Alternative: Group B does have less bullet points than group A. – claim Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30. Te sts of Normality

Bullet Point Reasoning

Groups for Bullet Point Reasoning Group A - Control Group B - Experiment

a

Kolmogorov-Smirnov Statistic df Sig. .208 21 .019 .317 24 .000

Statistic .941 .749

Shapiro-Wilk df 21 24

Sig. .229 .000

a. Lilliefors Significance Correction

Shapiro-Wilk test gives p-value > ( = 0.05), thus normality assumption is satisfied for Group A.

Normal Q-Q Plot of Bullet Point Reasoning Expected Normal

for groupBullet= Group A - Control

2

Most points are close to the line thus the assumption of normality is satisfied for the group A.

1 0 -1 -2 2

4

6

8

10

12

Observed Value Both Kolmogorov-Smirnov and Shapiro-Wilk test give p-value < ( = 0.05), thus normality assumption is not satisfied for Group B. Need to check Normal Probability plot for group B.

Normal Q-Q Plot of Bullet Point Reasoning Expected Normal

for groupBullet= Group B - Experiment

2

The graph shows curvature thus the assumption of normality is not satisfied for the group B.

1 0 -1

0

3

6

9

Observed Value

12

15

Since for group B normality is not satisfied, cannot use t-test. Will go with nonparametric tests. Need to use Mann-

De Ranks scriptives Groups for Bullet Groups for Bullet N Point Reasoning Bullet Point Reasoning Group A - Control Mean Bullet Point Reasoning Point GroupReasoning A - Control 21 5% Trimmed Mean Group B - Experiment 24 Median Total 45 Variance Std. Deviation Minimum Maximum Range Group B - Experiment Mean

Std. Error Mean Rank Statistic Sum of Ranks 6.9524 .55410 26.55 557.50 6.8915 19.90 477.50 7.0000

5% Trimmed Mean Median Variance Std. Deviation Minimum Maximum Range

6.448 2.53922 3.00 12.00 9.00 5.7917 4.5845 6.9988 5.4722 5.0000 8.172 2.85869 3.00 15.00 12.00

.58353

Te st Statisticsa

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed)

Bullet Point Reasoning 177.500 477.500 -1.733 .083

a. Grouping Variable: Groups for Bullet Point Reasoning

According to above table, Mann-Whitney test value = -1.733, P-value = 0.083/2 = 0.0415 (need to divide the P-value by 2 as we have one-tailed test where as SPSS gives you 2tailed test P-value. Since (P-value = 0.0415) < ( = 0.05), null hypothesis is rejected.

Conclusion: At 5% level, Group B does have less bullet points than group A.

Comparison Question #2 for Pre-test: On a scale from 1 to 5, where 1 represents not knowledgeable at all and 5 represents extremely knowledgeable, how knowledgeable are you with this topic? Null: For Q2 answers do not differ significantly for group A and B. Alternative: For Q2 answers do differ significantly for group A and B. – claim Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30. Te sts of Normality a

Pretest Q. 2

Groups for Pretest Group A - Control Group B - Experiment

Kolmogorov-Smirnov Statistic df Sig. .258 21 .001 .276 24 .000

Statistic .873 .741

Shapiro-Wilk df 21 24

Sig. .011 .000

a. Lilliefors Significance Correction

Both Kolmogorov-Smirnov and Shapiro-Wilk tests give p-value < ( = 0.05), thus normality assumption is not satisfied for both Groups. Need to check Normal Probability plots for both groups.

Normal Q-Q Plot of Pretest Q. 2 Expected Expected Normal Normal

groupPre= Group Control NormalforQ-Q Plot ofA -Pretest Q. 2

1.5

for groupPre= Group B - Experiment

1.0 2.0 0.5 1.5 0.0 1.0 -0.5 0.5 -1.0 0.0 -1.5 -0.5 -1.0

Most points are close to the line except one thus the assumption of normality is satisfied for the group B. 1.0 1

1.5

2.0

2.5

3.0

Observed Value4 3

2

Observed Value

3.5

4.0 5

Most points are close to the line thus the assumption of normality is satisfied for the group A.

42

Pretest Q. 2

5.00

Box plot shows outlier for group B. Even with the presence of outliers, normality is satisfied. Also this value is important for the analysis. Thus the decision is not to remove the

4.00

3.00

2.00

1.00 Group A - Control

Group B - Experiment

Groups for Pretest Group Statistics

Pretest Q. 2

Groups for Pretest Indepe ndeNnt Sample s TeMean Std. Deviation st Group A - Control 21 2.5238 .98077 Levene's Test for Equality of Variances t-test for Equality of Means Group B - Experiment 24 1.7500 .98907 F

Pretest Q. 2

Equal variances assumed Equal variances not assumed

.223

Sig. .639

Interval of the Difference Lower Upper

Sig. (2-tailed)

Mean Difference

2.629

43

.012

.77381

.29439

.18012

1.36750

2.630

42.303

.012

.77381

.29422

.18017

1.36745

t

df

Std. Error Difference

Std. Error Mean .21402 .20189 95% Confidence

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.639) > ( = 0.05), thus assumption of equal variances is satisfied. Inde pe nde nt Sample s Te st Levene's Test for Equality of Variances F Sig. Pretest Q. 2

Equal variances assumed Equal variances not assumed

.223

.639

t-test for Equality of Means t df Sig. (2-tailed) 2.629

43

.012

2.630

42.303

.012

According to above table, t-test value = 2.629, P-value = 0.012. Since (P-value = 0.012) < ( = 0.05), null hypothesis is rejected. Conclusion: At 5% level, Q2 answers do differ significantly for group A and B. Comparison Question #3 for Pre-test: On a scale from 1 to 5, where 1 represents not interested at all and 5 represents extremely interested, how interested are you in learning about analytic methodologies? Null: For Q3 answers do not differ significantly for group A and B. Alternative: For Q3 answers do differ significantly for group A and B. – claim Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

Te sts of Normality a

Pretest Q. 3

Groups for Pretest Group A - Control Group B - Experiment

Kolmogorov-Smirnov Statistic df Sig. .218 21 .010 .277 24 .000

Statistic .904 .810

Shapiro-Wilk df 21 24

Sig. .041 .000

a. Lilliefors Significance Correction

Both Kolmogorov-Smirnov and Shapiro-Wilk tests give p-value < ( = 0.05), thus normality assumption is not satisfied for both Groups. Need to check Normal Probability plots for both groups.

Most points are close to the line thus the assumption of normality is satisfied for the group A.

Normal Q-Q Plot of Pretest Q. 3 Expected Normal

for groupPre= Group A - Control

1 0 -1 -2 1

2

3

Observed Value

4

5

Most points are close to the line thus the assumption of normality is satisfied for the group B.

Normal Q-Q Plot of Pretest Q. 3 Expected Normal

for groupPre= Group B - Experiment

2.0 1.5 1.0 0.5 0.0 -0.5 -1.0 2.0

2.5

3.0

3.5

4.0

Observed Value

4.5

5.0

Box plot shows outlier for group A. Even with the presence of outliers, normality is satisfied. Also this value is important for the analysis. Thus the decision is not to remove the outlier.

Pretest Q. 3

5.00

4.00

3.00

2.00

1.00

3 Group A - Control

Group B - Experiment

Groups for Pretest Group Statistics Std. Error Inde pe nde nt s Te st Groups for Pretest N Sample Mean Std. Deviation Mean Pretest Q. 3 Group A - Control 21 3.4762 1.03049 .22487 Levene's Test for Group B - Experiment 24 2.8333 t-test for Equality .91683 of Means .18715 Equality of Variances F Sig. t df Sig. (2-tailed) Pretest Q. 3 Equal variances .201 .656 2.215 43 .032 assumed Equal variances 2.197 40.433 .034 not assumed

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.656) > ( = 0.05), thus assumption of equal variances is satisfied. Inde pe nde nt Sample s Te st Levene's Test for Equality of Variances F Sig. Pretest Q. 3

Equal variances assumed Equal variances not assumed

.201

t

.656

t-test for Equality of Means df Sig. (2-tailed)

2.215

43

.032

2.197

40.433

.034

According to above table, t-test value = 2.215, P-value = 0.032. Since (P-value = 0.032) < ( = 0.05), null hypothesis is rejected. Conclusion: At 5% level, Q3 answers do differ significantly for group A and B. Comparison Question # 1 for Pre-test: On a scale from 1 to 5, where 1 represents not interested at all and 5 represents extremely interested, how interested are you in participating in this experiment? Null: For Q1 answers do not differ significantly for group A and B. Alternative: For Q1 answers do differ significantly for group A and B. – claim Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30. Tests of Normality a

Pretest Q. 1

Groups for Pretest Group A - Control Group B - Experiment

Kolmogorov-Smirnov Statistic df Sig. .227 21 .006 .336 24 .000

Statistic .884 .820

Shapiro-Wilk df 21 24

Sig. .018 .001

a. Lilliefors Significance Correction

Both Kolmogorov-Smirnov and Shapiro-Wilk tests give p-value < ( = 0.05), thus normality assumption is not satisfied for both Groups. Need to check Normal Probability plots for both groups.

Most points are close to the line thus the assumption of normality is satisfied for the group A.

Normal Q-Q Plot of Pretest Q. 1 Expected Normal

for groupPre= Group A - Control

1.5 1.0 0.5 0.0 -0.5 -1.0 -1.5 2.0

2.5

3.0

3.5

4.0

Observed Value

4.5

5.0

Most points are close to the line thus the assumption of normality is satisfied for the group B.

Normal Q-Q Plot of Pretest Q. 1 Expected Normal

for groupPre= Group B - Experiment

2 1 0 -1 -2 2.0

2.5

3.0

3.5

4.0

Observed Value

4.5

5.0

5.00

Pretest Q. 1

4.50

Box plot does not show outliers for either group. This is a desired result for normality.

4.00 3.50 3.00 2.50 2.00 Group A - Control

Group B - Experiment

Groups for Pretest Group Statistics Std. Error Inde pe nde nt s Te st Groups for Pretest N Sample Mean Std. Deviation Mean Pretest Q. 1 Group A - Control 21 3.4762 .98077 .21402 Levene's Test for Group B - Experiment 24 3.3333 .76139 .15542 Equality of Variances t-test for Equality of Means F Sig. t df Sig. (2-tailed) Pretest Q. 1 Equal variances 2.674 .109 .549 43 .586 assumed Equal variances .540 37.570 .592 not assumed

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.109) > ( = 0.05), thus assumption of equal variances is satisfied. Inde pe nde nt Sample s Te st Levene's Test for Equality of Variances F Sig. Pretest Q. 1

Equal variances assumed Equal variances not assumed

2.674

.109

t-test for Equality of Means t df Sig. (2-tailed) .549

43

.586

.540

37.570

.592

According to above table, t-test value = 0.549, P-value = 0.586. Since (P-value = 0.586) > ( = 0.05), null hypothesis is not rejected. Conclusion: At 5% level, Q1 answers do not differ significantly for group A and B. Comparison Question #3 for Post-test: On a scale from 1 to 5, where 1 represents not at all and 5 represents extremely frequently, how frequently do you use structured analytic methods in solving intelligence related problems? Null: For Q3 answers do not differ significantly for group A and B. Alternative: For Q3 answers do differ significantly for group A and B. – claim Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30. Te sts of Normality a

Posttest Q. 3

Groups for Pretest Group A - Control Group B - Experiment

Kolmogorov-Smirnov Statistic df Sig. .277 21 .000 .297 24 .000

Statistic .860 .830

Shapiro-Wilk df 21 24

Sig. .006 .001

a. Lilliefors Significance Correction

Both Kolmogorov-Smirnov and Shapiro-Wilk tests give p-value < ( = 0.05), thus normality assumption is not satisfied for both Groups. Need to check Normal Probability plots for both groups.

Normal Q-Q Plot of Posttest Q. 3 Expected Normal

for groupPre= Group A - Control

1.5 1.0

Most points are close to the line thus the assumption of normality is satisfied for the

0.5 0.0 -0.5 -1.0 -1.5 2.0

2.5

3.0

3.5

4.0

Observed Value

4.5

5.0

Most points are close to the line thus the assumption of normality is satisfied for the group B.

Normal Q-Q Plot of Posttest Q. 3 Expected Normal

for groupPre= Group B - Experiment

2 1 0 -1 -2 2.0

2.5

3.0

3.5

4.0

Observed Value

4.5

5.0

Box plot does not show outliers for either group. This is a desired result for normality.

5.00

Posttest Q. 3

4.50 4.00 3.50 3.00 2.50 2.00 Group A - Control

Group B - Experiment

Groups for Pretest Group Statistics

Posttest Q. 3

Groups for Pretest Group A - Control Group B - Experiment

N 21 24

Mean 3.1429 3.5000

Std. Deviation .91026 .72232

Std. Error Mean .19863 .14744

Indepe nde nt Sample s Te st Levene's Test for Equality of Variances F Sig. Posttest Q. 3

Equal variances assumed Equal variances not assumed

.155

.696

t-test for Equality of Means t df Sig. (2-tailed) -1.466

43

.150

-1.444

38.063

.157

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.696) > ( = 0.05), thus assumption of equal variances is satisfied.

Indepe nde nt Sample s Te st Levene's Test for Equality of Variances F Sig. Posttest Q. 3

Equal variances assumed Equal variances not assumed

.155

t

.696

t-test for Equality of Means df Sig. (2-tailed)

-1.466

43

.150

-1.444

38.063

.157

According to above table, t-test value = -1.466, P-value = 0.150. Since (P-value = 0.150) > ( = 0.05), null hypothesis is not rejected. Conclusion: At 5% level, Q3 answers do not differ significantly for group A and B. Comparison Question #4 for Post-test: On a scale from 1 to 5, where 1 represents not at all and 5 represents a significant improvement, how much do you feel analytic methods improve intelligence analysis? Null: For Q4 answers do not differ significantly for group A and B. Alternative: For Q4 answers do differ significantly for group A and B. – claim Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30. Te sts of Normality a

Posttest Q. 4

Groups for Pretest Group A - Control Group B - Experiment

a. Lilliefors Significance Correction

Kolmogorov-Smirnov Statistic df Sig. .241 20 .003 .299 24 .000

Statistic .862 .812

Shapiro-Wilk df 20 24

Sig. .009 .000

Both Kolmogorov-Smirnov and Shapiro-Wilk tests give p-value < ( = 0.05), thus normality assumption is not satisfied for both Groups. Need to check Normal Probability plots for both groups.

Most points are close to the line thus the assumption of normality is satisfied for the group A.

Normal Q-Q Plot of Posttest Q. 4 Expected Normal

for groupPre= Group A - Control

1.0 0.5 0.0 -0.5 -1.0 -1.5 2.0

2.5

3.0

3.5

4.0

Observed Value

4.5

5.0

Normal Q-Q Plot of Posttest Q. 4 Expected Normal

for groupPre= Group B - Experiment

2

Most points are close to the line thus the assumption of normality is satisfied for the group B.

1 0 -1 -2 2.0

2.5

3.0

3.5

4.0

4.5

5.0

Observed Value

Group Statistics

Posttest Q. 4

Groups for Pretest Group A - Control Group B - Experiment

N 20 24

Mean 3.9000 3.5417

5.00

Std. Error Mean .21643 .13431

Box plot does not show outliers for either group. This is a desired result for normality.

4.50

Posttest Q. 4

Std. Deviation .96791 .65801

4.00 3.50 3.00 2.50 2.00 Group A - Control

Group B - Experiment

Groups for Pretest

Indepe nde nt Sample s Te st Levene's Test for Equality of Variances F Sig. Posttest Q. 4

Equal variances assumed Equal variances not assumed

1.345

t

.253

t-test for Equality of Means df Sig. (2-tailed)

1.456

42

.153

1.407

32.474

.169

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.253) > ( = 0.05), thus assumption of equal variances is satisfied. Indepe nde nt Sample s Te st Levene's Test for Equality of Variances F Sig. Posttest Q. 4

Equal variances assumed Equal variances not assumed

1.345

t

.253

t-test for Equality of Means df Sig. (2-tailed)

1.456

42

.153

1.407

32.474

.169

According to above table, t-test value = 1.456, P-value = 0.153. Since (P-value = 0.153) > ( = 0.05), null hypothesis is not rejected. Conclusion: At 5% level, Q4 answers do not differ significantly for group A and B. Comparison Question #6 for Post-test: On a scale from 1 to 5, where 1 represents not enough time at all and 5 represents more than adequate amount of time, do you feel you had enough time to complete this process? Null: For Q6 answers do not differ significantly for group A and B. Alternative: For Q6 answers do differ significantly for group A and B. – claim Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

Te sts of Normality a

Posttest Q. 6

Groups for Pretest Group A - Control Group B - Experiment

a. Lilliefors Significance Correction

Kolmogorov-Smirnov Statistic df Sig. .349 21 .000 .510 24 .000

Statistic .727 .401

Shapiro-Wilk df 21 24

Sig. .000 .000

Both Kolmogorov-Smirnov and Shapiro-Wilk tests give p-value < ( = 0.05), thus normality assumption is not satisfied for both Groups. Need to check Normal Probability plots for both groups.

Most points are close to the line thus the assumption of normality is satisfied for the group A.

Normal Q-Q Plot of Posttest Q. 6 Expected Normal

for groupPre= Group A - Control

0.6 0.3 0.0 -0.3 -0.6 -0.9 -1.2 -1.5 3.0

3.5

4.0

4.5

Observed Value

5.0

Most points are not close to the line thus the assumption of normality is not satisfied for the group B. We cannot use the ttest. Need to use

Normal Q-Q Plot of Posttest Q. 6 Expected Normal

for groupPre= Group B - Experiment

0.0 -0.5 -1.0 -1.5

3.0

3.5

4.0

4.5

Observed Value

5.0

De scriptiv e s Posttest Q. 6

Groups for Pretest Group A - Control

Statistic 4.4286 .557 .74642 3.00 5.00 4.8333 .232 .48154 3.00 5.00

Mean Variance Std. Deviation Minimum Maximum Group B - Experiment Mean Variance Std. Deviation Minimum Maximum

Std. Error .16288

.09829

Ranks Posttest Q. 6

Groups for Pretest Group A - Control Group B - Experiment Total

N 21 24 45

Mean Rank 19.36 26.19

Sum of Ranks 406.50 628.50

Test Statisticsa Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed)

Posttest Q. 6 175.500 406.500 -2.248 .025

a. Grouping Variable: Groups for Pretest

According to above table, Mann-Whitney test value = 175.5, P-value = 0.025. Since (P-value = 0.025) < ( = 0.05), null hypothesis is rejected. Conclusion: At 5% level, Q6 answers do differ significantly for group A and B. Comparison Question #7 for Post-test: On a scale from 1 to 5, where 1 represents no enjoyment at all and 5 represents thorough enjoyment, how much did you enjoy this process? Null: For Q7 answers do not differ significantly for group A and B.

Alternative: For Q7 answers do differ significantly for group A and B. – claim Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30. Tests of Normality a

Kolmogorov-Smirnov Groups for Pretest Statistic df Sig. Posttest Q. 7 Group A - Control .223 20 .010 Group B - Experiment .261 24 .000

Shapiro-Wilk Statistic df .809 20 .872 24

Sig. .001 .006

a. Lilliefors Significance Correction

Both Kolmogorov-Smirnov and Shapiro-Wilk tests give p-value < ( = 0.05), thus normality assumption is not satisfied for both Groups. Need to check Normal Probability plots for both groups.

Normal Q-Q Plot of Posttest Q. 7 Expected Normal

for groupPre= Group A - Control

1.0

Most points are close to the line thus the assumption of normality is satisfied for the group A.

0.5 0.0 -0.5 -1.0 3.0

3.5

4.0

4.5

Observed Value

5.0

Normal Q-Q Plot of Posttest Q. 7 Most points are close to the line thus the assumption of normality is satisfied for the group B.

Expected Normal

for groupPre= Group B - Experiment

2 1 0 -1 -2 1

2

3

4

5

Observed Value Group Statistics

Posttest Q. 7

Groups for Pretest Group A - Control Group B - Experiment

N 20 24

Mean 3.9000 3.4583

Std. Deviation .78807 .93153

Std. Error Mean .17622 .19015

Indepe nde nt Sample s Te st Levene's Test for Equality of Variances F Sig. Posttest Q. 7

Equal variances assumed Equal variances not assumed

.685

.412

t

t-test for Equality of Means df Sig. (2-tailed)

1.678

42

.101

1.704

41.984

.096

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.412) > ( = 0.05), thus assumption of equal variances is satisfied. Indepe nde nt Sample s Te st Levene's Test for Equality of Variances F Sig. Posttest Q. 7

Equal variances assumed Equal variances not assumed

.685

.412

t

t-test for Equality of Means df Sig. (2-tailed)

1.678

42

.101

1.704

41.984

.096

According to above table, t-test value = 1.678, P-value = 0.101. Since (P-value = 0.101) > ( = 0.05), null hypothesis is not rejected. Conclusion: At 5% level, Q7 answers do not differ significantly for group A and B.