Explicit Conceptual Models Synthesizing Divergent And Convergent Thinking

   

EXPLICIT CONCEPTUAL MODELS: SYNTHESIZING DIVERGENT AND CONVERGENT THINKING

SHANNON L. FERRUCCI

A Thesis Submitted to the Faculty of Mercyhurst College In Partial Fulfillment of the Requirements for The Degree of MASTER OF SCIENCE IN APPLIED INTELLIGENCE

DEPARTMENT OF INTELLIGENCE STUDIES MERCYHURST COLLEGE ERIE, PENNSYLVANIA MAY 2009    

   

DEPARTMENT OF INTELLIGENCE STUDIES MERCYHURST COLLEGE ERIE, PENNSYLVANIA

EXPLICIT CONCEPTUAL MODELS: SYNTHESIZING DIVERGENT AND CONVERGENT THINKING A Thesis Submitted to the Faculty of Mercyhurst College In Partial Fulfillment of the Requirements for The Degree of MASTER OF SCIENCE IN APPLIED INTELLIGENCE

Submitted By: SHANNON L. FERRUCCI

Certificate of Approval:

___________________________________ Kristan J. Wheaton Assistant Professor Department of Intelligence Studies

___________________________________ William J. Welch Instructor Department of Intelligence Studies

___________________________________ Phillip J. Belfiore Vice President Office of Academic Affairs

May 2009    

   

Copyright © 2009 by Shannon L. Ferrucci All rights reserved.

iii   

   

ACKNOWLEDGMENTS

I would like to thank Kristan J. Wheaton, my thesis advisor and primary reader, for his continued guidance and encouragement throughout the course of this work. I would also like to thank Professor Hemangini Deshmukh for her patience and assistance with the statistical analysis of this work, it was greatly appreciated.

iv   

   

ABSTRACT OF THE THESIS

Explicit Conceptual Models: Synthesizing Divergent And Convergent Thinking By Shannon L. Ferrucci Master of Science in Applied Intelligence Mercyhurst College, 2009 Professor Kristan J. Wheaton, Chair

Explicit conceptual modeling (ECM) within intelligence analysis is a topic on which very little specific research has thus far been done. However, when considering the complexity and depth of most intelligence requirements it becomes evident that consideration of this topic is both crucial and long overdue. This thesis examines what little literature exists on conceptual modeling within intelligence analysis, in addition to discussing relevant studies from other fields that help to shed light on the need for, and value of, incorporating this technique into intelligence analysis.

After examining the

relevant literature, an experiment was conducted to test the hypothesis that intelligence analysts who engage in ECM will generate better analytic products, as evaluated by thoroughness of process and accuracy of product, than analysts who do not. However, despite a wealth of literature strongly suggesting that ECM will improve analysis the results of this study’s experiment did not support that notion. The author ends by drawing conclusions from the experimental data highlighting the notion that ECM requires a combination of robust divergent and convergent thinking techniques to be successful. v   

   

TABLE OF CONTENTS Page COPYRIGHT PAGE………………………………………………………………...

iii

ACKNOWLEDGEMENTS………………………………………………………….

iv

ABSTRACT………………………………………………………………………....

v

TABLE OF CONTENTS……………………………………………………………

vi

LIST OF FIGURES………………………………………………………………….

ix

CHAPTER 1

2

3

INTRODUCTION……………………………………………………

1

Conceptual Models………………………………………………….. Explicit Modeling and Intelligence Analysis………………………..

2 3

LITERATURE REVIEW…………………………………………....

6

Constructivist Roots…………………………………………………. Mental Models and Intelligence……………………………………... Memory Limitations………………………………………………… Group Intellect………………………………………………………. Combating Groupthink……………………………………………… Related Mapping Disciplines………………………………………... Mind Maps…………………………………………………………... Concept Maps……………………………………………………….. Technology Aids…………………………………………………….. Learning Styles……………………………………………………… Hypotheses…………………………………………………………...

6 7 8 9 10 11 13 14 15 17 17

METHODOLOGY…………………………………………………...

18

Research Design……………………………………………………... Subjects……………………………………………………………… Preliminaries………………………………………………………… Control Group: Day 1……………………………………………….. Experimental Group: Day 1…………………………………………. Bubbl.us……………………………………………………………... Control Group: Day 2……………………………………………….. Experimental Group: Day 2………………………………………….

18 18 22 22 24 26 27 28

vi   

   

Data Analysis Procedures……………………………………………

29

RESULTS…....………………………………………………………

30

Significance Testing…………………………………………………. Pre- and Post-Questionnaire Results………………………………… Process – Conceptual Model Findings………………………………. Process – Logic/Quality of Supporting Evidence Findings…………. Product – Forecasting Findings……………………………………... Quality Of Supporting Evidence Vs. Forecasting Accuracy………... Product – Source Reliability and Analytic Confidence Findings……

30 31 36 40 41 42 46

CONCLUSIONS…………………………………………………….

48

Excessive Possibilities Confuse……………………………………... Importance of Convergence…………………………………………. Final Thoughts………………………………………………………. Future Research……………………………………………………...

49 51 52 54

BIBLIOGRAPHY…………………………………………………………………...

55

APPENDICES……………………………………………………………………….

57

Appendix 1: Experiment Sign-Up Form……………………………..

58

Appendix 2: IRB Research Proposal………………………………...

59

Appendix 3: Control Group Consent Form………………………….

66

Appendix 4: Experimental Group Consent Form……………………

67

Appendix 5: Research Question……………………………………...

68

Appendix 6: Important Supporting Information……………………..

69

Appendix 7: Experiment Answer Sheet……………………………...

70

Appendix 8: Control Group Expectation Sheet……………………...

71

Appendix 9: Experimental Group Expectation Sheet………………..

72

Appendix 10: Pre-Experiment Questionnaire………………………..

73

Appendix 11: Contact Information…………………………………..

74

Appendix 12: Conceptual Modeling Lecture………………………...

75

4

5

vii   

   

Appendix 13: Bubbl.us Instruction Sheet For Experimental Group…

77

Appendix 14: Bubbl.us Instruction Sheet For Control Group……….

79

Appendix 15: Structured Conceptual Modeling Exercise…………...

80

Appendix 16: Control Group Post-Experiment Questionnaire………

81

Appendix 17: Experimental Group Post-Experiment Questionnaire...

83

Appendix 18: Control Group Debriefing Sheet……………………...

86

Appendix 19: Experimental Group Debriefing Sheet………………..

87

Appendix 20: Significance Testing Results………………………….

88

 

viii   

   

LIST OF FIGURES

Page Figure 2.1

Example Conceptual Model

12

Figure 2.2

Example Mind Map

14

Figure 2.3

Example Concept Map

15

Figure 3.1

Subject Education Level

21

Figure 3.2

Subject Education Level By Group

21

Figure 3.3

Original Control Group Vs. Actual Control Group

23

Figure 3.4

Original Experimental Group Vs. Actual Experimental Group

25

Figure 3.5

Bubbl.us Screenshot

27

Figure 4.1

Pre- Vs. Post Experiment: Time Dedicated To Experiment

32

Figure 4.2

Control Vs. Experimental: Learning Style

34

Figure 4.3

Bubbl.us Screenshot With Concept/Connection Labels

37

Figure 4.4

Experimental Group: Average Concepts And Connections

38

Figure 4.5

Control Vs. Experimental: Average Concepts And Connections

38

Figure 4.6

Control Group Conceptual Model Example

39

Figure 4.7

Experimental Group Conceptual Model Example

40

Figure 4.8

Control Vs. Experimental: Accuracy Of Forecasting

42

Figure 4.9

Forecasting Accuracy: Top Vs. Bottom Half Process Rankings

44

Figure 4.10

Control Forecasting Accuracy By Process Ranking

44

Figure 4.11

Experimental Forecasting Accuracy By Process Ranking

45

Figure 4.12

Control Vs. Experimental: Source Reliability

46

Figure 4.13

Control Vs. Experimental: Analytic Confidence

47

ix   

1   

CHAPTER I: INTRODUCTION

In an introduction to The Jefferson Bible: The Life and Morals of Jesus of Nazareth, Forrest Church, Minister of Public Theology at the Unitarian Church of All Souls in New York City, tells the reader of an historical offer made by Thomas Jefferson to Congress.1 Jefferson’s offer consisted of selling his personal library to replace the volumes in the Library of Congress burned by the British during the War of 1812. While some might find the most interesting aspect of Jefferson’s proposal to be the reaction it elicited from members of Congress who were insulted by the specific makeup of the collection, Church places importance on a different aspect of the story altogether. According to Church, “Jefferson’s scheme of classification as formulated in the catalog that he submitted to Congress” was more telling than anything else.2 Furthering a method established by Francis Bacon in 1605, Jefferson categorized his books by “the process of mind employed on them.”3 Therefore, books having to do with philosophy were classified under reason, history books could be found under the label of memory and books focused on fine art could be located under a section entitled imagination.4 However, Jefferson did not stop there. Under each of the overarching categories mentioned above was a variety of intricate subdivisions that further organized Jefferson’s collection. Based on Church’s retrospective examination of the library, it                                                              ***This research partially funded by the Mercyhurst College Academic Enrichment Fund*** 1

Forrest Church, Introduction to The Jefferson Bible: The Life and Morals of Jesus of Nazareth, by Thomas Jefferson (Boston: Beacon Press, 1989), 1. 2 Church, The Jefferson Bible, 2. 3 Ibid. 4 Ibid. 

   

2   

appears that the true value of Jefferson’s system of categorization lies in its ability to provide a glimpse into the inner thoughts and beliefs of Jefferson himself. Due to the detail with which the library was constructed, Church was able to surmise Jefferson’s viewpoint on a variety of issues, particularly religion, based on the placement and relationship amongst books within Jefferson’s hierarchical structure.

Conceptual Models What Bacon in the early 1600s, and Jefferson in the early 1800s, were essentially doing through their systems of classification was attempting to make their individual mental models of the world around them explicit. Surprisingly enough, not only do the likes of Bacon and Jefferson develop such mental models, but each and every one of us carries out this same exercise on a variety of different levels numerous times per day. For example, we construct mental models of the route we take on the way to the grocery store and of our routine for getting ready in the morning. This implicit modeling is extremely interesting in the context of intelligence analysis, when considering that we also build models when faced with questions. Whether the issue is a simple one, such as what to do on our day off, or as complex as an intelligence requirement set forth by a decision maker, the human mind automatically attempts to model the question and arrive at possible preliminary answers. Oftentimes in doing this, we are able to recognize not only what we currently know about a given situation, but also what we think we need to know in order to arrive at a comprehensive answer. According to an article from the journal of Information Research by Kalervo Jarvelin, Academy Professor in the Department of Information Studies at the University    

3   

of Tampere in Finland, and T.D. Wilson, Professor Emeritus at the University of Sheffield in the United Kingdom: All research has an underlying model of the phenomena it investigates, be it tacitly assumed or explicit. Such models called conceptual frameworks, or conceptual models… may and should map reality, guide research and systematize knowledge. A conceptual model provides a working strategy, a scheme containing general, major concepts and their interrelations. It orients research towards specific sets of research questions.5 Obviously, the more complex the question, the more intricate the subsequent model tends to be. This is especially true of requirements posed to intelligence analysts, which often entail the understanding of multifaceted relationships between people, states, organizations, industries, etc. Therefore, the odds of any analyst being able to develop a complete model of an intelligence requirement on their first try are very slim. More often than not, analysts are able to fill in pieces of their model with information they already know, but are forced to fill in the rest with topics they recognize they need to understand more about.

Explicit Modeling and Intelligence Analysis The complexity of intelligence requirements leads to the core purpose of this study: determining the value of making these conceptual models explicit within the analytic process. In considering the scope of most intelligence requirements, it becomes obvious that the vast majority of related conceptual models will become too complex to be held in an individual’s memory, and hence would benefit from being made explicit. This is especially true when considering that conceptual models are not static, but actually quite amorphous, constantly evolving and adapting to new information and                                                              5

Kalervo Jarvelin and T.D. Wilson, “On Conceptual Models for Information Seeking and Retrieval Research,” Information Research 9, no. 1 (2003), http://informationr.net/ir/9-1/paper163.html (accessed January 15, 2009).

   

4   

improving knowledge on a specific topic. Consequently, at present, an exploration of explicit conceptual modeling’s (ECM) place within the field of intelligence is both crucial and long overdue. Within the intelligence community, this topic is primarily of interest to intelligence professionals holding managerial positions, intelligence educators, and individual intelligence analysts (both students and practitioners). For these groups the incorporation of ECM into the analytic process would likely be beneficial on a variety of fronts.

First, explicit modeling may increase efficiency in the analyst’s collection

process, in addition to aiding in the identification of knowledge gaps.

Second, by

organizing ideas and information in a simplistic, straightforward and graphic way, managers at the head of small analytic teams might more easily grasp what needs to be done, the best method for doing it, and the most efficient way to originally task project analysts. In addition, after initial areas of responsibility are assigned to each analyst it is likely that managers might find it easier to supervise analysts, due to the organizational foundation provided by the model. ECM may also be useful in helping analysts to assess their level of analytic confidence in the estimate produced. In addition, analysts may share, compare and discuss models amongst themselves and with other professionals. Finally, ECM would likely be useful for after the fact review and in providing a solid starting point for any related questions posed to an analyst in the future. However, while these are only some of the potential benefits stemming from the incorporation of ECM into the analytic process, this thesis will show that obtaining the abovementioned results is not easy.

   

5   

Furthermore, this study will call into question conventional wisdom regarding what makes a good explicit conceptual model. Taken as a whole, this thesis will argue that despite the relative dearth of studies focused specifically on conceptual modeling within the field of intelligence, literature and examples from other fields will shed light on the need for, and value of, incorporating this technique into intelligence analysis. As one study on conceptual modeling within the field of intelligence has stated, “Conceptual models both fix the mesh of the nets that the analyst drags through the material in order to explain a particular action or decision and direct him to cast his net in select ponds, at certain depths, in order to catch the fish he is after.”6

                                                             6

Graham T. Allison. “Conceptual Models and the Cuban Missile Crisis,” in The Sociology of Organizations: Classic, Contemporary and Critical Readings, ed. Michael Jeremy Handel, (Thousand Oaks: SAGE, 2003), 185. 

   

6   

CHAPTER II: LITERATURE REVIEW

The following is a review of literature relevant to conceptual modeling and intelligence analysis. The section begins with a discussion of constructivism’s bearing on conceptual modeling and an illustration of the importance of a given analyst’s mental model to the analysis produced.

Next is a discussion of the relationships between

intelligence requirements, human memory limitations, group intelligence and groupthink and the support they provide for the need to make our mental models explicit. This is followed by a segment on the various methods for making these models explicit, in particular mind maps, concept maps and explicit conceptual models. The benefit of using technological aids to assist in the creation of explicit conceptual models is also mentioned, as is the notion that the utility of ECM may be affected by varying individual learning styles. Finally, this section concludes with the author’s original hypotheses for this study.

Constructivist Roots The array of concepts and relationships between them, illustrated through conceptual models, is closely related to constructivist notions of knowledge formation. In particular, the work of the famed Swiss psychologist Jean Piaget is relevant, as his viewpoint holds that individuals continually construct cognitive models to make sense of the world around them by organizing and connecting their ideas, observations and experiences.7

Additionally, according to Piaget these cognitive models are always

                                                             7

John W. Santrock, Adolescence, 8th ed.(New York: McGraw-Hill, 2001), 102.

   

7   

evolving to include new information that aids individuals in furthering their understanding of the world around them.8 Piaget called the constructs for assembling these models schema or “a concept or framework that exists in the individuals’ mind to organize and interpret information.”9

While a discussion of the pros and cons of

constructivism theory is outside the reach of this paper, using this theory to aid in thinking about the development of models within our minds is actually quite useful.

Mental Models and Intelligence When taking constructivist theory and applying it to the field of intelligence analysis it becomes clear that each analyst’s cognitive, or mental model, is uniquely shaped by the context and purpose of the requirement posed to them, along with the summation of that individual’s prior experiences, schooling, cultural values, professional position and organizational standards.10

As stated in “Intelligence Analysis: Once

Again,” by Charles A. Mangio, of Shim Enterprise, Inc., and Bonnie J. Wilkinson of the Air Force Research Laboratory: Given the importance of the mental model in influencing and shaping the analysis (i.e., from problem exploration and formulation, to purpose refinement, through data acquisition and evaluation, and ultimately determining meaning and making judgments), it is not surprising how it influences the discussion of intelligence analysis.11 However, despite the importance of a well-defined and thorough mental model to an analyst’s subsequent analysis, it is rarely touched upon in intelligence literature.

                                                             8

Ibid. Ibid.  10 Charles A. Mangio and Bonnie J. Wilkinson, “Intelligence Analysis: Once Again” (paper presented at the annual international meeting of the International Studies Association, San Francisco, California, 26 March, 2008): 8. 11 Ibid. 9

   

8   

Memory Limitations Despite the lack of attention given to mental models in the intelligence community, the formulation of these models can significantly impact the process of intelligence analysis. However, eventually it becomes obvious that as the amount of concepts and relationships included in the analyst’s mental model continues to grow, it becomes difficult to store all of that information accurately in working memory. In “The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information,” George A. Miller, Professor Emeritus of Psychology at Princeton University, argues that we only have the ability to hold 7 things (plus or minus 2) in our mind at a given time without making mistakes in differentiation.12 Having said that, there are methods individuals can use to help them surpass these known limits, as well as there are a variety of exceptions to the rule in the first place. In Psychology of Intelligence Analysis, Richards Heuer, prior staff officer and contractor of the CIA for almost 45 years, discusses one such method for aiding analysts in exceeding memory constraints. Essentially, what Heuer describes is none other than making an individual’s mental model explicit: “The recommended technique for coping with this limitation of working memory is called externalizing the problem—getting it out of one’s head and down on paper in some simplified form that shows the main elements of the problem and how they relate to each other…Breaking down a problem into its component parts and then preparing a simple ‘model’ that shows how the parts relate to the whole. When working on a small part of the problem, the model keeps one from losing sight of the whole.”13 Regardless of the specific number of concepts that our working memory can handle, the idea that there is an upper limit is quite evident, as well as is the fact that most                                                              12

George A. Miller, “The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information,” Psychological Review 63 (1956): 81-97.  13  Richards J. Heuer, Psychology of Intelligence Analysis (Center for the Study of Intelligence, 1999), 27. 

   

9   

intelligence requirements will easily exceed that maximum. Limits on working memory, although one of the major concerns considered in the argument for making mental models explicit, are only one of the factors behind the need for engaging in the process of ECM. Another reason strengthening the argument for explicit modeling stems from the inherent complexity in intelligence requirements, which oftentimes leads analysts to work in groups in order to tackle compound issues and topics.

Group Intellect People often criticize the collective judgment of groups as unreliable, viewing individual conclusions as being much more sensible and sound. In fact, groups are often viewed as bringing out the worst in individuals, resulting in illogical and foolish behavior. However, in The Wisdom of Crowds, James Surowiecki, a staff writer at The New Yorker, actually defends group decisions, noting that the four conditions distinguishing wise crowds are: Diversity of opinion (each person should have some private information, even if it is just an eccentric interpretation of the known facts), independence (people’s opinions are not determined by the opinions of those around them), decentralization (people are able to specialize and draw on local knowledge), and aggregation (some mechanism exists for turning private judgments into a collective decision. If a group satisfies those conditions, its judgment is likely to be accurate.14 According to Surowiecki, the collective intelligence of groups often far surpasses the individual intelligences of the people making up that group.15 However, group work does entail its own unique set of problems, one of which is groupthink.

                                                             14 15

James Surowiecki, The Wisdom of Crowds (New York: Anchor Books, 2004), 10. Surowiecki, Wisdom of Crowds, XIII.

   

10   

Combating Groupthink Most classrooms and professional environments are made up of a mix of individuals ranging from those inclined to chime in to discussions and offer opinions ad nauseam to those who shudder at the thought of speaking up. While there can of course be a wide variety of reasons certain individuals are hesitant to actively participate in classroom discussion or workplace meetings, a common fear is that their responses will somehow be inadequate, causing them to embarrass themselves in front of others. In McKeachie’s Teaching Tips, Wilbert J. McKeachie suggests that, “Asking students to take a couple of minutes to write out their initial answers to a question can help. If a student has already written an answer, the step to speaking is much less than answering when asked to respond immediately.”16 Essentially, the notion is that even the most timid will contribute when simply asked to read off what they have already written down. While McKeachie, Professor Emeritus of Psychology at the University of Michigan, speaks solely of students, this same concept applies to professionals. By asking all individuals to jot down answers to a proposed question, then focusing on each person in turn and having them voice those ideas out loud, equal involvement is fostered. No one is allowed to passively soak up the information being offered by others, while at the same time a select few individuals are prevented from dominating the discussion. This method of systematically focusing on each group member’s opinion also helps to combat instances of groupthink. In Groupthink: Psychological Studies of Policy Decisions and Fiascoes, Irving L. Janis defines groupthink as, “A mode of thinking that people engage in when they are deeply involved in a cohesive in-group, when the                                                              16

Wilbert J. McKeachie, McKeachie’s Teaching Tips (Boston: Houghton Mifflin Company, 2002), 42.

   

11   

member’s strivings for unanimity override their motivation to realistically appraise alternative courses of action.”17 According to Janis, Professor of Psychology at Yale University prior to his death in 1990, there are three main categories of groupthink: group overestimations of its power and morality, closed-mindedness and pressures toward group uniformity.18 In situations where groupthink prevails, teams of individuals often have trouble successfully completing the requirements placed upon them, subsequently failing to meet their goals. By having all group members write down their thoughts and then repeat those thoughts aloud, the phenomenon of individuals keeping quiet so as not to voice unpopular or contrasting views is limited. Furthermore, this method also limits having only a select few outspoken individuals’ perspectives heard and considered. Therefore, the process of ECM is likely beneficial not only in surpassing memory limitations but also in combating groupthink, one of the most common problems plaguing group work.

Related Mapping Disciplines While ECM is the method used for explicitly visualizing information and knowledge in this study, a variety of related mapping disciplines with similar functions do exist. Two in particular that warrant a brief discussion are mind maps and concept maps, both of which are highly analogous to conceptual modeling. For the purposes of this study however, conceptual models were found to be the most functional and userfriendly method for experiment participants to learn and understand in a short time period. Based on the fact that the methods of mind and concept mapping are both                                                              17

Irving L. Janis, Groupthink: Psychological Studies of Policy Decisions and Fiascoes (Boston: Houghton Mifflin Company, 1982), 9. 18 Janis, Groupthink, 174-175.

   

12   

essentially “coined” exercises, their creation involves following a set of predetermined criteria (see below mind map and concept mapping sections for further detail). ECM on the other hand, simply focuses on the visualization of concepts and their relationships without the added emphasis on the specific construction of the model, allowing individuals the maximum freedom possible to organize the model in whatever way was most helpful to them. As a result, the method and design for conceptual model construction used within this thesis has been operationalized from the relevant literature (see the methodology section for further detail regarding development of the models). Please refer to Figure 2.1 below, taken from Bubbl.us, for an illustration of a conceptual model created by experiment participants in this study. Figure 2.1

Of course, taking a somewhat abstract concept and translating that into a concrete and measurable product, is not without its problems. First, this author’s interpretation of the physical creation of the conceptual models may differ from the interpretation of others. Additionally, whereas this author treats the notion of conceptual modeling as    

13   

distinct and unique, others may disagree, believing it to be simply a subset of a related mapping discipline.

Mind Maps The popular exercise of mind mapping that many individuals are familiar with today got its start in the 1960s with its originator, Tony Buzan. In “Mind Maps as Classroom Exercises,” John Budd, professor in the Industrial Relations Center at the University of Minnesota’s Carlson School of Management, provides a thorough description of the accepted format for creating such maps: As with a traditional outline, a mind map is based on organizing information via hierarchies and categories. But in a mind map, the hierarchies and associations flow out from a central image in a freeflowing, yet organized and coherent, manner. Major topics or categories associated with the central topic are captured by branches flowing from the central image. Each branch is labeled with a key word or image. Lesser items within each category stem from the relevant branches.19 Additionally, a strong emphasis is placed on the incorporation of colors and images into the creation of a mind map.20 Therefore, the essential function of a mind map is very similar to that of the explicit conceptual model, but the process for developing one is more formalized.

Please see Figure 2.2 below, taken from the TechKNOW Tools

website, for an illustration of a typical mind map.

                                                             19

John W. Budd, “Mind Maps as Classroom Exercises,” Journal of Economic Education (Winter 2004): 36. 20 Budd, “Mind Maps as Classroom Exercises.” 

   

14   

Figure 2.2

Concept Maps According to Alberto Canas, Associate Director of the Institute for Human and Machine Cognition, and Joseph Novak, known for his development of concept mapping in the 1970s: Concept maps are graphical tools for organizing and representing knowledge. They include concepts, usually enclosed in circles or boxes of some type, and relationships between concepts indicated by a connecting line linking two concepts. Words on the line, referred to as linking words or linking phrases, specify the relationship between the two concepts.21 The detailing of the relationship between concepts by naming links accordingly is very important to the notion of concept mapping and helps to distinguish this form of mapping                                                              21

Joseph Novak and Alberto Canas, The Theory Underlying Concept Maps and How to Construct and Use Them, Technical Report IHMC Cmap Tools 2006-01 Rev 01-2008, Florida Institute for Human and Machine Cognition, 2008. http://cmap.ihmc.us/Publications/ResearchPapers/TheoryUnderlyingConceptMaps.pdf (accessed August 8, 2008).

   

15   

from mind mapping and conceptual modeling. Another difference between concept maps and mind maps is that the latter are organized around one central concept, whereas the former tend to be organized around several. Like conceptual models, concept maps are intended to evolve over time as an individual’s understanding of a topic increases. However, once again, while concept mapping serves a very similar purpose to that of conceptual modeling, its construction is much more structured in nature. Please see Figure 2.3 below, taken from the cited work of Novak and Canas, for an illustration of a typical concept map. Figure 2.3

Technology Aids At this point, after a discussion of the abovementioned techniques for information visualization, one may be left to wonder if making such models explicit is hard to do. In truth, the answer to this question is no, in large part due to technological advances that do    

16   

away with much of the burden of creation for these models. In fact, the relatively recent emergence of technological aids designed to augment the creation of conceptual models, concept maps and mind maps has brought about an increased interest in these techniques. Compared to the traditional pencil and paper construction, software programs make the editing and formatting of such models considerably easier.

This evolution in

functionality encourages users to revise and expand their maps as their knowledge base regarding a specific topic grows and changes.22 Christina De Simone, in “Applications of Concept Mapping,” states that “by externalizing information as they create concept maps, students are better able to detect and correct gaps and inconsistencies in their knowledge.”23 However, the development and evolution of concept maps as a response to detailed questions and requirements can sometimes prove difficult, lengthy and disorganized when created by hand. De Simone, Assistant Professor of Education at the University of Ottawa, further states that many students in her classes “find electronic concept mapping …very useful, as it minimizes the cumbersome and time-consuming activity of erasing, revising, and beginning anew. It allows them greater freedom to adjust their conceptual thinking and mapped representations.”24 Technologically based conceptual modeling aids, of which a wide variety exist, are likely the most useful and advantageous tools for intelligence students and analysts to utilize in the construction of complex and fluid conceptual models.

                                                             22

Josianne Basque and Beatrice Pudelko. “Using a Concept Mapping Software as a Knowledge Construction Tool in a Graduate Online Course,” in Proceedings of ED-MEDIA 2003, World Conference on Educational Multimedia, Hypermedia &Telecommunications, Honolulu, June 23-28, 2003, ed. D. Lassner and C. McNaught (Norfolk: Association for the Advancement of Computing in Education, 2003), 2268-2274. 23 Christina De Simone, “Applications of Concept Mapping,” Journal of College Teaching 55, no. 1 (2007): 34. 24 De Simone, “Applications of Concept Mapping,” 35.

   

17   

Learning Styles However, even with the increased efficiency in creating conceptual models brought about through technological aids, it is important to note that some individuals may be better suited towards this type of exercise than others. Research conducted by Josianne Basque and Beatrice Pudelko, from the LICEF Research Center in Canada, showed that some graduate students claiming to be auditory learners found little to no utility in constructing a visual representation of knowledge in the form of a concept map.25 However, other students identifying themselves as visual learners claimed to better understand a topic when concepts were structured in such a visual way.26

Hypotheses Based on review of the above literature the following hypotheses were formed: First, intelligence analysts who engage in ECM will generate better analytic products, as evaluated by thoroughness of process and accuracy of product, than analysts who do not. Second, that the individual to group method employed for creation of the conceptual models in this study will affect, either positively or negatively, the models’ ability to aid in intelligence analysis.  

                                                             25 26

Basque & Pudelko, Using a Concept Mapping Software, 2268-2274. Ibid.

   

18   

CHAPTER III: METHODOLOGY

In order to test the stated hypotheses I conducted an experiment that examined the value of ECM as it applies to the quality of the analysis produced. The experiment was designed to determine if intelligence analysts who engaged in ECM would generate better analytic products, as evaluated by thoroughness of process and accuracy of product, in comparison to analysts who did not. The following methodology section will provide the details of this experiment.

Research Design In conducting my experiment, I divided the subjects into an experimental and a control group. Efforts were made to ensure that conditions in both groups were identical, with the exception of the addition of ECM in the experimental group. The experimental group was instructed to use a structured ECM approach, facilitated by the use of the open-source program Bubbl.us (a free, internet-based conceptual modeling program), to aid in their analysis. The control group on the other hand was not instructed to use any particular method in conducting their analysis, as this group served as a baseline from which to measure the experimental group against.

Subjects In order to better relate the findings of my experiment to the United States Intelligence Community, I chose to draw from the undergraduate and graduate student population at the Mercyhurst College Institute for Intelligence Studies (MCIIS). The aim

   

19   

of this program is to produce graduates qualified to enter the government or private sector as entry-level intelligence analysts and it was therefore considered to be an appropriate pool from which to draw a sample. Mercyhurst College houses the oldest institution in the world dedicated specifically to the study of intelligence analysis. The program offers coursework in the fields of national security, law enforcement and competitive business analysis. Students in both the undergraduate and graduate programs are subjected to a rigorous academic curriculum during their time at Mercyhurst. They are expected to meet certain foreign language proficiency and internship requirements and are often faced with accelerated project deadlines for real-world decision makers in the field of intelligence. MCIIS considers their students to be experts in the exploitation of open source information for analytic purposes. For these reasons, MCIIS turns out capable and well-trained entrylevel analysts with a wide variety of analytic skills and abilities. To obtain participants for my experiment I contacted professors within the Intelligence Studies program at Mercyhurst College via email to ask if they would be willing to let me briefly speak to their classes in order to recruit students. I then followed up in-person with all professors who responded to my emails positively to setup a schedule for class visits that was convenient for them. After establishing this schedule I made appearances in a variety of undergraduate and graduate intelligence classes where I gave a broad overview of the experiment and passed out individual signup sheets to interested students (see Appendix 1). These sheets asked for the student’s name, email address, class year and available time slots for experiment participation. Students were given a total of twelve time slots to choose from,

   

20   

with six slots on the first day of the experiment and six slots on the second day. The only requirement placed on students was that they choose at least two time slots, one on each day of the experiment. Additionally, of these two time slots students were asked to select one for the duration of ninety minutes and the other for the duration of thirty minutes. Students could either turn these sheets back into me on the spot or they could drop them off at their convenience to my worksite located within the Intelligence Studies building. After receiving all signup sheets students were divided into subgroups based on class year. Individuals within each subgroup were then randomly divided between the control and the experimental group in an attempt to control for the educational level of participants.

For example, after the subgroup of freshman who signed up for the

experiment was established, individuals within the group were randomly assigned to either the control or experimental group. Students were then notified of their designated time slots for participation via an email, which included information on the location of the experiment. The email also stated that participation would be rewarded with extra credit from select professors within the Intelligence Studies Department, in addition to free refreshments and pizza on the second day of the experiment. Finally, my contact data was included in the email so all participants would easily be able to access me if they had any questions or concerns in the days leading up to the experiment. Intelligence Studies students of all grade levels were welcome to participate as any information needed to complete the experiment would be provided during the sessions. Testing took place approximately three-quarters of the way through the fall term, unfortunately falling at a time when students were especially busy trying to meet the

   

21   

demands of their t coursew work. A tottal of 47 stuudents (25 control c and 22 2 experimeental) acctually partiicipated in, and a finishedd, this experriment. For a breakdow wn of participants byy educationaal level and group, g pleasse see Figurees 3.1 & 3.2 below. Figure 3.1

Subject Educattion Level 14

Number Of Subjects

15 10

9

10

7

6

5 1

0 Freshman Sophomore

Junior

Senior

1st Year  Graduate G

2 2nd Year  Graduate G

Year In SSchool Figure 3.2

Subject Ed ducation LLevel By Grroup 10 Number Of Subjects

10

8 5

5

4

4

4

2

1

0

Year In Scho ool

   

3 3

3 0

Con. C E Exp.

22   

Preliminaries Prior to conducting the experiment, I had to submit a detailed research proposal to the Mercyhurst College Institutional Review Board (IRB) for approval (see Appendix 2). It is Mercyhurst College policy that any student conducting research involving the use of human subjects be granted permission by the IRB. In order to receive a green light from the IRB students must provide a description of the proposed research, its purpose, and an explanation of any potential dangers (physical or psychological) that could befall individuals as a result of their participation in the experiment. However, not only did I need to secure the consent of the IRB I also needed the consent of each individual experiment participant. Therefore, on the first day of the experiment (for both control and experimental sessions) all participants were given a formal consent form upon arrival (see Appendix 3 for control group consent form and Appendix 4 for experimental group consent form). This form outlined what would be expected of them as participants, along with the fact that there were no foreseen dangers or risks associated with involvement in the experiment. The form also asked for basic contact information such as name, class year, and telephone number.

Control Group: Day 1 Control group participants were asked to attend two experiment sessions. The first session was slotted for thirty minutes and the next session, scheduled for one week later, was slotted for 90 minutes. Three control group sessions were run on both days of the experiment, for a total of six sessions, in order to make it as convenient as possible for subjects to schedule participation into their busy agendas.

Out of those who

originally signed up for the experiment and were assigned to the control group (37    

23   

inndividuals), a total of 25 2 actually attended a andd completed the experim ment. Please see Figure 3.3 below for a graaphic representation of this t breakdow wn by class year. Figure 3.3 ontrol Group p Vs. Actual Control Gro oup By Class Year Original Co 12 2 Number Of Subjects

12 10 8 6 4

10

9 7

5

5

4

2

3 3 1 1

2

Original  Con. Actual  Con.

0

Yeear In School

Both groups weree given the same questtion to analyyze, regardinng October 2008 prresidential elections e in Zambia Z (seee Appendix 5). 5 The conntrol group was w simply asked a too forecast the winner of the electionss and to provvide a list off the main piieces of eviddence thhat aided theem in their analysis. a Booth groups were w providedd with inform mation regaarding soource reliabiility and anaalytic confiddence as theyy were askedd to supply measures m off both inn their final product (seee Appendix 6). Also, both groups were w given a semi-strucctured annswer sheett with spacee for their name, n a pre--written foreecast with built-in b wordds of esstimative pro obability andd presidentiaal candidatess to choose from f and spaace for a bullleted discussion (seee Appendixx 7). The boottom of the answer sheeet also askedd them to ideentify

   

24   

their source reliability and analytic confidence on a scale from low to high and to provide the names of their professors offering extra credit for participation in the experiment. Lastly, participants were given a sheet of expectations for the second and final session of the experiment one week later (see Appendix 8) and were asked to fill out a short pre-experiment questionnaire (see Appendix 10).

They were also once again

provided with my contact information in case they encountered problems or had questions while working on their analysis during the course of the week (see Appendix 11).

Experimental Group: Day 1 Experimental group participants were also asked to attend two sessions. The first session was slotted for ninety minutes and the next session, scheduled for one week later, was slotted for thirty minutes. Three experimental group sessions were run on both days of the experiment, for a total of six sessions, once again to make scheduling more convenient for participants. Out of those who originally signed up for the experiment and were assigned to the experimental group (37 individuals), a total of 22 actually attended and completed the experiment. Please see Figure 3.4 below for a graphic representation of this breakdown by class year.

   

25   

Figure 3.4

Number Of Subjects

Original Experrimental Gro oup Vs. Actu ual Experime ental Group By Class  Year 15 11 1

10 10 5

8

7

6 4

4

Original Exp.

3

3 3

Actual Exxp. 

0 0 0

Ye ear In School

The experimental e l group wass given the same taskinng as the coontrol groupp (see A Appendix 5). This groupp was also provided p withh the same information i regarding soource reeliability and d analytic coonfidence ass the controll group (see Appendix 6), 6 along witth the saame answer sheet (see Appendix A 7).. Although this t group was w given thee same tasking as thhe control group g in regards to foreecasting the winner of the t electionss, they weree also reequired to usse ECM to assist a them inn completingg this endeavvor. Thereefore, I begaan by givingg a lecture, approximateely ten minuutes in lengtth, to faamiliarize ex xperimental group participants withh what conceeptual modeels are, how w they caan be used and a the propposed value of making them expliccit in the fielld of intelliggence (ssee Appendiix 12). Folllowing the lecture, I haad all particcipants sign into a com mputer w whereby I leed them thhrough a steep-by-step tutorial t of the program m Bubbl.us (see A Appendix 13)). Once eveeryone was comfortable c w how thhe program worked, with w I beggan a sttructured EC CM exercise..    

26   

Participants began the exercise by making individual lists of concepts they felt would be important to answering the question asked of them. After individual lists were completed, participants were asked to read their lists aloud one at a time. As concepts were read off, they were written on a whiteboard at the front of the room, creating one combined group list. For every time a concept was repeated a check mark was placed next to it in order to highlight the most commonly thought of concepts. Also, concepts that the group immediately recognized as useful but that were mentioned only once or twice were made note of as well. Due to limitations on time, participants were not asked to engage in a convergent thinking exercise as a group, whereby they would critically evaluate the master list before producing their own conceptual models.

Instead,

participants were simply asked to use Bubbl.us to construct a conceptual model based on their individual list and thoughts as well as that of the collaborative group list. Finally, participants were given a chance to briefly look at the way others around them had assembled their models and were then asked to electronically share what they had created with me through the collaboration function within Bubbl.us (see Appendix 15). Following this task participants were given a sheet of expectations for the second and final session of the experiment one week later (see Appendix 9) and were asked to fill out a pre-experiment questionnaire (see Appendix 10). Lastly, they were provided with a sheet of my contact information (see Appendix 11).

Bubbl.us Bubbl.us is a free, internet-based, conceptual modeling program. It was chosen for use in this experiment due to its extremely simple user-interface. Not only did the program encompass all the relevant functions necessary to complete the conceptual    

27   

m modeling seg gment of myy experimennt, it could easily be taaught and leearned withinn the m minimal amo ount of time I had durinng sessions. Basic funcctions includde the creatioon of buubbles and lines l to illusstrate conceppts and theirr relationshiips, internet--based sharinng of w work with oth her Bubbl.uss users, and exporting fiinished prodducts as phottos or embeddding thhem into a web w page. Please P see Fiigure 3.5 bellow for a saample producct taken from m the B Bubbl.us web bsite, illustraating a varietty of the program’s featuures Figure 3.5

Conntrol Group:: Day 2 On the second daay of the expperiment, thee control grooup was expected to arriive at w their coompleted ansswer sheets ready r to turnn in. All reseearch thheir designatted session with annd analysis was w to be doone prior to arriving at this t session. Since the control c groupp had not received any a training on Bubbl.us in the firstt session, I began b their second s sessioon by assking them to t login to a computer and a follow along a with me m as I taughht them the basic fuunctions of the t program m. However, they still received r no lectures detaailing conceeptual

   

28   

modeling background information and were not given any specifics regarding how to actually construct their models in Bubbl.us (see Appendix 14). After this group became familiar with the program they were asked to illustrate the concepts and relationships they found to be important over the course of the week in answering the question posed to them. This was done in order to draw a comparison between the quality of conceptual models made after background information was provided and those made with little to no prior instruction. Additionally, it compared the quality of conceptual models made pre-collection and updated throughout the analytic process with those made post-analysis. In bringing the session to a close participants were asked to fill out a post-experiment questionnaire (see Appendix 16) and were given a debriefing sheet thanking them for their time and further explaining the purpose of the experiment (see Appendix 18).

Experimental Group: Day 2 Once again, the experimental group was expected to arrive on the second day of the experiment with their research and analysis completed and a finished answer sheet ready to be handed in. Additionally, they were expected to have electronically updated the conceptual models made during the first session of the experiment throughout the course of the week to reflect their expanding knowledge base in regards to the question at hand. Therefore, after handing in their answer sheets this group was asked to fill out a follow-up questionnaire (see Appendix 17) and was then provided with the same debriefing sheet as the control group (see Appendix 19).

   

29   

Data Analysis Procedures Since the intent of this experiment was to test the value of ECM as it applies to analysis, control and experimental group results were compared in terms of quality and accuracy of process and product. To evaluate and compare the processes of the two groups, three MCIIS second year graduate students independently ranked the discussion section of each participant’s answer sheet from best to worst. Students were used in lieu of professors who tend to have unique grading styles, as the students all received the same training regarding what makes a sound analysis and were thus thought to be on more equal footing. All identifying information including the student’s name, class year and group were removed prior to evaluation.

Additionally, the students doing the

evaluating did not know the outcome of the elections at the time of ranking in order to keep measures of process and product independent of one another. The product measure was derived through a simple tally of whether or not the participant predicted the question correctly. Finally, the actual conceptual models created were compared in terms of complexity, based on how many concepts and connections between concepts each encompassed. All measurements and rankings were compiled into a Microsoft Excel Spreadsheet along with subject education level breakdowns and information from the pre- and post-experiment questionnaires. Statistical analyses were then undertaken to determine whether or not the experiment results were statistically significant.

The

control group was generally expected to fall lower in the graduate student process rankings than the experimental group and was also expected to be less accurate overall in terms of forecasting the outcome of the election.

   

30   

CHAPTER IV: RESULTS

The results of the ECM experiment generated a variety of interesting and surprising results. The next section will first provide a brief explanation of the statistical significance testing conducted throughout this thesis and will then detail the results derived from the analysis of pre- and post-experiment questionnaires. Next, findings from the experiment itself will be discussed as a function of process and product, with experimental and control group findings initially reported on individually, followed by a comparison of the groups to one another.

Significance Testing All significance tests related to this thesis were conducted at the 0.10 significance level (see Appendix 20). The reason behind setting what some may consider to be a rather lax level of significance is the fact that the research conducted in this thesis regarding ECM and intelligence analysis is exploratory in nature. According to G. David Garson, Professor of Public Administration at North Carolina State University, in Guide to Writing Empirical Papers, Theses, and Dissertations, “It is inappropriate to set a stringent significance level in exploratory research (a .10 level is acceptable in exploratory research).”27 Although debate remains lively amongst researchers regarding proper significance levels based on situation, this author felt that a 0.10 level was most appropriate when dealing with this particular set of research and data.

                                                             27

G. David Garson, Guide to Writing Empirical Papers, Theses, and Dissertations (New York: CRC Press, 2002), 199.

   

31   

Pre- and Post-Questionnaire Results Prior to the experiment participants were asked whether or not they thought they would be able to dedicate a sufficient amount of time to completing the experiment over the course of the next week. In response, 64.3% of control group participants thought that they would have ample time, 32.1% were not sure and 3.6% did not expect to be able to dedicate a sufficient amount of time to the experiment. On the other hand, 58.3% of experimental group participants expected to have enough time and 41.7% were unsure. When asked post-experiment whether or not they had actually been able to devote a sufficient amount of time to the experiment over the course of the past week, 68% of control group participants claimed they had, 16% were unsure and 16% said that they had not. In the experimental group 60% claimed to have had enough time to dedicate to the experiment, 20% were unsure and 20% had not. While the percentage of participants in both the control and experimental group claiming to have been able to dedicate a sufficient amount of time to completion of the experiment increased slightly from pre- to post-experiment, the percentage of those who claimed that they did not have a sufficient amount of time to dedicate to the experiment increased from pre- to post-experiment more substantially. Post-experiment 20% of the experimental group claimed that they had not had enough time (up from 0% preexperiment) and 16% of control group participants claimed the same (up from 3.6% preexperiment). As previously stated in the methodology section of this thesis the timing of the experiment fell during an extremely busy time in the participants’ trimester, serving as a limitation to this study as students had to balance the experiment with their class

   

32   

w work and oth her responsibbilities. Pleaase see Figurre 4.1 below w for a graphhic display of o this data. Figure 4.1

Pre‐ Vs. Post‐Experimen nt: Able To o Dedicate A  ufficient Am mount Of TTime To Exxperiment?? Su 20

64.3% %

68%

15 10 5

32 2.1%

58.3% % 4 41.7%

0% 60 16% 16%

3.6%

20% 20%

Yes Maybe No

0%

0 Con ntrol

Experrimental

Co ontrol

Experimental

Post‐Experim ment

Pre‐Experimen nt

t identify how Additionally, pree-experimennt all particcipants werre asked to w 1 beingg not intereested and 5 being extreemely innterested theey were in the study, with innterested. The T average response r forr both groups was a 3.5, illustrating that both coontrol annd experimeental groupss were on average a equually interestted in the experiment e at its onnset. The diifference bettween controol and experrimental group responsees to this queestion w not found was d to be statisstically signiificant at the 0.10 level (p-value ( = 0..851). When n asked the same questtion post-exxperiment, the t average response of o the coontrol group p increased to 3.8, while the averaage responsee of the expperimental group g reemained the same at 3.55. This show ws a slight average a increease in intereest on the paart of thhe control grroup from prep to post-experiment. However, once again the differennce in

   

33   

control and experimental group responses was not found to be statistically significant at the 0.10 level (p-value = 0.267). Another question asked of participants prior to their involvement in the experiment was how useful they feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.

On average, control group

participants responded with a 3.8 and experimental group participants responded with a 4.2. Although a significant difference at the 0.10 level was not found between control and experimental group responses to this question, the results did approach significance (p-value = 0.127). Faced with the same question post-experiment the average control group response increased to a 4.1, while the experimental group response maintained steady at 4.2. However, the difference in control and experimental group post-experiment responses was not found to be significant at the 0.10 level (p-value = 0.617). Results for this question show that the experimental group found structured approaches to the analytic process to be more useful than did the control group, both pre- and post-experiment. The control group’s feelings regarding the utility of structured approaches to the analytic process grew throughout the course of the experiment, whereas the experimental group’s did not. When experiment participants were asked to identify the learning style they most closely associated with, over half of both control and experimental group participants identified themselves as visual learners. Since ECM is a visual learning aid, it is likely that the exercise was generally more beneficial to those claiming to be visual learners than to those who chose an alternative learning style. Please see Figure 4.2 below for the

   

34   

fuull range of control and experimentaal group respponses to thee question reegarding leaarning sttyles. Figure 4.22

Control Vs. Experrimental G Group: Learning Style e 1 12.5% 3.4% %

Unsu ure

29.2% 31%

Kinesthetic &  Tactile

Experiimental  54.2%

Visu ual

Contro ol 58.6 6%

4.2% % 6.9 9%

Audito ory 0

10

20 0

ment both coontrol and experimental e l groups werre asked to gauge g Also, post-experim thheir level of understandinng of concepptual modeliing prior to the t study onn a scale of 1 to 5, w 1 being with g extremely low and 5 being extrem mely high. The averagge control group g reesponse to this t questionn was a 3.6,, whereas thhe average experimenta e l group respponse w a 3.2. While was W the difference d beetween contrrol and expeerimental reesponses waas not foound to be significant s att the 0.10 leevel, the resuults did apprroach signifiicance (p-vaalue = 0.187). b groups were also asked to ideentify their understandin u ng of Post-eexperiment both coonceptual modeling m folllowing the experiment, e using the same s scale. Post-experiiment thhe average response r forr both the control c and experimentaal group waas a 4.0 andd was thherefore not statisticallyy significant at the 0.10 level (p-valuue = 1.0). Results R from m this quuestion show w that postt-experimentt both contrrol and expeerimental grroup participants    

35   

claimed to have the same understanding of conceptual modeling, an increase for both groups from their pre-experiment knowledge on the topic. However, the experimental group claimed to have less of an understanding of conceptual modeling than the control group at the onset of the experiment, signaling that on average the experiment raised their knowledge of conceptual modeling more than it did the control group’s. In relation to the above question, post-experiment all participants were asked how often ECM had been a part of their personal analytic process prior to the experiment on a scale of 1 to 5, with 1 being never and 5 being every time they produced an intelligence estimate. On average the control group responded with a 2.8 and the experimental group responded with a 2.75. The difference between control and experimental group responses was not found to be statistically significant at the 0.10 level (p-value = 0.872). However, post-experiment both groups were also asked how often they plan to incorporate ECM into their personal analytic process in the future, using the same scale. In response to this question, the control group average was a 3.5 and the experimental group average was a 3.6. Once again, the difference between experimental and control group responses was not found to be statistically significant at the 0.10 level (p-value = 0.617).

Results from the above question highlight that although the control group

claimed to employ ECM in their analytic process prior to the experiment on average slightly more than the experimental group, the experimental group claimed that they will employ ECM in their analytic process on average more than the control group in the future. Post-experiment the experimental group was asked a series of four questions regarding their specific responsibilities within the experiment. First, the experimental

   

36   

group was asked to rate whether or not they found that ECM aided them in developing a more thorough and nuanced intelligence analysis in this experiment. In response, 33% of experimental group participants claimed that ECM definitely aided them in their analysis, and 66.7% claimed that it helped them somewhat, with no participants responding that it did not help at all. The experimental group was also asked post-experiment to rate how useful they found the conceptual modeling training provided at the beginning of the experiment to be, with 1 being not at all helpful and 5 being extremely helpful.

On average,

experimental group participants responded to this question with a 3.9. Additionally, the experimental group was asked how effective they found the conceptual modeling method used in this experiment, inclusive of both individual work and group collaboration, to be. The average response to this question was a 3.7. Finally, the experimental group was asked how useful they found the technology aid, Bubbl.us, to be in creating and updating their conceptual models, with 1 being not useful and 5 being extremely useful. Overall, experimental group participants found Bubbl.us to be quite valuable, averaging a response of 4.1.

Process - Conceptual Model Findings The 2007 Intelligence Community Directive Number 203 on analytic standards confirms that, “To the extent possible, analysis should incorporate insights from the application of structured analytic technique(s) appropriate to the topic being analyzed.”28 As such, the conceptual models resulting from the experiment were analyzed in terms of complexity by simply tallying the number of concepts and connections between concepts                                                              28

United States Government, Intelligence Community Directive Number 203, June 21, 2007, http://www.fas.org/irp/dni/icd/icd-203.pdf (accessed January 26, 2009).

   

37   

foound in each h model (please see Figuure 4.3 below w to see the distinction between b conncepts annd connectio ons). Contrrol group coonceptual models m averaaged 12.6 cooncepts and 12.9 coonnections, per p model. Figure 4.3 4

Concept 

Concept 

Connection

Connection 

Pre-an nalysis expeerimental grroup concepptual modelss averaged 25 conceptss and 28.9 connecttions, per model. m

Posst-analysis experimenta e al group conceptual models m

avveraged 30.9 9 concepts and a 31 connnections, perr model. Reesults of siggnificance teesting foor the numb ber of conceppts in pre- and a post-expperimental group g concepptual modelss was foound to be significant at the 0.100 level (p-vaalue = 0.056), howeverr the number of coonnections was w not founnd to be siggnificant (p-vvalue = 0.2331). As illuustrated beloow in Figure 4.4, bo oth the average number of conceptss and the aveerage numbeer of connecctions between conccepts increaased betweenn pre-analyssis and postt-analysis coonceptual models m w within the exp perimental group. g

   

38   

Figure 4.44

Exp perimatal G Group: Pre e‐ Vs. Post‐Analysis  Averagge Conceptts & Conne ections Per Conceptu ual  Mode el 40 4 30 3

30.9 2 25

28.9

31

Pre‐An nalysis Post‐A Analysis

20 2 10 1 0 Con ncepts

Connectiions

When n looking at a experimeental group conceptuall models as a whole (not g between prep and posst-analysis) the models averaged 277.9 conceptss and distinguishing 29.9 connectiions, per moodel. As illuustrated beloow in Figuree 4.5, when comparing these t differencce in compllexity exxperimental group averrages with control groupp averages the between the two t groups’ models becoomes quite obvious. o

Figure 4.5

Con ntrol Vs. Exxperimentaal Group: A Average  Concep pts & Conn nections Pe er Concepttual Modeel 29.9

27.9

30

Contrrol 20

12 2.6

12.9

10 0

Con ncepts

Connecttions  

 

Experrimental

39   

The control group’s models were much simpler, consisting on average of less than half the amount of concepts and connections between concepts present in experimental group models. Results of significance testing for the number of concepts in control versus experimental group conceptual models was found to be significant at the 0.10 level (p-value = 0.000), furthermore the number of connections was also found to be significant (p-value = 0.000). Please see Figure 4.6 below for an illustration of a typical conceptual model made by a control group participant and Figure 4.7 for an illustration of a typical conceptual model made by an experimental group participant. Figure 4.6

   

40   

Figure 4.7

Process – Logic/Quality of Supporting Evidence Findings Intelligence Community Directive Number 203 highlights the need for logical argumentation within analytic products, stating that, “Analytic presentation should facilitate clear understanding of the information and reasoning underlying analytic judgments.”29 As a result, three Intelligence Studies graduate students independently ranked the analytic products of all 47 participants from best to worst, based on the quality and logic of evidence supporting the analyst’s estimate. Based on the rankings assigned by each graduate student, the overall average ranking of control group participants was a 22.4 and the overall average ranking of experimental group participants was a 25.8. Therefore, control group participants scored approximately 3 points higher than did experimental group participants in terms of the reasoning used to substantiate their                                                              29

Ibid.

   

41   

estimates, implying that participants not using ECM to aid in their analysis were able to formulate slightly better supporting arguments than participants who did in fact use ECM. Correlation scores amongst the three graduate student rankers at the source of this finding were relatively high across the boards (0.76, 0.72 and 0.63), illustrating consistency in experiment participant scoring.30

As Jacob Cohen, an influential

statistician and Professor Emeritus at New York University before his death in 1998, discusses in Statistical Power Analysis for the Behavioral Sciences, according to convention, correlations above a 0.5 are traditionally considered to be large within the social sciences.31 This lends support to the Mercyhurst method for the evaluation of analytic products, as the ranking consistency of the three graduate student raters was high.

Product – Forecasting Findings According to Intelligence Community Directive Number 203, analytic products should “make accurate judgments and assessments.”32

Therefore, not only must the

quality of the analyst’s process be accounted for, but the correctness of estimates must be measured as well. In terms of forecasting the correct outcome of the October 2008 Zambian presidential elections, 68% of control group participants predicted accurately, whereas only 40.9% of experimental group participants did. Therefore, individuals who                                                              30

 The common measures of inter rater-reliability, known as Cohen’s and Fleiss’ Kappa, were not used when conducting tests of correlation in this study. This is due to the fact that both measures are designed for use in situations where the data is categorical (ex. yes vs. no) and were therefore felt by the author to be inappropriate measures for the type of data present (ordinal numbers). 31 Jacob Cohen, Statistical Power Analysis for the Behavioral Sciences (Philadelphia: Lawrence Erlbaum Associates, 1988). 32 United States Government, Intelligence Community Directive Number 203.

   

42   

did not use ECM E to aid in i their anallysis were abble to identiify the actuaal outcome of o the reesearch quesstion posed to them much more oftten than indiividuals whoo did incorpporate E ECM into th heir analyticc process.

Forecastingg result diffferences weere found to t be

sttatistically significant att the 0.10 leevel (p-valuee = 0.065). Please see Figure F 4.8 below b foor a graphic representatioon of this daata. Figure 4.8

Control V Vs. Experim mental Group:       Accurracy Of Forrecasting  68 8%

20

59.1% %

15 10

Contrrol 40.9%

32%

Experimental 

5 0

Inaccurate 

Acccurate 

Qu uality Of Suppporting Eviidence Vs. Forecasting F A Accuracy Finndings After looking at both the quuality of thhe evidence supporting the particippants’ annalysis and the t participaants’ forecasting accuraccy separatelyy, the opporttunity to com mpare thhe two findiings presentted itself. Therefore, T t followinng supplemeentary concluusion the reegarding graaduate student process rankings and a forecastting accuraccy, althoughh not directly relateed to the hyppotheses of this t experimeent, was thouught to be innteresting ennough too warrant meentioning at this time. Out of o curiosity, graduate student process rankings, essentiallly measuringg the quualitative strrength of thee individual’’s assessmennt, were com mpared againnst whether or o not

   

43   

the individual correctly forecasted the outcome of the election. To make this comparison the process rankings were simply split in half and a tally of the amount of individuals in the top half and bottom half who forecasted the elections correctly was conducted. This measurement was carried out three times: first for the group as a whole, second for just the control group and lastly for just the experimental group. The expectation following this comparison was that individuals in the top half of the graduate student process rankings would forecast the winner of the elections correctly considerably more often than individuals falling in the bottom half of the rankings. However, this was not found to be the case. In fact, results of the tally showed little difference in forecasting accuracy between those who were ranked better qualitatively than those who were not. When looking at the group as a whole, 14 individuals in the top half of the graduate student process rankings forecasted the outcome of the elections correctly and 9 individuals forecasted incorrectly, compared with an even split in the bottom half of 12 individuals forecasting correctly and 12 forecasting incorrectly. When looking at solely the control group, 9 individuals in the top half of the graduate student process rankings forecasted the outcome of the elections correctly and 4 individuals did not, compared to 8 individuals who forecasted correctly in the bottom half and 4 who did not. Finally, when looking at the experimental group on its own, there was an even split of 5 individuals in the top half of the graduate student process rankings who forecasted correctly and 5 who did not, compared to 4 individuals in the bottom half who forecasted correctly with 8 who did not. Please see Figures 4.9, 4.10 and 4.11 below for graphic representations of this data.

   

44   

Figure 4.9

Forecaasting Accu uracy: Top p Vs. Bottom Half Of  G Graduate S tudent Pro ocess Rankkings 14

12

12

15 9 10 5 0 Corrrect

Inccorrect

C Correct

I Incorrect

Bottom Half

Top Half

Figure 4.110

Contrrol Group FForecasting Accuracyy: Top Vs.  Botttom Half O Of Graduate Student Process  Rankinggs 10 8 6 4 2 0

9

8 4

Corrrect

4

Inco orrect

Correct

Bottom Haalf

Top Half

   

Incorrect

45   

Figure 4.111

Experim mental Gro oup Forecaasting Acccuracy: Top p  Vs. Bo ottom Half  f Of Graduaate Studen nt Process  Rankinggs 8

8 5

6

5

4

4 2 0 Correect

Inco orrect

C Correct

Top Half

I Incorrect

Bottom Haalf

Althou ugh a stricct methodollogy was not n applied in reachingg this partiicular coonclusion, making m it diffficult to ascertain the exxtent to whicch any extranneous variabble or variables has impacted it,, the point inn its most basic form rem mains the sam me. w are betteer writers, orr who Thereefore, this coonclusion sugggests that inndividuals who arre able to craft c more convincing arguments, are not neecessarily annymore likeely to foorecast correectly than inndividuals who w are lackking in thosse skills. This T notion is an offshoot of th he general argument a m made in Philiip Tetlock’ss Expert Political Judgm ment, w which basicaally states thhat the way in which we w reason orr think abouut things is more im mportant thaan our backggrounds and accomplishm ments or eveen our belieff systems.33 How w think, then appears too be more im we mportant thaan what we think t when it i comes to being b prroficient forrecasters.

                                                             33

Philip E. Tetlo ock, Expert Poolitical Judgmeent (Princeton: Princeton University Press, 2005).  2

   

46   

Product – Soource Reliabbility and Annalytic Confi fidence Findiings Intelliigence Com mmunity Dirrective Num mber 203 sttates that analytic a products perly describbe quality annd reliabilityy of underlyying sourcess” and “prooperly shhould “prop caaveat and ex xpress uncerrtainties or confidence c inn analytic juudgments.”344 As a result, all exxperiment participants p w were asked to assess both b their soource reliabiility and anaalytic coonfidence on o a scale from f low too high.

Coontrol groupp findings regarding r soource

reeliability illu ustrate that 4% 4 of particcipants claim med low souurce reliabiliity, 80% claaimed m medium and 16% claim med high. Findings F for the experim mental grouup show 4.5% of participants claimed c low w reliability, 68.2% claiimed medium m and 27.3% claimed high. A Although peercentages reveal that approximaately 11% more experimental group g participants claimed c to haave high souurce reliabiliity than did control grouup participannts, it t note that this t differennce is a function of just two t particippants. As a result r iss necessary to soource reliabiility findinggs were not found to bee significant at the 0.10 level (p-vallue = 0.451). Pleasse see Figuree 4.12 below w for a graphhic representaation of this data. Figuree 4.12

Contro ol Vs. Experimental G Group:       Source Reliability                     80 0% 20

68.2% Co ontrol

15 10 5

27.3% 16%

4% % 4.5%

0 Lo ow

Meedium

High

                                                             34

Ibid.

   

Exxperimental

47   

In teerms of anallytic confideence, 12% of control grooup participants claimedd low annalytic conffidence, 80% % claimed medium m and 8% claimed high. Onn the other hand, h 222.7% of ex xperimental group partticipants claaimed low confidence, 63.6% claaimed m medium and 13.6% claim med high. Although A peercentages reeveal that appproximatelyy 6% m more experim mental groupp participantts claimed too have high analytic connfidence thaan did coontrol group p participants, it is necesssary to notee that this diifference is a function of just onne participaant. As a result, r findinngs for anaalytic confiddence were not found to t be siignificant at the 0.10 levvel (p-value = 0.745). Pllease see Figgure 4.13 below for a graaphic reepresentation n of this dataa. Figure 4.133

Control Vs. Experime C ental Grou up:       dence Anallytic Confid 80%

20

63.6%

Contro ol

15

Experim mental

10 5

22 2.7% 8 13.6% 8%

12%

0 Low

Medium

H High

   

48   

CHAPTER V: CONCLUSIONS

As previously stated, the purpose of this study was to determine the value of ECM within the analytic process. This was accomplished by requiring the experimental group to incorporate the use of conceptual models, created through a structured individual to group approach, into their analytic process. The control group on the other hand was simply asked to analyze the question posed to them, using no particular method. While the control group correctly predicted the outcome of the elections 68% of the time, the experimental group forecasted the outcome correctly only 40.9% of the time. This result was found to be statistically significant at the 0.10 level (p-value = 0.065).

Additionally, although the difference was marginal, a larger percentage of

experimental group participants claimed to have high source reliability and analytic confidence than did control group participants.

Therefore, the experimental group

members did considerably poorer in terms of correctly forecasting the result of the elections, but felt they had more reliable sources and were more confident in their assessment. However, neither source reliability (p-value = 0.451) nor analytic confidence (p-value = 0.745) results were found to be statistically significant at the 0.10 level. Even so, in terms of product measures only, these results paint a bleak picture of the role of ECM within the analytic process. Turning to process measurements, results of graduate student rankings placed control group participants, on average, roughly 3 points higher than experimental group participants in terms of the quality and logic of the evidence used to support their analysis. Correlation scores amongst the three graduate student rankers were relatively    

49   

high across the boards (0.76, 0.72 and 0.63), illustrating consistency in experiment participant scoring.

Furthermore, the experimental group’s conceptual models were

appreciably more complex than the control groups in terms of the amount of concepts and relationships between concepts. This result was found to be statistically significant at the 0.10 level (p-value = 0.000). Therefore, although the experimental group’s conceptual models appear to be more complex and thorough than the control group’s models, they scored lower on average in regards to the reasoning used to substantiate their estimates. Once again, these results appear largely to invalidate any suggested value of ECM within intelligence analysis.

Since the literature strongly suggests that ECM will improve

analysis, what could account for these counter-intuitive results?

Excessive Possibilities Confuse Sheena S. Iyengar and Mark R. Lepper in, “When Choice is Demotivating: Can One Desire Too Much of a Good Thing?,” state that, “It is a common supposition in modern society that the more choices the better—that the human ability to manage, and the human desire for, choice is infinite.”35 Traditionally, research has tended to support the concept that having some choice produces better outcomes than having no choice. However, a growing body of literature concludes that when the amount of choices available becomes too large, people have a very hard time managing that complexity. As a result, Iyengar, professor in the Management Department of the Columbia Business School, and Lepper, professor of psychology at Stanford University, conducted a field experiment at an upscale grocery store whereby they observed the outcome of consumers visiting one of two tasting booths. One tasting booth displayed only 6 jams,                                                              35

Sheena S. Iyengar and Mark R. Lepper, “When Choice is Demotivating: Can One Desire Too Much of a Good Thing?,” Journal of Personality and Social Psychology 79, no. 6 (2000): 995.

   

50   

while the other displayed a variety of 24 different flavored jams. Iyengar and Lepper’s findings showed that initially, shoppers who encountered the booth with 24 flavors were more attracted to the display (stopping 60% of the time) than shoppers who encountered the booth with only 6 (stopping only 40% of the time).36 Additionally, even though one booth displayed only 6 flavors, whereas the other displayed 24, there were no significant differences in the amount of jams sampled by visitors to each of the different booths.37 Finally, almost 30% of consumers who stopped at the 6 flavor booth bought a jar of jam, while only 3% of consumers who stopped at the booth with 24 flavors did.38 This suggests that although individuals originally found the booth with the plethora of flavors to be more attractive it hampered their ability and motivation to make a choice when it came time to purchase the product. A similar point is made in Expert Political Judgment, when Tetlock discusses a series of scenario exercises tested on a group of experts comprised of individuals he refers to as hedgehogs (those who know one big thing) and foxes (those who know many little things).39 Essentially participants were provided with an exhaustive variety of possible future scenarios in regards to a particular country and were asked to forecast the scenario that was most likely. This presentation of scenarios did not substantially affect the predictions of hedgehogs who were quite easily able to reject scenarios that they believed would not actually happen.40 However, the foxes, being more open-minded, found it very difficult not to consider even the strange or implausible scenarios.41                                                              36

Iyenger and Lepper, “When Choice is Demotivating,” 997. Ibid. 38 Ibid.  39 Tetlock, Expert Political Judgment, 190. 40 Ibid. 41 Ibid. 37

   

51   

Therefore, for this group in particular, the danger of attributing limited resources to the contemplation of a plethora of possibilities did little more than send them on a wild goose chase. This illustrates that “foxes become more susceptible than hedgehogs to a serious bias: the tendency to assign so much likelihood to so many possibilities that they become entangled in self-contradictions.”42

Importance of Convergence Conventional wisdom has long been a proponent of the process of divergent thought, focusing on the need for thinking outside the box, maintaining an open mind and encouraging an ever-increasing flow of ideas. In fact, until recently, convergent thinking has perpetually received a bad rap. However, research has finally begun to unearth the benefits of a combined approach including both divergent and convergent thinking. “In Praise of Convergent Thinking” by Arthur Cropley, states that, “Convergent thinking is oriented toward deriving the single best (or correct) answer to a clearly defined question…Divergent thinking, by contrast, involves producing multiple or alternative answers from available information.”43 Cropley, visiting professor of psychology at the University of Latvia for the past eleven years, argues that divergent thinking is essential to the creation of novel ideas, but that convergent thinking is then vital to the exploration of those ideas.44 Truly utilitarian creative thought, says Cropley, can only be achieved through the generation of ideas through divergence, followed by the criticism and evaluation of those ideas through convergence.45

                                                             42

Ibid. Arthur Cropley, “In Praise of Convergent Thinking,” Creativity Research Journal 18, no.3 (2006), 391. 44 Cropley, “In Praise,” 398. 45 Ibid. 43

   

52   

According to Michael Handel, as quoted by Stephen Marrin, “While the absence of competition and variety in intelligence is a recipe for failure, its institution does not guarantee success.”46

Handel, joint founding editor of the journal Intelligence and

National Security, further notes that while divergent thinking exercises lead to an increased number of opinions for consideration, they are not able to aid in ascertaining the best alternative.47 Richard Betts, director of the Institute of War and Peace Studies, and the director of the International Security Policy Program at Columbia University, makes a similar point in “Analysis, War, and Decision: Why Intelligence Failures Are Inevitable.” Betts states that, “To the extent that multiple advocacy works, and succeeds in maximizing the number of views promulgated and in supporting the argumentative resources of all contending analysts, it may simply highlight the ambiguity rather than resolve it.”48

Essentially, both Handel and Betts acknowledge that while divergent

thinking methods may indeed be useful and necessary within intelligence analysis, they are not without their limitations. In specific, the generation of numerous ideas alone does not automatically result in better answers, highlighting the need for a combination of both divergent and convergent approaches to intelligence analysis.

Final Thoughts The jam experiment and scenario exercises discussed above tie directly into the findings of this experiment, showing that although experimental group conceptual models were significantly larger than control group conceptual models, the control group did                                                              46

Michael I. Handel, “Intelligence and the Problem of Strategic Surprise,” The Journal of Strategic Studies (September 1984), 268. In Stephen Marrin, “Preventing Intelligence Failures by Learning from the Past,” International Journal of Intelligence and CounterIntelligence 17, no. 4 (2004), 665. 47 Ibid.  48 Richard K. Betts, “Analysis, War, and Decision: Why Intelligence Failures Are Inevitable,” World Politics 31 (1978): 76.

   

53   

significantly better than the experimental group in forecasting the correct outcome of the research question posed. As a result of the individual to group conceptual modeling method employed in this study, the generation of a multitude of ideas seemed to do little more than confuse and overwhelm experimental group participants. Faced with such a large number of concepts, experimental group participants appeared to struggle to make sense of the relevant relationships and to identify adequately the information most important to answering the question. At this juncture, the discussion of convergent and divergent thinking discussed in detail above, becomes relevant. Experimental group participants, having been involved in a structured individual to group divergent thinking exercise, were left with many more options to consider than control group participants who simply set off to research the question on their own. Traditionally, this would be considered an excellent outcome as the divergent thinking exercise appeared to work, leaving the experimental group with a much wider array of concepts to consider. However, this experiment found that while this is true, it is not enough. Divergent thinking on its own appears to be a handicap, without some form of convergent thinking to counterbalance it. Therefore, it is likely that experimental group participants, once faced with the plethora of ideas generated by the group, would have greatly benefitted from a structured convergent thinking exercise before creation and development of their conceptual models. The goal of this exercise being to critically evaluate the ideas proposed, possibly eliminating those concepts that were clearly off base and prioritizing what was left into a meaningful arrangement.

One way this convergent aspect is often accomplished in

groups is based on the amount of individual members within that group. For example,

   

54   

often times a group with four team members will find a way to organize and break down information into four separate sections, thereby allowing them to assign one section to each team member. While this is certainly not the ideal way to engage in convergent thinking, it is likely better than not incorporating it at all. However, at this point the best method for engaging in convergent thinking within the process of ECM has yet to be determined.

Future Research Future research should explore the importance of incorporating both divergent and convergent thinking into the method for the construction of conceptual models. While divergence was sufficiently accounted for in this study, due to time limitations convergence was not. Therefore, it would be interesting to see the effect of taking the individual to group conceptual modeling method employed in this study one-step further. As Surowiecki, McKeachie and Janis tell us, starting the process at the individual level and then moving into group collaboration is very valuable. However, at this juncture, once all possibilities the group can think of have been accounted for, it is necessary for the group to narrow the scope of the conceptual model into only the most relevant concepts and relationships and then organize them accordingly. Therefore, additional research is needed to establish the best method for carrying out this task.

   

55   

BIBLIOGRAPHY

Allison, Graham T. “Conceptual Models and the Cuban Missile Crisis.” In The Sociology of Organizations: Classic, Contemporary and Critical Readings, ed. Michael Jeremy Handel. Thousand Oaks: SAGE, 2003. Basque, Josianne and Beatrice Pudelko. “Using a Concept Mapping Software as a Knowledge Construction Tool in a Graduate Online Course,” in Proceedings of ED-MEDIA 2003, World Conference on Educational Multimedia, Hypermedia &Telecommunications, Honolulu, June 23-28, 2003, ed. D. Lassner and C. McNaught (Norfolk: Association for the Advancement of Computing in Education, 2003). Betts, Richard K. “Analysis, War, and Decision.” World Politics 31 (1978). Budd, John W. “Mind Maps as Classroom Exercises.” Journal of Economic Education (Winter 2004).  

Church, Forrest. Introduction to The Jefferson Bible: The Life and Morals of Jesus of Nazareth, by Thomas Jefferson, 1-31. Boston: Beacon Press, 1989. Cohen, Jacob. Statistical Power Analysis for the Behavioral Sciences (Philadelphia: Lawrence Erlbaum Associates, 1988). Cropley, Arthur. “In Praise of Convergent Thinking.” Creativity Research Journal 18, no. 3 (2006). De Simone, Christina. “Applications of Concept Mapping.” Journal of College Teaching 55, no. 1 (2007). Garson, G. David. Guide to Writing Empirical Papers, Theses, and Dissertations (New York: CRC Press, 2002). Heuer, Richards J. Psychology of Intelligence Analysis (Center for the Study of Intelligence, 1999).

Iyengar, Sheena S. and Mark R. Lepper. “When Choice is Demotivating: Can One Desire Too Much of a Good Thing?.” Journal of Personality and Social Psychology 79, no. 6 (2000). Janis, Irving L. Groupthink: Psychological Studies of Policy Decisions and Fiascoes. Boston: Houghton Mifflin Company, 1982.

   

56   

Jarvelin, Kalervo and T.D. Wilson. “On Conceptual Models for Information Seeking and Retrieval Research.” Information Research 9, no. 1 (October 2003), http://informationr.net/ir/9-1/paper163.html (accessed January 15, 2009). Mangio, Charles A. and Bonnie J. Wilkinson. “Intelligence Analysis: Once Again.” Paper presented at the annual international meeting of the International Studies Association, San Francisco, California 26 March, 2008. Marrin, Stephen. “Preventing Intelligence Failures by Learning from the Past.” International Journal of Intelligence and CounterIntelligence 17, no. 4 (2004). McKeachie, Wilbert J. McKeachie’s Teaching Tips. Boston: Houghton Mifflin Company, 2002. Miller, George A. “The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information.” Psychology Review 63 (1956). Novak, Joseph and Alberto Canas. The Theory Underlying Concept Maps and How to Construct and Use Them, Technical Report IHMC Cmap Tools 2006-01 Rev 012008, Florida Institute for Human and Machine Cognition, 2008. http://cmap.ihmc.us/Publications/ResearchPapers/TheoryUnderlyingConceptMap s.pdf (accessed August 8, 2008). Santrock, John W. Adolescence, 8th ed. New York: McGraw-Hill, 2001. Surowiecki, James. The Wisdom of Crowds. New York: Anchor Books, 2004. Tetlock, Philip E. Expert Political Judgment. Princeton: Princeton University Press, 2005. United States Government. Intelligence Community Directive Number 203 (June 21, 2007), http://www.fas.org/irp/dni/icd/icd-203.pdf (accessed January 26, 2009).

   

57   

APPENDICES

   

58   

Appendix 1: Experiment Sign-Up Form

Full Name: _______________________________________________________ Email Address: ____________________________________________________ Class Year: ________________________________________________________

Please sign up for at least 1 time slot in column A & 1 time slot in Column B 20 October 2008

Time Slot A

27 October 2008

11:00 – 12:30

1:00 – 1:30

2:00 – 3:30

4:00 – 4:30

5:00 – 6:30

7:00 – 7:30

Time Slot B

Please sign up for at least 1 time slot in column A & 1 time slot in Column B 20 October 2008

Time Slot A

27 October 2008

1:00 – 1:30

11:00 – 12:30

4:00 – 4:30

2:00 – 3:30

7:00 – 7:30

5:00 – 6:30

Time Slot B

Even though you are signing up for multiple spots, you will only be asked to come in once on the 20th & once on the 27th Contact Information: Shannon Ferrucci [email protected] (315) 525-3967

 

   

59   

Appendix 2: IRB Research Proposal

Date Submitted:

Advisor’s Name (if applicable):

9/24/2008

Kristan Wheaton

Investigator(s):

Advisor’s Email:

Shannon Ferrucci

[email protected]

Investigator Address:

Advisor’s Signature Of Approval:

5117 Belle Village Drive East

[X]

Erie, PA 16509

Investigator(s) E-mail:

Title Of Research Project:

[email protected]

Conceptual Modeling And Intelligence Analysis

Investigator Telephone Number:

Date Of Initial Data Collection:

315-525-3967

October 20, 2008

   

60   

Please describe the proposed research and its purpose, in narrative form:

The purpose of this study is to assess whether or not explicit conceptual modeling improves collection and the subsequent analysis The more complicated a question is and the more concepts that play a part in the answering of that question the harder it is to recall all of those concepts simply from memory. Therefore, putting these concepts and their relationships to each other down on paper can be extremely useful. Explicit conceptual modeling prior to collection should help to improve the efficiency of the collection process as the model provides you with a basis for what types of information to look for. Also, since the conceptual model is not static, but a fluid diagram that evolves as you learn more about a certain topic, the model should help to highlight and minimize gaps in knowledge. These improvements in collection should then improve the subsequent analysis. Additionally, this experiment is designed to test a specific method for developing conceptual models. The approach starts at the individual level and then moves to the group collaboration level. This method supports the age old idea that two minds are better than one. Furthermore, this method should limit the problems associated with group think by encouraging equal participation from all individuals, even those who may be less inclined to voice their opinions or participate in group settings.

Indicate the materials to be used: Consent Form Debriefing Form Conceptual Modeling Training Information Research Question Free Online Conceptual Modeling Software Analytic Confidence And Source Reliability Information Format For Written Product Writing Utensils Post-Test Questionnaire Procedure:    

61   

One week prior to the start of the experiment, I will make an appearance in various undergraduate and graduate intelligence studies classes in order to promote my experiment and have students sign up. The students will be asked to provide their availability on two separate dates. I will then email them a designated time slot for both dates. Date and time assignments, as well as group assignments, will be random. On the first day of the experiment, the control group will show up for half an hour. They will be provided with a research question regarding upcoming 2008 Zambian presidential elections and a semi-structured format for a written intelligence product. I will also give them explanations of both source reliability and analytic confidence. At the end of the half hour they will be sent home and given one week to research and analyze the question posed to them. The experimental group on the other hand will be asked to come in for an hour and a half on the first day of the experiment. They will be provided with the same research question as the control group, will be given the same format for a written intelligence product and will also receive the same information regarding source reliability and analytic confidence. However, this group will undergo a small training session regarding what conceptual modeling is and how to create one on the same piece of free conceptual modeling software, such as Mindomo. After the training session the participants will each be asked to make a list of concepts off the top of their heads that they feel are relevant to the research question I provided them with. Next, we will go through and make a master list, consolidating all of the participants’ individual lists, highlighting concepts that were commonly found, significant differences in opinion and uncommon but highly useful concepts. Finally, each individual will then be asked to use this master list to create a conceptual model using software that shows the perceived relationships between concepts. Participants will then discuss the models they have created with the individual sitting next to them in order to share their ideas and see how another person visualized the same information. They will then print out a copy for themselves and a copy for me. On the second day of the experiment, a week from the first day, both the experimental and control groups will come back in. The experimental group will come in for half an hour. They will hand in their written analysis and an updated conceptual model that they have modified to reflect what they learned through their research. After doing this they will answer a short post-experiment questionnaire and will then be debriefed. The control group on the other hand will come in for an hour and a half on this day. They will hand in their written analysis and will then simply be told to visualize the concepts that were important in answering the research question using bubbles and lines. After completion they will hand in their visualization, complete a post-experiment questionnaire and then be debriefed.

   

62   

Participants who successfully complete all experiment responsibilities will receive extra credit from intelligence professors and pizza and soda will be offered to all those who participate in the study. Three second-year intelligence studies graduate students, who were not experiment participants, will then evaluate the written analyses using the same criteria that students are graded against in the Intelligence Communications class that these graduate students have already successfully completed.

1. Do you have external funding for this research (money coming from outside the College)? Yes[ ] No[X] Funding Source (if applicable): N/A

2. Will the participants in your study come from a population requiring special protection; in other words, are your subjects someone other than Mercyhurst College students (i.e., children 17-years-old or younger, elderly, criminals, welfare recipients, persons with disabilities, NCAA athletes)? Yes[ ] No[X]

If your participants include a population requiring special protection, describe how you will obtain consent from their legal guardians and/or from them directly to insure their full and free consent to participate. N/A

Indicate the approximate number of participants, the source of the participant pool, and recruitment procedures for your research: I plan to have approximately 60-80 participants. I intend to recruit undergraduate and graduate students in the intelligence studies department through sign-up sheets which I will pass around to students when I visit their classes to promote the experiment.

Will participants receive any payment or compensation for their participation in your research (this includes money, gifts, extra credit, etc.)? Yes[X] No[ ] If yes, please explain: Students will obtain extra credit from the intelligence professors willing to grant it for participation in an experiment and all participants will be offered pizza and refreshments at the end of the experiment.    

63   

3. Will the participants in your study be at any physical or psychological risk (risk is defined as any procedure that is invasive to the body, such as injections or drawing blood; any procedure that may cause undue fatigue; any procedure that may be of a sensitive nature, such as asking questions about sexual behaviors or practices) such that participants could be emotionally or mentally upset? Yes[ ] No[X]

Describe any harmful effects and/or risks to the participants' health, safety, and emotional or social well being, incurred as a result of participating in this research, and how you will insure that these risks will be mitigated: None.

4. Will the participants in your study be deceived in any way while participating in this research? Yes[ ] No[X]

If your research makes use of any deception of the respondents, state what other alternative (e.g., non-deceptive) procedures were considered and why they weren't chosen: N/A

5. Will you have a written informed consent form for participants to sign, and will you have appropriate debriefing arrangements in place? Yes[X] No[ ]

Describe how participants will be clearly and completely informed of the true nature and purpose of the research, whether deception is involved or not (submit informed consent form and debriefing statement): Prior to the start of the experiments, participants will be provided with a general overview of what will occur during the session as well as the consent form, which will also describe what is expected of them. Following the experiment participants will be asked to fill out an administrative questionnaire and will then be provided with a

   

64   

debriefing statement that will explain how the results from the session will be used (please see forms at the end of this proposal).

Please include the following statement at the bottom of your informed consent form: “Research at Mercyhurst College which involves human participants is overseen by the Institutional Review Board. Questions or problems regarding your rights as a participant should be addressed to Mr. Tim Harvey Institutional Review Board Chair; Mercyhurst College; 501 East 38th Street; Erie, Pennsylvania 16546-0001; Telephone (814) 824-3372.”

6. Describe the nature of the data you will collect and your procedures for insuring that confidentiality is maintained, both in the record keeping and presentation of this data: Names are not required for my research and thus no names will be used in the recording of the results or the presentation of my data. Names will only be used to notify professors of participation in order for them to correctly assign extra credit.

7. Identify the potential benefits of this research on research participants and humankind in general. Potential benefits include: For participants: An opportunity to practice the intelligence analysis skills they have learned in the classroom in an experiment aimed at testing the value of explicit conceptual modeling as it applies to the quality of the analysis produced. Students are often asked to complete short written intelligence assignments with quick turnaround times in Intelligence Studies courses. This experiment hopes to validate a particular method for the creation of conceptual models, which if used by intelligence students should increase efficiency in collection and accuracy in analysis.

For the Intelligence Community: Currently, collection takes up quite a large amount of an analyst’s time due to information overload. This experiment hopes to demonstrate that pre-collection conceptual modeling will not only make the collection process more efficient, it will also    

65   

help to minimize gaps in knowledge as those gaps will be recognized earlier on. This process will then in turn help to improve the analysis stemming from the collection.

   

66   

Appendix 3: Control Group Consent Form

The purpose of this research is to test the value of variations in analytic approaches as they apply to the quality of the analysis produced. Your participation involves the development of a short analytic product, the completion of a data visualization exercise and the filling out of a post-experiment questionnaire. This process will require your onsite attendance today and one week from today during the pre-determined timeslot that was designated to you. In total time spent onsite should not exceed two hours, however, some participation on your own time is required throughout the week. Your name WILL NOT appear in any information disseminated by the researcher. Your name will only be used to notify professors of your participation in order for them to assign extra credit. There are no foreseeable risks or discomforts associated with your participation in this study. Participation is voluntary and you have the right to opt out of the study at any time for any reason without penalty. I, ____________________________, acknowledge that my involvement in this research is voluntary and agree to submit my data for the purpose of this research.

_________________________________

__________________

Signature

Date

_________________________________

__________________

Printed Name

Class

Telephone Number: ____________________________________ Researcher’s Signature: ___________________________________________________

If you have any further question about this research you can contact me at [email protected] Research at Mercyhurst College which involves human participants is overseen by the Institutional Review Board. Questions or problems regarding your rights as a participant should be addressed to Tim Harvey; Institutional Review Board Chair; Mercyhurst College; 501 East 38th Street; Erie, Pennsylvania 16546-0001; Telephone (814) 824-3372. [email protected]    

67   

Appendix 4: Experimental Group Consent Form

The purpose of this research is to test the value of explicit conceptual modeling as it applies to the quality of the analysis produced. Furthermore, the following experiment will test the effectiveness of a structured individual to group conceptual modeling method. Your participation involves an instruction period and training exercise, the development of a short analytic product with accompanying model and the filling out of a post-experiment questionnaire. This process will require your onsite attendance today and one week from today during the predetermined timeslots that have been designated to you. In total, time spent onsite should not exceed two hours, however, some participation on your own time is required throughout the week. Your name WILL NOT appear in any information disseminated by the researcher. Your name will only be used to notify professors of your participation in order for them to assign extra credit. There are no foreseeable risks or discomforts associated with your participation in this study. Participation is voluntary and you have the right to opt out of the study at any time for any reason without penalty. I, ____________________________, acknowledge that my involvement in this research is voluntary and agree to submit my data for the purpose of this research.

_________________________________

__________________

Signature

Date

_________________________________

__________________

Printed Name

Class

Telephone Number: ____________________________________ Researcher’s Signature: ___________________________________________________

If you have any further question about Conceptual Modeling or this research, you can contact me at [email protected] Research at Mercyhurst College which involves human participants is overseen by the Institutional Review Board. Questions or problems regarding your rights as a participant should be addressed to Tim Harvey; Institutional Review Board Chair; Mercyhurst College; 501 East 38th Street; Erie, Pennsylvania 16546-0001; Telephone (814) 824-3372. [email protected]    

68   

Appendix 5: Research Question

Due to the 19 August 2008 death of Zambia’s President Mwanawasa, early presidential elections will take place on 30 October 2008 in accordance with the Zambian constitution that requires new elections to be held within 90 days of a president's untimely departure from office. Who will win this upcoming Zambian presidential election (Rupiah Banda, Michael Sata or Hakainde Hichilema) and why?

   

69   

Appendix 6: Important Supporting Information Source Reliability: Source Reliability reflects the accuracy and reliability of a particular source over time. Sources with high reliability have been proven to have produced accurate, consistently reliable, information in the past. Sources with low reliability lack the accuracy and proven track record commensurate with more reliable sources. o

In this experiment source reliability will be measured on a low - high scale conveying the reliability of the sources used for that piece of intelligence/report.

o

For more information regarding internet source reliability please refer to: http://www.library.jhu.edu/researchhelp/general/evaluating/

Analytic Confidence: Analytic Confidence reflects the level of confidence an analyst has in his or her estimates and analyses. It is not the same as using words of estimative probability, which indicate likelihood. It is possible for an analyst to suggest an event is virtually certain based on the available evidence, yet have a low amount of confidence in that forecast due to a variety of factors or vice versa. o

In this experiment Analytic Confidence will be measured on a low - high scale.

o

For more information regarding factors contributing to the assessment of analytic confidence see the Peterson Table of Analytic Confidence provided below.

   

Peterson Table Of Analytic Confidence Assessment

  Use Of Structured Method(s) In Analysis Overall Source Reliability Source Corroboration/Agreement: Level Of Conflict Amongst Sources Level Of Expertise On Subject/Topic & Experience Amount Of Collaboration Task Complexity Time Pressure: Time Given To Make Analysis

   

70   

Appendix 7: Experiment Answer Sheet

NAME:

FORECAST: It is (likely, highly likely, almost certain) that (Rupiah Banda, Michael Sata, Hakainde Hichilema) will win the 30 October 2008 Zambian presidential elections. BULLETED DISCUSSION:

SOURCE RELIABILITY (CIRCLE ONE): HIGH

LOW

MEDIUM

ANALYTIC CONFIDENCE (CIRCLE ONE): HIGH

LOW

MEDIUM

NAME OF PROFESSOR(S) GIVING EXTRA CREDIT:

   

71   

Appendix 8: Control Group Expectation Sheet EXPECTATIONS FOR 27 OCTOBER 2008 GROUP B

•

Completed answer sheet ready to be turned in o Inclusive of: Name, Forecast, Bulleted Discussion, Source Reliability, Analytic Confidence, Name of Professor(s) Giving Extra Credit

•

Complete short data visualization exercise onsite, based on past week of collection and analysis

•

Order of Events for 27 October 2008 (45 minute maximum) o Complete short data visualization exercise & hand in o Hand in answer sheet o Answer short post-experiment questionnaire o Pass out debriefing sheets

                     

   

72   

Appendix 9: Experimental Group Expectation Sheet EXPECTATIONS FOR 27 OCTOBER 2008 GROUP A

•

Completed answer sheet ready to be turned in o Inclusive of: Name, Forecast, Bulleted Discussion, Source Reliability, Analytic Confidence, Name of Professor(s) Giving Extra Credit

•

Updated Bubbl.us Conceptual Model o Over the course of the week please update your models as you learn more about the topic (ex. fill in knowledge gaps in the model, highlight areas that turned out to be more important than originally thought, place less focus on those areas that turned out not to be as important as originally thought). Allow your model to evolve alongside your analysis o I will ask you to sign into the computer and share the final version of your conceptual model with me once you arrive on the 27th

•

Order Of Events for 27 October 2008 (30 minute maximum) o Hand in answer sheet o Share final conceptual model o Answer short post-experiment questionnaire o Pass out debriefing sheets

   

73   

Appendix 10: Pre-Experiment Questionnaire

  Thank you for agreeing to participate in this study! Please take a few moments to answer the following questions. Your feedback is greatly appreciated.

1. Do you feel as though you will be able to dedicate a sufficient amount of time to working on this experiment over the next week? Yes

Maybe

No

2. What type of a learner do you primarily consider yourself to be? Auditory

Visual

Kinesthetic & Tactile

Unsure

3. Please rate how interested you are in this study, with 1 being not interested and 5 being extremely interested. 1

2

3

4

5

4. Please rate how useful you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful. 1

2

3

4

5

5. What do you think the purpose of this experiment is?

6. What are your reasons for participating in this experiment?  

   

74   

Appendix 11: Contact Information

CONTACT INFORMATION

Please feel free to get a hold of me during the week if you have any questions or problems! Thank you again for your participation.

Name: Shannon Ferrucci Email: [email protected] Telephone: (315) 525-3967 Location: CIRAT Lab        

 

   

75   

Appendix 12: Conceptual Modeling Lecture •

•

•

•

Everybody builds implicit models on a daily basis o We build models of the world around us ƒ For ex: when we drive to the grocery store we model the route in our minds and when we get ready each morning we model our routine o We also build models when we are faced with questions ƒ We try to come up with preliminary answers and conclusions ƒ However, we often recognize gaps in our knowledge signaling that we need more information o Our models are unique to each of us ƒ They draw from our experiences, interests, opinions, etc… ƒ For ex: If I were to ask all of you, a room of intelligence students, what first comes to mind when faced with the question of what intelligence is, your answers would likely be very different than if I went out into the community and asked the same question of the first ten people I saw. Models can become extremely complex o Intelligence requirements (questions or topics posed to the analyst by a decision maker) often entail understanding complex relationships between people, states, organizations, industries etc… o Therefore, it is very rare that an analyst will ever develop a complete model of the requirements right off the bat. Generally, an analyst is able to fill in pieces of their model with things they already know but are forced to fill in the rest with topics, which they recognize they need to understand more about. Use of Conceptual Models o Conceptual models highlight key concepts and their relationships to each other o These models appear to be a useful way for intelligence professionals to model knowledge and come to terms with compound requirements o Conceptual modeling within the field of intelligence must include both what an individual already knows about a topic and also what that individual thinks he or she needs to know about it in order to sufficiently answer the requirements posed Don’t be afraid to explore o The first construction of the conceptual model is not factual but exploratory ƒ We may identify areas that we think are important to answering the requirement, but until we have collected that information we have no way of knowing whether or not they truly are  

 

76   

ƒ

However, when this happens it should not be viewed as a setback or waste of time • Exploring various concepts helps the analyst recognize what is critical to answering the question and what is not

•

•

•

•

to a requirement, it often provides context, background and understanding of the requirement Importance of relationships o Concepts alone are not enough, relationships amongst concepts must be included in the model as well ƒ Imagine that you are modeling a drug trafficking organization. To learn about the suppliers and the runners separately is a good start. However, you must then learn about the relationship between the two. Need for explicit models o When dealing with such complex models as is usually necessary within the field of intelligence, the need to make the models explicit becomes obvious o Many psychological studies have been done suggesting that there are upper limits on our working memory ƒ Often it is cited that an individual can only hold 7 things (plus or minus 2) in their memory at any given time ƒ While there are certain exceptions to this rule, it is obvious that most intelligence models will eventually become too complex to be stored solely in memory and must instead be made explicit Importance of model to analysts o Share and compare models with fellow analysts and professionals o Assess level of confidence in analysis produced o From a managing standpoint can help in tasking a team of analysts and divvying up responsibilities o Aids efficiency in collection process and helps to identify gaps in knowledge o Useful for after the fact review o Provides a good starting point for future related questions

   

77   

Appendix 13: Bubbl.us Instruction Sheet For Experimental Group Experiment Day 1 – Group A Bubbl.us Instructional Sheet •

Please sign into computers and go to Bubbl.us

•

Click start brainstorming

•

Click save on the right hand side of the screen o Fill out the create account steps and hit submit o This will allow you to save and share you project with me later on

•

To start you will see that there is a single bubble in the center of the screen o If you click on the text saying start here you can replace that with whatever words or concept you deem appropriate o In this case, since it is the 1st bubble it is important to start by entering the specific requirements you need to answer based on the research question provided to you

•

Now if you simply place your cursor over the center of the bubble you will see a choice of 6 icons. Let’s start on the top left hand corner. o If you click on the cross with arrows you can move the bubble anywhere you like on the screen o Moving to the top right, if you click on the X your bubble will disappear (to get it back simply click the undo button on the top left of your screen) o Clicking on the middle icon on the right hand side of the bubble allows you to create a new sibling bubble (i.e. a bubble that does not spring from the 1st bubble, but is entirely separate) o The blue circles icon allows you to show directional relationships through the use of arrowed lines. By clicking on the icon and dragging your cursor to the sibling bubble you just made in the previous step you can see an example of this o Clicking on the middle bottom icon of one of the bubbles you have created allows you to make a child balloon (i.e. a bubble that does spring from a previous bubble, generally these bubbles have some sort of direct relationship to each other, with the concept in the child balloon being a sub-concept of the original parent balloon) o Lastly, clicking on the bottom left hand icon allows you to change the color of the balloon

   

78   

•

Just a few more tips: o If you click the center button at the upper left, your entire conceptual model will be centered on the page o Also, if you would like to print your conceptual model you can click the set print area button at the upper left. This will help you to ensure your entire conceptual model is within the printable area of the page o Also, to zoom in and out you can scroll up and down on your mouse or hit the plus and minus buttons on the upper left

•

Now take about 5 minutes to familiarize yourself with the software on your own o To do this begin to craft a practice conceptual model on important things to consider when buying a new car (ex. gas mileage) o Practice using the different icons that we just went over and try to incorporate each function into your conceptual model at least once o I will walk around to offer suggestions and take questions

   

79   

Appendix 14: Bubbl.us Instruction Sheet For Control Group Experiment Day 2 – Group B Bubbl.us Instructional Sheet •

Please sign into computers and go to Bubbl.us

•

Click start brainstorming

•

Click save on the right hand side of the screen o Fill out the create account steps and hit submit o This will allow you to save and share you project with me later on

•

To start you will see that there is a single bubble in the center of the screen o If you click on the text saying start here you can replace that with whatever words or concept you deem appropriate

•

Now if you simply place your cursor over the center of the bubble you will see a choice of 6 icons. Let’s start on the top left hand corner. o If you click on the cross with arrows you can move the bubble anywhere you like on the screen o Moving to the top right, if you click on the X your bubble will disappear (to get it back simply click the undo button on the top left of your screen) o Clicking on the middle icon on the right hand side of the bubble allows you to create a new sibling bubble (i.e. a bubble that does not spring from the 1st bubble, but is entirely separate) o The blue circles icon allows you to show directional relationships through the use of arrowed lines. By clicking on the icon and dragging your cursor to the sibling bubble you just made in the previous step you can see an example of this o Clicking on the middle bottom icon of one of the bubbles you have created allows you to make a child balloon (i.e. a bubble that does spring from a previous bubble, generally these bubbles have some sort of direct relationship to each other, with the child balloon being subordinate to the original parent balloon) o Lastly, clicking on the bottom left hand icon allows you to change the color of the balloon

•

Just a few more tips: o If you click the center button at the upper left, your entire conceptual model will be centered on the page o Also, to zoom in and out you can scroll up and down on your mouse or hit the plus and minus buttons on the upper left

   

80   

Appendix 15: Structured Conceptual Modeling Exercise Structured Conceptual Modeling Exercise

•

Give students 2 minutes to identify requirements and create a list of concepts they feel are relevant to those requirements o What do they think they need to learn more about in order to answer the question? ƒ Ex. Governmental Type And Structure o What are the big, moving pieces? ƒ Ex. Level Of Candidate Support

•

Come together as a group and consolidate individual lists into group list on board o Go around the room with each student reading off their list o Emphasize commonalities with plus signs o Highlight legitimate differences in opinion as food for thought o Take note of “AHA” moments ƒ Very important concept that only a select few thought of, but all recognize as essential to the question

•

Go back onto Bubbl.us and create your own conceptual model combining your individual list and thoughts with that of the collaborative group list o Remember to start with requirements and build from there o Highlight relationships between concepts and directional flow of those relationships where applicable

•

Briefly look at the way someone sitting next to you has set up their conceptual model o Take away ideas for your own o Offer suggestions or alternatives

   

81   

Appendix 16: Control Group Post-Experiment Questionnaire Follow-Up Questionnaire B

Thanks for your participation! Please take a few moments to answer the following questions. Your feedback is greatly appreciated.

1. Please rate your understanding of conceptual modeling prior to this study, with 1 being extremely low and 5 being extremely high. 1

2

3

4

5

2. Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high. 1

2

3

4

5

3. Please rate how often explicit conceptual modeling has been a part of your personal analytic process prior to this experiment, with 1 being never and 5 being every time you produce an intelligence estimate. 1

2

3

4

5

4. Please rate how often you plan to incorporate explicit conceptual modeling into your personal analytic process in the future, with 1 being never and 5 being every time you produce an intelligence estimate. 1

2

3

4

5

5. Were you able to dedicate a sufficient amount of time to working on this experiment over the past week? Yes Maybe No

   

82   

6. Please rate your interest in the study after having completed the experiment, with 1 being not interested and 5 being extremely interested. 1

2

3

4

5

7. Based on your experience in this experiment, how useful do you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful. 1

2

3

4

5

8. What do you think the purpose of this experiment was?

9. Please provide any additional comments you may have regarding conceptual modeling in general or any particular part of this experiment.

   

83   

Appendix 17: Experimental Group Post-Experiment Questionnaire Follow-Up Questionnaire A

Thanks for your participation! Please take a few moments to answer the following questions. Your feedback is greatly appreciated.

1. Please rate your understanding of conceptual modeling prior to this study, with 1 being extremely low and 5 being extremely high. 1

2

3

4

5

2. Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high. 1

2

3

4

5

3. Please rate how useful you found the conceptual modeling training provided at the beginning of this experiment to be, with 1 being not at all helpful and 5 being extremely helpful. 1

2

3

4

5

4. Please rate how often explicit conceptual modeling has been a part of your personal analytic process prior to this experiment, with 1 being never and 5 being every time you produce an intelligence estimate. 1

2

3

4

   

5

84   

5. Please rate how often you plan to incorporate explicit conceptual modeling into your personal analytic process in the future, with 1 being never and 5 being every time you produce an intelligence estimate. 1

2

3

4

5

6. Please rate whether or not you found that explicit conceptual modeling in this experiment aided you in developing a more thorough and nuanced intelligence analysis. Definitely

Somewhat

Not At All

7. Please rate how effective you think the conceptual modeling method used in this experiment, inclusive of both individual work and group collaboration, was. 1

2

3

4

5

8. Please rate how useful you found the use of the technology aid Bubbl.us to be in creating and updating your conceptual models, with 1 being not useful and 5 being extremely useful. 1

2

3

4

5

9. Were you able to dedicate a sufficient amount of time to working on this experiment over the past week? Yes

Maybe

No

10. Please rate your interest in the study after having completed the experiment, with 1 being not interested and 5 being extremely interested. 1

2

3

4

   

5

85   

11. Based on your experience in this experiment, how useful do you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful. 1

2

3

4

5

12. What do you think the purpose of this experiment was?

13. Please provide any additional comments you may have regarding conceptual modeling in general or any particular part of this experiment.

   

86   

Appendix 18: Control Group Debriefing Sheet Participation Debriefing B

Thank you for participating in this research. I appreciate your contribution and willingness to support the student research process.

This experiment was designed to test the specific part of the analytic process termed conceptual modeling. Currently there has been little research done on the topic of conceptual modeling within the field of intelligence analysis, and this study hopes to take the first of many steps in establishing the importance of explicit conceptual modeling within the analytic process.

Within the Intelligence Community collection takes up a significant amount of an analyst’s time due to information overload stemming from both open and classified sources. This experiment hopes to demonstrate that pre-collection conceptual modeling will not only make the collection process more efficient, it will also help to minimize gaps in knowledge as those gaps will be recognized earlier on. This process should then in turn help to improve the subsequent analysis.

If you have any further questions about conceptual modeling or this research you can contact me at [email protected].

   

87   

Appendix 19: Experimental Group Debriefing Sheet Participation Debriefing A

Thank you for participating in this research. I appreciate your contribution and willingness to support the student research process.

The purpose of this study was to test the value of explicit conceptual modeling as it applies to the quality of the analysis produced. Furthermore, this experiment tested the effectiveness of a structured individual to group conceptual modeling method. Currently there has been little research done on the topic of conceptual modeling within the field of intelligence analysis, and this study hopes to take the first of many steps in establishing the importance of explicit conceptual modeling within the analytic process.

Within the Intelligence Community collection takes up a significant amount of an analyst’s time due to information overload stemming from both open and classified sources. This experiment hopes to demonstrate that pre-collection conceptual modeling will not only make the collection process more efficient, it will also help to minimize gaps in knowledge as those gaps will be recognized earlier on. This process should then in turn help to improve the subsequent analysis.

If you have any further questions about conceptual modeling or this research you can contact me at [email protected].

   

88   

Appendix 20: Significance Testing Results  

***The following results are based on a 0.05 level of significance. However, due to the fact that this research is exploratory in nature, a 0.10 level of significance was deemed most appropriate and is therefore reflected in the text of this thesis.***

Results: Is there a difference between control and experimental for source reliability? Null: there is no difference between control and experimental for source reliability. Alternative: there a difference between control and experimental for source reliability. Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30. 3.00

23

24

Response

22 25

Box plot shows outliers for control group. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outliers.

2.50

2.00

1.50

1.00

1 Control

Experimental

Group

   

89   

Expected Normal

Normal Q-Q Plot of Response for group= Control

2

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

1 0 -1 -2 1.0

1.5

2.0

2.5

3.0

Observed Value

Normal Q-Q Plot of Response Expected Normal

for group= Experimental Most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

1 0 -1 -2 1.0

1.5

2.0

2.5

3.0

Observed Value Group Statistics

Response

Group Control Experimental

N 25 22

Mean 2.1200 2.2273

   

Std. Deviation .43970 .52841

Std. Error Mean .08794 .11266

90    Independent Samples Test Levene's Test for Equality of Variances F Response Equal variances assumed Equal variances not assumed

t-test for Equality of Means

Sig.

2.234

t

.142

Mean Sig. (2-tailed) Difference

df

-.760

45

.451

-.10727

-.751

41.052

.457

-.10727

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.142) > ( variances is satisfied.

= 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances F Response Equal variances assumed Equal variances not assumed

2.234

t-test for Equality of Means

Sig.

t

.142

Mean Sig. (2-tailed) Difference

df

-.760

45

.451

-.10727

-.751

41.052

.457

-.10727

According to above table, t-test value = -0.76, P-value = 0.451. Since (P-value = 0.451) > (

= 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for source reliability. Is there a difference between control and experimental for analytic confidence? Null: there is no difference between control and experimental for analytic confidence. Alternative: there a difference between control and experimental for analytic confidence. Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

   

91 

Response for Analytical Confidence

  24 25

3.00

47 45 46

Box plot shows outliers for both groups. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outliers.

2.50

2.00

1.50 21

1.00

30 29 27

Control

28

Experimental

Group

Expected Normal

Normal Q-Q Plot of Response for Analytical Confidence for group= Control 2

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

1 0 -1 -2 1.0

1.5

2.0

2.5

3.0

Observed Value

Expected Normal

Normal Q-Q Plot of Response for Analytical Confidence for group= Experimental 1.5 1.0

Most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

0.5 0.0 -0.5 -1.0 -1.5 1.0

1.5

2.0

2.5

3.0

Observed Value    

92    Group Statistics

Response for Analytical Confidence

Group Control Experimental

N 25 22

Mean 1.9600 1.9091

Std. Error Mean .09092 .13009

Std. Deviation .45461 .61016

Independent Samples Test Levene's Test for Equality of Variances F Response for Equal variances Analytical Confidence assumed Equal variances not assumed

Sig.

2.287

t-test for Equality of Means t

.137

df

Sig. (2-tailed)

Mean Difference

.327

45

.745

.05091

.321

38.491

.750

.05091

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.137) > ( variances is satisfied.

= 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances F Response for Equal variances Analytical Confidence assumed Equal variances not assumed

Sig.

2.287

.137

t-test for Equality of Means t

df

Sig. (2-tailed)

Mean Difference

.327

45

.745

.05091

.321

38.491

.750

.05091

According to above table, t-test value = 0.327, P-value = 0.745. Since (P-value = 0.745) > (

= 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for analytic confidence. Is there a difference between control and experimental for forecast results? Null: there is no difference between control and experimental for forecast results. Alternative: there a difference between control and experimental for forecast results. Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

   

93 

Response for Forecast Results

  2.00

Box plot shows no outliers for both groups.

1.80 1.60 1.40 1.20 1.00 Control

Experimental

Group

Expected Normal

Normal Q-Q Plot of Response for Forecast Results for group= Control 0.25 0.00

Most points are not close to the line thus the assumption of normality is not satisfied for the group Control.

-0.25 -0.50 -0.75 -1.00 1.0

1.2

1.4

1.6

1.8

2.0

Observed Value

Expected Normal

Normal Q-Q Plot of Response for Forecast Results for group= Experimental 0.75

Most points are not close to the line thus the assumption of normality is not satisfied for the group Experimental.

0.50 0.25 0.00 -0.25 -0.50 1.0

1.2

1.4

1.6

1.8

2.0

Observed Value

   

94   

Cannot use independent samples t- test as normality is not satisfied. Wilcoxon Rank Sum test, non-parametric test.

Need to use

Descriptive Statistics N Response for Forecast Results Group

Mean

Std. Deviation

Minimum

Maximum

47

1.5532

.50254

1.00

2.00

47

1.4681

.50437

1.00

2.00

Ranks Response for Forecast Results

Group Control Experimental Total

N

Mean Rank 26.98 20.61

25 22 47

Sum of Ranks 674.50 453.50

Test Statisticsa

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed)

Response for Forecast Results 200.500 453.500 -1.844 .065

a. Grouping Variable: Group

Wilcoxon Rank Sum test value = -1.844, P-value = 0.065. Since (P-value = 0.065) > (

= 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for forecast results. Is there a difference between control and experimental for question, “Please rate how interested you are in this study, with 1 being not interested and 5 being extremely interested.”? Null: There is no difference between control and experimental for question, “Please rate how interested you are in this study, with 1 being not interested and 5 being extremely interested.”

   

95   

Alternative: There is a difference between control and experimental for question, “Please rate how interested you are in this study, with 1 being not interested and 5 being extremely interested.” Will be using t-test for independent samples.

Response for Q. 1 in Pre-Experiment

Testing normality assumption as sample sizes are < than 30. 5.00 4.50

Box plot shows no outliers for both groups.

4.00 3.50 3.00 2.50 2.00 Control

Experimental

Group for Questionnaire Results

Expected Normal

Normal Q-Q Plot of Response for Q. 1 in Pre-Experiment for groupq= Control 0.5 0.0 -0.5 -1.0 -1.5 -2.0 2.0

2.5

3.0

3.5

4.0

Observed Value

   

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

96   

Expected Normal

Normal Q-Q Plot of Response for Q. 1 in Pre-Experiment for groupq= Experimental 2

Most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

1 0 -1 -2 2.0

2.5

3.0

3.5

4.0

4.5

5.0

Observed Value Group Statistics

Response for Q. 1 in Pre-Experiment

Group for Questionnaire Results Control Experimental

N

Mean 3.5357 3.5000

28 24

Std. Deviation .63725 .72232

Std. Error Mean .12043 .14744

Independent Samples Test Levene's Test for Equality of Variances F Response for Q. 1 in Pre-Experiment

Equal variances assumed Equal variances not assumed

.516

Sig.

t-test for Equality of Means t

.476

df

Sig. (2-tailed)

Mean Difference

.189

50

.851

.03571

.188

46.352

.852

.03571

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.476) > ( variances is satisfied.

= 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances F Response for Q. 1 in Pre-Experiment

Equal variances assumed Equal variances not assumed

.516

Sig.

t-test for Equality of Means t

.476

df

Sig. (2-tailed)

Mean Difference

.189

50

.851

.03571

.188

46.352

.852

.03571

According to above table, t-test value = 0.189, P-value = 0.851. Since (P-value = 0.851) > (

= 0.05), null hypothesis is not rejected.  

 

97   

Conclusion: At 5% level, there is no difference between control and experimental for question, “Please rate how interested you are in this study, with 1 being not interested and 5 being extremely interested.” Is there a difference between control and experimental for question, “Please rate your interest in the study after having completed the experiment, with 1 being not interested and 5 being extremely interested.”? Null: There is no difference between control and experimental for question, “Please rate your interest in the study after having completed the experiment, with 1 being not interested and 5 being extremely interested.” Alternative: There is a difference between control and experimental for question, “Please rate your interest in the study after having completed the experiment, with 1 being not interested and 5 being extremely interested.” Will be using t-test for independent samples.

Response for Q. 1 in Post-Experiment

Testing normality assumption as sample sizes are < than 30. 5.00

Box plot shows outliers for Experimental group. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outlier.

4.00 3.00 2.00 26 1.00 Control

Experimental

Group for Questionnaire Results for Post

   

98   

Expected Normal

Normal Q-Q Plot of Response for Q. 1 in Post-Experiment for grouppostq1= Control 2

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

1 0 -1 -2 2.0

2.5

3.0

3.5

4.0

4.5

5.0

Observed Value

Expected Normal

Normal Q-Q Plot of Response for Q. 1 in Post-Experiment for grouppostq1= Experimental 0.5 0.0

Except one, most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

-0.5 -1.0 -1.5 -2.0 1.0

1.5

2.0

2.5

3.0

3.5

4.0

Observed Value Independent Samples Test Levene's Test for Equality of Variances F Response for Q. 1 in Post-Experiment

Equal variances assumed Equal variances not assumed

.367

Sig. .548

t-test for Equality of Means t

df

Sig. (2-tailed)

Mean Difference

1.124

43

.267

.26000

1.107

38.079

.275

.26000

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.548) > ( variances is satisfied.

   

= 0.05), thus assumption of equal

99    Group Statistics

Response for Q. 1 in Post-Experiment

Group for Questionnaire Results for Post Control Experimental

N

Mean 3.7600 3.5000

25 20

Std. Deviation .72342 .82717

Std. Error Mean .14468 .18496

Independent Samples Test Levene's Test for Equality of Variances F Response for Q. 1 in Post-Experiment

Equal variances assumed Equal variances not assumed

.367

Sig. .548

t-test for Equality of Means t

df

Sig. (2-tailed)

Mean Difference

1.124

43

.267

.26000

1.107

38.079

.275

.26000

According to above table, t-test value = 1.124, P-value = 0.267. Since (P-value = 0.267) > (

= 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for question, “Please rate your interest in the study after having completed the experiment, with 1 being not interested and 5 being extremely interested.” Is there a difference between control and experimental for question, “Please rate how useful you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.”? Null: There is no difference between control and experimental for question, “Please rate how useful you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.” Alternative: There is a difference between control and experimental for question, “Please rate how useful you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.” Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

   

100 

Response for Q. 2 in Pre-Experiment

  28 2726 25

5.00

Box plot shows outliers for Control group. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outlier.

4.00 3.00 2.00 1 2

1.00

Control

Experimental

Group for Questionnaire Results for Pre

Expected Normal

Normal Q-Q Plot of Response for Q. 2 in Pre-Experiment for grpreq2= Control Except one, most points are close to the line thus the assumption of normality is satisfied for the group Control.

1 0 -1 -2 1

2

3

4

5

6

Observed Value

Expected Normal

Normal Q-Q Plot of Response for Q. 2 in Pre-Experiment for grpreq2= Experimental 1.0

Most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

0.5 0.0 -0.5 -1.0 -1.5 3.0

3.5

4.0

4.5

5.0

Observed Value

   

101    Independent Samples Test Levene's Test for Equality of Variances F Response for Q. 2 in Pre-Experiment

Equal variances assumed Equal variances not assumed

.378

t-test for Equality of Means

Sig.

t

.541

df

Sig. (2-tailed)

Mean Difference

-1.554

50

.127

-.38690

-1.595

48.358

.117

-.38690

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.541) > ( variances is satisfied.

= 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances F Response for Q. 2 in Pre-Experiment

Equal variances assumed Equal variances not assumed

.378

Sig.

t-test for Equality of Means t

.541

df

Sig. (2-tailed)

Mean Difference

-1.554

50

.127

-.38690

-1.595

48.358

.117

-.38690

According to above table, t-test value = -1.554, P-value = 0.127. Since (P-value = 0.127) > (

= 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for question, “Please rate how useful you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.” Is there a difference between control and experimental for question, “Based on your experience in this experiment, how useful do you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.”? Null: There is no difference between control and experimental for question, “Based on your experience in this experiment, how useful do you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.” Alternative: There is a difference between control and experimental for question, “Based on your experience in this experiment, how useful do you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.” Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

   

102   

Box plot shows outliers for Control group. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outlier.

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

Most points are close to the line thus the assumption of normality is satisfied for the group Experiment.

Group Statistics Group for Questionnai re Results for Post N

Mean

   

Std. Deviation

Std. Mean

Error

103   

Response for Q. 2 in Control 25 Post-Experiment Experiment 20 al

4.0800

.81240

.16248

4.2000

.76777

.17168

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means

Response for Q. 2 Equal in Post-Experiment variances assumed Equal variances assumed

F

Sig.

t

df

Sig. (2tailed)

.123

.727

-.504

43

.617

-.508

41.758 .614

not

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.727) > ( variances is satisfied.

= 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means

Response for Q. 2 Equal in Post-Experiment variances assumed

F

Sig.

t

df

Sig. (2tailed)

.123

.727

-.504

43

.617

   

104   

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means

Response for Q. 2 Equal in Post-Experiment variances assumed Equal variances assumed

F

Sig.

t

df

Sig. (2tailed)

.123

.727

-.504

43

.617

-.508

41.758 .614

not

According to above table, t-test value = -0.504, P-value = 0.617. Since (P-value = 0.617) > (

= 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for question, “Based on your experience in this experiment, how useful do you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.” Is there a difference between control and experimental for question, “Please rate your understanding of conceptual modeling prior to this study, with 1 being extremely low and 5 being extremely high.” Null: There is no difference between control and experimental for question, “Please rate your understanding of conceptual modeling prior to this study, with 1 being extremely low and 5 being extremely high.” Alternative: There is a difference between control and experimental for question, “Please rate your understanding of conceptual modeling prior to this study, with 1 being extremely low and 5 being extremely high.” Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

   

105   

Box plot shows no outliers for both groups.

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

Most points are close to the line thus the assumption of normality is satisfied for the group Experiment.

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means    

106   

F Response for Q. 3 Equal variances .137 in Pre- assumed Experiment Equal variances

Sig.

t

df

Sig. (2tailed)

.713

1.340

43

.187

1.335

40.189 .189

not assumed Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.713) > ( variances is satisfied.

= 0.05), thus assumption of equal

Group Statistics Group for Questionnaire Results for Q. 2, 3, and 4 N Response for Q. 3 in Control 25 Pre-Experiment Experimental 20

Mean

Std. Deviation

Std. Mean

3.5600

1.00333

.20067

3.1500

1.03999

.23255

Error

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means

F Response for Q. 3 Equal variances .137 in Pre- assumed Experiment Equal variances not assumed

   

Sig.

t

df

Sig. (2tailed)

.713

1.340

43

.187

1.335

40.189 .189

107   

According to above table, t-test value = 1.34, P-value = 0.187. Since (P-value = 0.187) > (

= 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for question, “Please rate your understanding of conceptual modeling prior to this study, with 1 being extremely low and 5 being extremely high.” Is there a difference between control and experimental for question, “Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high.”? Null: There is no difference between control and experimental for question, “Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high.” Alternative: There is a difference between control and experimental for question, “Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high.” Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30. Both Box plots show outliers. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outliers.

   

108   

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

Most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means

Response for Q. Equal 3 in Post- variances Experiment assumed

F

Sig.

t

df

Sig. tailed)

.944

.337

.000

43

1.000

.000

43.000 1.000

Equal variances not assumed

Here need to check if the assumption of equal variances is satisfied.    

(2-

109   

According to Levene’s test (P-value = 0.337) > ( variances is satisfied.

= 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means

Response for Q. Equal 3 in Post- variances Experiment assumed

F

Sig.

t

df

Sig. tailed)

.944

.337

.000

43

1.000

.000

43.000 1.000

Equal variances not assumed

(2-

According to above table, t-test value = 0.00, P-value = 1. Since (P-value = 1) > (

= 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for question, “Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high.” Is there a difference between control and experimental for question, “Please rate how often explicit conceptual modeling has been a part of your personal analytic process prior to this experiment, with 1 being never and 5 being every time you produce an intelligence estimate.”? Null: There is no difference between control and experimental for question, “Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high.” Alternative: There is a difference between control and experimental for question, “Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high.” Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.    

110   

Box plot shows no outliers for both groups.

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

Most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

Independent Samples Test    

111   

Levene's Test for Equality of Variances t-test for Equality of Means

F Response for Q. Equal variances .002 4 in Pre- assumed Experiment Equal variances

Sig.

t

df

Sig. tailed)

.968

.162

43

.872

.162

41.211 .872

(2-

not assumed Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.968) > ( variances is satisfied.

= 0.05), thus assumption of equal

Group Statistics Group for Questionnaire Results for Q. 2, 3, and 4 N Response for Q. 4 in Control 25 Pre-Experiment Experimental 20

Mean

Std. Deviation

Std. Mean

2.8000

1.04083

.20817

2.7500

1.01955

.22798

Error

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means

F Response for Q. Equal variances .002 4 in Pre- assumed Experiment Equal variances not assumed

   

Sig.

t

df

Sig. tailed)

.968

.162

43

.872

.162

41.211 .872

(2-

112   

According to above table, t-test value = 0.162, P-value = 0.872. Since (P-value = 0.872) > (

= 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for question, “Please rate how often explicit conceptual modeling has been a part of your personal analytic process prior to this experiment, with 1 being never and 5 being every time you produce an intelligence estimate.” Is there a difference between control and experimental for question, “Please rate how often you plan to incorporate explicit conceptual modeling into your personal analytic process in the future, with 1 being never and 5 being every time you produce an intelligence estimate.”? Null: There is no difference between control and experimental for question, “Please rate how often you plan to incorporate explicit conceptual modeling into your personal analytic process in the future, with 1 being never and 5 being every time you produce an intelligence estimate.” Alternative: There is a difference between control and experimental for question, “Please rate how often you plan to incorporate explicit conceptual modeling into your personal analytic process in the future, with 1 being never and 5 being every time you produce an intelligence estimate.” Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30. Box plot shows outliers for Experimental group. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outlier.

   

113   

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

Except on point, most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means

Response for Q. Equal 4 in Post- variances Experiment assumed Equal variances assumed

F

Sig.

t

df

Sig. tailed)

.086

.770

-.504

43

.617

-.501

39.631 .619

not

Here need to check if the assumption of equal variances is satisfied.    

(2-

114   

According to Levene’s test (P-value = 0.77) > ( variances is satisfied.

= 0.05), thus assumption of equal

Group Statistics Group for Questionnaire Results for Q. 2, 3, and 4 N Response for Q. 4 in Control 25 Post-Experiment Experimental 20

Mean

Std. Deviation

Std. Mean

3.4800

.77028

.15406

3.6000

.82078

.18353

Error

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means

Response for Q. Equal 4 in Post- variances Experiment assumed Equal variances assumed

F

Sig.

t

df

Sig. tailed)

.086

.770

-.504

43

.617

-.501

39.631 .619

(2-

not

According to above table, t-test value = -0.504, P-value = 0.617. Since (P-value = 0.617) > (

= 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for question, “Please rate how often you plan to incorporate explicit conceptual modeling into your personal analytic process in the future, with 1 being never and 5 being every time you produce an intelligence estimate.” Conceptual Modeling Results: Null: Number of bubbles for Pre and Post experimental CM are not different.

   

115   

Alternative: Number of bubbles for Pre and Post experimental CM are significantly different. Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30. 36

80.00

Bubbles

60.00

Box plot shows outliers for both groups. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outliers.

21

40.00

20.00

0.00 Pre

Post

Group Tests of Normality a

Bubbles

Group Pre Post

Kolmogorov-Smirnov Statistic df Sig. .183 24 .036 .193 23 .026

Statistic .927 .863

Shapiro-Wilk df 24 23

Sig. .086 .005

a. Lilliefors Significance Correction

Kolmogorov-Smirvov test gives p-values < ( = 0.05), thus normality assumption is not satisfied for both samples. So take a look at Shapiro-Wilk test. P-value for group Pre is > (( = 0.05), thus normality assumption is satisfied for group Pre. Shapiro-Wilk test gives p-values < ( = 0.05), thus normality assumption is not satisfied for group Post. Need to look at Normal probability plot for group Post.

   

116   

Normal Q-Q Plot of Bubbles Expected Normal

for group= Pre

2

Most points are close to the line thus the assumption of normality is satisfied for the group Pre.

1 0 -1 -2 0

10

20

30

40

50

60

Observed Value

Normal Q-Q Plot of Bubbles Expected Normal

for group= Post

2 1 0 -1 -2 0

20

40

60

80

Most points are close to the line with no pronounced curvature thus the assumption of normality is satisfied for the group Post.

Observed Value Group Statistics

Bubbles

Group Pre Post

N 24 23

Mean 23.0000 30.9130

Std. Deviation 11.90542 15.60569

   

Std. Error Mean 2.43018 3.25401

117    Independent Samples Test Levene's Test for Equality of Variances F Sig. Bubbles

Equal variances assumed Equal variances not assumed

1.637

.207

t

t-test for Equality of Means df Sig. (2-tailed)

-1.960

45

.056

-1.948

41.143

.058

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.207) > ( variances is satisfied.

= 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances F Sig. Bubbles

Equal variances assumed Equal variances not assumed

1.637

.207

t

t-test for Equality of Means df Sig. (2-tailed)

-1.960

45

.056

-1.948

41.143

.058

According to above table, t-test value = -1.96, P-value = 0.056. Since (P-value = 0.056) > (

= 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, Number of bubbles for Pre and Post experimental CM are not different. Null: Number of lines and arrows for Pre and Post experimental CM are not different. Alternative: Number of lines and arrows for Pre and Post experimental CM are significantly different. Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

   

118   

Lines and Arrows

70.00

36

Box plot shows outliers for Post group. Even with the presence of outliers, normality is satisfied (See below). Also this value is important for the analysis. Thus the decision is not to remove the outliers.

60.00 50.00 40.00 30.00 20.00 10.00 0.00 Pre

Post

Group Tests of Normality a

Lines and Arrows

Group Pre Post

Kolmogorov-Smirnov Statistic df Sig. .128 24 .200* .200 23 .018

Statistic .942 .895

Shapiro-Wilk df 24 23

Sig. .181 .020

*. This is a lower bound of the true significance. a. Lilliefors Significance Correction

Kolmogorov-Smirvov test gives p-values > ( = 0.05), thus normality assumption is satisfied for group Pre. Kolmogorov-Smirvov test gives p-values < ( = 0.05), thus normality assumption is not satisfied for Post. So take a look at Shapiro-Wilk test for group Post. Shapiro-Wilk test gives p-value < ( = 0.05), thus normality assumption is not satisfied for group Post. Need to look at Normal probability plot for group Post.

   

119   

Normal Q-Q Plot of Lines and Arrows Expected Normal

for group= Post

2

Most points are close to the line with no pronounced curvature thus the assumption of normality is satisfied for the group Post.

1 0 -1 -2 0

10

20

30

40

50

60

70

Observed Value Group Statistics

Lines and Arrows

Group Pre Post

N 24 23

Mean 26.5000 30.9565

Std. Deviation 12.39916 12.77952

Std. Error Mean 2.53097 2.66471

Independent Samples Test Levene's Test for Equality of Variances F Sig. Lines and Arrows

Equal variances assumed Equal variances not assumed

.142

t

.708

t-test for Equality of Means df Sig. (2-tailed)

-1.213

45

.231

-1.213

44.757

.232

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.708) > ( variances is satisfied.

= 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances F Sig. Lines and Arrows

Equal variances assumed Equal variances not assumed

.142

.708

   

t

t-test for Equality of Means df Sig. (2-tailed)

-1.213

45

.231

-1.213

44.757

.232

120   

According to above table, t-test value = -1.213, P-value = 0.231. Since (P-value = 0.231) > (

= 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, Number of lines and arrows for Pre and Post experimental CM are not different. Null: Number of bubbles for Control and Experimental CM are not different. Alternative: Number of bubbles for Control and Experimental CM are significantly different. Will be using t-test for independent samples.

Bubbles

Testing normality assumption as sample sizes are < than 30. 80.00

60

60.00

53

40.00 23 20.00

0.00 Control

Box plot shows outliers for both groups. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outliers.

Experimental

Group Tests of Normality a

Bubbles

Group Control Experimental

Kolmogorov-Smirnov Statistic df Sig. .202 24 .012 .191 47 .000

Statistic .920 .893

Shapiro-Wilk df 24 47

Sig. .057 .000

a. Lilliefors Significance Correction

Kolmogorov-Smirvov test gives p-values < ( = 0.05), thus normality assumption is not satisfied for both samples. So take a look at Shapiro-Wilk test. P-value for group Control is > (( = 0.05), thus normality assumption is satisfied for group Control. Shapiro-Wilk test gives p-values < ( = 0.05), thus normality assumption is not satisfied for group Experimental. Need to look at Normal probability plot for group Experimental.    

121   

Normal Q-Q Plot of Bubbles Expected Normal

for groupce= Experimental

4

Most points are close to the line with no pronounced curvature thus the assumption of normality is satisfied for the group Experimental.

2 0 -2 -4 0

20

40

60

80

Observed Value Group Statistics

Bubbles

Group Control Experimental

N 24 47

Mean 12.5833 26.8723

Std. Deviation 3.46306 14.25942

Std. Error Mean .70689 2.07995

Independent Samples Test Levene's Test for Equality of Variances F Sig. Bubbles

Equal variances assumed Equal variances not assumed

17.117

.000

t

t-test for Equality of Means df Sig. (2-tailed)

-4.821

69

.000

-6.504

55.753

.000

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.000) < ( variances is not satisfied.

   

= 0.05), thus assumption of equal

122    Independent Samples Test Levene's Test for Equality of Variances F Sig. Bubbles

Equal variances assumed Equal variances not assumed

17.117

.000

t

t-test for Equality of Means df Sig. (2-tailed)

-4.821

69

.000

-6.504

55.753

.000

According to above table, t-test value = -6.504, P-value = 0.000. Since (P-value = 0.000) < (

= 0.05), null hypothesis is rejected.

Conclusion: At 5% level, Number of bubbles for Control and Experimental CM are significantly different. Null: Number of lines and arrows for Control and Experimental CM are not different. Alternative: Number of lines and arrows for Control and Experimental CM are significantly different. Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

Lines and Arrows

70.00

59

60.00 50.00 40.00 30.00

23

20.00 10.00 0.00 Control

Experimental

Group

   

Box plot shows outliers for both groups. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outliers.

123    Tests of Normality a

Lines and Arrows

Group Control Experimental

Kolmogorov-Smirnov Statistic df Sig. .129 24 .200* .174 46 .001

Statistic .942 .943

Shapiro-Wilk df 24 46

Sig. .182 .025

*. This is a lower bound of the true significance. a. Lilliefors Significance Correction

Kolmogorov-Smirvov test gives p-values > ( = 0.05), thus normality assumption is satisfied for group Control. Kolmogorov-Smirvov test gives p-values < ( = 0.05), thus normality assumption is not satisfied for Experimetnal. So take a look at Shapiro-Wilk test for group Experimental. Shapiro-Wilk test gives p-value < ( = 0.05), thus normality assumption is not satisfied for group Experimental. Need to look at Normal probability plot for group Experimental.

Normal Q-Q Plot of Lines and Arrows Expected Normal

for groupce= Experimental

4

Most points are close to the line with no pronounced curvature thus the assumption of normality is satisfied for the group Experimental.

2 0 -2 -4 0

10

20

30

40

50

60

70

Observed Value Group Statistics

Lines and Arrows

Group Control Experimental

N

Mean 12.9167 29.3043

24 46

   

Std. Deviation 3.88885 12.03859

Std. Error Mean .79381 1.77499

124    Independent Samples Test Levene's Test for Equality of Variances F Sig. Lines and Arrows

Equal variances assumed Equal variances not assumed

17.268

t-test for Equality of Means t df Sig. (2-tailed)

.000

-6.475

68

.000

-8.428

60.097

.000

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.000) < ( variances is not satisfied.

= 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances F Sig. Lines and Arrows

Equal variances assumed Equal variances not assumed

17.268

.000

t

t-test for Equality of Means df Sig. (2-tailed)

-6.475

68

.000

-8.428

60.097

.000

According to above table, t-test value = -8.428, P-value = 0.000. Since (P-value = 0.000) < (

= 0.05), null hypothesis is rejected.

Conclusion: At 5% level, Number of lines and arrows for Control and Experimental CM are significantly different.