Page 1 Concept Mapping As On-site Assessment Of Bms Pbl

Page 1 Concept Mapping as On-site Assessment of BMS PBL

1. Project Summary (Abstract) The School of Medicine at Atma Jaya Catholic University Indonesia has been implementing problem-based learning (PBL) as the main teaching and learning process since 2006. The Biomedical Sciences (BMS) occupy two unit blocks in the first semester, with each unit block lasting up to six weeks. The faculty members involved in BMS blocks are from multidisciplinary backgrounds, including biology, chemistry, biochemistry, physiology, and physics. Assessment of PBL has some problems including the practicality of on-site assessment, the matched assessment with the PBL philosophy, and the multidisciplinary faculty involved in designing PBL cases. PBL consists of two group meetings. At the end of the first meeting, the students produce a drawing scheme that consists of the idea and learning objectives the group wants to achieve and those the faculty have already set. The second meeting is a group’s sharing of what each of the members has been studying . This meeting produces a drawing scheme with all the knowledge that the group has discussed. Between the first and the second meeting, students have to study individually with various learning resources. The steps in the group meetings require students to put a structured cognitive learning process into diagrams. Therefore, incorporation of concept mapping in PBL is reasonable and concept mapping may potentially be the on-site assessment of PBL. The concept map has been claimed as a tool related to meaningful learning. Developed in accordance with Ausubel’s assimilation theory , meaningful learning has become a focal tenet in cognitive acquisition. Research on concept maps as assessments and learning processes has been widely explored . The main comprehensive review on the issues and problems of concept maps as assessment tools was written by Ruiz-Primo and Shavelson . Concept maps as assessment tools need to have an explicit task, a format for the students’ response, and a scoring system. For the latter requirement, determining the validity and reliability of the scoring system has been problematic. This proposal requests support to research the application of concept maps in BMS PBL and to develop a rubric scoring system based on the stakeholders’ view of concept mapping as an on-site PBL assessment (the term ‘stakeholders’ refers students, PBL facilitators, and PBL case designers). The rubric scoring system will not merely be based on the numerical calculation of how many nodes, links, or correct propositions that may be applied to the concept map. Regarding the issue of the scoring system, this proposal intends to conduct an in-depth development of a valid and reliable rubric to assess PBL concept maps. This proposal aims to search for pieces of evidence to support the implementation of concept maps as on-site PBL assessments and to develop a rubric to assess the concept map in BMS PBL. A mixed-methodology study will be utilized to answer the following research questions: 1. What are students’ perceptions in learning through PBL with or without concept mapping? 2. Are there any differences in exam scores between PBL students with or without concept mapping? 3. What are the stakeholders’ perceptions of concept mapping as an on-site PBL assessment? 4. What kind of rubric can score a concept map in BMS PBL according to the stakeholders’ points of view? Quantitative data will consist of students’ exams scores and qualitative data will be gathered from interviews and focus groups made up of stakeholders. Findings from this research will contribute to PBL progression by incorporating concept mapping. The produced rubric may be potentially applied to other PBL subsets, such as case-based

Page 2 Concept Mapping as On-site Assessment of BMS PBL

learning and bed-side teaching in clinical education. Furthermore, the rubric may provide another instance of concept mapping as an assessment tool.

2. Project Description 2.1. Project Objectives (Goal) The goals of this proposal are to investigate the stakeholders’ perceptions of concept mapping in BMS PBL and to develop a rubric that can be used to assess concept mapping as onsite PBL assessment. To investigate the effect of concept mapping in PBL, we will collect data from students who learn through PBL with and without concept mapping. The rubric will be developed based on the stakeholders’ views. ‘Stakeholders’ refer to students, case designers, and PBL facilitators. The produced rubric will be used to assess concept mapping in BMS PBL and the stakeholders’ comments on using the rubric will be collected. A mixed-method study will be utilized to answer the following research questions: (1) What are students’ perceptions in learning through PBL with or without concept mapping? (2) Are there any differences in exam scores between PBL students with or without concept mapping? (3) What are the stakeholders’ perceptions of concept mapping as an on-site PBL assessment? (4) What kind of rubric can score a concept map in BMS PBL according to the stakeholders’ points of view? Findings from this research will contribute to understanding concept mapping as both a learning tool and as an on-site PBL assessment. BMS is a multidisciplinary subject that can cause difficulty for first-year students at schools of medicine, but concept mapping as a potentially meaningful learning tool may help students to overcome the difficulties. The rubric produced from this research will be based on stakeholders’ views and will be practical as an on-site PBL assessment. The department will benefit from this rubric as a valid and reliable assessment tool. Other subsets of PBL could also benefit from the application of the rubric design. 2.2. Introduction Students enter medical schools with the conception that they can memorize information. PBL supposedly brings more meaningful learning to the classroom experience and intends to help students overcome the tendency to rely on rote memorization. PBL uses problems to approach teaching and the learning process . Using PBL, students can move toward knowledge acquisition through problem-solving. Problem-based learning is started by exposing students to a real problem rather than the traditional lecture . 2.2.1. PBL at the School of Medicine Atma Jaya Catholic University Indonesia Our curriculum, in which we use PBL and other teaching and learning methods, is often called a hybrid model . Curriculum at our school is divided into unit blocks. Each of the unit blocks has four approaches to teaching and learning: PBL, lecture, laboratory activities, and medical skills laboratory. Each of the unit blocks has (1) a certain theme, (2) up to five PBL activities, and (3) occupies up to six weeks. BMS in the first semester occupies two unit blocks: BMS 1 and BMS 2. BMS 1 discusses cells and nutrition; it incorporates biology, chemistry, microbiology, and physics. BMS 2’s theme is genetics and metabolism. Biology, chemistry, biochemistry, and physiology topics contribute to BMS 2. In PBL, students meet twice a week in groups made up of nine to eleven students. The PBL group is arbitrarily chosen by the department and its composition changes after one semester. Each group is facilitated by a faculty member,

Page 3 Concept Mapping as On-site Assessment of BMS PBL

known as a “facilitator.” The facilitator has the task of making sure that the group can achieve the learning objectives set by the case designer or faculty. The facilitator is not a person who provides answers if the students are unable to answer a question themselves; instead, he or she guides the group in purposeful cooperation with the problem presented. In the first meeting, students would solve a problem by using their prior knowledge. The purpose of this first attempt is to activate prior knowledge and to build a problem solving scheme. At the end of the first meeting, students formulate learning objectives for self-directed learning activities. The facilitator must make sure that the learning objectives set by the faculty are included in students’ learning objectives. The self-directed study can be done individually through various learning resources or by an informal group meeting of two to five students. The time allocated for the self-directed study is two days. Subsequently, students meet again and share their self-directed studies, exchange their points of view, and clarify issues that may be unclear to the group. At the end of the second meeting, the group will produce a scheme of the shared information. PBL at our school incorporates what we call “7 Jumps in PBL” and was adapted from Maastricht University . The 7 jumps are: (1) identifying and clarifying the terminologies embedded in the scenario, (2) defining the problems to be discussed, (3) brainstorming possible problem-solving steps based on students’ prior knowledge, (4) arranging tentative explanations by using a concept or a problem-solving organizer, (5) defining the learning objectives, (6) selfdirected study, and (7) sharing information in the group. The first meeting ends with the fifth step, while the second meeting primarily consists of the last step. 2.2.2. The concept map in PBL. A concept map links prior knowledge and new knowledge in a meaningful way. "… A concept map is a schematic device for representing a set of concept meanings embedded in a framework of propositions." . At the end of the first PBL meeting, students will have a tentative scheme through which they can learn and solve the problem. Based on this tentative scheme and the learning objectives, students conduct self-directed learning for two days. At the second meeting, students share their knowledge based on their self-directed study. The group starts with their previous scheme and from there develop the wider and deeper scheme. The schemes produced by the students are mainly connected concepts without propositions. The drawing scheme in PBL may represent the students’ cognitive structure, but the concept map has more potential, due to the inclusion of propositions; the important difference between the general schemes and the concept map. “Propositions are two or more concept labels linked by words in a semantic unit.” . The concept map has nodes (concepts) and linking words to form propositions. The concept map created by the end of the first and second meetings will help students in specific ways. At the end of the first meeting, the concept map functions as an “advance organizer” that arranges tentative solutions based on their prior knowledge and as a reminder of what they agree to learn further. By the end of the second meeting, the concept map functions as a visual summary to what they have learned and discussed. The concept map will help students in PBL to connect each idea among integrated knowledge and thus solve the PBL case. Some studies discussed the impact of PBL and concept mapping . Hsu and Hsieh (2005) investigated the effect of concept mapping in PBL students versus traditional (lecture-based) students in an advanced nursing course. The authors found that the score in the levels of hierarchy of concept maps is higher for PBL students. Johnstone and Otis investigated the application of concept mapping in PBL and proved that students using concept mapping as their learning strategy performed better than students who did not use concept mapping. The authors encouraged medical

Page 4 Concept Mapping as On-site Assessment of BMS PBL

students in PBL to use concept maps as part of their learning “to plan their work, to make explicit their own mental network and to prepare materials for later formal assessment. The maps are not themselves assessed, but they are aids for revision and inter-linking of knowledge” (p. 85). This quote implies that the authors did not apply concept mapping as an assessment tool. Unlike Novak and Gowin’s (1984) hierarchical concept map, concept maps in some disciplines do not always show hierarchical relationship . Yin et al. summarized five forms of concept maps: linear, circular, hub/spokes, tree, and network concept maps. Moni et al. included a network concept map that was produced by a group of medical students as an assessment of the cardiovascular system. The nature of each discipline directs the form of the concept map. Biomedical sciences incorporate biology, chemistry, physics, biochemistry and physiology. Chemistry, physiology, and physics often incorporate network concept maps . Thus, BMS PBL may therefore adopt a network concept map. The potential usage of concept mapping in BMS PBL may require additional effort. Novak (1998) wrote, Unfortunately, we have found some of the greatest resistance to use of concept maps among medical students for whom meaningful learning is essential if they are to perform competently. Their previous successes with rote-learning approaches make them very insecure in moving to meaningful learning strategies. (p. 195)

In PBL, students have had experience drawing tentative explanations using diagram, without any propositions. PBL case designers and facilitators have been trained in using diagrams for use during PBL meetings, but not using concept maps. The active endeavors that students put in PBL with concept mapping need to be appreciated by the faculty members. On-site assessment is one way to appreciate students’ effort. However, concept mapping as an on-site PBL assessment needs to be both valid and reliable. Furthermore, the faculty members involved in the course as case designers and facilitators need evidence supporting whether their efforts in incorporating concept maps as an on-site PBL assessment are valuable. 2.3. Statement of Problems 2.3.1. The Problems with PBL assessment Boud and Filletti highlighted “the need for good qualitative studies” (p. 10) in their introduction of the second edition of The Challenge of Problem-Based Learning. They suggest areas in PBL that need to be investigated more; one of them is the “appropriate problem-solving methods for student assessment” (p. 10). Boud and Filletti also recommend finding answers for: “How can we best measure problem-solving processes and outcomes before and after graduation? What difference can we confidently say it makes to the learners, the faculty, the institution and/or the profession in the short- and long-term?” (pp. 10-11). These statements bring an indication of necessity of rigorous research in PBL assessment. PBL assessment is divided into two main categories: process-oriented and outcomeoriented assessments . Process-oriented assessment often consists of formative rather than summative assessment, while outcome-oriented assessment includes both formative and summative assessments. Several common PBL assessments are peer assessment, facilitator assessment, self-assessment, free recall, modified essay question (MEQ), short answer question (SAQ), multiple-choice question (MCQ), case-based examination, standardized patient simulation (SMP), patient management problems (PMP), portfolio, and log books . SMP and PMP are often

Page 5 Concept Mapping as On-site Assessment of BMS PBL

applied in PBL clinical studies. Swanson et al. (1997) argue that portfolios and log books are more appropriate as curriculum evaluations rather than as students’ assessment tools. Although peer, facilitator, and self-assessments are in accordance with PBL philosophy, some investigations show that both students and faculty are reluctant to do these types of assessments and the validity of these assessments is questioned . These types of assessments are acceptable to assess attitude and skills (team-work), but not the knowledge acquisition in PBL . The School of Medicine at Atma Jaya Catholic University Indonesia assesses formative and summative written exams mainly by MEQ and MCQ. A formative written exam is not an obligation for each unit block, thus the faculty may only choose to do a summative written exam at the end of the unit block. On the other hand, on-site PBL assessment (as part of process-oriented assessment) assesses both on skills (team work) and attitude based on peer, facilitator, and selfassessments. There is a gap between on-site and written assessments. This gap in the lack of onsite PBL knowledge assessment may deter students in PBL meetings. Students may not find meaningful learning during on-site PBL. It is possible that students choose to put their maximum effort in the summative written exam rather than in the PBL meeting. We realize that we do need to assess the on-site PBL knowledge acquisition and problem-solving process. The lack of standardized on-site PBL assessment that particularly assesses students’ knowledge acquisition and problem solving skills is the main problem. Standardized assessment tools have to consider validity and reliability. We need to think about practicability, too, in designing an on-site PBL assessment because the PBL facilitator has to be able to apply the tool directly in a PBL group meeting. The facilitator has important tasks to make sure that the group learns and achieves at least the learning objectives set by the faculty. At the same time, the facilitator has to assure that each student participates in the discussion, shows attentiveness, and understands the learning process during PBL. 2.3.2. The concept map and on-site PBL assessment Segers summarizes three characteristics of assessments that match with the PBL philosophy: (1) the assessment should be based on the problem presented, (2) the assessment is an acquisition and application of knowledge, and (3) the tool can assess integrated knowledge. The nature of concept mapping to show both knowledge acquisition and problem-solving skills supports its potential application as a PBL assessment. The concept map in PBL is obviously generated from the problem and it has potential to show knowledge integration. The concept map has been used in a wide range of disciplines as an assessment tool . However, concept mapping as an on-site PBL assessment has not yet been investigated. What evidence do we need to support implementing concept mapping in PBL? This question needs to be answered before concept mapping could be implemented as an on-site PBL assessment. Both students and faculty members want convincing reasons as to why they need concept mapping as an on-site PBL assessment. To find a valid and reliable assessment tool, a rigorous study and design of the assessment needs to be done. The stakeholders’ perspectives need to be revealed in order to design a trustworthy assessment tool. Their participations in this proposed research will answer not only the PI’s research questions, but also their own concerns about creating a rubric to assess concept mapping as an on-site PBL assessment. One of the obvious problems of concept mapping as an assessment tool is the scoring system. What items should be scored from the concept map? How to score the concept map? More importantly, do we need to score the concept map numerically? “Scoring was in many respects irrelevant, for we were looking for qualitative changes in the structure of children’s concept maps. But because we live in a numbers-oriented society, most students and teachers want to score

Page 6 Concept Mapping as On-site Assessment of BMS PBL

concept maps” . This proposal intends to investigate the stakeholders’ view in creating a rubric, which will combine qualitative criteria and a simple numerical scoring system to assess the concept map. 2.4. Theoretical Framework: 2.4.1. Constructivism, Metacognition, and PBL Bodner, Klobuchar, and Geelan argue about the traditional theory of learning versus constructivist theories of learning. In the traditional theory, human minds contain images that can be true if they correspond to reality. A teacher is the only source of knowledge in teaching and learning processes according to the traditional theory. Knowledge is transferred to the students by the instructor. In this theory, assessment is based on the comparison of the students' answers to the instructor's answers. Students’ answers are ‘true’ when they are exactly the same as the instructor’s answers. On the contrary, the constructivist believes Knowledge is constructed in the mind of the learner. … . It is only the implications of this phrase that cause trouble because this assumption leads one inexorably along the path to a corollary assumption: Knowledge is seldom transferred intact from the mind of the teacher to the mind of the student. A second, more radical, form of the constructivist theory has been summarized as follows: Useful knowledge is never transferred intact.

The quote implies that the constructivist believes in a continual process throughout the learner’s experience. It also means that the constructivist values the utility of knowledge: “… Knowledge should no longer be judged in terms of whether it is true or false, but in terms of whether it works" . Based on these assumptions, teaching shifts toward the learner’s choice or the student-centered approach. Teachers’ roles change to facilitators, “…from teaching by imposition to teaching by negotiation" . Learning is shifted toward searching for the utility of the knowledge and toward discovery . PBL is an example of discovery learning. “PBL derives from the theory that learning is a process in which the learner actively constructs knowledge.” . PBL is an instructional method that incorporates the problem and problem-solving skills through small group discussion . Gijselaers summarizes three principles of learning theory that become the framework for PBL. The principles are learning happens in a constructive way, is metacognitive, and is influenced by social and contextual factors. Metacognition is thinking about one’s own thought. PBL involves real problems as the trigger of learning, thus solving problems is one of the learning goals. To solve these problems, students need to incorporate metacognitive skills, which consist of planning, monitoring, and evaluating skills . PBL’s “7 Jumps” adopts the same principle of metacognition to solve problems . Planning skills are included in the first and the second steps: identifying key terminologies and defining problems. Monitoring skills are in the third to the fifth steps; those are brainstorming the solution, arranging the tentative solution, and defining learning objectives. The sixth step, which is self-directed learning, includes all of the metacognitive skills, assuming that students plan how to solve the problem, matching (monitoring) their self-study to the learning objectives, and evaluating their study by consulting the various learning resources. During the second meeting of PBL, which is the seventh step, students share their self-directed study to the group. This last step incorporates at least monitoring and evaluating skills through sharing and assessing the information gathered.

Page 7 Concept Mapping as On-site Assessment of BMS PBL

PBL utilizes problems or real cases as triggers for knowledge acquisition. This fact concerns mediated learning, in which social and contextual factors arbitrate the prior and the new knowledge. Vygotsky argues that learning and knowledge development are processes of mediation . He suggests that in mediated learning there is the Zone of Proximal Development (ZPD): It is the distance between the actual developmental level as determined by independent problem solving and the level of potential development as determined through problem solving under adult guidance or in collaboration with more capable peers .

Daniels elaborates the ZPD principle into factual implications in pedagogy. One of the important implications is the possibility of the ZPD to increase scientific understandings. The importance of the interplay between the scientific concepts derived in theoretical learning and the spontaneous concepts formed in empirical learning is central to this account of development. If the two forms do not “connect,” then true concept development does not take place. Thus, theoretically driven content-based teaching that is not designed to connect with learners’ everyday empirical learning will remain inert and developmentally ineffective. (p. 314)

Thus, Daniels emphasizes how social and contextual factors are important to connect scientific concepts and real problems. Further implications of the ZPD in a teaching method or an instructional design are cooperation and collaboration . “Adult guidance” or “more capable peers” that are required by Vygotsky should be set in a teaching and learning process where learners are situated to contribute knowledge to each other. For Vygotsky, this situation should become a scientific practice on a day-to-day basis between learners and adult guides or peers. Subsequently, this good habit will become common place every time learners want to solve problems. In this sense, the ZPD will increase the learning opportunity. PBL meetings have the potential to accommodate this practice. The facilitator as the adult guidance collaborates with peers in PBL meetings to solve real problems in scientific practices. 2.4.2. Constructivism, Meaningful Learning, and PBL Ausubel developed the theory of meaningful learning . Ausubel made clear the distinction between meaningful learning and rote learning . Although there are two distinctions, Ausubel emphasizes that rote learning and meaningful learning lie on either end of a continuum line. It is almost impossible that a student learns new knowledge through completely rote learning or completely meaningful learning . Meaningful learning incorporates non-arbitratry, substantive, and non-verbatim new knowledge. On the contrary, rote learning incorporates arbitrary, nonsubstantive, and verbatim new knowledge. Meaningful learners choose the appropriate knowledge that connects to the prior knowledge (non-arbitrary), while rote learners choose to memorize the knowledge (arbitrary). Substantive learning involves one’s awareness to choose the information that relates to the real problems that the learner encounters. The meaningful learner successfully incorporates non-arbitrary and substantive information and thus he or she is able to generate nonverbatim new knowledge. On the other hand, the rote learner falls into memorization and nonsubstantive information; both of these facts lead the rote learner to a technical definition or verbatim information without knowing its essence . The differences between meaningful and rote learning bring us to the three conditions that are required to successfully incorporate information through meaningful learning . Two of these conditions are totally controlled by learners; these are the possession of prior knowledge and the

Page 8 Concept Mapping as On-site Assessment of BMS PBL

consciousness to choose new meaningful material. The other requirement for meaningful learning is that the material to be learned must be meaningful. Bretz emphasizes that only the last requirement of meaningful learning comes purely from the teacher or instructor. PBL has been claimed as one of the meaningful approaches to learning . “PBL facilitates a constructivist approach to learning. When generating learning issues, students make use of existing or prior knowledge to identify what they still need to learn.” . That is to say, meaningful learning occurs through ascertaining students’ prior knowledge. The second requirement of meaningful learning comes from the learner; PBL as a group discussion may enhance the motivation among learners . PBL is also an approach which integrates knowledge from more than one discipline. This is in agreement with one key concept of Ausubel’s theory of learning. Novak calls the principles “Seven key concepts in Ausubel learning theory.” One principle is “Integrative reconciliation” , which provides more meaningful learning through an integration of several knowledge-domains. 2.4.3. Concept Mapping and Meaningful Learning The concept map is a representative way to show meaningful relationships between concepts . Concepts are defined as regularities in events or objects designated by some arbitrary label. ... Most concepts acquire meaning through propositions, which are two or more concepts linked together, such as a gram is a unit of mass. Concepts grow in meaning as an individual learns more new propositions in which a given concept is embedded. (p. 607)

Novak and Gowin emphasize the importance of propositions in the concept map. According to them, learners incorporate prior knowledge in the form of concepts that are already familiar to them. But, when they go further as meaningful learners, they need to organize those concepts. In the concepts organization, propositions will be needed to identify the relationships among concepts. By creating concept maps, students may realize that some concepts have not fully developed. Therefore, concept mapping is a powerful tool to recognize a learner’s prior knowledge and possible misconceptions. Concept mapping can give information toward: “(1) the concepts already present in a student’s mind (initial concepts); (2) the conceptual linkages between the concepts (context); (3) the evolution that takes place as a consequence of teaching/learning activities (conceptual change)” (Regis, 1996, p. 1084). In relation with other learners, Bretz points out that, “the power of concept maps lies in their abilities to not only represent the meaning of knowledge to a particular person but to visually represent it to another person” (p. 8). Furthermore, the act of constructing and reconstructing the concept map with proper propositions produces a non-arbitrary, substantive, and non-verbatim meaningful learning experience as oppose to rote learning . To call attention to its importance, the concept map also signifies metacognitive representations of the learners through a concrete, two-dimensional diagram with cross linking structures . Research in teaching and learning chemistry showed the importance of concept mapping as part of meaningful learning. Pendley et al. (1994) connected concept mapping and meaningful learning by constructing concept maps from students’ interviews in physical/analytical chemistry about the concepts related to chromatography. Nakleh (1994) developed two types of concept maps based on the initial and final interviews from the research participants about acid-base titration. The author recommended using concept mapping in pre- and post-lab as tools to understanding the conceptual change before and after the lab activities. Sket and Glazar (2005)

Page 9 Concept Mapping as On-site Assessment of BMS PBL

applied the concept map in teaching organic chemistry to show the interconnection of organic reactions of hydrocarbons, organic halides and organic oxygen-containing compounds. The authors designed the concept map (Scheme 1, Sket & Glazar, 2005, p. 473) so that students could read vertically or horizontally. The vertical side represented the changes of carbon’s oxidation number from – 4 to + 4. The vertical side could be read from top to bottom as oxidation reactions or the reverse order as reduction reactions. The horizontal side showed functional groups transformations. The concept map could also be read from any part of the reactions. According to Novak and Gowin a concept map should (1) be hierarchically arranged, (2) have labels on the links between concepts, and (3) have cross-links that identify relations between branches of concepts. Meanwhile, according to Deese’s association theory (as cited by Ruiz-Primo & Shavelson, 1996, p. 571) the concept map can be a network of concepts. The nature of the concept map depends on the nature of the knowledge domain . Thus, the concept map is not always in a hierarchical form. “Network theory, then, places requirements on concept mapping very similar to those in Ausubel’s theory with the important exception that maps do not have to be hierarchical.” . The argument about hierarchical or non-hierarchical concept mapping enters the discussions when one wants to apply concept mapping as an assessment tool. 2.4.5. Concept Mapping as an Assessment Tool Ruiz-Primo and Shavelson (1996) argue that the concept map can be used as an assessment tool to measure students’ cognitive structures. They emphasize the terminology of the concept map as an assessment and not merely as a test. They argue, “If, however, the purpose is to use concept maps alone as a procedure for describing a student’s knowledge structure on a numeric scale, concept map tests would be a more appropriate term” . The authors continue with the three main criteria to consider the concept map as an assessment: “(a) a task that invites students to provide evidence bearing on their knowledge structure in a domain, (b) a format for the students’ response, and (c) a scoring system by which students’ concept maps can be evaluated accurately and consistently” (p. 573). A task is a set of explanations or information that students need to construct to show their knowledge. A response format is the product that will be assessed; this can be a drawn concept map, an oral explanation, and/or a computer-generated map. Ruiz-Primo and Shavelson summarize three scoring systems: scoring on the component of students’ maps, scoring based on comparison between students’ and the expert’s concept maps, and the combination of the first and the second systems. Novak and Gowin’s scoring system and the modification of their system have well-defined scoring systems that can score the component of students’ concept maps. But, these scoring systems are based on hierarchical concept maps. Ruiz-Primo and Shavelson cited the work of Lomask as a good scoring system by comparing a student’s concept map to the expert’s concept map. Lomask (as cited by Ruiz-Primo & Shavelson, 1996, p. 582-583) compares the counting of terms and strengths of links in a student’s versus expert’s concept maps in percentage. This scoring system may work for both hierarchical and nonhierarchical concept maps. A further issue with scoring systems is whether it is appropriate to only assess students’ concept maps numerically as opposed to the more qualitative approach to assessing concept maps. Stuart suggests using a more qualitative approach toward the assessment of concept maps: If concept maps are to be of greater utility, for educational research and as an aid to teaching and learning, a more holistic and qualitative scoring technique needs to be developed. This will not be easy. The maps are both idiosyncratic and rich. Schemes focusing on the linguistic structure of the propositions; on the nature of the interrelationships between branches of the map; and on the

Page 10 Concept Mapping as On-site Assessment of BMS PBL

sequence of nodes in maps need to be developed. To continue to rely on numerical scores, whether the different components are aggregated or treated separately, is to risk missing research data of great value as well as diagnostic data used to help the pupil. (p. 80)

A more qualitative approach in the concept map scoring system is close to the quality of propositions in concept mapping. “… [We] favor scoring criteria that focus more on the adequacy of the propositions over those that focus simply on counting the number of map component (i.e., nodes and links).” . Some researchers apply more qualitative approaches to assess concept mapping. Kinchin et al. provided guidance to assess “annotations” or brief written explanations in the concept map. Nicoll et al. developed a three-tier system for analyzing concept maps in their research purpose. The authors stated that the system could be used to assess students' concept maps. However, the "three-tier" (utility, stability, and complexity) is less practical for the classroom’s or group’s assessment. Regis and Albertazzi developed an action research in the application of the concept map. The authors focused on changes in "content and organization of concept maps over time..." (p. 1088). The authors wrote that this application was congruent with constructivism. In medical education, West’s group used the concept map as an assessment tool for postgraduate medical students. The authors compared two scoring systems, structural and relational scoring systems. The structural scoring system was adopted from Novak and Gowin’s scoring system, which calculated the concept-link, cross-link, hierarchy, and examples. The relational scoring system was adopted from McClure’s scoring system , which scored the propositions. West et al. claimed that a relational scoring system was more qualitative and adaptable to the concept map’s scoring system toward postgraduate medical students, while the structural method was appropriate to test the knowledge development of pre- and post- instructional designs toward postgraduate medical students. The idea of using a rubric to assess concept mapping is in accordance with the investigation toward a more qualitative scoring system for the concept map. Ruiz-Primo and Shavelson indicated that the concept map as an assessment tool should be practical. Thus, a rubric is also in accordance with practicality. Dorough and Rye suggested using a rubric in the concept map assessment. They suggested including criteria in a concept mapping rubric: (1) number of relevant concepts, (2) number of valid propositions, (3) branching, (4) number of appropriate cross-links or the degree of integrative and meaningful connections between concepts, and (5) number of examples of specific concepts (p. 39). Stoddart et al. applied a scoring system with a rubric to assess elementary science inquiry based on three main variables of each propositions: accuracy, explanation, and complexity. The authors provided detailed information for each of the variables. For example, variable accuracy had four categories: scientific accuracy, common knowledge, inaccurate statements, and affective statements. In the conclusion, the authors wrote: “The rubric could also be modified by adding a variable to measure the relevance of the concept map to the curriculum.” (p. 1236) Moni et al. developed a rubric to assess concept mapping in physiology education for dentistry students. The authors initially developed the rubric to assess the concept map based on the researchers’ points of view of what should be assessed in the concept map. The authors then collected faculty members’ understanding on the criteria developed, and conducted a survey to the students about how the students understood the criteria. Moni et al. (2005) revised the rubric based on their study on the faculty members’ and students’ perceptions. The revised rubric had the same criteria as before, but details and the language in each of the criteria was considered to be more user friendly. The authors wrote an explicit task for students to create concept maps based on

Page 11 Concept Mapping as On-site Assessment of BMS PBL

pathophysiology and pharmacology information. The students worked in groups of four. The response format was a written concept map. The scoring system was based on three criteria: comprehensive coverage, logical sequence, and overall presentation. The produced rubric was implemented to assess the following year’s enrollment of the same course . This article showed a valuable example of developing a scoring system based on the users’ perceptions. The authors considered the language and word choices that could be understood by both students and faculty members clearly. Although the authors fulfilled the three criteria in concept mapping as an assessment tool—task, response format, and scoring system—, they did not discuss the validity issue. It is important to establish validity and reliability of the concept map as an assessment tool before reporting the results of the assessment to the students and faculty members. Ruiz-Primo and Shavelson (1996) summarized twenty one research articles to argue about validity and reliability of concept mapping as an assessment tool. Reliability refers to the consistency or generalization of the measurement. Reliability was measured through inter-rater agreement and stability (retest reliability, or reliability over time). “Validity refers to the extent to which inferences to students’ cognitive structures, on the basis of their concept map scores, can be supported logically and empirically” . Content validity is commonly established by an expert panel. Convergent validity can be customized by comparing concept mapping with other assessment methods in the course. Instructional sensitivity supports the idea of validity by comparing students’ concept maps between the pre- and post-instruction. Ruiz-Primo and Shavelson identified three components when considering concept mapping as an assessment tool: the task, the response format, and the scoring system. Concept mapping in PBL assessments may follow at least two of the criteria. The task of concept mapping in PBL is embedded in two PBL meetings. The groups will produce (response format) paper-based or computer-based concept maps in the first and second PBL meeting. The scoring system for concept mapping in PBL has not yet been established. One part of the proposal’s goal is to produce a rubric to be applied in the scoring of concept mapping in on-site assessment of BMS PBL. To develop this system, “…, the possible scoring strategies require research to evaluate their ability to provide useful and valid information about students’ cognitive structures in a content domain” . The issue of validity will be resolved by the stakeholders’ focus groups, while the reliability issues will be resolved by the inter-rater reliability of the rubric in the last phase of this study. 2.5. Research Questions The research questions of this study are, 1. What are students’ perceptions in learning through PBL with or without concept mapping? 2. Are there any differences in exam scores between PBL students with or without concept mapping? 3. What are the stakeholders’ perceptions of concept mapping as an on-site PBL assessment? 4. What kind of rubric can score a concept map in BMS PBL according to the stakeholders’ points of view? The hypothesis of the second research question is that there will be significant difference in exam scores between PBL students with and without concept mapping. However, we realize that we may find that there is no significant difference in exam scores between PBL students with and

Page 12 Concept Mapping as On-site Assessment of BMS PBL

without concept mapping. If so, this result will be confounded with the result of research question one whether students find that concept mapping is worth doing. 2.6. Research design/Methodology The proposed study will follow the requirements for research with human subjects as indicated by the Committee of Ethics at Atma Jaya Catholic University Indonesia. This research will consist of a mixed-method study and will be done in three years (three phases). Data will be obtained from both quantitative and qualitative methods. 2.6.1. Qualitative Data Participants’ interviews and focus groups will be the main qualitative data resources. The interviews will be semi-structured with cohort year 1 students after the BMS 2 final exam. The major focus of the interviews will be the students’ learning experiences through PBL and how they prepare for the exams. The qualitative data from cohort year 1 students will provide input to the advancement of the PBL approach, and provide input to BMS 1 and BMS 2 summative exams. The data will also be compared with the cohort year 2 students’ focus groups. The interview will be about 45 minutes in length. Each participant will receive an IDR 50,000 gift card after the interview. These interviews will contribute to answer research question number one: What are students’ perceptions in learning through PBL with or without concept mapping? In this case, cohort year 1 constitutes the group that learns through PBL without concept mapping. There will be two kinds of focus groups: a students’ focus group and a faculty members’ (case designers and facilitators) focus group. Focus groups will be held in the second and the third phases. Focus groups in the second phase will have the main task of producing a rubric, which will be used to assess concept maps. Focus groups in the third phase will evaluate the implementation of the rubric to assess concept mapping as an on-site PBL assessment. Krueger and Casey argue that focus groups can help decision making and guide program developments through several group discussions. Thus, in the second phase, each type of focus group will meet twice. The goal of the first meeting will be to learn about the stakeholders’ perceptions of concept mapping as an on-site PBL assessment tool, and the goal of the second meeting will be to develop a rubric to assess concept mapping in PBL. In the third phase, focus groups will meet to evaluate the implementation of the rubric produced by the prior focus groups. The complete cycle of stakeholders’ focus group meetings is in accordance with constructivism from the Fourth Generation Evaluation . The stakeholders’ points of view are the main focus to evaluate a program and to make further decision. Time allocated for one focus group meeting will be 90 to 120 minutes. Each member will be offered an IDR 100,000 gift card to show appreciation for his/her participation. Bringing the stakeholders’ perceptions into a discussion about the concept map as an assessment tool in PBL is essential to developing a valid and reliable rubric. Ruiz-Primo and Shavelson say in their conclusion: “We also believe that the participation of science educators, experts, and instructors in designing a concept map assessment is essential” (p. 595). Through focus groups, standards of assessment may also be clarified. “It became clear that faculty member understandings about standards of criteria were greatly enhanced by discussion around exemplars.” . The focus group meetings will expectantly discuss the practicality issue of developing the rubric because the members are the executors. Findings from focus groups will answer research questions three and four: What are the stakeholders’ perceptions of concept mapping as an on-site PBL assessment? and What kind of rubric can score a concept map in BMS PBL according to the stakeholders’ points of view?

Page 13 Concept Mapping as On-site Assessment of BMS PBL

Interviews and focus groups will be audio-recorded. Prior to interviews and focus groups, the PI will provide informed consent forms. The participants will be asked to sign the informed consent forms. The audio-recorded data from interviews and discussions in focus groups will be transcribed. A freeware text analysis software (WeftQDA from http://www.pressure.to/qda/) will be used to assist the categorization of the transcribed data. 2.6.2. Quantitative Data Data will be taken from three kinds of exam scores: final BMS 1, final BMS 2, and National Benchmark Examination (NBE). Final BMS 1 and BMS 2 are summative exams of each unit block held at the end of BMS 1 and BMS 2 unit blocks. The NBE is a national exam to evaluate the second year students at a school of medicine. The NBE includes topics in the areas of biochemistry, physiology, and biology. The NBE is created by the National Committee of Medical Education, whose members are experts from representative medical schools. The NBE is held locally from May to July. The exams will be submitted to the committee to be assessed. The committee will send the scores to our school within one month. Cohort year 1 and year 2 students will take the three exams mentioned above and the results will be compared. To make sure that cohort year 1 and cohort year 2 students are homogenous, two one-sided t-tests (Lewis & Lewis, 2005) will be administered to compare students’ entrance exams in biology. The MANOVA test will be used to compare the exam scores. The quantitative data analysis will answer research question two: Are there any differences in exam scores between PBL students with or without concept mapping? 2.6.3. Sampling Participants for quantitative data collection will be cohort year 1 and year 2 students who will take three exams (Final BMS 1, Final BMS 2, and NBE). Participants in the interview will be selected purposefully based on students’ grades from BMS 1 and BMS 2 of cohort year 1. This method will generate ten to fifteen participants. Participants for focus groups in cohort year 2, as well as cohort year 3 students, will be selected by snowballing. The selection is based on PBL facilitators’ and peers’ recommendations. The selection of the faculty members’ focus groups will be based on a purposeful sampling from all faculty members involved in BMS PBL. The main selection is the faculty members who are involved in BMS 1 and BMS 2 case designing and PBL facilitators in BMS 1 and BMS 2. Each focus group will have eight to ten members. 2.7. Project Timeline This project will be started in August 2010 and will end in July 2013. It is a three-phase project as follows: • First year/ first phase: (August 2010 – July 2011) During the first phase, we will establish the validity of BMS 1 and BMS 2 final exams through an expert panel. We will design concept mapping training protocols for faculty members and students and establish the pilot studies of the protocols. These training protocols will be used in the second phase. Cohort year 1 students will take final BMS 1 in October 2010 and final BMS 2 at the end of December 2010. The exams’ scores will constitute a part of the quantitative data. Interviews will be administered for the purposefully selected participants of cohort year 1 in January to February 2011. This will constitute the qualitative data of cohort year 1, representing PBL without concept mapping.

Page 14 Concept Mapping as On-site Assessment of BMS PBL

•

Second year / second phase: (August 2011 – July 2012) The second phase will begin with training in concept mapping for faculty members and students. We will implement concept mapping in cohort year 2 BMS PBL as learning processes. The researchers will design two focus groups’ guidelines: students’ and faculty members’. Focus groups will meet during January and February 2012, in which both focus groups will meet twice. The main task of these focus groups is to discuss the stakeholders’ perceptions about concept mapping in PBL and to produce a rubric to be applied in concept mapping as the on-site BMS PBL assessment. Cohort year 2 students will take final BMS 1 and BMS 2. Meanwhile, cohort year 1 students will take NBE in May 2012. •

Third year / third phase: (August 2012 – July 2013) The last phase of this study will be an evaluation on the rubric implementation to BMS PBL students of cohort year 3. By doing this, we will establish reliability through facilitators’ scores (inter-rater reliability). Furthermore, we will have cohort year 3 students’ and faculty members’ focus groups. The main task of these focus groups is to determine perceptions of the rubric to be used in assessing concept mapping. The rubric will be revised wherever necessary. Cohort year 2 students will take the NBE in May 2013. This will complete the quantitative data. At the end of the project a final report will be submitted. 2.8. Expected Outcomes This study will search for evidence of concept mapping implementation in BMS PBL. The findings will be valuable in understanding students’ learning through PBL particularly through concept mapping as cognitive acquisition in PBL. This research will identify stakeholders’ views of concept mapping as an on-site PBL assessment. These results will encourage discussions and further research to search for effective approaches in PBL. The proposed research will produce a rubric to assess concept mapping as the on-site BMS PBL assessment and apply the rubric in the last phase of the project. These findings will support the rubric’s validity and reliability to assess concept mapping in the on-site BMS PBL. 2.9. Project Evaluation Dr. Gandes Retno Rahayu will be an external evaluator of this project. Dr. Rahayu is a medical educator at the School of Medicine Gadjah Mada University, Indonesia. She has been trained in medical education at Dundee University, Scotland, UK. She is an experienced PBL trainer. Her major research is in PBL assessments, particularly in the pre-clinical education. She is familiar with mixed-method research. Dr. Rahayu will evaluate the project annually. She will have full access to an audit trail of fieldnotes, methodology chosen, data collection, data analysis, and research findings. Dr. Rahayu will write annual reports, in the first and second phases, and a final evaluation at the end of the project. 2.10. Dissemination Preliminary findings from the first and second phases will be presented through posters and oral presentations in national education as well as medical education meetings and regional medical education meetings (Asia Pacific Medical Education Conference, APMEC, Asia Medical Education Association, AMEA, and Asia Medical Education Expo, AMEE). The reports will be submitted to medical education journals and journals in educational research (i.e. Medical

Page 15 Concept Mapping as On-site Assessment of BMS PBL

Education, Medical Teacher, Academic Medicine and International Journal of Research and Method in Education). The rubric produced from this study will be proposed to the dean of the School of Medicine at Atma Jaya Catholic University Indonesia to be applied to BMS unit blocks. A recommendation with a simple guideline to produce the similar rubric will be promoted to the faculty members who involved in other unit blocks to help them producing specific rubrics that they need in their on-site PBL assessments. 2.11. Expertise of PI The PI and Co-PI have been trained as PBL facilitators. The PI has worked as the PBL coordinator and facilitator at the School of Medicine at Atma Jaya Catholic University Indonesia for two years. The PI has been trained in research methodology, both quantitative and qualitative. The PI has experience in mixed-method research. These experiences will be useful for the proposed project. The PI has presented posters and oral presentations in medical education communities nationally (Pertemuan dan Ekspo Pendidikan Kedokteran Indonesia, PEPKI) and regionally (AMEA and APMEC). 2.12. Intellectual Merit The proposed research will apply a mixed-methodology to investigate the evidences of concept mapping in students’ learning and the possibility of using concept mapping as an on-site PBL assessment. The focus group, which will design a rubric to assess concept mapping, will represent the contribution of all stakeholders: students, facilitators, and case designers. The research will therefore be enriched with not only the expertise of the PI, but also the expertise of the stakeholders. In this case, the PI has proposed a creative way to design a rubric that will originate from, and be used by, the stakeholders. The project will make an important contribution in understanding how concept maps can be used as an on-site PBL assessment. In particular, this project will benefit the BMS unit block in perceiving students’ learning through concept mapping in BMS PBL. The possibility to assess concept mapping in each PBL meeting is one contribution to the BMS unit block’s formative assessment. The result of this study will allow further achievement and research in PBL assessment. This research will also promote PBL progression. Appropriate disseminations through national and regional conferences will build and expand the use of concept maps as an assessment tool in PBL. Disseminations through journals or publications will expand the community to more than medical educators. A recommendation to use the rubric produced from this study will support the on-ste PBL assessment at our school. 2.13. Broader Impacts The proposed study will benefit PBL students because they will know that their cognitive effort will be assessed. This study will help the School of Medicine at Atma Jaya Catholic University Indonesia by providing a valid and reliable tool to assess the on-site PBL knowledge acquisition through concept mapping. This will improve the assessment tools in PBL by not simply assessing skills and attitude.

Page 16 Concept Mapping as On-site Assessment of BMS PBL

The proposed research will apply a “bottom-up” rubric design through stakeholders’ focus groups. This will make a valuable contribution as a model to a curriculum design, which may start from the executors of education and learners. The rubric produced from the stakeholders’ focus group may stimulate further research in rubric design for concept mapping in a higher level of medical education. Examples are in the second and the third year PBL, in which students learn about pathology, and in clinical education through case-based learning and bed-side teaching. The rubric may also inspire the application of concept mapping as the summative assessment in the BMS unit block. To the extent of stakeholders’ focus groups, and in particular for faculty members’ focus groups, this research will contribute to the team working of the academic staffs. This will improve relationships among faculty members and enhance a supportive working climate in the institution. To the extent of networking, this research may contribute to the medical education community nationally and regionally through proper disseminations.

Page 17 Concept Mapping as On-site Assessment of BMS PBL

3. Project Budget 3.1. Itemized Budget* Item Personnel PI Co-PI Evaluator Fringe Benefits PI Co-PI Equipment

Description

First Year

10% of annual salary 10% of annual salary

IDR IDR IDR

5,400,000.00 IDR 5,400,000.00 IDR 9,000,000.00 IDR

5,670,000.00 IDR 5,670,000.00 IDR 9,000,000.00 IDR

5,940,000.00 5,940,000.00 9,000,000.00

IDR IDR

1,890,000.00 IDR 1,890,000.00 IDR

1,984,500.00 IDR 1,984,500.00 IDR

2,079,000.00 2,079,000.00

for interviews and transcription

Travel PI Domestic and foreign Co-PI Domestic and foreign Evaluator Domestic Participants' Support Costs Interview participants 10-15 participants Focus Groups students focus group and faculty member focus group Other Direct Costs Material and supplies stationary Publication costs / subscription Documentation / Dissemination Total Direct Costs Indirect Costs (20%) Total Proposed Budget Annually Total Proposed total first year to third Budget year budget

Second Year

Third Year

IDR 10,000,000.00 IDR 15,000,000.00 IDR 18,000,000.00 IDR 21,600,000.00 IDR 15,000,000.00 IDR 18,000,000.00 IDR 21,600,000.00 IDR 5,000,000.00 IDR 6,000,000.00 IDR 6,000,000.00 IDR

750,000.00

IDR

4,000,000.00 IDR

2,000,000.00

IDR

5,000,000.00 IDR

4,000,000.00 IDR

3,000,000.00

IDR

7,000,000.00 IDR

7,000,000.00 IDR

7,000,000.00

IDR 81,330,000.00 IDR 81,309,000.00 IDR 86,238,000.00 IDR 16,266,000.00 IDR 16,261,800.00 IDR 17,247,600.00 IDR 97,596,000.00 IDR 97,570,800.00 IDR 103,485,600.00 IDR 298,652,400.00

* $1 = IDR 9,700

3.2. Budget Justification • Personnel This proposed budget will cover ten percent of the annual salary for the PI and the Co-PI. The increment is five percent annually. The fringe benefit is thirty-five percent of the proposed salary. •

Equipment The equipment for interviews and focus groups are audio recorders and a transcription tool that will be covered by this proposed fund.

Page 18 Concept Mapping as On-site Assessment of BMS PBL

•

Travel Travel funds will cover domestic and international travels for the PI and the Co-PI to attend and present the research findings annually. A travel fund will also be provided annually for the evaluator. •

Participants’ Support Costs The interview participant in the cohort year 1 will be offered a gift card of IDR 50,000 for a 30 to 45 minute interview. Total participants for the interview are fifteen students at most. The calculation is 15 students X IDR 50,000 = IDR 750,000. For the second phase, there will be four focus group meetings. Two of them are students’ focus groups and the other two are faculty members’ focus groups. Each of the focus groups will have a maximum ten members. Each of the members will be offered a gift card of IDR 100,000. Therefore, the calculation is 2 focus group types X 2 meetings X 10 members X IDR 100,000 = IDR 4,000,000. In the third phase, there will be one students’ focus group and one faculty members’ focus group; each of the focus groups will meet once. Therefore, the calculation is 2 focus groups X 1 meeting X IDR 100,000 = IDR 2,000,000.

Page 19 Concept Mapping as On-site Assessment of BMS PBL