Elearn 2003 Quiz Engine

The Walden’s Paths Quiz Engine Avital Arora, Emily Barker, Unmil P. Karadkar, Pratik Dave, Luis Francisco-Revilla, Richard Furuta, Frank Shipman, Suvendu Dash, Zubin Dalal Center for the Study of Digital Libraries and Department of Computer Science Texas A&M University College Station, TX 77843-3112, USA Phone: +1-979-845-3839 {avital, eluke, unmil, pratik, l0f0954, furuta, shipman, suvendud, zubin}@csdl.tamu.edu Abstract: The Walden’s Paths project is developing tools to facilitate the inclusion of Web-based materials in classroom education. It currently includes components to create, present and manage paths that recontextualize information on the Web. In this paper we present the Quiz Engine, a fullfeatured online testing system that extends the repertoire of our tools to include Web-based assessment of the knowledge gained by the students. It allows teachers to author the different types of questions commonly used in paper-based testing and generates interactive as well as passive tests with various characteristics. It is designed to support teachers with little experience in Web technologies to create and administer online tests in a variety of settings for aiding students’ selfassessment while learning or for administering graded examinations.

Introduction Periodic evaluation is essential to encourage students to acquire and apply the knowledge, information and the skills that they learn. Evaluation helps students understand their strengths and weaknesses and helps teachers assess the effectiveness of their teaching methods and focus their attention on areas where the students need improvement. Teachers employ various evaluation mechanisms to verify that students are gaining the required proficiency: they administer tests and quizzes, assign homework, lab work and involve students in projects and demonstrations that apply the knowledge and skills gained. Online tests and quizzes with automated grading mechanisms (Browne 2002, Hall et al. 1999, Woit & Mason 2000) minimize the time needed for grading and can provide quick feedback for the students’ benefit. Meaningful and timely response is of utmost importance if students are to benefit from the evaluation (Browne 2002, Flewelling et al. 2001). Adaptive online assessment mechanisms like the QUIZIT system (Tinoco et al. 1997) tailor the difficulty level of the quizzes to maximize the benefits for individual students. Web-based testing systems are more accessible to students as they transcend the space and time barriers imposed by most conventional testing mechanisms. Students may take practice tests at their convenience from a location and at a time of their choice. Martinez stresses that the Web offers the perfect technology and environment for precision learning because learners can be uniquely identified, content can be personalized for them individually, and subsequent progress can be monitored, supported, reported and assessed cost effectively (Martinez 2002). Research suggests that online testing is an effective mechanism for evaluating student knowledge of the subject matter and that there is a good correlation between the scores obtained by students on paper-based and online tests (Kumar 1999). Online testing is also an effective preparatory mechanism for in-class tests (Hall et al. 1999). In spite of the advantages of online evaluation many teachers are reluctant to take the plunge, mostly because they are unfamiliar with the technology that goes into creating and managing these systems (Arif, 2001, Matuga 2001). The process of setting up online assessment and evaluation activities is completely different from that followed in a traditional classroom setting. Commercially available software such as WebCT (WebCT 2003) and Blackboard (Blackboard 2003) support teachers in creating, managing and hosting online courses by integrating the syllabi, multimedia course content and assignments and providing other allied services. Walden’s Paths (Shipman et al. 1999), our suite of tools, supports teachers with varied technological backgrounds in tailoring Web-based information for classroom use. The Web is a vast, unedited information repository with little publishing control and contains information that may be inaccurate or inappropriate for use in an educational setting. Even when correct, Web materials are often authored for purposes other than education and can leave students confused about the context and vocabulary. Walden’s Paths is motivated by the belief that the

material already available on the Web can be recontextualized for use in educational settings. It provides teachers a way to focus student exploration, to add explanation and context to materials for use in the classroom environment. The next section of this paper provides a brief overview of Walden’s Paths. We then present our motivation and rationale for the quiz system, followed by the design and operational overview. We conclude with a discussion of the various issues that we plan to address in the near future to enhance the value of the Quiz Engine.

Walden’s Paths Walden’s Paths (http://www.csdl.tamu.edu/walden/) supports the reuse of Web-based information via the metaphor of a linear, guided path. Guided paths were envisioned as far back as Vannevar Bush’s Memex (Bush 1945). Systems implementing guided paths have been investigated with limited user communities since the 1980s (Trigg 1988, Zellweger 1989). Walden’s Paths is a suite of tools that supports creation, presentation and maintenance (Francisco-Revilla et al. 2001) of linear paths over Web-based materials. Teachers (or more generally, path authors) create paths by searching for relevant Web pages and adding the ones that they find interesting into the authoring interface. They may then add text or HTML annotations to reflect their perspective on the topic addressed by each included page. These annotations may elucidate the page content, highlight points of interest, provide instructions to students or ask questions that the students must answer. Students (or more generally, path readers) view the paths from any of the standard Web browsers via the Path Server. Path readers typically begin browsing by choosing a path from a list presented to them. Figure 1 illustrates the navigation interface while browsing a path. The Path Server displays the Web pages along with the contextualizing annotations as organized by the teacher. The bottom part of the browser window displays the Web page, as it would appear without the Path Server interface. The annotation added by the teacher is displayed in the top-right part of the window. Readers navigate the path using controls displayed in the top-left portion. They may view the pages on the path sequentially by clicking on the “Next” and “Back” images or jump to other pages on the path by scrolling to and clicking on the “paw” corresponding to the position of the page. While following the path, readers are free to follow links embedded in the Web pages to explore the surrounding information space. Following these links takes the reader “off the path”. While the reader is browsing the information space off the path the control widgets described above are replaced by a single “Back to path” image that links back to the page from which the reader wandered off the path as shown in figure 2. This feature provides readers with an easy way to return to the path once they are done exploring. Thus a student may browse through pages not included in the path, secure in the knowledge that she can return to the path with a single click. By providing context to the vast resources on the Web, the Path Server supports guided navigation as well as exploratory learning.

Figure 1: The Walden’s Path Server

Figure 2: Browsing off the path

Our experience with Walden’s Paths has shown that soon after they start using the system, teachers and students alike shift focus from the underlying metaphor of the path to the subject matter presented and the contents of the Web pages (Furuta et al. 1997). We attribute this “fading” of the technology to the simplicity of the metaphor and of the interface.

Motivation for a Developing a Quiz System Some of the teachers who used Walden’s Paths early on annotated their paths with questions that the students would answer upon reading each page. The annotations for the pages shown in figures 1 and 2 contain questions of this type for a path that described the Kitt Peak Observatory and compared it to others. Students were asked to turn in their answers to these questions on paper, which the instructor graded by hand. The teacher successfully mixed digital and paper media to achieve his curricular goal of evaluating students’ understanding of the material he had included in the path. A quiz system with automated grading support frees teachers of the burden of grading some kinds of questions, for example, multiple choice and true-or-false questions. It aids teachers in evaluating students’ understanding of the materials that they have learned from the paths and can provide instantaneous results and immediate feedback. Teachers may present quizzes with randomly ordered questions to address issues of academic dishonesty where students may peek at their neighbors’ monitors (Browne 2002). Over time, teachers may build a large database of questions to serve as a question bank and quizzes may be generated by randomly picking questions from this database. Thus, different (but comparable) tests can be generated for individual students to further discourage cheating on tests. Various statistics and trends regarding students’ performance on the test may be automatically compiled and emailed to the teacher. Teachers may easily analyze the results generated by the quiz to identify problem areas and common sources of errors. The feedback obtained in the form of the students’ performance helps instructors analyze and improve various aspects of their teaching, such as the course contents, style of delivery or the nature of interaction in the class. In the case of Walden’s Paths, such analysis may suggest improvements to a path’s content. If a large number of students are found answering a particular question incorrectly the instructor may go back and examine the path to see if the concept needs additional illustration or contextualization. A quiz system also aids students in fine-tuning their learning process. Students can choose to attempt practice quizzes on their own initiative, to test how well they are prepared for an upcoming exam or to diagnose the effectiveness of their study patterns. Practice quizzes, which may be modeled along the pattern of the exams, help students prepare for their final examinations. Students may initiate a quiz as they are browsing a path to reassure themselves that they have grasped the essence of the path (or of some pages on the path). The quiz system provides immediate feedback, especially in quizzes taken for self-evaluation. The quiz system may randomly select questions for a given quiz, thus exposing the students to a variety of questions on the same topic. As the database of questions grows larger a quiz can be taken more than once to answer a possibly different set of questions. The immediate feedback offered by the Walden’s Paths Quiz Engine will allow students to quickly identify their mistakes and if necessary view the relevant path(s) for more information. Although graded paper tests provide some feedback, the response is typically delayed and teachers rarely have the time to provide detailed explanations to each student or to point them to additional resources individually. The students may or may not take the initiative to find out what went wrong. The quiz mechanism allows teachers to provide a Web resource where students may get additional information regarding the question. With immediate feedback and the availability of additional resources students are more likely to view this information, hopefully resulting in better performance on the actual tests. The quiz facilitates easy accessibility since an online quiz removes distance barriers and caters to the student's time commitments, location, preferences and convenience. Quizzes could also be used as the primary evaluation mechanism for a course taught after hours, possibly for part-time students or in a distance learning class. We conducted an informal survey of teachers in various academic disciplines at K-12 and university levels to gain an insight into the factors that affect their decisions regarding use of online testing systems. Some of these teachers use online testing extensively while others had little experience in this area. Most teachers surveyed use the Web for disseminating course materials to their students. Some teachers noted that most tools do not support the features required to answer their questions. For example, one teacher includes questions that require students to draw graphs, write complex mathematical formulae and derive equations on her tests. While developing tools that will support rich interactions is a long-term goal, we have chosen to begin by supporting the types of questions that are most commonly used on tests in an effort to address the issues faced by a wider community of teachers.

Figure 3: The Question Authoring Interface

Figure 4: The Quiz Generation Interface

The Walden’s Paths Quiz Engine The Quiz Engine has been developed with simplicity of use as the key design principle with a view to catering to audiences with varied technological expertise. The Quiz Engine is a generic quiz authoring and management system. While designed to work with Walden’s Paths, it could be easily used to generate quizzes in isolation, and presumably, with other software. The process of generating quizzes using the Quiz Engine consists of two activities, authoring questions and choosing questions to form tests. Authoring questions is a long-term process as teachers may add questions about various paths over a period of time. These questions are stored in central database that acts as a repository for generating quizzes for all teachers. When teachers need to generate quizzes they specify various criteria to select a set of questions from the question database. They may customize a generated quiz by adding and deleting specific questions. The Quiz Engine saves this quiz and returns a URL for it. A teacher may send the URL to her students for taking the quiz or publicize the links for practice quizzes on her class Website. By separating the process of creating questions from that of combining these to form tests, the Quiz Engine allows teachers more flexibility in generating quizzes as quizzes created for a path at any time include questions from the pool of all questions that are currently available. The authoring interface displayed in figure 3 allows teachers to enter questions into the database. A teacher begins by identifying the path and the page(s) containing the material to which this question refers. She types the question and categorizes it as either a general question—a question regarding general principles related to the topic of the path—or a topic-specific question—one that deals with details specific to a topic or a sub-topic within the path. The ability to distinguish between these two categories is useful during quiz generation. She then decides upon the format of the question. The Quiz Engine currently supports questions of four types: questions with a specific word or a specific phrase as the correct answer multiple-choice questions, with a single correct answer multiple-choice questions, with multiple correct answers questions with longer, essay-type answers The Quiz Engine is capable of automatically evaluating questions in the first three formats. For the multiplechoice questions, the teacher specifies the various options and indicates which of these are the correct answers. True-or-false questions are treated as instances of multiple-choice questions with one correct answer. Answers to essay-type answers are e-mailed to the teacher for evaluation and must be manually graded. She assigns a difficulty level between 1 (easy) to 5 (difficult) to each question. This enables her to form quizzes of varying composition of easy or challenging questions. The teacher may provide a URL to an image or an illustrative figure as a part of the question. The image may be an integral part of the question or it may act as a clue to the answer. For example, the question “Name the person shown in the accompanying figure” requires that there be an image associated with it. On the other hand, the fill-in-the-blank question, “_____ live in floating lodges.”, may be illustrated with a picture of a beaver to provide students with a visual clue. The teacher may also include a URL that provides additional information about the topic of the question.

Figure 5: The Interactive (short) Quiz

Figure 6: The Passive (long) Quiz

The Quiz Engine validates the teacher’s choices and alerts her if invalid data is entered. Informative tool tips associated with the various input fields give her a better idea about the expected data for each field. The teacher may preview the data entered to ensure accuracy before it is added to the question database and edit the added questions using an interface similar to that used for authoring. To create a quiz, the teacher invokes the Quiz Generation Interface illustrated in figure 4. This interface accepts her preferences and uses these to generate a draft proposal for the quiz. The teacher specifies the path for which she is generating the quiz and names it for easy identification. She then enters the number of questions she would like to include in the draft quiz. The size of the final quiz may vary as teacher may add or delete questions from the system generated draft. The Quiz Engine e-mails the student responses to the teacher if she so prefers, the most common case of this may be a quiz that includes questions with long answers. The teacher may choose to randomize the order in which questions are displayed to the students as they take the test. While all students answer the same questions, this makes it considerably more difficult to copy the answers from nearby students’ computer displays. She may also choose to prioritize questions that least recently used questions to ensure that all questions in her question bank are used. Quizzes to be used for grading require that students authenticate themselves, while the practice quizzes may be taken anonymously. Teachers may also want students to identify themselves if the Quiz Engine is monitoring the progress of individual students to tailor questions at the appropriate difficulty level. The current prototype does not monitor students’ progress on practice tests. The teacher may broaden the scope of the quiz by including questions authored by other teachers. This feature allows teachers who teach different sections of a class to pool resources or to include questions authored by another teacher who has taught the class earlier. A quiz can be tailored further to focus on the general or specific concepts by including questions from various categories described earlier in desired proportions. A quiz that focuses on the general principles may include a higher percentage of overview questions, while a quiz that expects students to know details may consist mostly of sub-topic-specific questions. The Quiz Engine returns a draft quiz based on the teacher’s preferences. She may then preen the quiz by deleting individual questions and adding new ones. The quiz may be either interactive or passive. As shown in figure 5, interactive quizzes display one question at a time. These quizzes evaluate the student input for each question and display the result with any suggestions or additional information provided by the teacher before continuing to the next question. As one can guess, interactive quizzes are best used as a tool for self-assessment or as preparatory tests. It is difficult to impose time constraints as the time taken to complete these quizzes varies depending upon whether they view the additional information provided by the teacher. During the quiz, students may also access an overview page that displays the result of all questions they have answered so far. A passive quiz, as shown in figure 6, displays all questions at a time and students may answer the questions in any order they choose. All questions are graded after the student indicates that he has finished answering the quiz and he receives no feedback while progressing through the questions. The additional information entered by the teacher is not displayed either. These quizzes are the digital analogues of in class, closed book, time delimited exams. Indeed, a teacher may print out this quiz and administer it as a paper-based test.

Discussion and Future Work With educational practices and tools changing at a great pace, Nichols (Nichols 2002) discusses how the old dichotomy of traditional face-to-face and distance education is becoming obsolete as more diverse mixed-mode forms of education delivery emerge. Teachers who teach a class in the classroom often make course notes or discussions available over the Web. They may use e-mail as the primary communication tool outside the classroom or may use Web-based discussion groups to encourage curricular discussion beyond class hours. Technology can be used to foster a deeper approach to learning by actively engaging students in the learning process and thus placing the focus and responsibility for learning upon the students themselves (Goodwin et al. 2001). As the responsibility shifts to the students, they must also be empowered to successfully fulfill their responsibilities. Currently, the students may only take quizzes that are created by the teachers. However, motivated students may wish to track their progress more frequently or at a finer grain than that of the path. Providing support for student-created custom quizzes may engage them better in the learning process. For example, a student may generate a quiz to test his grasp of the pages that he has visited so far on the current path. Students may invoke quiz sessions while navigating along a path, possibly by clicking on a “Take Quiz Now” button, to generate a quiz on the pages that they have viewed so far. Providing support for such features requires that the Path Server be augmented to record the student sessions and pass it to the Quiz Engine for generating the quiz. Integrating the Path Server and the Quiz Engine may also help change the nature of the paths themselves. Teachers may add question pages as a part of their paths. These pages may include a random or a specific question relating to the materials that they have seen so far. These pages have the potential to introduce a wide variety in interaction while reading the paths. A question page may announce itself to students by being visually different from information pages or it may masquerade as a regular path page and surprise the student when it is viewed. Answering the question may be mandatory or optional and answering a question incorrectly could redirect the student to the information page that contains the correct answer. Teachers could use surprise question pages to counter the issue of students being easily distracted during online learning sessions, a problem identified by Bergstrom (Bergstrom 1997). Ensuring that students actually complete their assigned readings and tasks is another important problem that must be addressed (Mason & Woit 1998). In an effort to ensure that students are indeed paying attention, teachers may grade the answers to these surprise questions to simulate pop quizzes over the paths and thus enhance the online learning experience. As quiz questions are closely related to contents of pages included in the paths, teachers may find it easier to author questions for the quiz while creating the paths. While they may add questions to existing paths, the coauthoring of paths and quiz questions will help strengthen the narratives of the paths as well the variety of questions for a more in-depth assessment. The current system ties every question to a specific page on the given path. However, it is easy to imagine more complex relationships between questions and pages. An overview question may relate to concepts expressed in the path as a whole or to a part of the path (a few pages on the path). It is essential to remember that students may not always view the pages serially by their ordering in the path. Answers to some questions may vary depending upon the order in which a student viewed the pages. For example, a teacher may create a path consisting of individual images that may convey different impressions depending upon the order of their viewing. In this case, the teacher may ask questions the answers to which may vary depending upon the order in which the pages are viewed. They may also author several questions that may only be asked if a student views the pages in a specific order. Providing a simple interface that supports expression of complex relationships between questions and the materials presented by the paths is a difficult task. While technologically savvy teachers may not flinch from expressing intricate relationships via complex conditional expressions, providing a simple interface with lesser expressivity will be more useful to a larger community of teachers. We are also striving to support a wider variety of questions. Some types of questions that are commonly used in paper-based testing environment are not suitable for online testing (Arif 2001), for example, questions with long, essay-type answers. We provide a work around by e-mailing the answer to the teacher, who must grade it manually. While the teachers may accept this system for tests that are graded, they may find it extremely inconvenient to receive email for each long question answered during self-assessment quizzes.

Conclusion The reluctance to adopt online testing mechanism stems from various factors. Teachers are accustomed to administering paper-based tests and may not be unwilling to migrate as it implies learning new techniques and technology that they are unfamiliar with. Teachers may not be able to replicate all question types in paper-based settings in online testing environments. While it lacks some features of paper-based testing, the online systems make up by providing automated grading, ease of analyzing student responses to detect patterns and student-specific adaptation for effective learning support. Providing appropriate social as well as technological solutions will encourage teachers and students to incorporate online testing in their educational environments. An online assessment system that ties in with the Walden’s Paths system can serve multiple advantages to both, the students and the teachers. With the addition of the Quiz Engine, Walden’s Paths can be used for delivering curricular content, engaging the learner in the topic at hand, facilitating, even forcing, learning and assessing the knowledge gained. The Quiz Engine may work as an alternative to or as a support tool for traditional paper-based quizzes. The design and development of the tool has helped us gain a perspective of the complex processes underlying effective student evaluation. We are channeling the understanding gained from the development of the initial prototype into addressing these issues via social and technological solutions.

References Arif, A. (2001). Learning from the Web: Are Students Ready or Not?. In the Journal of International Forum of Educational Technology & Society and IEEE Learning Technology Task Force, Oct. 2001. http://ifets.ieee.org/periodical/vol_4_2001/arif.html (accessed April 25, 2003). Bergstrom, S. (1997). Distance learning technology for aviation training. In Aviation Industry Computer Based Training Committee working papers, June 1997. http://www.aicc.org/docs/working/dlwp2doc.zip (accessed April 25, 2003). Blackboard Inc. (2003). Blackboard Course Management System 5.1. Blackboard Inc. http://www.blackboard.com/ (accessed April 25, 2003). Browne, R. (2002). Automated Tutorial and Assignment Assessment. In the Journal of International Forum of Educational Technology & Society and IEEE Learning Technology Task Force, Jan. 2002. http://ifets.ieee.org/periodical/vol_1_2002/browne.html (accessed April 25, 2003). Bush, V. (1945). As We May Think. The Atlantic Monthly, Vol. 176, No. 1, July 1945. 101-108. Flewelling, J. and Snider, D. (2001). CARLA (Computer-Assisted Recorded Language Assessment) – Software for Evaluating Student Oral Responses, ICTE Educational Technology Resource Library. Francisco-Revilla, L., Shipman, F., Furuta, R., Karadkar, U. and Arora, A. (2001). Managing Change on the Web. In Proceedings of First ACM/IEEE-CS Joint Conference on Digital Libraries, June 2001, Roanoke, VA. 67-76. Furuta, R., Shipman, F, Marshall, C., Brenner, D. and Hsieh, H. (1997). Hypertext Paths and the World-Wide Web: Experiences with Walden’s Paths. In Proceedings of Hypertext ’97, April 1997, Southampton, UK. 167-176. Goodwin, C., Graham, M. and Scarborough, H. (2001). Developing an asynchronous learning network. In the Journal of International Forum of Educational Technology & Society and IEEE Learning Technology Task Force, Oct. 2001. http://ifets.ieee.org/periodical/vol_4_2001/scarborough.html (accessed April 25, 2003). Hall, R., Pilant, M. and Strader, A. (1999). The Impact of Web-based Instruction in an Applied Statistics Course. In Proceedings of the International Conference on Mathematics/Science Education and Technology (M/SET 99), Mar. 1999. http://onlinetesting.tamu.edu/docs/AACE-MSET1135.pdf (accessed April 25, 2003). Kumar, A. (1999). On changing from written to online tests in Computer Science I: an assessment. In ACM SIGCSE Bulletin, Proceedings of the 4th Annual SIGCSE/SIGCUE on Innovation and Technology in Computer Science Education, vol. 31, Sept. 1999. 25-28. Matuga, M. (2001). Electronic Pedagogical Practice: The Art and Science of Teaching and Learning On-line. In the Journal of International Forum of Educational Technology & Society and IEEE Learning Technology Task Force, July 2001. http://ifets.ieee.org/periodical/vol_3_2001/matuga.html (accessed April 25, 2003). Martinez, M. (2002). Beyond classroom solutions: new design perspectives for online learning excellence. Discussion Summaries in the Journal of International Forum of Educational Technology & Society and IEEE Learning Technology Task Force, Jan. 2002. http://ifets.ieee.org/periodical/vol_1_2001/martinez.html (accessed April 25, 2003). Nichols, M. (2002). Principles of best practice for 21st century education. In Discussion Summaries in the Journal of International Forum of Educational Technology & Society and IEEE Learning Technology Task Force, Apr. 2002. http://ifets.ieee.org/periodical/vol_2_2002/discuss_summary_april2002.html (accessed April 25, 2003). Shipman, F., Furuta, R., Hsieh, H., Francisco-Revilla, L., Karadkar, U., Rele, A., Shenoy, G. and Brenner, D. (1999). Using the Internet in the Classroom: Variety in the Use of Walden's Paths. In Proceedings of ED-MEDIA 99: World Conference on Educational Multimedia, Hypermedia, and Telecommunications, June 1999, Seattle, WA. 335-340.

Tinoco, L., Fox, E. and Barnette, N. (1997). Online Evaluation in WWW-based Courseware. In the Proceedings of the 28th SIGCSE technical symposium on Computer Science Education, San Jose, CA. 194-198. Trigg, R.H. (1988). Guided Tours and Tabletops: Tools for Communicating in a Hypertext Environment. In ACM Transactions on Office Information Systems, 6 (4). 398-414. WebCT (2003). WebCT Course Management System 3.8. WebCT, Inc., Lynnfield, MA. http://www.webct.com/ (accessed April 25, 2003). Woit, D. and Mason, D. (2000). Enhancing student learning through on-line quizzes. In Proceedings of the 31st SIGCSE technical symposium on Computer Science Education, Austin, TX. 367-371. Zellweger, P. T. (1989). Scripted Documents: A Hypertext Path Mechanism. In Proceedings of the ACM Hypertext 89 Conference, New York, NY. 1-26.

Acknowledgements This material is based upon work supported by the National Science Foundation under grant numbers DUE-0085798, DUE-0121527 and IIS-0219540.