Philosophy Paper

Evaluation for Improvement According to Dr. Reeves and Hedberg (2003), anyone who wants to develop or manage interactive learning systems (ILS) must be skilled at handling various evaluation activities before making decisions. Evaluation is an important component of the process of producing an interactive learning system. For example, before producing an online learning program, a developer may ask content experts, interview teachers, and observe learners using prototype versions of the program to collect his/her needed evaluation data (Reeves & Hedberg, 2003). I agree that evaluation is a management tool that helps to make mid-course corrections and document the successes of a program. The effective evaluative information will help program producers/implementers to determine if the project is on-track and also more effectively learn from their reflecting experience. As a project manager, instructional designer, and implementer of interactive learning systems, he/she is keenly interested in knowing how to do his/her work better. Evaluation provides a mechanism for stakeholders to understand what works, what doesn’t work, and why (Kirshstein, & Quiñones, 1998). Evaluation is sometimes thought as an intrusive requirement that takes time away from the process of producing/implementing a program. From my point of view, evaluation is not an additional work, but a critical part of the process. The team of an interactive learning system should be actively engaged in collecting the information they need to make decisions, and in interpreting and using the evaluative data. As a result, the process of gathering data can become part of the change process. I believe that the validity and usefulness of evaluation results are enhanced by how skillful an instructional designer or

1

project manger in conducting evaluative acts such as interviewing, testing and observing etc. We have learned that effective evaluation should be planned and implemented within the context and nature of specific program (Kromrey, Hogarty, Hess, RendinaGobioff, Hilbelink & Lang, 2005). Furthermore, we should move evaluation from being a stand-alone monitoring process to an integrated and valuable part of an interactive learning system planning and delivery. There are a variety of evaluation models that have been developed. Most often, I would suggest mixing of multiple evaluation models, i.e. “Multiple Methods evaluation model”. Using multiple methods is not just adding simple two or more methods into one solution. We need to choose appropriate approaches for a specific purpose. For instance, the case of Kromrey et al. (2005) showed that only using multiple measures can meet the requirements of their university’s online instruction evaluations. Currently, online courses gain many attentions. The complexities of the Webbased learning and teaching derive from integrating technologies (such as multimedia applications and social networking) and an “its use to support the new modes of teaching and learning. (Kromrey et al., 2005)” From my point, an online course should be evaluated on three stages. Firstly, on the planning stage, the evaluator works with project manager and designers to review course documents including course syllabi, instructional design plans, and online course content. The second stage is implementing stage, and the evaluator would deliver surveys and interviews to faculties to collect their reflections. At the end of the course, evaluation would focus on students’ responses. The evaluator would deliver surveys including both selected-response and open-ended items

2

to students and faculties. Finally, the evaluator would collect all participant students’ performance grades or examinations’ results to analyze the online course’s academic impactions. To analyze all these collected data, the evaluator may use quantitative methods. As an evaluator, he/she should be comfortable with and believe in using a wide range of appropriate methods to gather information. Most often, an interactive system is evaluated by both quantitative and qualitative approaches. References Kirshstein, R., & Quiñones, S. (1998). An educator's guide to evaluating the use of technology in schools and classrooms. Washington, D.C.: American Institutes. Available at http://www.ed.gov/pubs/EdTechGuide/index.html Kromrey, J.D., Hogarty, K.Y., Hess, M.R., Rendina-Gobioff, G., Hilbelink, A., &Lang, T.R. (2005). A comprehensive system for the evaluation of innovative online instruction at a research university: foundations, components and effectiveness. College Teaching and Learning Conference, Lake Buena Vista, Florida, U.S. Reeves, T.C., & Hedberg, J.G. (2003). Interactive learning systems evaluation. Englewood Cliffs, NJ: Educational Technology.

3