Wutoh_et_al-2004-journal_of_continuing_education_in_the_health_professions.pdf

The Journul @Continuing Educution in rhe Hedth Projiessions, Volume 24, pp. 20-30. Printed in the U.S.A. Copyright 02004 The Alliance for Continuing Medical Education, the Society for Medical Education, the Society for Academic Continuing Medical Education, and the Council on CME, Association for Hospital Medical Education. All rights reserved.

Original Article

elearning: A Review of Internet-Based Continuing Medical Education Rita Wutoh, MD, MPH, Suzanne Austin Boren, MHA, and E. Andrew Balas, MD, PhD

Abstract

Introduction: The objective was to review the effrct oflnternet-based continuing medical education (CME) interventions on physician pegormance and health care outcomes. Methods: Data sources included searches of MEDLINE (1966 to January 20041, CINAHL (1982 to December 2003), ACP Journal Club (1991 to JulyIAugust 2003), and the Cochrane Database of Systematic Reviews (third quartel; 2003). Studies were included in the analyses if they were randomized controlled trials of Internet-based education in which participants were practicing health care professionals or health professionals in training. CME interventions were categorized according to the nature of the intervention, sample size, und other information about educational content and form&. Results: Sixteen studies met the eligibility criteria. Six studies generated positive changes in participant knowledge over traditional formats; t v d y three studies showed a positive change in practices. The remainder of the studies showed no difference in knowledge levels between Internet-based interventions and traditional formats f o r CME. Discussion: The results demonstrate that Internet-based CME programs are just as efective in imparting knowledge as traditional formats of CME. Little is known as to whether these positive changes in knowledge are translated into changes in practice. Subjective reports of change in physician behavior should be confirmed through chart review or other objective measures. Additional studies need to be peqormed to assess how long these new learned behaviors could be sustained. elearning will continue to evolve as new innovations and more interactive modes are incorporated into learning Key Words: Computer-assisted instruction, computers, continuing medical education (CME), continuing professional development, family physicians, graduate medical education, Internet, randomized controlled trials Introduction Dr. Wutoh: Assistant Professor, Howard University College of Pharmacy, Nursing and Allied Health Sciences, Division of Allied Health Sciences, Washington, DC; Ms.Boren: Associate Director, Center for Health Care Quality, School of Medicine, University of Missouri, Columbia, Missouri; and Dr. Balm: Dean and Professor, School of Public Health, Saint Louis University, St. Louis, Missouri.

,4lthough health professionals dutifully attain con1.inuing medical education (CME) credits, the CME goal of optimizing physician performance and outcomes has not been met.’,*Despite increases in knowledge evidenced by testing before and after CME activities, most CME courses fail to change physician practices.’-3Eckdahl et al.4suggest that physicians’ knowledge decreases after graduation from medical school, and new medical information fails to be integrated into routine practice. These shortcomings in teaching and Practice have long been recognized and addressed by

This study was supported in part by a grant from the Agency for Health Care Research and Quality (ROI HS 10472). Reprint requests: Rita Wutoh, MD, MPH, Assistant Professor, Howard University College of Pharmacy, Nursing and Allied Health Sciences, Division of Allied Health Sciences, 6th and Bryant Streets, NW, Washington, DC 20059.

20

Wutoh et al. the development of innovative and interactive health education programs.*s5 Nontraditional formats such as problembased small-group learning and electronic education have had limited success in imparting knowledge and changing the behavior of the health care profe~sionall-~; yet one of the latest innovations holds promise in the area of Internetbased CME.6 Lower costs and c ~ n v e n i e n c eare ~,~ advantages sometimes associated with Internetbased education, but there are others. Internetbased programs are becoming more interactive, often using multimedia formats to transfer knowledge to meet individual preferences,8-10and the newer Internet formats for learning can complement and reinforce traditional medical teaching.*,11,12 Although these approaches offer potential for improvement, there are many claims that Internet-based CME is no more effective than traditional methods in imparting knowledge.I2,l3 There are few randomized controlled trials of the effects of computer-assisted instruction in medical education; most studies are demonstrations or comparative r e p o r t ~ . ' ~ J ~ J ~ In the United States, access to technology has increased dramatically during the past 20 years,2 and the learning formats of health professionals have slowly changed with this access. In 2001, only 2.7% of physicians used the Internet for CME.16 More recently, on-line use by physicians was estimated at 31 %, with CME the second most common use of the Internet and younger female physicians the most active category of physician-learners. Physicians are searching the Internet to solve patient pr0b1ems.I~If Internet programs are not effective, then ways for improvement need to be identified. The question herein is whether Internetbased CME increases the level of knowledge over traditional formats and whether Internet-based formats of leaming can be more effective in changing behavior. The literature offers few review articles found on Internet or distance learning for medical edu~ a t i o n . ' ~Those , ' ~ that are available indicate that distance learning courses compare favorably with

classroom instruction and on-line participants express high satisfaction.12Few studies examine whether on-line programs change health professional behavior.I2In addition, few studies focus on individual learning preterences; instead, they group and average participant commentary and scores, and there is little documentation of gender and racial differences in regard to learning preferences and computer use with regard to continuing In addition and contrasting with the recent Harris et a1.18 report that young women are surpassing men in their use of on-line CME, Bernhardt et al.19 revealed no association between age, sex, or race/ethnicity and a health professional's perceived effects of a World Wide Web-based continuing education program. The health professionals working in a university system preferred traditional classroom learning formats over Web-based continuing education. l 9

Practical Significance With a dearth of randomized controlled trials and a lack of individualized learning there are many gaps in the research of Internetbased CME. There is a need for more analysis of current programs and randomized controlled trials to determine whether current versions of online CME are effective in changing behavior to ensure provision of the latest standards of care. This article reviews scientific studies of Internet-based CME to answer three questions: Are Internetbased CME programs effective? Are Internetbased CME programs more effective than formal CME? What is particularly effective within Internet-based CME in changing physician behavior or patient outcomes?

Methods In the systematic review of the literature, three major sources of original research were explored. These sources included randomized controlled trials, meta-analyses of randomized controlled trials, and pre/post studies of clinical interventions.

21

Internet-Based Continuing Medical Education Data Extraction

Eligibility Criteria

We used a standardized abstraction format to collect data from each article that met the eligibility criteria. For each trial, the abstractor noted the intervention (persons targeted, timing and periodicity, rules shaping the intervention), sample, primary measures of effect, and reported differences in process and outcome (observed ratios for the intervention and control groups, p values). Studies were analyzed to assess which intervention reported a positive outcome or improved performance or indicated negative or inconclusive results. The analyzed interventions were grouped into one of the following three categories, according to intervention type: e-mail, World Wide Web, and hybrid (CD-ROM plus Web).

The eligibility criteria for studies in this review were (1) randomized controlled clinical trials, metaanalysis, or retrospective study comparing the outcomes of specific educational interventions and control interventions in a group of health care professionals and (2) focus on Internet, Web, or software applications for the education and training of health care professionals in the areas of nursing, dentistry, pharniacy, allied health, or medicine.

Data Sources Systematic database searches were conducted electronically to identify eligible studies. Initially, MEDLINE (1966 to January 2004), CINAHL (1982 to December 2003), ACP Journal Club (1991 to July/August 2003), and the Cochrane Database of Systematic Reviews (third quarter, 2003) were searched. Each Internet-related word and health care specialty, including the following words, was combined without restriction to language: education, graduate education, continuing education, nursing education, graduate nursing education, continuing nursing education, medical education, graduate medical education, continuing medical education, pharmacy education, graduate pharmacy education, continuing pharmacy education, dental education, graduate dental education, continuing dental education, health education, hospital education department, distance education, Internet, Web, or software. Manual searches were not performed.

Results Literature searches identified 86 studies; however, 70 studies did not meet OUT criteria for inclusion. Twelve studies focused on patient education, 40 studies used computer software, and 18 studies were not an eligible study design. A total of 16 studies met all of the inclusion criteria (Table 1). Fifteen studies used an objective assessment of knowledge through the use of pre- and post-testing of both intervention and control groups for e-mail or Web-based programs.11.20 Six of 16 studies tested knowledge levels weekly or by unit (immediately) before conducting final examinations.s~8~'2~'5~20~21 Only one study conducted post-testing4 to 6 months after the intervention had been conducted.22 In addition to objective assessments, 10 studies included subjective evaluations as part of their assessment.3~x~11~12~20-25 Some subjective items included enhancement of learning through multimedia, ease of use, and learning preferences of Web modules over print materials or didactic instruction.10,12,23,24 In addition, negative comments were elicited in a few studies regarding barriers to using the e-mail or the Web, such as time constraints and ease of use. Three studies reported compliance with Web one of these studies exam-

Study Selection and Evaluation We included only those articles that met the following criteria: randomized controlled trial, metaanalysis, or retrospective study of Internet- or Web-based educational interventions (such as electronic mail [e-mail] and curriculum modules); studies that did not coerce participants; and participants who were practicing health care professionals or health care professionals in training.

10323-25

22

Lecture group vs. Web group with required activities and mastery quizzes

Web-based tutorial vs. self-study using content-equivalent print materials to teach care after myocardial infarction

Introduce new communications technologies+-mail, news, and gopher-into dermatology course Neuroscience course using a software application and Internet vs. a conventional lecture format Intervention group: discussion group with a facilitator and two psychiatrists vs. control group

Maki et al., 20002"

Bell and Mangione, 20009

Huntley and Conrad, 1994'O

Chan et al.. 1999'

There was no statistically significant difference between the pre- and post-test scores for the two groups (p = .5 1); there was no interaction effect between the allocation group and time on the MCQ scores (p = .33)

Model 2: significant differences were seen between intervention and control groups on the Body Shape Questionnaire (F = 5.78, p < .02) and the Eating Disorders Inventory Drive for Thinness (F = 4.29, p = .044) Web students performed better on the unit examinations than lecture students; performance on cumulative final examinations did not show any effect of the differential use of FAQ pages Number of question answers and guideline passages viewed were associated with a significantly higher post-test score; viewing graphic evidence was not associated with higher post-test scores (p = .93); female gender and prior low experience with the Web were correlated with significantly less use of randomized controlled trials 16% felt computer literacy was enhanced; 82% appreciated access to information; 42% felt time constraints; 25% found communication forms intimidating Normalized test scores were over 5 points higher for students in the multimedia sessions (p < .Ol)

Web with educational content on eating disorders

Dev et al., 199915

Goldberg and McKhann, 200012

Effects

Intervention

Study, Year

Table 1 Description of Studies Included

23 physicians

40 university students

88 second-year medical school students

166 medical residents

277 university students

61 university students

Sample

Interactive

Mixed

Interactive

Mixed

Mixed

Interactive

Didactid Interactive/ Mixed

5s. 2 a

P

h)

Printed materials or Web-based tutorial system on myocardial infarction

Computer learning module and interactive small-group seminar

Web-based oncologic curricular modules in basic science and clinical correlation

Bell et al., 20002?

Carr et al., 199914

Mehta et al., 1 99825

practical, or combined scores; intervention groups ENT rotations showed significant improved performance on written and practical examinations; students' comments were very positive No statistically significant difference between the 164 second-year medical students pretests and post-tests between the two groups, but improvements were seen individually between pre- and post-tests; 50% of both cohorts reported that the Web had enhanced their learning

students on

58 medical

practice and internal medicine residents

162 family

Mixed

Mixed

Interxctiw

Interactive 127 medical students

Didactic group vs. Internet group

Lipman et al., 20015

Interactive

On-line case discussion vs. control group

Marshall et al., 20018

40 family physicians

537 health care Interactive professionals (RD, MD, NP, etc.)

Immediate intervention group improved significantly more than the waiting list group on all three outcomes: knowledge (3.0 vs. 1.4; p < . O I L confidence (2.6 vs. 0.6; p < .Ol), and communication practices (0.21 vs. -0.1; p < . O l ) More intervention participants thought they had become aware of new techniques or relevant research (58.8%) and had made changes in their practice (64.7%) than those in the comparison group (38.5% and 30.8%, respectively) Case analysis grades from external reviewers were higher in the group with Internet component vs. traditional course (3.0 vs. 2.6, respectively; p < .005) Immediate scores on post-tests were similar between paper and Web groups (p > .2), but Web groups had greater learning efficiency (p = .04): Web groups were more satisfied with learning (p < .0001); after 4-6 months, knowledge of Web groups had decreased to the same extent as paper (p = .12) Post-instruction scores between intervention groups showed no significant differences among written,

Curriculum: case-based modules, listserv discussion group, hypertext links to resources

Didactid Interactive/ Mixed

Kemper et al., 2002'

Table 1 continued Description of Studies Included

Sample

~~

Effects

~~

Intervention

~

Study, Year

~

s

!g. 2

3

% ! trl

% n, 3 n

09

E. s

a. x

5 c?s

2

7

3 s 2n,

Intervention

Self-study vs. direct face-to-face intervention vs. telehealth teaching

CD-ROM and Web to teach dermatologic office procedures

52 physicians

Interactive

Mixed

121 first-year medical students

Mixed

Mixed

103 second-year dental students

No statistically significant difference between mean test scores or mean test times between conventional and Web-based groups (p < .05) Post-tests showed no significant differences between didactic and telehealth students, but significant differences were seen between self-study and telehealth groups and self-study and didactic groups for all five skills (p < .05) Multimedia group: students spent more time on material and after analysis still performed better than those using print materials (p < .001); there was a correlation between the eye information and time spent on material (r2 = .61, p < .0001); 78% students preferred multimedia version to print and thought their learning was more effective with multimedia Physicians in experimental groups using computermediated instruction performed significantly better on knowledge evaluations of dermatologic office procedures (p = .OOO); participants were satisfied with the instruction

Didactid Interactive/ Mixed

42 physical therapists

Sample

Effects

ENT = otorhinolaryngology; FAQ = frequently asked questions; MCQ = multiple-choice questions; MD = medical doctor; NP = nurse practitioner; RCT = randomized controlled trial; R D = registered dietitian.

Curran et al., 20OOz1

Grundman et al.. 200026 Print vs. multimedia

Barden et al., 2OOOZ7

Komolpis and Johnson, Orthodontic Web site was 200224 developed (RCT)

Study, Year

Table 1 continued Description of Studies Included

K

2

Internet-Based Continuing Medical Education naires completed at the last lecture, 93% had activated their e-mail; of those, 87% used their e-mail and 81% used news and only 65% used gopher. However, on written commentary, the majority of respondents indicated an appreciation for access to information. Seven of 44 felt that they had enhanced computer literacy. When questioned on the negative aspects of using Internet tools, respondents cited time constraints, intimidation, and difficulty getting started."' Four years later, in 1998, a trial using problem-based small-group learning via the Internet was conducted with family physicians. Twentythree family physicians participated in the intervention using a listserv service. These physicians had been in practice for an average of 13 years, with only 1.3 years experience using the Web. There was no statistically significant difference between the intervention group and the control group. Moreover, there was no statistically significant difference between the pretests and the post-tests (no difference with the intervention). Finally, there was no significant interaction effect for group allocation on post-test scores. However, this study demonstrated that it was feasible and inexpensive to organize problem-based smallgroup learning using the Internet.8 After another 3 years, a similar study was conducted using family physicians who had already been using a listserv to communicate on various medical issues. This study used on-line case discussions over the course of 2 weeks per case to get family practitioners to access the latest research and to change their practices in preventive medicine. The intervention group tended to be younger, with an average of 3 years less in practice. On post-testing, more intervention group participants had accurate knowledge on seven of eight items. Both groups improved their preventive measure screening over the course of the intervention; however, intervention group members had a greater sense that they were aware of newer techniques and research, and 64.7% felt that they had made changes in their practice compared with the control group.*

ined the use of rewards and compliance to assess motivation of the ~ t u d e n tThis . ~ study also examined when the study aid was used in relation to the final e~arnination.~ The assessment provided some insight as to the level of comfort with the study aid before examinations, utility of the study aid, and the motivation for use of the study aid. Fourteen studies were randomized clinical trials and two studies were modified quasi-experiments.20%21 The overwhelming majority of these studies were conducted among professional students, but three studies involved physicians as participants.2*'x2'Only one study examined the correlation of Web use with gender and previous low experience with the Web.9 This study found female gender and low previous Web experience to be negatively correlated with use of the Internet intervention9 In addition, three studies examined whether participants believed that Web instruction had changed their practice behaviors.',8,21Marshall et al.x found that 64.7% of online case discussion participants believed that they had changed their practice behaviors compared with 30.8% of control group participants. Kemper et a1.l reported that intervention groups achieved significant improvements in communication practices compared with controls.

E-mail Only three studies used e-mail or listserv for CME interventions.2.8,'0The first study, conducted in 1994, was designed to introduce electronic communication tools to medical students. The Internet tools used were e-mail, newsgroup, and gopher (a bulletin board interface).1° E-mail was best used to communicate on a one-to-one basis between student and teacher. Newsgroup provided the dissemination of notes, announcements, class quizzes, etc. to be posted for several individuals to access. Lastly, gopher was used for posting course schedules, previous examinations, syllabi, and any other material for extended periods of time. Ninety percent of participants used their e-mail accounts. From the 63% of question-

26

Wutoh et al. World Wide Web

i c ~ . ~ Three , ~ ~of ,the~ six ~ studies , ~ ~ found ~ ~ that Web-based interventions allowed participants to score higher on post-tests than those who used print material^.^*^^^^*^ The results on learning efficiency, however, were inconclusive. Two of the studies showed that learning efficiency was higher in the Web intervention groups.22,28 Another study found that the Web group spent more time on the A third study found no difference between the Web and print groups with regard to time spent learning.% On written commentaries by participants, Webbased interventions were generally ranked higher in value than paper materials in three s t ~ d i e s . ~ ~ ~ ~ ~ Many participants using Web interventions preferred multimedia versions and felt a greater satisfaction with Web learning in three s t ~ d i e s . ~ ~ , ~ ~ , ~ ~

The rest of the studies, 13 in all, incorporated the Web into the curricula for health professional stud e n t ~ . ~ , ~ ,One ~ ~ study, , ~ ~ in , ~1999, ~ , ~initially ~ - ~ ~ used CD-ROM multimedia programs, but the program was later converted to the Web with educational content on eating disorder^.'^ In Dev et al.'sls study, participants were female students at risk of developing eating disorders from two large universities on the west coast. The study used a combination of media such as newsgroup, educational materials, individual exercises, and a journal over an 8- to 10-week period for selfmonitoring of attitudes and behaviors to create a supportive community of young women and to change behavior that could place a young woman at risk of developing an eating disorder. The first model using a CD-ROM showed a small statistically significant risk reduction and a compliance of 53%. Compliance was low because the process of accessing the program was cumbersome owing to low numbers of computers with CD drives and having to download the program each time. l5 Models 2 through 4 used converted versions of the program on the Web. Ease of use improved dramatically as communication occurred via Webbased newsgroups. Thus, compliance with the program increased to 63.5%. In addition, significant differences were seen between intervention and control groups at baseline and follow-up psychiatric measures on the Body Shape Questionnaire and the Eating Disorders Inventory. Model 3 implemented a more structured Web intervention than model 2; readings, postings, and journal entries were required. However, compliance with newsgroups requirements was higher than in model 2. Initial analyses showed decreases in weight concerns and disordered eating behaviors and attitudes.15

Web versus Didactic Six studies compared Web-based programs with didactic or traditional lecture fonnats.5~12,'4,20,2"25,27 Only one study showed a statistically significant overall or final positive advantage of a Web-based learning program over a didactic or lecture format.I2This study showed normalized virtual learning environment scores being consistently higher than mean normalized conventional lecture hall scores for all lectures (p < .01).12Atwo-wayanalysis of variance demonstrated an effect for the type of lecture format received [F(1,144) = 6.696, p < .01].12In three other studies, a positive outcome of Web courses over didactic instruction was seen on a unit b a s i ~ ~ however, ~ ' ~ . ~ ~on; final examinations, the advantage was lessened, and both interventions were equally effective in improving performance on final examinations.5~12~14~20~2s~27 Subjective commentary from all studles was favorable toward Web intervention^.^^^^^^^,^^,^^,^^ Students remarked that Web courses were more effective and that Web courses enhanced

Web versus Print

Discussion Six randomized controlled trials evaluated Web interventionscompared with print materials for conveying knowledge about a myriad of medical top-

Several randomized controlled trials of Internetbased CME programs show that they are as effec-

27

Internet-Based Continuing Medical Education influences must be examined to counter them and promote more effective formats of learning. There are three steps to undertake in examining Internet-based CME:

Lessons for Practice Internet-based CME programs are as effective as selected other CME in imparting knowledge.

1. Examine what has been successful in traditional formats of CME and incorporate them into Internet-based CME.

Little is known as to whether positive changes in knowledge are translated into changes in practice.

2. Perform more studies using sound scientific design (e.g., randomized controlled trials) to examine if Internet-based programs are effective in changing physician behavior using objective measures of assessment.

Additional studies need to be performed to assess how long newly learned behaviors are sustained.

3. Design additional Internet-based educational programs that cater to individual needs based on market analysis.

tive as selected other CME in transferring knowledge to the health provider.2,3~s~14~20~22-2s~27 Three There are many advantages to accessing studies report participants feeling that they had, as Internet-based CME programs. Aggressive mara result, changed their practice behavior^.^,^,^' keting needs to be initiated to encourage more Another study, however, indicated that changes in health professionals to overcome barriers to techthe behavior of participants were not sustained nology. Moreover, Internet programs should be beyond 4 to 6 months after the intervention.22The customized to the individual; Internet-based CME behaviors and knowledge levels of participants should provide the user with medical information decreased to the levels of traditional didactic forof interest to the user in a format that suits the user. mats.22This brings the following questions to In the customization of services and medical informind: Can Internet-based CME programs change mation for the health care provider, it may be posbehavior? Do practitioners need to enrol in minisible to increase knowledge and change behavior residencies to have sustained change in behavior? to produce the best outcome for the patient. It is possible that as Internet-based CME programs apply the same curricula as traditional formats of CME, they apply the same deficiencies, rendering both programs ineffective. As such, References new, innovative, interactive programs need to be 1. Davis D, Thomson O’Brien M, Freemantle N, created and tested to see if they are effective. Wolf F, Mazmanian P, Taylor-Vaisey A. Several influences drive CME. These influImpact of formal education. Do conferences, ences need to be examined if change is to be workshops, rounds and other traditional conimplemented and Internet-based CME is to gain tinuing education activities change physician larger support. For example, attending meetings behavior or health care outcomes? JAMA is still the most preferred format for physicians 1999; 282(9):867-874. obtaining CME in the United States.I6There are 2. Chan D, Leclair K, Kaczorowski J. Problemsocial, economic, and marketing influences based small group learning via the Internet involved in the domination of this format. These among community family physicians: a ran-

28

Wutoh et al. domized controlled trial. MD Computing 1999; 16(3):54-58.

education. Acad Med 2000; 75( 10):1025-1028.

3. Kemper K, Amata-Kynvi A, Sanghavi D, Whelan J, Dvorkin L, Woolf A, Samuels R, Hibberd P. Randomized trial of an Internet curriculum on herbs and other dietary supplements for health care professionals. Acad Med 2002; 77(9):882-889.

14. Carr MM, Reznick RK, Brown DH. Comparison of computer-assisted instruction and seminar instruction to acquire psychomotor and cognitive knowledge of epistaxis management. Otolaryngol Head Neck Surg 1999; 121(4):430-434.

4. Eckdahl C, Karlsson D, Wigertz 0, Forsum U. A study of the usage of a decision-support system for infective endocarditis. Med Inf Internet Med 2000; 25( 1):1-1 8.

15. Dev P, Winzelberg AJ, Celio A, Taylor CB. Student bodies: psycho-education communities on the Web. Proc AMIA Annu Symp 1999; 510-514.

5. Lipman A, Sade R, Glotzbach A, Lancaster C, Marshall M. The incremental value of Internet-based instruction as adjunct to classroom instruction: a prospective randomized study. Acad Med 2001; 76(10):1060-1064.

16. Brown TT, Proctor SE, Sinkowitz-Cochran RL, Smith TL, Jarvis WR. Physician preferences for continuing medical education with a focus on the topic of antimicrobial resistance: Society for Healthcare Epidemiology of America. Infect Control Hosp Epidemiol 2001 ; 22:656-660.

6. Slotte V, Wangel M, Lonka K. Information technology in medical education: a nationwide project on the opportunities of the new technology. Med Educ 2001; 35:990-995.

17. Casebeer L, Bennett N, Kristofco R, Carillo A, Centor R. Physician Internet medical information seeking and on-line continuing education use patterns. J Contin Educ Health Prof 2002; 22:33-42.

7. Maki WS, Maki RH. Evaluation of a Webbased introductory psychology course: 11. Contingency management to increase use of on-line study aids. Behav Res Methods Instrum Computers 2000; 32(2):240-245.

18. Harris JM, Novalis-Marine C, Harris R. Women physicians are early adopters of online continuing medical education. J Contin Educ Health Prof 2003; 23:221-228.

8. Marshall JN, Stewart M, Ostbye T. Smallgroup CME using e-mail discussions. Can it work? Can Fam Physician 2001; 4737-563.

19. Bernhardt JM, Runyan CW, Bou-Saada I, Felter EM. Implementation and evaluation of a Web-based continuing education course in injury prevention and control. Health Promotion Pract 2003; 4(2): 120-1 28.

9. Bell DS, Mangione C. Design and analysis of a Web-based guideline tutorial system that emphasizes clinical trial evidence. Proc AMIA Annu Symp 2000; 56-60.

20. Maki RH, Maki WS, Patterson M, Whittaker PD. Evaluation of a Web-based introductory psychology course: I. Learning and satisfaction in on-line versus lecture courses. Behav Res Methods Instrum Computers 2000; 32(2):230-239.

10. Huntley AC, Conrad SJ. Internet tools in the medical classroom. Med Educ 1994; 28508-5 12. 11. Merisotis JP, Phipps RA. What’s the difference? Outcomes of distance vs. traditional classroom-based learning. Change 1999; 3 1:13- 17.

21. Curran VR, Hoekman T, Gulliver W, Landells I, Hatcher L. Web-based continuing medical education (11): evaluation study of computermediated continuing medical education. J Contin Educ Health Prof 2000; 20:106-119.

12. Goldberg HR, McKhann GM. Student test scores are improved in a virtual learning environment. Adv Physiol Educ 2000; 23( 1):59-66.

22. Bell DS, Fonarow GC, Hays RD, Mangione CM. Self-study from Web-based and printed guideline materials. A randomized, controlled

13. Adler MD, Johnson KB. Quantifying the literature of computer-aided instruction in medical

29

Internet-Based Continuing Medical Education trial among resident physicians. Ann Intern Med 2000; 132(12):938-946.

26. Grundman J, Wigton R, Nickol D. A controlled trial of an interactive, Web-based virtual reality program for teaching physical diagnosis skills to medical students. Acad Med 2000; 75(0ct S~ppl):S47-S49.

23. Baumlin KM, Bessete MJ, Lewis C , Richardson LD. EMCyberSchool: an evaluation of computer-assisted instruction on the internet. Acad Emerg Med 2000; 7(8):959-962. 24. Komolpis R, Johnson RA. Web-based orthodontic instruction and assessment, J Dent Educ 2002; 66(5):650-658.

27. Barden W, Clarke H, Young N, McKee N, Regehr G. Effectiveness of telehealth for teaching specialized hand-assessment techniques to physical therapists. Acad Med 2000; 75( 1O):S43-S46.

25. Mehta MP, Sinha P, Kanwar K, Inman A, Albanese M, Fahl W. Evaluation of Internetbased oncologic teaching for medical students. J Cancer Educ 1998; 13(4):197-202.

28. Papa FJ, Aldrich D, Schumacker RE. The effects of immediate online feedback upon diagnostic performance. Acad Med 1999; 74( 10):S 16-S 18.

30