Md Techpapersi

Technology and Child Development, Part I: A Ten-Year Review of Reviews

Submitted to: The Center for Child Well-being

The Center for Child Well-being

prepared by: The Public Health Informatics Research Laboratory

June 14, 2001

Technology and Child Development, Part I: A Ten-Year Review of Reviews

Technology and Child Development, Part I A Ten-Year Review of Reviews Atkinson, N.L., Silsby, J., Gold, R.S., Koeppl, P.T., Chokshi, A.N., & Gutierrez, L.S.

ABSTRACT In an effort to understand the value and use of technology in fostering child development and well-being, the researchers identified and synthesized literature reviews, research syntheses, and meta-analyses that focused on the positive effects of technology on children in settings where they learn. This process confirmed that this topic continues to be a controversial one, even though computer-based technology and “e-learning” are receiving credit for the increased productivity and economic success of the United States. With each advance in communication technology, educators and policy makers need to confirm the value of technology in terms of immediately tangible outcomes. While improved standardized test scores have been the most common outcome of interest, much empirical evidence shows a causal relationship between computer-based technology and student achievement in a wide variety of subjects, as well as other social and emotional outcomes. Not only are these benefits focused on the individual child, they are present in the learning environment and are changing the way children learn and interact with their teachers and peers. Technology today allows children to be active participants and collaborators, rather than passive recipients in their own learning. Unfortunately, a paucity of experimental research was found to describe the full potential of technology to influence child well-being. In two papers, we examined the evidence available on the relationship between emerging technologies and holistic dimensions of child development and well-being. Reviews and research syntheses published from 1987 through 2000 provide the raw material for these two reports that address the following questions: 1) What impact does technology used in educational settings have on child development? 2) Which uses of technology hold the greatest promise for improving child wellbeing outcomes? 3) What future research is needed to understand the impact of technology on children according to cognitive, social, emotional, and physical perspectives? This first paper focuses on literature reviews and meta-analyses and answers the first and third questions above. The second paper focuses on primary research studies and answers the first two questions. In this paper, our synthesis of the evidence leads us to answer the first question by saying that empirical, regional, and national research supports the positive impact of educational

June 14, 2001

Page 1

Technology and Child Development, Part I: A Ten-Year Review of Reviews

technology on academic performance and specific cognitive and social/emotional elements. Regarding the second question, researchers working in this area throughout the 1990s have noted the preponderance of research conducted on test scores and knowledge gain, as well as the limited focus of most research on the “technology,” isolated from how it is designed and integrated into the curriculum. More research is needed to understand its impact on other areas of child development and the contextual factors influencing effectiveness. Based on the recommendations of the synthesized reports, the paper concludes with a research agenda that outlines areas for basic, formative, and applied research.

INTRODUCTION While educators, and others, have long been excited by the potential of technology for improving cognitive and other developmental outcomes, the form and function has not always been up to the task. However, computer-based technology has evolved over the past three decades from a close-ended tutoring tool to an interactive, virtual learning environment where students and teachers can access and synthesize limitless amounts and types of information. With these new data, we must examine whether educational computer technology is beginning to live up to its promise. During the 1980s, computer-based instruction emphasized close-ended drill-and-practice and tutorial software strategies to teach students pre-determined content and skills. In the early to mid 1990s, software developers shifted toward more learner-centered approaches. Computers became tools for learner-centered practices rather than closeended delivery systems. Computers also helped teachers move from isolated learning activities to applications that involved students working and collaborating in groups. Since the late 1990s to the present, technological development has grown at an unprecedented rate, especially as schools have gained nearly universal access to the Internet. Internet access further facilitated teachers’ abilities to engage students in selfdirected, real-world learning activities, but it has also required extensive changes in classrooms and school administration. Access to the global network of multi-media information and online learning communities requires extensive planning and funding for technology infrastructure and professional development of teachers. This paper and its companion paper focus on empirical research exploring the impact of educational technology on child development and learning. In this paper, we organize and integrate research findings, with special attention to 11 meta-analyses and comprehensive literature reviews conducted over the past decade. We then interpret these research findings in conjunction with areas of inquiry that have not been sufficiently investigated. Additionally, we recommend a research agenda and suggest additional creative approaches to research design and methodology using emerging technologies. The goal of these research recommendations will be to understand the positive impacts that technology has had on cognitive, socio-emotional, and physical dimensions of child well-being.

June 14, 2001

Page 2

Technology and Child Development, Part I: A Ten-Year Review of Reviews

COMPUTER TERMINOLOGY AND BACKGROUND This section assembles the computer terminology and abbreviations used throughout this paper. It also reviews the statistical concept of an “effect size” for measuring the typical change associated with the use of an intervention. A List of Terminology and Related Abbreviations The table below lists several terms that are used in this paper, providing common definitions. Some of the definitions emerged from the meta-analyses and literature reviews that are included in this paper and its companion paper. Table 1: Terms and Definitions Technology

Media Computer-assisted instruction (CAI) Computer-based instruction (CBI) Asynchronous communication Synchronous communication Constructivist Learning Environments Constructivist Learning Theory Cognitive Tools

Assistive Technology

June 14, 2001

Any object or process of human origin that can be used to convey media (includes books, television, films, computers, the Internet and more). “Technologies are the tools that allow people to share their knowledge representations with others” (Reeves 1998). Includes a product, a tool, as well as the process of using the tool (Pierce 1994). All means of communication, whatever its format, including print, graphics, animation, audio, and motion pictures (Reeves 1998). Any instruction that uses computers in teaching. Computer-assisted instruction in which the computer delivers the lesson. One-way communication that takes place over time. In reference to the Internet, messages and information may be exchanged among groups of people via message boards, listservs, email, and newsgroups. Two-way communication that takes place in “real time.” In reference to the Internet, communication exchange among small to large groups of people via chat rooms, discussion groups, and special events online. A place where learners may work together and support each other as they use a variety of tools and information resources in their guided pursuit of learning goals and problem-solving activities (Wilson 1996). The process of how students create meaning and knowledge in the world. Tangible or intangible technologies that enhance the cognitive abilities of human beings during thinking, problem-solving, and learning. They help learners organize, restructure, and represent what they know. Examples are databases, spreadsheets, expert systems, communications software, online collaborative knowledge construction environments, multimedia construction software, and computer programming languages (Reeves 1998). Types of technology used with persons with disabilities that compensate for cognitive, sensory, motor, and communicative limitations (Pierce 1994).

Page 3

Technology and Child Development, Part I: A Ten-Year Review of Reviews

Effect Size Primary research studies that assess the effectiveness of a technological innovation on enhancing child development and well-being usually compare outcomes or achievement between two or more groups of children who either were or were not exposed to some technology. Other primary research studies look at whether students exposed to a technological enhancement in instruction, either in the classroom or in another instructional setting, have improved outcomes. Investigators ask whether the groups differ in how much and how well they have gained skills, attitudes, knowledge, or beliefs that positively impact the development of the children who comprise the groups. Given that the studies we identified vary widely, it is essential that we have a reliable common examination on potential gains as a yardstick of making comparisons. Familiar statistics like t-tests and F-tests can tell us whether differences in average test scores are statistically significant, or whether the observed differences could have arisen by chance. Statistics can further provide confidence limits for the difference of two means. Such tools, although important in answering questions about scientific hypotheses and in drawing other inferences from empirical data, do not provide easily interpreted measures of the differences seen among groups that received different levels of exposure to technologies, or different methods of instruction. A unifying metric, called an “effect size,” offers a way to compare groups or outcomes that differ from each other by creating a common measure that relates practical gains of an intervention when compared to a control group, regardless of what the intervention is. As Emerson and Mosteller (1998a) state, the effect size “is sometimes more practical; it may help us begin to address the issue of ‘practical effect,’ just as t-tests address ‘statistical significance’” (45). An effect size is not a statistical test per se, but it provides an excellent way to report differences in outcomes when comparing groups. For the purposes of this paper, an effect size applies to averages of scores on examinations conducted to determine the benefits of technology on child development and well-being.

June 14, 2001

Page 4

Technology and Child Development, Part I: A Ten-Year Review of Reviews

Effect Size To calculate an effect size for the benefits of technology on child development and well-being, we begin by finding the difference between the average examination score for the outcomes in the group that was exposed to the technology, and the average score for the outcomes in the control groups (or in some other comparison group). The effect size is then obtained by dividing this difference in averages by the standard deviation of the scores from the control group. Thus, an effect size is a measure of the average gain (or loss) in benefit associated with the innovation or technology, with the measure reported relative to the natural variability of the test scores from students who were not exposed to the technology. The effect size, then, is an average gain in an experimental group measured in standard deviation units of the scores in the control group.

Effect Size

=

Mean Score Experimental Group – Mean Score Control Group ---------------------------------------------------------------------------Standard deviation of the control group

Cohen (1988) gives a guide to the magnitude of effect sizes in the social sciences. An effect size of 0.20 is considered small, an effect size of 0.50 is considered medium, and an effect size of 0.80 is considered large.1 Although 0.80 is considered a large effect size, effect sizes can be larger than one. Where possible, effect sizes for the summarized studies are included.

Study Design and Materials Our searches provided more than 350 potentially relevant research articles, reports, books, and book chapters appearing since 1990. These articles are categorized in the table below. Information in the abstracts, introduction, and tables of contents narrowed our focus to 119 articles, reports, and chapters. The first paper relies on reviews and systematic syntheses of research about the effect of technology on children. We found 11 of these articles that were published since 1987. We also focused on primary research articles since 1990 that met each of the following criteria for inclusion: 1. Assesses the positive impact of technology on child development and well-being. 2. Uses subjects from conception through age 18. 1

The effect sizes listed are considered the standard of measure if the calculations are based on t-tests (es = effect size). The standard measure for magnitude of an effect size based on ANOVA results is slightly different. With ANOVA results, the statistic is F, and the magnitude of the effect is considered small if es = 0.10, medium if es = 0.25, and large if es= 0.40.

June 14, 2001

Page 5

Technology and Child Development, Part I: A Ten-Year Review of Reviews

3. 4. 5. 6. 7.

Uses impact on child well-being as a primary basis for evaluation. Provides data that enable the comparison of two or more treatment groups. Evaluates impact in the context of environments where children work and live. Reports work done in the United States and Canada. The article is asset-based and focuses on positive outcomes (negative outcomes of technology use, such as carpal tunnel among computer users, were considered to be outside the search criteria).

Table 2 below summarizes the categories of articles reviewed; Appendix 1 provides additional details and the rationale for this particular classification. Table 2: Results from Searches for Literature About the Affect of Technology on Child Health, Development, and Well-being Description Articles identified by searches and other means Articles identified for reading and extraction of information

Number of Articles 350 119

Mutually exclusive categories of articles Research syntheses and review articles since 19871

11

Primary controlled research since 1990

41

Qualitative and quasi-experimental research since 1990

19

Descriptive, theoretical, philosophical, advocacy

36

Statistical methods and methods for research synthesis and evaluation Subtotal 1

12 119

We limited our literature search to reports appearing since 1990, except for research syntheses. To be consistent with the methodology by Emerson and Mosteller, we limited searches of research syntheses to reports appearing since 1987.

SUMMARY OF 11 LITERATURE REVIEWS AND META-ANALYSES The following literature reviews and meta-analyses are listed in chronological order. We focused on the findings related to the following research questions: 1) What impact does technology used in educational settings have on child development? 2) What future research is needed to understand the impact of technology on children according to cognitive, social, emotional, and physical perspectives?

June 14, 2001

Page 6

Technology and Child Development, Part I: A Ten-Year Review of Reviews

1) Kulik, J., and Kulik, C.L. (1991). Effectiveness of Computer-Based Instruction: An Updated Analysis. James Kulik and Chen-Lin Kulik have conducted many meta-analyses on the effect of computer-based instruction (CBI) on student achievement from the elementary school level to college and adult training levels. Their 1991 report updates their 1987 report and integrates the findings from 254 studies that compared student learning in classes taught with and without CBI. They found that the average effect for CBI was to raise test scores by .30 standard deviations. This means that in a typical study, the performance of CBI students was .30 standard deviations higher than the performance of the control students. Students who used CBI liked their classes better (average effect size=.28, based on n= 22 studies) and had more positive attitudes towards computers (average effect size=.34, n=19). Another major finding was that computer use reduced instruction time by twothirds of that required by traditional teaching methods in 29 out of the 32 studies that reported results of time spent on instruction. The researchers also examined possible study characteristics that may have biased their findings. Effects were larger in studies conducted in four-week time frames, as opposed to entire semesters or academic school years (24-36 weeks). This difference may exist because learning outcomes after shorter periods of time were more sensitive to the computer intervention, than were the outcomes measured later at the end of a semester or school year. This suggests that the novelty of CBI may wear off, with students going back to traditional study habits and responding to computer instruction in much the same way they respond to traditional teacher instruction. Effects were also larger in studies where different teachers taught experimental and control classes. These results may be explained by differences in the teachers or by the Hawthorne effect in classes getting the novel treatment. Fifty-three of the 254 studies were newer studies, not included in Kulik and Kulik’s previous 1987 meta-analysis. Of the new studies, 16 measured achievement gain at the elementary school level and 9 at the high school level, with effect size magnitudes ranging from -.42 to .88. For elementary school studies, the average effect size for CBI was .27 (standard error=.07), compared to .21 (se=.07) for high school studies.

2) Pierce, P.L. (1994). Technology Integration Into Early Childhood Curricula: Where We’ve Been, Where We Are, Where We Should Go. Center for Literacy and Disability Studies, University of North Carolina, Chapel Hill. Pierce (1994) reviewed the literature related to the use of technology with young children, ages birth through five years, to describe its use, its impact, and suggestions for improving use, both with very young children and young children with disabilities. She asserted that the preschool classroom had changed in the last decade because of the inclusion of preschool children with disabilities—due to social policy and legislation

June 14, 2001

Page 7

Technology and Child Development, Part I: A Ten-Year Review of Reviews

such as the Individuals with Disabilities Education Act (IDEA)—and because of the integration of technologies into the curriculum. In this review, technologies included television, videos/interactive, videodisc, and computers and software found being used as part of the curricula for young children. In all cases, the author found that early research dealt with concerns with the use of technology-based materials and its effect on skill development. As concerns proved to be unfounded, research then evolved to explore ecological questions, such as, how to most successfully use technology in the classroom. We focused on findings that show the positive impact of technology via computers and software. We also focused on findings that provide guidance on how technology can be successfully integrated into preschool settings with positive outcomes on child development, for children with and without disabilities. Research evidence supported the impact of computer use on several developmental domains. These included: • • • •

• • •

Eye-hand coordination Cognition (i.e., memory, spatial problem solving, logical problem solving, selflearning, self-organization, concentration) Oral language (i.e., number of spoken words a minute, number of foreign language words learned, amount of communication to teach other students) Literacy (i.e., letter naming, beginning word recognition, engagement in literacyrelated activities—such as making lists and story reading and writing, number of words written, elaboration in stories, second language writing abilities, interpreting symbols, letters, and words) Mathematics (i.e., shape recognition, counting, sorting) Social/emotional development (i.e., self-efficacy, self-esteem, overall satisfaction, cooperative learning, comfort in a technology-driven society, improved interpersonal relationships) Creativity and artistic abilities

The author described a number of specific findings about the influence of computer technology on child development on the above domains. Several researchers found that computer use helped children develop higher thought processes, moving them from concrete to symbolic representational thought. At the computer, both preschool children without disabilities and preschool children with disabilities had greater language production as measured by the number of spoken words per minute, than when they were participating in other learning activities. In terms of social and emotional development, the cooperative learning that occurred during computer use was attributed to the preference of preschool students to work in pairs or small groups. Further, students’ increased self-esteem was related to improved interpersonal relationships. Another finding was that early computer use decreased gender differences seen among older children in computer use and comfort, suggesting that preschool computer use could be important to ensuring that girls are as prepared as boys to pursue scientific and technical careers.

June 14, 2001

Page 8

Technology and Child Development, Part I: A Ten-Year Review of Reviews

Research cited also demonstrated the benefits of assistive communication technology for children with disabilities by helping those children to develop according to the “ABC Model” (Augmenting abilities and Bypassing or Compensating for disabilities). Conducted largely with hearing impaired and deaf children, research in this area has shown that technology has benefited the following developmental domains: • • • • • •

Social skills Higher peer acceptance Social interaction Reading and writing skills Word recognition and identification Communication through American Sign Language

The author suggested several mechanisms by which technology was able to improve the development of children with disabilities. For children previously taught in selfcontained settings, technology may expand the range of educational experiences by providing vicarious experiences. It may also help children with severe physical impairments feel some control over their environments. Using software and computer games improved the social skills in preschool children with social deficits and speechlanguage impairments by serving as a point for joint attention and social interaction among children using the materials in a collaborative way. Further, computers link action and language closely in instruction and conjoin multiple methods of presenting information in a way that can help children, with disabilities or not, learn to read and write. Simply using technological applications was not sufficient to achieve positive developmental gains. Pierce cited several factors that enhanced the positive effect of technology: type of teaching strategy, type of learning activity, instructional design features, parental involvement, and establishing clear guidelines and support for integration in the school setting. On the subject of teaching strategy, Pierce compared studies on the impact of drill-andpractice to open-ended constructivist teaching strategies in both written language and mathematical development. Drill-and-practice activities were used more exclusively early in the application of computers in preschool settings. Drill-and-practice methods proved to increase early concepts in written language and mathematics—letter naming, beginning word recognition, shape recognition, counting, and sorting. However, constructivist strategies were better able to help preschool students to learn more complex skills and proved to motivate students to learn. For example, when constructive writing procedures were employed to teach literacy while accomplishing real tasks, such as making lists or writing stories at a word processor, children wrote stories of greater quantity (number of words) and quality (elaborate narrative) than when they used pen and paper.

June 14, 2001

Page 9

Technology and Child Development, Part I: A Ten-Year Review of Reviews

Certain learning activities also enhanced the impact of technology on child development. Students had greater skill development when they created materials delivered via technology because they knew others would see and hear the results of their work. Students also learned better when using the computer as a tool to accomplish real purposes; the computer supported an active learning environment rather than serving as the focus of learning. Students learned better on computers when activities were introduced over time and when they were able to have time to explore software by themselves. Teachers could teach more effectively with computer or videodisc supplemental materials because they could be stopped to allow for group discussion. The author recommended a variety of computer software for the best results in the preschool classroom. Besides programs that focus on early skills, such as letter recognition and counting, the teacher should also have open-ended tools such as writing and drawing software. They should also have playful exploratory programs that teach concepts with entertaining animated graphics and positive feedback to foster success. Lastly, CD-ROM storybooks that feature instantaneous animation, sound, and voice output should be part of the preschool software library. In general, these programs should have child-friendly controls and graphics; be flexible to a variety of educational needs and goals; be colorful, animated, and responsive; and have teacher control options. Parental and school setting factors also promoted positive technology use. Parental support and instruction fostered preschool students’ abilities to make decisions and to control their environment. Clear curricular goals, operational guidelines, and teaching training for educational technology allowed schools to purchase appropriate software, plan for effective classroom use, and provide for the maintenance of hardware and software. Over time, integrating technology successfully will transform instruction from teacher-centered to learner-centered and from an emphasis on lower-level skills to higher-level analysis and problem solving.

3) U.S. Congress, Office of Technology Assessment. Technology: Making the Connection.

(1995).

Teachers and

The U.S. Office of Technology Assessment (OTA) issued this report in 1995 to answer the question, “How can schools use technology more effectively?” While the report focuses on individual teacher, system, and policy barriers to integrating technology, the authors review the evidence supporting the effectiveness of educational technology, the issues in effectiveness research, and recommend directions for future research. The OTA analysis of meta-analyses revealed that studies have consistently demonstrated that computer-assisted instruction is either equivalent or superior to conventional instruction. Effect sizes ranged from .26 and .66 standard deviations in these studies, indicating a sizable improvement on many achievement measures. The studies they reviewed also consistently reported positive attitudinal effects among students using educational technology.

June 14, 2001

Page 10

Technology and Child Development, Part I: A Ten-Year Review of Reviews

Understanding effectiveness in this area requires taking into account conceptual, methodological, and timeliness issues. Conceptually, finding out whether one technology can beat another is less important than understanding the mix of technology, content, and pedagogy that affects learning positively. Researchers must also consider how the technology is used in the classroom context and for which types of student a particular application is most suited. Concerning research methodology, those looking at educational technology need to understand the difficulty of conducting controlled research in the school setting, as well as the challenge of measuring the full range of appropriate outcomes. Unlike laboratory research, quantitative research in the classroom is hampered by problems in finding comparable comparison groups, teasing out the effects of technology from the student’s entire learning experience, and ensuring that teachers make sure control groups do not benefit from advances in the experimental approach. According to OTA, measures of the impact of technology must reach beyond existing measures of student achievement to other areas, such as higher-order thinking, that many believe are positively affected by new technologies. Measures must also look beyond cognitive gains because student achievement is related to other indicators of child well-being—students’ self-concept, attitudes about school and learning, and ability to work collaboratively. Timeliness refers to the ability to conduct effectiveness research quickly enough so that the appropriate technologies are identified and made available to students. Given the rapid pace of development and slow pace of research, some technologies are obsolete by the time they are proven valuable. Rather than focusing on specific technologies, OTA recommended that future research examine the context in which technologies improve teaching and learning for children over time. The outcome of these studies would be recommendations on how to design technology environments, which instructional approaches work best in conjunction with certain subject matter and technologies, and the role of the teacher in integrating technology into the classroom.

4) Statham, D.S., and Torell, C.R. (1996). Computers in the Classroom: The Impact of Technology on Student Learning. In a cooperative research project of the Consortium Research Fellows Program, the U.S. Army Research Institute, and the Boise State University College of Education, researchers analyzed literature prior to 1996 on the efficacy of computer use with elementary, secondary, and at-risk students. Although the report included nonexperimental research in its reviews of primary and secondary sources of data, we focused on only the experimental and meta-analytic studies cited. Several of the articles profiled examine the impact of technology in elementary and secondary education. Studies showed that children in kindergarten, third grade, and eighth grade improved their writing skills by using a word processor compared to

June 14, 2001

Page 11

Technology and Child Development, Part I: A Ten-Year Review of Reviews

children who did not. Computer-using students also experienced a more studentcentered, work-focused, collaborative learning environment than their counterparts. In a study of math-related software, fifth grade students made gains in math achievement after using the software, but seventh and eighth graders did not. The author of this mathrelated software study attributed these findings to teachers in the higher grades using the software as a supplement rather than integrating it in a meaningful way into their curricula. Elementary school children using geography-related software also did not demonstrate improved knowledge recall compared to a control group. The authors found studies that revealed the potential of improving the cognitive development of at-risk students, who “have at least average potential to learn but their academic achievement in the core areas of learning—reading, writing, and arithmetic—fall far short of their potential” (23). Both low achieving fourth grade students, at-risk seventh grade students, and remedial eleventh grade students improved their writing skills significantly after using educational technology. The authors also reviewed evidence from secondary sources. One meta-analysis by Kulik and Kulik (1991) showed that 81 percent of the studies demonstrated higher examination averages among students in CBI classes than in conventionally taught classes. In another meta-analysis, researchers found that two thirds of the 32 studies on the effect of word processing on writing quality found that access to word processing during writing instruction improved the quality of students’ writing, especially among students with basic (low) writing ability (Bangert-Drowns, 1993). Another meta-analysis by Fletcher-Flinn and Gravatt (1995) found that students in CAI classes had scores .24 standard deviations higher than comparison students. Most interesting, effect sizes (Fletcher-Flinn & Gravatt, 1995) tended to be highest among students in kindergarten and preschool (.55), followed by those in elementary school (.46) and high school (.32). The lowest effect sizes were reported at the college/university level (.26) and among adults in training situations (.22). In special education classes, CAI reported the largest effect sizes (.56). The report provided several overall conclusions: • • •

•

When computer access is sufficient and computer technology is employed appropriately, student learning is improved. To maximize the number of students who succeed, students who are most likely to show the greatest gains—those who are educationally at-risk of failure—must be allocated computer time. While computer use has shifted from teaching programming to teaching computer literacy and providing drill-and-practice sessions, computers offer their greatest benefit when used for enrichment and as work tools. This leads to the development of higher level information-seeking and problem solving skills. Giving students the opportunity to learn via the computer empowers them to take an active, participatory role in learning.

June 14, 2001

Page 12

Technology and Child Development, Part I: A Ten-Year Review of Reviews

5) President’s Committee of Advisors on Science and Technology. (1997). Report to the President on the Use of Technology to Strengthen K-12 Education in the United States. The President’s Committee of Advisors on Science and Technology held a panel on educational technology. As background to the panel’s work, the committee reported on results of four meta-analyses examining the effectiveness of traditional computer-based instruction. The analysis of the literature found that the majority of outcome measures were standardized test results. Evidence demonstrated that traditional applications most benefited students from lower socioeconomic status homes and those who were low achieving. Additionally, students learned faster, enjoyed classes more, and had more positive attitudes towards computers. Researchers have had various criticisms about the state of educational technology research. Some researchers have questioned the methodologies and experimental designs of these studies, as well as the amenability of these studies to meta-analytic aggregation. Further, constructivist approaches—which are currently emphasized for developing complex critical thinking skills—have not been subjected to enough experimental research to prove that these applications achieve positive educational outcomes. Although researchers and educators have written about constructivism and conducted case studies and reviews of constructivist applications, no rigorous studies have been conducted to tease out the underlying sources of positive effects. One challenge is the lack of well-defined, well-accepted metrics for the comparative evaluation of educational outcomes within a constructivist context. The Panel believed three areas of research should be supported at the Federal level: 1. Basic research in learning-related disciplines and educationally relevant technologies. 2. Exploratory research for developing new software, content, and technologyenabled pedagogy. 3. Empirical studies to determine which approaches to the use of technology are most effective for what sorts of learning.

6) Reeves, T.C. (1998). The Impact of Media and Technology in Schools. The Bertelsmann Foundation. This report summarized the evidence for the effectiveness and impact of media and technology used in the schools (kindergarten through twelfth grade) according to two different approaches. Findings were organized in terms of the effectiveness of learning “from” technology as opposed to learning “with” technology.

June 14, 2001

Page 13

Technology and Child Development, Part I: A Ten-Year Review of Reviews

Learning “from” technology was referred to as computer-based instruction, integrated learning systems, and instructional television. It is analogous to learning content “from” a text book. Although these types of technologies were easy to deliver, and their effect on student achievement could be measured, the question of whether they enabled learning more than traditional classroom methods remained unresolved. Modest and inconsistent differences have been found in comparisons of technology and teachers as mechanisms for instruction. The greater value of technology-based tutoring was found in its ability to motivate students, decrease instruction time, and increase equity of access. Learning “with” media and technology meant using computer-based cognitive tools—such as databases, spreadsheets, expert systems, communications software, and programming languages—to facilitate critical thinking and higher order learning. Learners used these tools to analyze the world, access and interpret information, and represent what they know to others. For example, students can explore actual data to answer their own questions; this inquiry-based approach allows more active and complex learning. Cognitive tools required learners to think in meaningful ways about how to use an application’s features to represent what they know. Students constructed knowledge rather than reproducing it. Cognitive tools were learner-centered and controlled, not technology driven or teacher-controlled. Learning with technology was more productive but not emphasized enough in the schools. The author conducted a search of the ERIC database in 1997 and found 250 publications related to the use of multi-media in the schools. The vast majority of the publications were based on the perception that multi-media technology is something students learn “from” rather than “with.” Several multi-media tools and projects that take a constructivist approach to learning with technology were reviewed. Research on four projects are summarized below: • LOGO, a programming language or “cognitive tool” that students use to develop problem-solving skills, has had mixed results. Early studies conducted in the late 1980s were not able to demonstrate the prediction that it would enable students (thirdand fifth-grade students working in groups of 2 to 4 students) to develop generalizable problem-solving skills. Newer versions of LOGO have been developed recently, such as LEGO/Logo, that involve real objects that students can program with LOGO instructions. Early qualitative studies have demonstrated positive results in increasing the relevance of math and science concepts. • The Jasper Woodbury Series, a video and interactive videodisc curriculum, has been shown to improve performance in mathematical and scientific knowledge, higher level problem-solving skills, solving word problems, and creativity among students in grades 5 and up. • The CoVis Collaboratory is a high school level learning environment that combines the objects and tools of constructivism with communication and visualization tools

June 14, 2001

Page 14

Technology and Child Development, Part I: A Ten-Year Review of Reviews

that enable collaboration among science students in a socio-cultural context. Its key components are a project-based science learning pedagogy, visualization tools for open-ended inquiry, and network environments for synchronous and asynchronous communication and collaboration. • The Apple Classroom of Tomorrow (ACOT) is a learner-centered education project ,begun in 1985 by Apple Computers, Inc. Research found that elementary and middle school students participating in ACOT “became socially aware and more confident, were able to explore information and represent it in many forms, communicated effectively about complex processes, used technology routinely and appropriately, became independent self-starters, knew their expertise and shared it spontaneously, worked collaboratively, and developed a positive orientation to the future” (37). In discussing the future of media and technology in the schools, Reeves presented the controversy about whether empirical evidence demonstrated that media and technology were any more effective than other instructional approaches. Some argued that media and technology were merely vehicles for delivering instructional methods. It is the instructional methods—based on pedagogy, design, and student activities—that accounted for learning. Reeves attributed insufficient empirical evidence about the effectiveness of media and technology to the fact that “most research studies confound media and methods.” Kozma (1991) recommended altering the research emphasis from questions about whether media and technology impact learning to questions concerning the ways in which media and technology can be used to influence learning for particular students, tasks, and contexts.

7) Lou, Y., Abrami, P.C., and Muni, S. (1998). Effects of Social Context When Learning with Computer Technology: A Series of Meta-Analyses. At the American Educational Research Association (AERA) annual meeting in 1998, Lou, Abrami, and Muni (1998) presented dissertation research findings on a quantitative synthesis of the literature. The findings focused on the effect of the number of computer users (small groups of 2 to 5 people or individuals) on individual achievement, group task performance, task behaviors, and attitudes. A total of 447 effect sizes were calculated from 103 studies involving 18,319 learners. The effect of small group learning on individual achievement tended to be larger than individual learning in several situations. For example, group learning was greater when group size was as small as two members (effect size of .18), when students have group work experience (effect size of .31), and when working with drill-and-practice tutorial programs (effect size of .19). Group learning was also more effective in raising individual achievement among relatively high or low ability students (effect sizes of .22 and .30, respectively) and female students (effect size of .50). Small group learning had a positive effect on student attitudes toward learning the subject, toward group work, and toward classmates. They found that individual learning with computers was as effective

June 14, 2001

Page 15

Technology and Child Development, Part I: A Ten-Year Review of Reviews

as group learning when students worked with computer programs as exploratory environments or as tools for other learning, and for students who were males or of medium ability.

8) Schacter, J. (1999). The Impact of Education Technology on Student Achievement. Milken Exchange on Education Technology, Milken Family Foundation. The Milken Exchange on Education Technology analyzed 5 large-scale studies and 2 illustrative smaller scale studies to outline the impact of education technology on learning. The large studies were selected for their scope, comprehensive samples, and generalizability to local, state, and national audiences. The smaller studies were chosen to reveal the promise that newer technologies may afford. We reviewed 4 of the largescale studies. We did not review the study by Sivin-Kachala and Bialo here because we reviewed a more recent report by these authors later in this report (see #11 on page 20). Overall, the author concluded that the studies profiled show that students with access to CAI, integrated learning systems technology, simulations, software teaching higher order thinking, collaborative networked technologies, or design and programming technologies showed positive gains in achievement. Evidence for achievement gains have been found in researcher-constructed tests, standardized tests, and national tests. The author found that the evidence base was clear at the empirical level in review of the 1994 study by Kulik. The author extracted information from James Kulik’s 1994 article. They found that 11 meta-analyses in studies of computer-based instruction showed experimental subjects demonstrated a percentile gain in achievement of between 9 and 22 points over the control groups. Five of the 11 meta-analyses focused on findings in studies looking at the elementary or secondary level and found a 10 to 16 percentile gain among the experimental group over the control group (see Table 3). Table 3: Table excerpted from Kulik (1994) * Meta-Analysis Bangert-Drowns, Kulik, & Kulik (1985) Burns & Bozeman (1981) Hartley (1978) Kulik, Kulik, & Bangert-Drowns (1985) Niemiec & Walberg (1985)

Instructional Level

Number of Studies Analyzed

Percentile Gain over Control Group

Secondary

51

10

44

14

33

16

Elementary

44

16

Elementary

48

14

Elementary & Secondary Elementary & Secondary

* Kulik, J.A. (1994). Meta Analytic Study of Findings on Computer-Based Instruction.

June 14, 2001

Page 16

Technology and Child Development, Part I: A Ten-Year Review of Reviews

When reviewing another of the 4 articles, the author examined the evidence at the state level in a 1999 study by Dale Mann. Mann conducted a comprehensive study in West Virginia looking at how fifth-grade students’ achievement was affected by participation in a Basic Skills/ Computer Education (BS/CE) program (Mann 1999). The findings showed that the more time students spent in the program translated into improved Stanford 9 test scores, especially for lower achieving students. Further, a cost benefit analysis of this program showed that BS/CE was more cost effective in improving student achievement than class size reduction, increased instructional time, and cross age tutoring programs. A third study, by Wenglinsky, concerned a 1998 national survey examining the effects of computer use on higher math achievement among a national sample of fourth-grade students (N=6,227) and eighth-grade students (N=7,146). Controlling for socioeconomic status, class size, and teacher characteristics, the study found that math achievement and professional development were positively related to computer use (fourth graders used computers primarily from math/learning games, and eighth graders used them for simulations and applications). Eighth-grade students showed gains up to 15 weeks above grade level as measured by the National Assessment of Educational Progress (NAEP), and fourth grade students showed gains up to 3 to 5 weeks ahead of students who did not use technology. Professional development on computers for teachers was related to student gains in math scores of up to 13 weeks above grade level. A fourth study Schacter reviewed was on an 8-year analysis of the Computer Supported Intentional Learning Environment (CSILE). CSILE has demonstrated that collaborative communication applications were effective in improving standardized scores and higherlevel cognitive skills. CSILE students surpassed students in control classrooms on standardized reading, language, and vocabulary tests. CSILE students also surpassed controls on measures such as depth of understanding, depth of expectations, reflection, expectations for knowledge growth, and identifying conceptual difficulties.

9) Culp, K.M., Hawkins, J., and Honey, M. (1999). Review Paper on Educational Technology, Research, and Development. The Center for Children and Technology, Education Development Center. The Center for Children and Technology reviewed the recent history of research related to educational technology and described key research themes. They placed research in the context of the technological change; specifically, advances in technology changed the research questions being asked and the methods required to answer them. In the 1970s and 1980s, empirical studies were tied to the type of technology used by the students. With advances in connectivity, visual display, multi-media capabilities, and speed, more recent research focused less on improvement in standardized test scores and more on how technology helps students develop critical and creative thinking. They wrote that research was also needed to describe how technology use was mediated by factors related to the teacher, the classroom, and the socio-cultural setting of the school.

June 14, 2001

Page 17

Technology and Child Development, Part I: A Ten-Year Review of Reviews

The authors outlined 14 thematic areas for future research, of which 10 have direct application to examining the impact of technology on child development. Many of these areas are currently being practiced and evaluated in model programs throughout the country, but they have little complete evaluation or empirical research. Each area contained several research questions and required research methods that extend beyond traditional approaches. These 10 thematic areas were: • Making real-world connections. This research area involves students using the Internet to participate in ongoing investigations of real-world issues and problems. Students are active producers of knowledge rather than passive recipients. Research is needed to build a stronger understanding of how projects can best be designed to help students learn and to help them discern worthwhile sources of information. • Engaging in complex analysis. As students have increased access to complex datasets and other primary source materials, research is needed to understand how this has changed teaching and learning and how these tools can be presented in schools. Researchers also need to identify the types of new tools and environments to develop to support student exploration. • Home/school/community connections. This area refers to expanding environments for learning across multiple contexts beyond schools—in homes, communities, museums, and libraries. Research needs to be conducted to show how technology might help schools establish stronger connections with students’ homes, the local community, and other social institutions that positively influence children’s development. • Teacher learning and professional community. Continuing education and professional development for teachers may be addressed through the use of technology. University-based distance learning centers are supporting many teachertraining programs, but work also needs to be done to explore online peer-to-peer learning situations for teachers and online teacher communities. • Reorganizing the education workplace. Technology can also help teachers, administrators, and students become more efficient and improve administration and coordination functions. An example is the use of intranets among faculty to facilitate information exchange and work processes. • Equity and access/gender/special education. Development of culturally- and educationally-appropriate technologies to address the specific needs identified by special populations and to ensure equitable access to those technologies. Research is needed to further understand the differential impact of technologies on various subpopulations. • Emerging technologies and challenging content. Relatively more research and development investment has been devoted to science and mathematics, compared to the humanities. Future technology development should reflect the needs of the educational community, and be based on matching disciplines and content areas with the appropriate emerging technologies. • Supercomputing. Advanced technologies, such as powerful computers, high speed networks, and sophisticated software are now more available at the school and district levels. Students can explore complex systems and virtual environments and communities, which can help them understand phenomena like never before.

June 14, 2001

Page 18

Technology and Child Development, Part I: A Ten-Year Review of Reviews

Research is needed to explore how students can be best supported and how technology can facilitate high-level learning experiences. • Telementoring. The capacity for developing online mentoring via email, chat rooms, and other Internet resources is increasing rapidly, and it presents new challenges. Research and development is needed to structure telementoring programs for different purposes, facilitate introductory and goal-setting processes, sustain communication, and find ways to achieve tangible results of working relationships. • Computer-assisted instruction. CAI allows the development of more sophisticated systems that respond to specific cognitive strengths and weaknesses of learners based on information they provide. Therefore, educational materials are “customized” more closely to learner needs. Research is needed to determine the impact of CAI on knowledge transfer, sustained knowledge, and optimal combinations of CAI with other forms of instruction.

10) Valdez, G., McNabb, M., Foertsch, M., Anderson, M., Hawkes, M., and Raack, L. (2000). Computer-Based Technology and Learning: Evolving Uses and Expectations. The North Central Regional Educational Laboratory. The report analyzed the literature and presented available evidence from each phase of computer-based technology that had a positive effect on learning. It then discussed the significance of these findings for educators as they make technology-related decisions. This literature review and recommendations were organized according to three distinct phases in technology uses and expectations: Phase 1: Phase 2:

Phase 3:

Print Automation. Relies on drill-and-practice to teach segmented content and/or skills in a close-ended, linear fashion (learning “from”). Expansion of Learning Opportunities. Computers are tools for learnercentered practices, rather than content delivery mechanisms (for learning “with”). Helps teachers move from largely isolated learning activities to applications that involve working in groups. Data-Driven Virtual Learning. Makes classrooms more effective through increasing connectivity, and makes schools more effective through sophisticated data-driven decision-making. Teachers use access to improve lesson plans and meet accountability expectations. For students, this technology offers a range of instructional opportunities that enhances the curricula they experience.

The authors presented a detailed grid of variables related to computer-based technology in learning environments and how they were manifested in each evolutionary phase. These variables included areas, such as, the roles of students and teachers in an engaged learning environment and the relationship of standards, conceptual integrity, and authentic tasks to instructional content. For example, the variable of “technology connectivity” was limited to electronic print in the first phase, where information was transferred in a discrete format such as that provided by diskette. In the second phase,

June 14, 2001

Page 19

Technology and Child Development, Part I: A Ten-Year Review of Reviews

this evolved to electronic print with multi-media and networking capacity; connectivity was tied to a hard drive. In phase three, unlimited information transfer and online collaboration was attained through the vast multi-media and global telecommunications infrastructure. A description of computer-based technology in the first phase revealed that the primitive computers of that era limited instructional software to simpler, segmented content taught through drill-and-practice. Despite the drawbacks of this instructional strategy, research showed that technology had a positive impact on student achievement and test scores. In the second phase, advances in sophisticated computer-based technologies (such as CDROM’s and digital technology) and content utilizing multi-media (such as sound, pictures, video, graphics, charts, maps, etc.) began to offer huge amounts of information. Computers became tools for learner-centered practices, teachers emphasized collaborative activities, and students had greater opportunities to investigate and answer complex questions. Research demonstrated that technology positively impacted teacherstudent interaction, cooperative learning, and problem solving and inquiry. In addition, most students considered computer activities to be highly motivating and interesting. Effectiveness of educational technology depended upon the match between the goals of instruction, characteristics of the learners, the software design, the technology, and the implementation decisions made by teachers. Subsequently, in the third phase, the Internet made access to amounts and types of information limitless, requiring students to develop more advanced thinking skills for sorting, evaluating, and synthesizing the information. Research continued to suggest that computer-based educational technology led to student achievement, as well as gains in higher-order skills, such as critical thinking, problem solving, and synthesis. Further classroom-based research was indicated to allow for a theory-based research synthesis. The overall conclusions of this review and analysis were that technology: • can make learning more interactive, • enhances enjoyment of learning, • customizes curriculum to learners’ developmental needs and personal interests, • captures and stores data for informing decision-making, • encourages collaboration among family members and the school community, and • improves methods of accountability and reporting. 11) Sivin-Kachala, J., and Bialo, E.R. (2000). Report on the Effectiveness of Technology in Schools, Software & Information Industry Association. The Software & Information Industry Association periodically prepares reports summarizing leading research on the effectiveness of technology in K-12 and higher education to provide software developers and publishers with information that will “enable them to improve educational technology so it continues to have a significant positive impact” (14). In this seventh edition of the report, they continued to find that

June 14, 2001

Page 20

Technology and Child Development, Part I: A Ten-Year Review of Reviews

technology, as a learning tool, can improve teaching and learning. However, they also asserted that promoting desired outcomes depends on the type of learner, use of effective instructional design strategies, and successfully integrating technology-based applications into the learning environment. The current literature review included research conducted with college students, adult learners, and children age 18 and younger. We focused on the findings that relate to children. The report found positive effects of technology in the following areas of early childhood education: • • • • • • •

Intelligence, non-verbal skills, structural knowledge, long-term memory, and complex manual dexterity (preschool students). Verbal skills (preschool students). Academic skills, memory growth, and visual perception (Head Start preschool students). Concepts of left and right (kindergarten students). Word identification, picture-word identification, passage comprehension (kindergarten). Attitudes toward reading (kindergarten students). Positive self-concept (Head Start preschool students).

The report also found positive effects of technology in the following areas of child development among children in elementary and secondary classrooms: • • • • • • • • • • • •

Language development. Reading (phonological awareness, reading comprehension, vocabulary, and reading age). Spelling (spelling age). Writing (writing quality, writing maturity, focus/organization, mechanics, persuasive writing, and number of words). Mathematics (problem, data, and concept analysis, problem solving, and solving word problems). Science (declarative knowledge, chemistry knowledge, meteorology knowledge, and dissection skills). Social studies knowledge. Foreign language (grammar and vocabulary). Logo and programming languages (reasoning skills, logical thinking, problem solving, verbal creativity and metacognitive processes). Career education (readiness to make educational and vocational decisions). Self-concept (feelings of success in school, self-esteem, and self-confidence). Positive attitudes toward various curriculum areas: o Language arts (writing, using the library, and spelling practice). o Mathematics (math in general, problem solving, business planning, and geometry). o Science (interest in a science career, perception of science, chemistry, finding science fun and important, and curiosity).

June 14, 2001

Page 21

Technology and Child Development, Part I: A Ten-Year Review of Reviews

o Social studies (academic intrinsic motivation, and self-efficacy). At-risk students also demonstrated positive outcomes from exposure to educational technology in the following areas: • • • • • • • • •

Vocabulary development and reading comprehension (learning disabled elementary students). Reading skills (low performing ninth grade students). Writing quality (learning disabled elementary students). Knowledge of fractions (learning disabled high school students). Automaticity, or the ability to recognize words instantly while reading (elementary age students needing extended math practice). Math skills (low performing ninth grade students). Language development (special education preschoolers). Time writing (emotionally disturbed students in grades 6 to 12). Decreased math anxiety (low ability sixth grade students).

Positive gains in the above areas were often attributed to software design characteristics. The evidence-based findings supported the following design features: • • • • • • • • •

Learner control over the amount and sequence of instruction (although lowachieving students may need more structure and guidance). Immediate corrective feedback. Embedded cognitive strategies, such as repetition, paraphrasing, cognitive mapping, illustrative examples, and pictorial information. Embedded conceptual change strategies that move students to more accurate understanding of concepts. Instructional scaffolding (gradually decreasing the level of help available or increasing the task complexity). Animation and video, accompanied by narration if possible. Captioning to support video and audio. Content-related graphics. Navigation map showing the linkages and hierarchical structure of information.

The report also found that the learning environment adapted when technology was introduced. Learning became more individualized and student-centered, cooperative learning was encouraged, and teacher-student interaction increased. Certain characteristics helped learning environments to maximize the benefits of educational technology, including district level support and leadership, teacher training, peer support among computer-using educators, smaller class size, and adequate funding for hardware and software. Educators should provide learning activities to familiarize students with software tools. They should provide self-directed experiences and activities that encourage self-expression. Lastly, educators should provide collaborative learning activities where students can benefit from personal interaction among class members.

June 14, 2001

Page 22

Technology and Child Development, Part I: A Ten-Year Review of Reviews

PRINCIPLE FINDINGS Several of the research reviews noted the evolution in educational technology and the research assessing its effectiveness. The findings were therefore organized to reflect the evolution from examining effectiveness to examining the design of instruction and the influence of the learning environment.

Effectiveness of Educational Technology The research was clear that technological applications have been, and continue to be, effective in promoting learning and social and emotional assets among children, from preschool to high school and beyond. All of the studies that met the screening criteria supported the effectiveness of educational technology to positively affect child development cognitively, socially, and emotionally. One study, by the Milken Exchange on Education technology, concludes that these findings were generalizable for student achievement at the state and national level (Schacter 1999). It is important to note that many of the research syntheses cited literature published in the 1970s and 1980s, prior to the time frame for this current literature review. This suggests that much of seminal work in this area (e.i., concerns with the impact of technology and examinations of its impact on skills) is several years old and that the literature search criteria used in the current meta-analysis may not have been broad enough to encompass other important empirical research related to technology and child well-being. Much of the findings from the early 1990s focused on student achievement as measured by standardized test scores. This research showed that computer-based instruction relying heavily on close-ended learning activities and using behavioral-based branching software to teach content and/or skills, increased student achievement as measured by standardized tests (Kulik and Kulik 1991; Reeves 1998; Valdez et. al. 2000). On average, students in elementary and high schools who used computer-based instruction scored in the 62nd percentile on achievement tests compared to students in control conditions without computers, who scored in the 50th percentile (Kulik and Kulik 1991). Students also learned more in less time with CBI than with more traditional approaches. (Kulik and Kulik 1991; Reeves 1998). Evidence also supported that educational technology positively impacted student achievement in specific subject matter. Young children, aged 5 and younger, using educational technology have demonstrated superior achievement in cognition, oral language, written language (Statham and Torell 1996), and mathematics (Pierce 1994), compared to their peers taught using traditional strategies. Research with elementary and secondary students also demonstrated superior results when using technology for learning reading, spelling, writing, mathematics, science, social studies, foreign languages, computer programming, and career education (Statham and Torell 1996; Sivin-Kachala and Bialo 2000).

June 14, 2001

Page 23

Technology and Child Development, Part I: A Ten-Year Review of Reviews

In addition to standardized test scores, research must examine how technology impacts other more specific areas (OTA 1995). Several researchers examining the cognitive gains in such areas have furthered understanding by looking at the particular cognitive mechanisms affected by technology use. Among young children, research has shown positive impact on several general cognitive abilities, such as memory, concentration, spatial and logical problem solving, creativity, and artistic abilities (Sivin-Kichala and Bialo 2000). In the development of language, researchers have also found that educational technology positively influenced young children’s non-verbal skills, verbal skills, number of words spoken, and amount of communication with peers. Technology also supported the development of written language skills (reading and writing) in early childhood, both early skills (letter naming, word recognition), as well as more advanced skills (engagement in literacy-related activities and elaboration in written stories) (Pierce 1994). Like written language, technology applications have supported the development of early math skills (shape recognition, counting, and sorting) in young children (Pierce 1994). Research with elementary and secondary students has also demonstrated a positive impact on several areas of cognitive development, including: reading (e.i., phonological awareness and comprehension vocabulary); writing (e.i., writing quality, focus, mechanics and persuasive writing); mathematics (e.i., analysis skills and problem solving); science (e.i., declarative knowledge and dissection skills); and computer programming (e.g., logical thinking and metacognitive processes) (Statham and Torell 1996; Sivin-Kachala and Bialo 2000). While many of these outcomes were at the lower end of Bloom’s taxonomy of educational objectives (Bloom 1956)—knowledge, comprehension, and application—several addressed higher levels of abstraction, such as analysis, synthesis, and evaluation (Reeves 1998; Statham and Torell 1996; Valdez et al. 2000). For example, Reeves (1998) described a study where students who developed hypermedia programs about the American Civil War were able to see how historical patterns and perspectives affected current views of history, unlike control students, who had trouble even recalling historical content. Statham and Torell (1996) reported findings for Apple Computers of Tomorrow, where the greatest gains among participating students compared to controls was their regular use of inquiry, collaboration, and problem-solving skills. Valdez and colleagues (2000) summarized research showing that computers and ancillary electronic devices enabled students to manipulate data and visualize processes in a way not possible before; having these capabilities facilitated experimentation of actual and hypothetical concepts. Research has also consistently found that learning through technology was intrinsically appealing to children, and it promoted positive self-concept, self-esteem, the ability to collaborate, and improved interpersonal skills (Valdez et al. 2000). Not only were these indicators of child well-being themselves, they were related to student achievement (OTA 1995). The use of computer technology in education had positive effects on student attitudes, stimulated increased teacher/student interaction, and encouraged cooperative learning, collaboration, and problem-solving and inquiry skills (Statham and Torell 1996). Students were more motivated and had more positive attitudes towards learning when it includes CBI (Kulik and Kulik 1991; Reeves 1998). More recently, evidence

June 14, 2001

Page 24

Technology and Child Development, Part I: A Ten-Year Review of Reviews

showed that computer use decreased social isolation by encouraging collaborative learning and access online to peers, experts, and learning communities (Culp et al. 1999). These findings were interesting in light of an often-repeated concern that technology use will isolate children. Research was scarce concerning the knowledge, attitudes, and skills supporting children’s physical development and well-being. Some studies mentioned that evidence exists that technology supports the young children’s development of eye-hand coordination (Pierce 1994) and manual dexterity (Sivin-Kachala and Bialo 2000). These findings suggested that technology applications for fitness and health promotion among children were either non-existent or that current research or literature reviews on the topic were non-existent. A cursory review of reports of computer-based health interventions revealed that much of the research was dated prior to 1990 or was conducted with adult populations (Atkinson 1997). In the report to the President on the use of educational technology, the committee of advisors found that computer applications most benefited poor and low-achieving students (1997). Three literature syntheses specifically analyzed research related to atrisk and low-achieving children (Sivan-Kachala and Bialo 2000; Statham and Torell 1996) and children with disabilities (Pierce, 1994). When comparing different categories of learners using word processing software to improve writing skills, researchers found the largest effect sizes among special education students (effect size of .56), compared to an effect size of .55 for preschool and kindergarten students, .46 for elementary school students, and .32 for high school students (Statham and Torell 1996). A later research synthesis by the Software and Information Industry Association reached the conclusion that CBI appeared to have maximum benefit for low-achieving students, and other students requiring increased structure and instructional support (Sivan-Kachala and Bialo 2000). In a review of research on the benefits of assistive communication technology for children with disabilities, Pierce (1994) found that technology had positively impacted several domains that were found in research done with children without disabilities— reading and writing skills, social interaction and social skills, and higher peer acceptance. These findings suggested that educational technology affords the opportunity to enable the children that would otherwise be left behind to reach their potential. Few of the studies addressed gender differences in technology use and effectiveness, except one by Pierce (1994). She found that computer use in early childhood decreased differences in computer use and comfort between older boys and girls (Pierce, 1994). Given the emphasis on promoting science and technology careers among girls, these findings suggested that early use of technology may be necessary to enable all children to pursue the same career opportunities. Further research is needed to reveal if early computer use will also promote computer use and comfort and science careers among other groups underrepresented in the sciences, such as minority populations and people with disabilities. Despite the promise of technology, discussed previously, the articles we reviewed also revealed that technology was sometimes not effective in significantly improving

June 14, 2001

Page 25

Technology and Child Development, Part I: A Ten-Year Review of Reviews

outcomes for children, compared to traditional learning strategies (Kulik and Kulik 1991; Pierce 1994; Reeves 1998; Sivin-Kachala and Bialo 2000; Statham and Torell 1996). In these cases, the authors acknowledged the negative evidence but pointed to the weight of the positive evidence that technology was effective. They also pointed to the possibility that sometimes the measures used in the studies were not valid indicators of learning (Reeves 1998). They asserted that research must turn to focus on the circumstances that support effective use of technology in learning environments. Because the focus of this paper was on the positive effect of technology on child wellbeing and development, the methodology was not sensitive to revealing negative issues or outcomes related to technology. Instead, we focused on the instructional factors and learning environment factors that increased the likelihood of positive outcomes, as this is the direction much of the research on technology effectiveness appears to be heading. The next two sections of the findings describe our findings on instructional factors and learning environment factors related to technology effectiveness.

Instructional Factors Affecting Educational Technology Effectiveness Properly implemented, computer technology in education has a significant, positive effect on student achievement, as measured by test scores, subject area grades, and with all levels of students (Stratham and Torell 1996). Therefore, one must examine the instructional design of the program/intervention, how it is integrated into the learning environment, and the needs of different learners, rather than focus on the communications technology—software, videodisc, CD-ROM, DVD, Internet, etc.—itself (Culp et al. 1999; OTA 1995; Reeves 1998; Valdez et al. 2000). Much of the discussion on instructional factors was related to the instructional design of the educational technology. This centered largely on the difference between drill-andpractice learning activities and activities structured with a constructivist approach. Drilland-practice or computer-based tutorial approaches to education have received the most attention and funding in school-based settings, were accepted by more teachers than other technologies, and were widely supported by administrators, parents, and policy-makers (Reeves 1998). The overall value of computer-based “drill-and-practice” instruction rested in its capacity to motivate students, increase equity of access for those with special needs to education delivery systems, save costs, and enable students to learn faster (Reeves 1998). Drill-and-practice computer activities have significantly increased preschool students’ understanding of early math concepts and early literacy skills (Pierce 1994). However, drill-and-practice was designed to develop the lowest level cognitive gains in instructional sequences (e.i., information recall rather than evaluation or synthesis of information into knowledge). Therefore, these methods should be used in combination with other methods that are learner-centered and lead to the development of higher-level skills. As opposed to the behavioral approach to learning through computer-based drill-andpractice technology, higher-level cognitive approaches used technology to build problem-

June 14, 2001

Page 26

Technology and Child Development, Part I: A Ten-Year Review of Reviews

solving skills and to achieve learner autonomy (Valdez et. al. 1999). For higher-level learning to take place, computers must be used less for drill-and-practice and more as open-ended thinking tools and content resources (Statham and Torell 1996). This approach facilitated learning through increased teacher-student interaction, discovery, cooperative learning, and problem solving and inquiry (Statham and Torell 1996; Pierce 1994). Studies have shown that young children talk, draw, and write more with openended, rather than, drill-and-practice software (Pierce 1994). Open-ended constructivist approaches may offer less research than drill-and-practice applications because they tend to be newer and their effects are more difficult to categorize and measure than test scores on standardized tests or measuring recall of factual information (OTA 1995; President’s Committee of Advisors on Science and Technology 1997). However, some tests of critical thinking may be used to measure the effectiveness of constructivist approaches. Regarding other elements of instructional design, the authors of the Software & Information Industry Association report offered the most systematic discussion of design characteristics associated with positive gains by young learners (Sivin-Kachala and Bialo 2000). These elements included learner control, corrective feedback, the use of animation and video, narration and captioning to support visual elements, and a clear navigational map of information hierarchy and structure. Other authors mentioned design features related to positive learner outcomes. For example, multi-media applications were recognized as presenting information in a variety of ways so different types of learners would benefit (Pierce 1994; Valdez et al. 2000). For children with disabilities, video and audio design provided modeling and vicarious experience (Pierce 1994). Being able to start, stop, and repeat information—as allowed in videodisc and computer programs—enabled teachers to integrate group discussion into the learning experience (Pierce 1994). Instructional factors included how the teacher integrated the educational technology materials into the learning environment. Early in the use of computer-based materials, students learned from content and activity-specific software (Reeves 1998). Later, students used technology to enhance skills and knowledge acquisition; they learned “with” the technology. Learning “with” media and technology was more productive than learning “from” it; the computer supported the learning environment, rather than providing the focus of learning (Pierce 1994; Reeves 1998). For example, using computers to conduct literacy-related activities (e.i., making lists), rather than to do simple letter recognition activities, accelerated young children’s “emergent literacy” and facilitated adoption and reinforcement of writing skills prior to and in conjunction with reading skills (Pierce 1994). Both younger and older students who used computer-based tools to analyze, access information, and share knowledge with others learned to construct knowledge and control their knowledge seeking (Pierce 1994; Reeves 1998). Some research suggested that the use of media and technology was more productive (e.i., increases learning) when children work in groups collaboratively (Sivin-Kachala and Bialo 2000; Lou et al. 1998). Young children also tended to prefer working in pairs than alone at the computer (Pierce 1994). In some cases, however, individual student-use of computers could be more effective, such as situations where less time is available for task

June 14, 2001

Page 27

Technology and Child Development, Part I: A Ten-Year Review of Reviews

completion (Lou, Abrami and Muni 1998). Sivin-Kachala and Bialo (2000) described several studies examining the effect of collaborative activities on technology-based learning. They found that students working in pairs tended to learn more; however, students were most likely to learn in collaborative activities if they also received training in collaborative learning. The findings provided some understanding of other research showing improved social interactions from computer use. Despite the positive influence of collaborative activities, teachers must decide when students do learning activities alone or together. Ultimately, matching computer instructional applications to educational goals and objectives will enable teachers to decide on the best materials and how to use them. Since teachers have a range of learning objectives, they will need software that addresses the demands of that range (Statham and Torell 1996). For example, Pierce (1994) recommended software that focused on early skills like letter recognition and counting, as well as, open-ended tools and exploratory programs for young children. Further, students learned better when they were allowed to become familiar with computer hardware and software over time and when they had the opportunity to explore software by themselves (Pierce 1994; Sivin-Kachala and Bialo 2000). These findings suggested that teachers should include computer skill development activities in their lesson plans as they build from basic learning activities to higher-level activities involving computer tools.

Ecological Factors Affecting Educational Technology Effectiveness The attributes of the technology and the design of the instruction (software and learning activities), while important considerations, are only a part of the picture. One should also consider the readiness and support of the learning environment for educational technology: the teacher, the school, the family, and the community. Teachers are key to the success or failure of integrating technology into the classroom. Research supported that teacher differences influence success. For example, the metaanalysis by Kulik and Kulik (1991) revealed that effect sizes were larger when different teachers taught experimental and control classes, suggesting that differences among teachers influenced the effectiveness of educational technology. Teachers must also be prepared and willing for the changes that come with technology integration. Integrating technology successfully changes instruction from teachercentered to learner-centered (OTA 1995; Pierce 1994). Teachers can leave fact-finding to the computer and spend their time as content experts—arousing curiosity, asking questions and stimulating debate and discussion. Some teachers might be uncomfortable with this shift, which may represent a lack of control to them (OTA 1995). However, technology tools can free teachers’ time so they can spend more time interacting with students and working with more students individually (OTA 1995). Teacher training experience will help teachers realize the benefits to them and their students. The amount of teacher training was significantly related to the achievement of

June 14, 2001

Page 28

Technology and Child Development, Part I: A Ten-Year Review of Reviews

students receiving computer-based learning. Students whose teachers had 10 or more hours of technology-related training outperformed those whose teachers had 5 or fewer hours of training (Ryan 1991). Professional development and support of teachers as “facilitators,” with respect to students learning “with” technology, is absolutely critical for creating a successful and interactive, learner-centered environment (Reeves 1998). Training should also help teachers establish clear curricular goals so they can obtain appropriate technology-based materials and plan for effective use (OTA 1995; Pierce 1994). Besides typical training, teachers can access distance learning and online communities of peer teachers to support their educational technology efforts (Culp et al. 1999). Schools must realize that adequate access to computer technology is crucial to positive learning outcomes, for both typically developing and at-risk students (Statham and Torell 1996). For technology to be a successful learning tool, there must be a critical mass of accessible and varied resources appropriate to learning needs. One computer for every four to five students is necessary for students to be able to use technology in a manner that results in significant gains (Statham and Torell 1996). This means that schools must have adequate funding for hardware and software and have computers in the classroom where teachers can use and integrate them readily into learning (Sivin-Kachala and Bialo 2000). Once schools commit to supporting adequate hardware and software materials, they must support the development of curricular goals and teacher training related to educational technology so that they can identify and obtain the appropriate materials (Pierce 1994; Reeves 1998; Sivin-Kachala and Bialo 2000). They must also commit to staff and budgetary support for maintenance of hardware and software (Pierce 1994). Schools can also play a lead role in expanding the learning environment beyond the school walls and into the home and community (Culp et al. 1999). Technology might help schools establish connections with students’ homes, the local community, and other social institutions that positively affect children’s development. Involving others, particularly parents, will ensure that children get the support they need.

CONCLUSIONS Given these findings, the following conclusions should be viewed within the context of potential limitations. Whenever secondary analyses are conducted, interpretation is constrained because the analysis is one step removed from clearly understanding the intent and constraints upon which the original data were generated. Further, a quick review of the sources of original data illustrate that these papers originated in work by private foundations, academic research centers, and governmental agencies. While we believe there is no substantial risk, there is always the possibility that advocacy plays a role in the aggregation and reporting of data. Another limitation is related to our search criteria, which focused on literature examining the efficacy and effectiveness, rather than the negative outcomes, of computer applications. Therefore, the results were generally positive in nature because the “negative” findings were limited to lack of evidence for a positive effect emerged. Finally, there is a limitation posed by publication bias in

June 14, 2001

Page 29

Technology and Child Development, Part I: A Ten-Year Review of Reviews

research syntheses. We attempted to mitigate that bias to the extent possible by not limiting our searches to published data alone and—to the extent possible—believe we succeeded. In summary, the research was clear that educational technology resulted in improved outcomes for children. Ensuring that technology impacts the child as planned and expected requires attention to: (1) how it will be used to support the curricular goals and (2) careful selection of hardware and software. Teachers will be successful in their efforts, given appropriate training and support from their schools and the community. As technology continues to advance, the question of effectiveness and successful integration must continually be addressed. Based on the review of literature and metaanalyses, we recommend the following research be pursued: Basic research to: • • • • •

Understand how increased access to information via the Internet and to computer tools has changed teaching and learning. Understand learning-related disciplines and educationally relevant technologies. Develop reliable and valid measures to assess the full range of educational technology effectiveness. Develop methodological strategies for empirical studies that are sensitive to the challenges of conducting research in the classroom and to the fast pace of technology development. Provide enough classroom-based research to allow for a theory-based research synthesis.

Formative research to: • • •

Develop new software, content, and technology-enabled pedagogy for the full range of developmental areas. Understand how to design projects that take advantage of Internet technology. Identify new tools and environments that support student exploration.

Applied research to: • • • •

Determine how to identify and use the capabilities of media and technology to facilitate learning. Further understand the differential impact of technologies on various subpopulations of learners. Determine how to tailor the use of technology to specific students, tasks, and environments. Determine how technology can be used to influence child development, while taking into account the technology, variation among students, the task, and the context.

June 14, 2001

Page 30

Technology and Child Development, Part I: A Ten-Year Review of Reviews

• •

Formulate an agenda for assessing an entire theory base relevant to this area. Demonstrate how technology can help schools build connections with families, community organizations, and other social institutions involved with children.

June 14, 2001

Page 31

Technology and Child Development, Part I: A Ten-Year Review of Reviews

APPENDIX I: SEARCHES, INCLUSION CRITERIA, AND MATERIALS We modeled our literature review methodology on the methodology used by Emerson and Mosteller (1998a; 1998b) on two studies of interactive multimedia in college teaching. Compared to the previous methodology, we expanded the library databases in which we searched to include those that cataloged a broader range of social, emotional, and physical research. We also used “child” as a key search term as well as “development” and “technology.” Literature Searches Our computer search used nine large library databases: • • • • • • • • •

Educational Resources Information Center (ERIC) Psychological Abstracts (PsycINFO) Medline/Pubmed Social Science Citation Index Dissertation Abstracts International Family Studies National Technical Information Service (NTIS) Library of Congress University of Maryland System Catalog (VICTOR)

Initial searchers used the keyword phrases “child development” and “technology.” Later searches combined “child development” with “Internet” or “computer.” We also searched combining “child health” or “child well-being” with “technology,” “Internet,” or “computer.” In addition to the computer database searches, we conducted several other searches to identify recent literature. We searched recent volumes of several journals in order to include current work that may not already have been cataloged in standard computer databases. We also reviewed the reference lists of the articles we read, and these often led us to additional articles and reports. We gained useful information and relevant source materials through contact with investigators and organizations. We also searched the Internet websites of governmental and non-governmental organizations involved in child education, health, and psychology to identify current reports on children and technology: • • • • • • • •

Benton Foundation Bertelsmann Foundation The Children’s Partnership Center for Children and Technology Educational Testing Service Intercultural Development Research Association (IDRA) Milken Exchange on Educational Technology The Nemours Foundation: Center for Children’s Health Media

June 14, 2001

Page 32

Technology and Child Development, Part I: A Ten-Year Review of Reviews

• • • •

Office of Technology Assessment (archive) President's Committee of Advisors on Science and Technology, Panel on Educational Technology Software and Information Industry Association U.S. Department of Education

Inclusion Criteria Our literature search had two main targets—primary empirical research and reviews about the effect of technology on positive child development—and some secondary targets. We included unpublished reports based on presentations at conferences, many of which were identified through our search of ERIC documents. We limited the searches to reports appearing since 1990, except for research syntheses. To obtain background information, we searched for research syntheses, meta-analyses, and literature reviews in assessing the effect of technology on children. Consistent with the article by Emerson and Mosteller, we limited searches of the research syntheses to those appearing since 1990. We also focused on primary research articles since 1990 that met each of the following criteria for inclusion: • • • • • • •

Assesses impact of technology on child development and well-being. Uses subjects from conception through age 18. Uses impact on child well-being as a primary basis for evaluation. Provides data that enable the comparison of two or more treatment groups. Evaluates impact in the context of environments where children work and live. Reports work done in the United States and Canada. The article is asset-based and focuses on positive outcomes (negative outcomes of technology use, such as carpal tunnel among computer users, were considered to be outside the search criteria).

This investigation included research carried out in a wide variety of academic disciplines, because we wanted to look for patterns that cross disciplinary boundaries. The disciplines included education, psychology, health, and sociology. We sought to distinguish qualitative, case study, and quasi-experimental research about programs and applications from experimental research. We also distinguished research that compared two or more interventions (often with one of the treatment groups being a noncomputer control group) from studies that simply described an experience with new uses for computers or new computing environments.

Rejection Categories

June 14, 2001

Page 33

Technology and Child Development, Part I: A Ten-Year Review of Reviews

In addition to inclusion criteria, a set of rejection categories was established to eliminate those articles that did not directly fit into the scope of this literature review. If an article or study seemed promising for inclusion, it was examined to see how it related to the inclusion criteria listed above. Determination as to the appropriateness of fit to those criteria was based on the categories listed below. Those that could not meet the inclusion criteria in relation to these categories were not included in the study. We eliminated from consideration any source paper that was: •

not demonstrative of the impact of technology on child development only descriptive research (not technology-focused) • out of our intended time/date range • lacking in scientific rigor • published in popular literature only • irreproducible research • based only on opinion • published only in conference proceedings and did not include primary data • based entirely on discussion only (i.e., policy paper) • based on the use of technology to facilitate research on another subject area • a newsletter or reference guides • focused solely on childcare worker/parent/caregiver, not on a directly observable technological impact for child development • television based and focused • focused on using technology only to facilitate research on children (i.e., new computers help the Head Start program keep better track of expenditures) • about technology outside our definition of technology appropriate for this report • focused entirely on medical technology • an examination of the impact of technology within the context of a deficit model (i.e., negative effects of technology on child well-being) • focused on measurement issues rather than health promotion Materials and Methods Our searches provided more than 350 potentially relevant research articles, reports, books, and book chapters appearing since 1990. Information in the abstracts, introduction, and tables of contents narrowed our focus to 119 articles, reports, and chapters. One of us read all of these and completed forms designed to extract information about each potential source. This information enabled us to decide whether each article met the inclusion criteria for primary research, for research syntheses, or for neither. •

The first paper relies on reviews and systematic syntheses of research about the effect of technology on children. We found 11 of these articles that were published since 1987. The second paper focuses on our review of research articles that compare computer-based interventions with more traditional interventions or no intervention. We found 41 articles that report on such primary research. We found 19 other articles similar to these except that their research was not empirical and did not allow a calculation of effect size. Many

June 14, 2001

Page 34

Technology and Child Development, Part I: A Ten-Year Review of Reviews

of the articles were qualitative in nature. Those articles provided additional background information, but we did not include them in our main category of primary research. We read and extracted information from other types of articles. These included 36 articles since 1990 that were primarily philosophical, theoretical, or descriptive of a particular aspect of using technology to positively impact children’s development, health, or well-being. They also included several articles and reports on national surveys. Finally, we included 12 articles that focused on statistical and evaluation issues for conducting research on technology-related effects on children.

June 14, 2001

Page 35

Technology and Child Development, Part I: A Ten-Year Review of Reviews

REFERENCES Atkinson, N. (1997). Perceptual Factors that influence the adoption of an interactive CD-ROM application for health education. Unpublished doctoral dissertation, December, University of Maryland, College Park, M.D. Bloom, B.S. (1956). Taxonomy of Educational Objectives: The Classification of Educational Goals: Handbook I, Cognitive Domain. New York: Longmans, Green. Bangert-Drowns, R.L. (1993). The word processor as an instructional tool: A metaanalysis of word process in writing instruction. Review of Educational Research, 63(1), 69-93. Cognition and Technology Group at Vanderbilt. (1991). Assessing the outcomes of an innovative instructional program: The 1990-1991 implementation of the “Adventures of Jasper Woodbury” (Tech. Report No. 91-1). Nashville, TN: Vanderbilt University, Learning Technology Center. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates. Coley, R.J., Cradler, J, & Engel, P.K. (1997). Computers and Classrooms: The Status of Technology in U.S. Schools (Policy Information Report). Princeton, NJ: Educational Testing Service. Culp, K.M., Hawkins, J., & Honey, M. (1999). Review Paper on Educational Technology Research and Development. Technical report #43. New York: Center for Children and Technology. Emerson, J.D., & Mosteller, F. (1998a). Interactive multimedia in college training. Part I: A ten-year review of reviews. Educational Media and Technology Yearbook, 23, 43-58. Emerson, J.D., & Mosteller, F. (1998b). Interactive multimedia in college training. Part II: Lessons from research in the sciences. Educational Media and Technology Yearbook, 23, 59-75. Fletcher-Flinn, C.M., & Gravatt, B. (1995). The efficacy of computer assisted instruction (CAI: A meta-analysis. Journal of Educational Computing Research, 12(3), 219242. Kozma, R.B. (1991). Learning with media. Review of Educational Research, 61(2), 179211. Kulik, C., & Kulik, J. (1991). Effectiveness of computer-based instruction: An updated analysis. Computers and Human Behavior, 7, 75-94. Lou, Y., Abrami, P.C., & Muni, S. (1998). Effects of social context when learning with computer technology: A series of meta-analyses. Poster presented at the Annual Meeting of the American Educational Research Association, San Diego, CA. Mann, D., Shakeshaft, C., Becker, J., & Kottkamp, R. (1999). West Virginia's Basic Skills, Computer Education Program: An analysis of student achievement. Santa Monica, CA: Milken Family Foundation. Pierce, P.L. (1994). Technology Integration Into Early Childhood Curricula: Where We’ve Been, Where We Are, Where We Should Go. Chapel Hill, NC: Center for Literacy and Disability Studies.

June 14, 2001

Page 36

Technology and Child Development, Part I: A Ten-Year Review of Reviews

President’s Committee of Advisors on Science and Technology. (1997). Report to the President on the Use of Technology to Strengthen K-12 Education in the United States. Washington, DC. Reeves, T.C. (1998). The Impact of Media and Technology in Schools. Georgia: The Bertelsmann Foundation. Ryan, A.W. (1991). Meta-analysis of achievement effects of microcomputer applications in elementary schools. Educational Administration Quarterly, 27(2), 161-184. Schacter, J. (1999). The Impact of Education Technology on Student Achievement: What the Most Current Research Has to Say. Santa Monica, CA: Milken Exchange on Education Technology. Sivin-Kachala, J., & Bialo, E.R. (2000). 2000 Research Report on the Effectiveness of Technology in Schools, 7th Edition. Washington, DC: Software & Information Industry Association. Statham, D.S., & Torell, C.R. (1996). Computers in the Classroom: The Impact of Technology on Student Learning. Boise, ID: Consortium Research Fellows Program, U.S. Army Research Institute, and Boise State University. U.S. Congress, Office of Technology Assessment. (1995). Teachers and Technology: Making the Connection. (OTA-EHR-616). Washington, D.C.: U.S. Government Printing Office. Valdez, G.M., McNabb, M., Foertsch, M., Anderson, M., Hawkes, M., & Raack, L. (2000). Computer-Based Technology and Learning: Evolving Uses and Expectations. Oakbrook, IL: North Central Regional Educational Laboratory. Wilson, B.G. (Ed). (1996). Constructivist Learning Environments: Case Studies in Instructional Design. Englewood Cliffs, NJ: Educational Technology.

June 14, 2001

Page 37