Aera Tica Formative Design

Using a Formative Experiment (a.k.a. “design experiment”) to Investigate Internet Reading Comprehension David Reinking , Clemson University & J. Gregory McVerry, University of Connecticut What are the defining characteristics of this approach?

WHAT IS FORMATIVE (OR DESIGN) EXPERIMENT?

Intervention-centered: An approach to education research that closely aligns theory, research, Investigates systematically designed activities in classrooms or in other authentic environments of practice practices (Hoadley, 2005) to investigate instructional interventions Theoretical: tests, refines, or develops useful theories that directly inform practice (Eisnehart & Borko, 1993) while acknowledging the Goal oriented: complex ecologies of a classroom (Brown, 1992). aims to accomplish a specific, valued goal Iterative and adaptive: Involves successive cycles of data collection and analysis to discover how an intervention might be adapted to be more effective, efficient, appealing,and Workable. Transformative: promotes positive changes in the practice and experience of practitioners and their clientele •Methodologically flexible: employs whatever research methodologies serve its ends, but most often associated qualitative and mixed methods. •Pragmatic: focuses on workability, advancing democratic ideals, and more interested in reducing ignorance than finding truth (see Wagner, 1993)

Three Year Research Plan • Year 1: Laying the groundwork. Develop/administer online survey of Internet use (n ~1200) Identify 50 high-end, savvy Internet users Collect analyze verbal protocols of Internet comprehension strategies Purpose: To create a data-based taxonomy of Internet comprehension skills and dispositions to inform the content of our intervention in Year 2: Internet Reciprocal Teaching • Year 2: Investigating Internet Reciprocal Teaching IRT) using a formative experiment Purpose: To investigate how IRT could be implemented successfully in diverse classrooms toward achieving our pedagogical goal and to identify factors that enhanced or inhibited workability and effectiveness to inform Year 3. • Year 3: Investigating IRT using conventional experimental methods Purpose: To investigate IRT’s effectiveness under controlled conditions and with fidelity of implementation.

Framework for conceptualizing, conducting, and reporting a formative experiment What is the pedagogical goal and what theoretical and empirical work justifies its value and suggests how it might be accomplished? Our goal: increase Internet reading comprehension strategies to improve reading (on line and off-line), academic engagement, and achievement. Theoretical/empirical work: adolescent literacy, motivation, drop-out prevention, cultural differences, digital/new literacies.

Comparing Methods

and

Experimental Controlled Variation Laboratory Metaphor Guding Question What’s best? Comparison (x vs. y) Stance Positivism Philosophy Quantitative Data Met. Imperative Internal validity Theor. Imperative General laws Casual relations Operative goal Ino for practice Generalizations Pawns Participants Proto researchers Chess-playing staistician

Research Questions

TICA Year 2: Formative experiment

What factors enhance or inhibit the intervention from achieving the pedagogical goal? How can the intervention be modified to achieve the pedagogical goal more effectively, efficiently, and workably in a way that is appealing to teachers and students? What unanticipated effects (positive or negative) does the intervention bring about? Has the environment changed in the process of implementing the intervention?

Phase 1: Recruit teachers; negotiate participation in study Phase 2: Thick description of school and classroom environment (qualitative data) Phase 3: Baseline data in relation to goal (primarily quantitative) -Internet-use survey (from Year 1) -On-line Reading Comprehension Assesment -School Success Profile (SSP)

Population What classroom intervention has potential to achieve the pedagogical goal and in what theoretical work is it grounded? Internet Reciprocal Teaching: teachers model strategies, students become expert and informants (peer teaching), privileges struggling readers, strategies emerge while seeking meaningful information, addresses all aspects included in taxonomy of Internet comprehension. Empirical work: e.g., Palinscar and Brown’s (1984) work on reciprocal teaching

Six schools: 3 (rural or small city) in South Carolina and two (urban) in Connecticut with a percentage of students at risk of dropping out of school Intervention/control classroom in each school Schools had high Contexts vary; for example: oSpecial/regular education oAccess to technology oTechnological expertise of teachers

-Assessment of Motivation for Online and Offline -Reading (AMOOR), theory based instrument developed and validated by the research team Phase 4: Intervention phase (qualitative and quantitative data) Phase 5: Post-assessment (quantitative measures rom phase 3) and retrospective analysis of qualitative data

Naturalistic

Formative/design

Studied Accommodated Lens, jazz Ecology, engineering What is? What could be? Ideology of practice Selection/modifcation Constructivism (x/y) Qualitative Pragmatism Trustworthiness Mixed methods Soc/cult understanding Ecological validity Thick descriptions Workability Reflections Working theory Agents Suggestions Butteryfly-chasing Partners ethnographer Deal making mixed methodologists

Methodological Issues and Recommendations Issue: F/D experiments mitigate, but do not completely overcome, pressures for teachers/schools to raise achievement scores. Issue: F/D experiments require careful attention to building professional relationships with schools/teachers (see Cole & Knowles, 1993). Issue: Large-scale F/D experiments can produce results that are generalizable to diverse teaching contexts (see Firestone, 1993). Issue: F/D experiments are fundable within federal grant programs. Issue: What is the relation between F/D experiments and conventional experimental research (e.g., see McCandless et al., 2003)? Recommendation: More federal funding is needed to support largescale F/D experiments. Recommendation: F/D experiments may be seen as more legitimate to experimentalists when framed as a precursor to a conventional expiment. Recommendation: Researchers using F/D experiments need to more clearly establish methodological frameworks for data collection and analysis and to establish standards.