Msp Proposal 0831974 Rites Core Questions[5]

Answers to Questions Regarding MSP Proposal 0831974 RITES: Rhode Island Technology Enhanced Science Program The goals of RITES are admittedly ambitious: to profoundly change the STEM culture in Rhode Island—as it pertains to K-16 educational agendas—over the next five years. Moreover, we hope that our project will not only serve the state of Rhode Island by advancing student proficiency in science and, as a result, support the state’s pursuit of economic success in the technologically driven milieu of the 21st century, but also provide a template for large scale, technology enhanced initiatives for STEM reform elsewhere in the country. In the following section we present a synopsis of what we hope RITES will accomplish, which will provide a focus for our responses to the points raised by NSF reviewers. These responses are based upon extensive discussions among, and contributions from, all partners (e.g., K-16 STEM educators, RI Dept of Education, RI Economic Development Corporation, and the Concord Consortium). As a guide to our responses to the concerns of the reviewers, we provide an organizational diagram (see Figure) of the project that shows the flow of personnel and resources among the various components of the project. As can be seen from that figure, the emphasis is not only on the short courses, which although essential to the project are not the heart of RITES. Instead, the heart of this project is the seamless integration of all segments of the Rhode Island educational community (i.e., teachers and school systems, higher education science departments and schools of education, the various state offices that monitor and support STEM education, and the private sector) into a STEM culture that is in sync with the technological future for the region. In that context, the involvement of all relevant components of the state’s STEM infrastructure in a coordinated effort to affect change is a key aspect of the project. As well, the nature of the partnership between the precollege community and scientists from academia is also an important feature. In particular, we envision the RITES partnerships as true learning communities, in which all members (i.e., school educators, scientists, and administrators) contribute equally to the project’s goals. For example, important components of the short courses are teacher customizing of toolbased materials (such as probes and models) according to their curricular needs. This engenders a true sense of ownership of the modules they will construct and empowers teachers to contribute to the project. Moreover, through partnering with teachers—instead of simply instructing them— Rhode Island College, University of Rhode Island, Community College of Rhode Island, and Brown University faculty will profit from teacher insights and experience. Lastly, we should point out that other activities besides the short courses will be available through the project to affect change. Secondary teachers will learn much about science content, and best practice instruction, in the short courses. Nevertheless, they may also learn much about science pedagogy from their extracurricular experiences in RITES. Some examples of other resources include the website, the use of Collaborative Learning Environment (Sakai), visitations to schools by higher ed science and educator teachers, and involvement in the research agendas of higher ed faculty. For the convenience of the reader, here we provide brief descriptions of components of RITES, as shown in the simplified sketch of the organization (see Figure). Additional detail is contained in the answers which follow; we would be pleased to supply additional specifics, if it would be helpful in evaluating the proposal. Partnership Leadership Team (PLT). The PLT sets project policy and reviews progress, receives reports from the Evaluation Team (The Education Alliance at Brown) and reports to the Rhode Island Technology Enhanced Science Program

Answers to Questions

page 1

National Advisory Committee and NSF. In addition to members listed in the proposal, it also includes Dr. Peter N. Woodberry, Ph.D., Dean of Business, Science and Technology, Community College of Rhode Island. School interests will be represented in the PLT through two representatives from the School Council. The Council will consist of representatives of all participating school pairs. Finally, in addition to the participation of individual teachers and school systems, the RI Department of Education (RIDE) is committed to working closely with the project. In particular, Co-PI Peter McLaren, RIDE Science-Tech Specialist, will continue to be involved in all aspects of the preparation of short courses and coordination of activities between higher education and secondary schools.

Figure. Simplified Organization of RITES RITES Members. Higher education members include Rhode Island College, University of Rhode Island, Community College of Rhode Island, and Brown University. Secondary education members are the high school and middle school pairs that develop acceptable two-year plans for optimal use of project resources. Each plan features increasing the number and diversity of students proficient in the state science assessment and choosing STEM careers and calls for increasing the quality and diversity of science teachers. Five school pairs start in year one, then 13 each in years 2-4. Research and Evaluation Teams. The Education Alliance at Brown will provide external evaluation. In addition, the Alliance will partner with researchers from all partners to undertake related educational research. Rhode Island Technology Enhanced Science Program

Answers to Questions

page 2

Design Team. This team is responsible for the intellectual work of the project. It will coordinate the design and implementation of short courses and the work of the Resource Teams. It will also design a seminar that introduces everyone to the project—its methods, innovations, and strategies—and design and conduct the Leader Workshops. There will be four three-person Resource Teams consisting of a teacher, scientist, and educational researcher. Each group develops some short courses in its area and collects, describes, and puts into a common electronic form, the best learning materials. All short course leaders learn about the project, its resources and strategies in Leader Workshops, which are specific to a discipline area and prepares a leader for several future short courses. RITES project staff members and consultants help with these plans. Ten to fifteen teacher/PD experts become short course leaders by taking the Leader Workshop. Some teacher members join the Resource Teams. Resources. An important and innovative aspect of the RITES project is the access to a wide range of STEM assets throughout the state and elsewhere. The RI Center for Excellence in STEM Education (the STEM Center) servers as a repository for STEM activities and is housed at RIC and directed by Co-PI Glenisson de Oliveira. The RI Economic Development Corporation (RIEDC) hosts the Governor’s STEM Education Program Manager, David Cedrone, who is responsible for coordinating statewide public/private STEM education and workforce development initiatives and has been committed half-time to support the RITES project. The Concord Consortium is an integral part of RITES, and is involved in all aspects of the proposed work. Their president is a Co-PI, and the Consortium has primary responsibility for the development and delivery of many of the most innovative features in the workshops and courses, such as virtual labs and customizable probes. Rhode Island is fortunate to have many STEM education-related projects, funded by NSF and other agencies, in place. We have been in close communication with the principals for them—indeed, many of the members of the PLT are participants in them—and intend to incorporate them in RITES as appropriate. The project structure was designed to achieve the following objectives: Balance fidelity and flexibility. We want to avoid imposing solutions that are not needed or will not work. This why we insist that each school pair come up with their own plan, informed by the gap analysis, and why individual teachers can complete the program different ways. This flexibility is constrained, however, by the overall commitment to: alignment with state standards and assessments; guided inquiry as realized by the materials; the development of the materials and courses in groups that always include members of the core design team; the establishment of a common set of readings; the requirement that all participants (incl. faculty and collaborators) complete a version of the introductory short course; and involvement of project leaders in the development of materials and short courses. Full utilization of the partners' expertise. We have taken care that both pre-college teachers and post-secondary faculty are able to contribute meaningfully to all the substantive aspects of the project. The focus on materials development provides a way for busy teachers and faculty to make lasting contributions in limited time. The pervasive use of technology improves communication and increases personal impact. Representation and responsibility. RITES decision-making will be transparent and democratic. All partners are represented on the Leadership Team to which the PIs, working

Rhode Island Technology Enhanced Science Program

Answers to Questions

page 3

groups, and administrative team will report. There are clear lines of responsibility and shared staffing to ensure a unity of policy and balance of responsibilities. Project Design 1. Reviewers noted that a 25% increase in proficiency is admirable, but may be unrealistic. Please comment providing details on how you intend to achieve this goal and, in particular, note the benchmark/metrics that will be utilized that will indicate annual progress towards meeting the goal. The optimistic, though not unrealistic, proficiency benchmark (in the proposal it was that 50% of students would reach proficiency--the reviewers interpreted this as a 25% increase) will be used as a barometer of progress towards our overarching goal of creating significant and sustainable gains in science proficiency throughout the state. Data support our sense that a well-designed intervention can achieve comparable gains. For example, Tatar et al. (2008) report on gains from SimCalc used in a very similar albeit much shorter computer-based intervention, which evidenced test score improvements that suggest the increases we hope to achieve. By the end of each project year, we expect proficiency will increase among participating school pairs representing the year-to-year program cohorts. With five school pairs participating in the first cohort, the expected increase in proficiency will reflect a modest influence on statewide results. However, the participating schools are projected to realize an approximate 5% increase given 75% engagement of teachers, increased use of guided inquiry instructional practices, increased content knowledge across participating teachers/schools and implementation of school wide science action plans. In subsequent years, as each cohort of 13 school pairs is added and program implementation and elements mature and become established, the annual increase among participating schools is expected to range between 5-10% such that a 25% increase among participating schools is realized by the end of the project period. This expectation for annual gains is consistent with Tatar et al’s (2008) results for systems that are highly imbalanced, as is the case with the excessive underachievement in Rhode Island science. Further gains after the initial impact are expected to be much less dramatic. Because RITES directly helps districts align curricula with standards, and addresses the standards where the greatest depth of knowledge is required, we expect the impact will be significant in the short period of five years. According to participants of a Mathematics Summit hosted by the STEM Center in May 2008, none of the math curricula from the districts represented (90% were present) are well aligned to state standards, and presumably the same is true for science. Professional development and teacher training programs have not been well aligned either (100% of the institutions of higher learning were represented at the summit). RITES will meet the specific needs highlighted by these schools, which will be further informed by a gap analysis indicating how they can best align their courses. As importantly, RITES is particularly focused on inquiry, which leads students to use deep analytical thinking rather than recollection or simple algorithms to solve problems. Achievement of our goals will be monitored against benchmarks established across four critical dimensions or components of this program: schools, teachers, students, and structural role groups. During the design phase incorporated into the early part of Year 1, the PLT will articuRhode Island Technology Enhanced Science Program

Answers to Questions

page 4

late benchmarks and specify appropriate indicators within each component. Baseline indicators for each benchmark will then be established, with annual indicator data collected to profile progress and continue to identify gaps or target areas for program attention. Working with the external evaluator to organize, analyze and use these indicators to monitor design, implementation and ongoing program development, the PLT will adjust program components to assure progress towards meeting the proficiency goal. The program benchmark components and prospective indicator data elements are outlined in Table 1. Structural changes considered therein via the respective role groups, loosely refer to the effect of RITES on the larger RI STEM community. Many of the metrics exist, and others will be developed by the project. Further considerations underlying the development of the project design are summarized in the remainder of this response. Table 1 Methods for Assessment of RITES SCHOOL TEACHERS SYSTEMS

STUDENTS

STRUCTURAL ROLE GROUPS

NECAP (8th & 11th grade) School leadership Time on task – RITES course modules (per IP Teacher-led STEM initialogin) tives

Level of content awareness NECAP scores

Use of best practices

Graduation rates

Teacher PD logs (pool)

Inquiry skills (as moniStudent enrollment and tored by class observademographic patterns in tion, NECAP, and Con- Development of STEM STEM courses cord probes) learning communities

Involvement in profes- Participation in AP and Successful transition Involvement of parents sional development similar courses across grades 8-11 Level of education (i.e., Alignment to degrees) and experience (GAP analyses)

Involvement of private GSEs Increased participation sector (in sponsoring in STEM courses STEM internships, etc.)

Changes in STEM and Support of extracurricu- Choice of STEM ma- education curricula in RI higher ed system lar STEM activities jors in college Increase in percentage of teachers in STEM curricula Two-level test items

Expansion of statewide STEM partnerships and culture and, into college, majors in STEM

Common Assessment Items (mixed into local assessment) System dynamics

Rhode Island Technology Enhanced Science Program

Answers to Questions

page 5

The Partnership and Its Management/Governance Plan 1. The Community College of Rhode Island is listed as a supporting partner. Data in the Supplementary documents indicates that the college currently plays no role in teacher preparation in science. One faculty member from CCRI will help develop a short course. No member from CCRI is on the Leadership Team. What role will the college have in the project? In particular please provide details indicating how the college is involved in the comprehensive statewide effort to improve PK-16 STEM education in Rhode Island. The Community College of Rhode Island (CCRI) currently serves more than 16,800 students with more than 80 programs of study to explore their career interests and become scientists, teachers, occupational therapists, laboratory clinicians, engineers, computer programmers, and more. The college also offers a wide range of science, mathematics, engineering, and technology courses and programs, as well as preparatory classes for students interested in pursuing education careers. These courses and programs are well articulated with programs at Rhode Island College and the University of Rhode Island. The CCRI student body has considerable diversity in ages, cultures, and experiences. More than 22% of CCRI's students are members of minority groups, an increase of 94% since 1993. This figure now mirrors the overall population of the State. CCRI students are overwhelmingly firstgeneration (70%) and predominately female (63%), with an average age of 27; most are enrolled in a degree or certificate program (85%), and enrolled on a part-time basis (64%). A vast majority (87%) work 20 to 30 hours a week while pursuing their education; 14% speak English as a second language; and 10% are single parents of children. Many CCRI graduates also continue their education at URI and RIC. In 2006, 887 students transferred to those institutions. Should underrepresented students be attracted to STEM education while at CCRI, they will continue to benefit from the same RITES models to which they were exposed at CCRI. CCRI is a supporting partner to the RITES proposal and was involved in all aspects of developing the RITES proposal. At the time of proposal preparation we had not identified the CCRI representative on the Leadership Team. Since then we have done so: Peter N. Woodberry, Ph.D, Dean of Business, Science and Technology. As a supporting partner, CCRI will make important contributions to Rhode Island's statewide effort to improve PK-16 STEM education in our primary areas: Priming STEM pre-service teacher programs at the University of Rhode Island and Rhode Island College with candidates from underrepresented groups. CCRI engages students in a Teacher Assistant program through the Department of Human Services and supports their development and eventual transfer into bachelor degree track programs at URI and RIC. Priming the STEM workforce/student pipeline with candidates from underrepresented groups who develop an interest in STEM programs at CCRI and either directly enter the workforce or transfer to STEM bachelor degree programs at the University of Rhode Island and Rhode Island College and elsewhere. Rhode Island Technology Enhanced Science Program

Answers to Questions

page 6

Informing the design of ITES professional development offerings with strategies developed by CCRI faculty to meet the needs of students who were neither engaged nor successful in traditional K-12 STEM programs, Applying the instructional materials developed by teachers in RITES professional development short courses to student instruction at CCRI. 2. “Pairs of middle and high school will develop a unified science improvement plan.” How will the critical junctures that typically occur between middle and high school be addressed in these plans? How will these science improvement plans assure that high school students are fully prepared to enter and succeed in college? RITES science teachers from middle and high school pairs will be drawn together in several ways. As described in our proposal, our efforts at each school will begin by supporting each school's science faculty in their efforts to determine their needs as indicated by their students' previous performance on the NECAP examination. We will use curriculum alignment tools that have already been developed by the state to support teachers at each school decide on actions that augment their school's improvement plan and focus attention on science achievement. These plans will eventually provide a coherent picture of science instruction at each school level that will be available for review on the RITES online community. In addition to serving as resources for professional development planners, this collection will provide each school team with a growing understanding of their role in pre-NECAP science instruction, and has the potential for supporting close curricular coordination between each high school and its feeder middle school. Alignment of middle school and high school efforts will also occur in the professional development short courses. Because of the strong focus on improvement at the school level, each short course will provide opportunities for participating teachers to develop modules, clarify, teach, and assess concepts critical to their students' performance on the NECAP. As these modules are vetted, they will also be shared between partners online so that each team's efforts contribute to the community's knowledge base. 3. There is currently one K-12 co-PI from Johnston schools on the proposal. Johnston will be a member of the first cohort. How will school leaders and senior personnel from other partner schools be incorporated into the decision making as more schools become involved over the five years? Once underway, RITES will set up a School Council that represents all participating schools and will elect two members to the Leadership Team. Until that time, Johnston will represent school interests to the Leadership Team. The School Council will be a forum where all schools will be able to discuss program implementation and outcomes, resources, alignment, and all issues pertaining to their interests in RITES. The School Council will be supported with electronic listservs and collaborative learning environments (CLE) such as Sakai to foster communication. 4. One of the criteria for selection is that schools may apply to join the project if 75% of the teachers agree to participate. What other criteria will be used to select the schools? If after the school has joined the project there is significant teacher attrition or the school leadership is no longer supportive, so that far less than 75% of the teachers are

Rhode Island Technology Enhanced Science Program

Answers to Questions

page 7

attending professional development what strategies will be in place to deal with the situation? We are committed to statewide change and to the engagement of all Rhode Island schools and teachers in the RITES program. As part of this, we will look for an institutional commitment from each school pair that joins the project that includes most teachers and substantial buy-in on the part of the schools’ administrations. After extensive conversations with school administrators, RIDE, and other projects, we selected 75% as a benchmark. Some teachers will refuse to participate, and that is their right. Some feel that they do not need RITES or are already involved with other PD programs. The 75% benchmark is a good indicator of program acceptance and good administration. If it is not being met by schools, we will have to assume that some program element or elements are not functioning as planned. We will try to understand what the problems are and fix them. There may be a breakdown in communication, scheduling may be inconvenient, there may be technical problems, the short courses might be unrewarding for teachers, or incentives may be wrong. Whatever the problems are, our administrative structure and assessment team should make the Leadership Team aware of them—long before they become chronic—and we have the administrative will and flexibility to address the problems. Since our goal is to provide support and resources to improve all RI students’ proficiency in science, we do not want to create barriers to school participation, especially to schools that may have the most potential for improvement. Consequently, districts will have time during their participation in RITES to increase the number of participating teachers to the 75% goal if they miss it in the first year. We hope that by working closely with the teachers at each school who initially commit to being involved, other teachers will be included at deeper levels as each school’s science improvement plans are implemented and their collection of resources grows. The project is committed to serving a balance of rural, urban, and suburban schools. This means that the numbers of schools from each category that join each year will reflect the state ratios for student enrollments in these three categories of schools. School admission to the RITES program will require a complete and appropriate plan for a pair of schools, one middle school and one high school. This plan will include articulation between the schools, technology access, individual plans for teachers, and other criteria that will be set by the Leadership Team. Developing a plan for the school pairs will require RITES staff collaboration with the candidate schools—it is unlikely that schools will complete a plan without prior RITES staff work, so the project will be able to shape the applicant pool from year to year to ensure balance. Teacher Quality, Quantity and Diversity 1. RITES does not need to focus on quantity or diversity in teacher preparation but will focus on teacher quality. “RITES will have a major impact on teacher preparation…” “URI and RIC will undertake reviews of their pre-service programs to align them with the approach and materials used in RITES.” On page 13 the statement is that IHEs participating in RITES “are likely to integrate project materials into undergraduate programs…” Are these statements contradictory? Please provide details on the preservice improvements that will be adopted to improve teacher quality. How will preRhode Island Technology Enhanced Science Program

Answers to Questions

page 8

service courses be evaluated to demonstrate that they are improved? How will the improvement in teacher quality be demonstrated? RIC and URI are committed to implementing RITES in their programs, and have institutional support at the highest level. As a result of RITES, URI and RIC will improve their teacher preparation program and RITES evaluation will document the extent of this improvement. We expect to see in higher education exactly the same increases in the use of inquiry, implementation of research-based instructional strategies, and pervasive use of technology that is being planned for precollege teachers. RITES evaluation includes indicators of impact on preservice courses. In addition, we expect that the same approach utilized in the short courses for teachers will be adapted by faculty in all undergraduate and preservice courses. The project will encourage these changes informally by engaging faculty in the project and through the formal top-down reviews of the preservice programs. The informal approach will include: *All faculty engaged by the project will learn about project materials and pedagogy by taking some form of the introductory short course and reading the literature selected by the Design Team. *Faculty will be paired with teachers in working groups and in offering the short courses. This will further deepen their understanding of the new approaches. *Faculty will have their own area in the online community where they can create, post, and critique customizations of the modules for use in their teaching. While RIC and URI are committed to implementing RITES in their programs, and have institutional support at the highest level, supporting partners have not made the same type of commitment. RITES is open to include faculty from all eleven Rhode Island IHE’s in the short courses, and in those cases adoption of models, philosophy, etc, is likely. The statements highlighted by reviewers are not contradictory, because the second statement is general to more IHE’s than RIC and URI. 2. RITES does not need to launch a special program to increase the diversity because there are already two such programs in existence. Please provide details of the collaboration that is promised with the two programs “to ensure alignment with its inquirybased, technology enabled approach, materials and strategies.” Please provide evidence that RITER and RECRUIT will provide students with the undergraduate programs that will assure teacher quality. In addition to the previously referenced program at CCRI, which recruits and supports students from under-represented groups to enroll in teacher assistant programs and ultimately transfer to teacher pre-service programs at RIC and URI, Rhode Island has two grant-funded programs that support under-represented populations becoming STEM teachers. The RITER and RECRUIT projects support students from traditional (K-12 feeder programs) and non-traditional channels (mid-career transition) who aspire to enter teaching. These programs are led through the University of Rhode Island and the Rhode Island Department of Elementary and Secondary Education (RIDE). Both organizations are Core RITES Partners. The RITES project will engage and collaborate with RITER and RECRUIT to promote STEM teaching opportunities for prospective Rhode Island Technology Enhanced Science Program

Answers to Questions

page 9

students in these programs. This collaboration will increase the diversity of the pipeline of future STEM teachers in Rhode Island. Project RITER (Rhode Island Teacher Education Renewal) is a partnership composed of all eight approved teacher preparation programs in Rhode Island, the arts and science faculties at these institutions, three high need school districts, two state education agencies (Rhode Island Department of Elementary and Secondary Education and Rhode Island Office of Higher Education), and a business partner (TBA Consultants). This project represents a commitment to build on current efforts to reform teacher education to strengthen the preparation of the next generation of Rhode Island teachers. Project RITER’s goal is to increase the knowledge of subject matter, the ability to integrate technology into instruction, and the knowledge of diverse communities and effective strategies for working with students from poverty, English Language Learners, and students with disabilities. The project effects changes in the curriculum, assessment, and clinical experience of teacher education programs, in the professional development and mentoring of district based teacher induction programs, and in student learning in PK-12 schools and a program for recruitment and development of a non-traditional program for teacher certification in STEM areas. At a statewide level, RITER supports a non-traditional route to certification and implementation in high needs areas. The Rhode Island Aspiring Teachers program is a collaborative pilot project among school districts and Institutions of Higher Education to recruit and prepare individuals who have at least five years of relevant experience in high need teaching areas, an undergraduate degree in the area of certification or equivalent and are interested in changing careers and becoming teachers. For this program, high need teaching areas as defined by the RI Department of Elementary/Secondary Education include mathematics, physics, chemistry, and secondary special education. 3. Teams of teachers, faculty from disciplines etc. “[w]ill assemble a collection of excellent materials that can provide proven inquiry based learning experience that are linked with the professional standards.” A conclusion is that each pair of schools may select entirely different materials. How will the short courses for teachers accommodate the different choices in classroom instructional materials selected by the schools? That is, how will it be assured that all teachers at the different grade levels have a deep understanding of the science/applied mathematics and technology and how to adapt their knowledge and skills to teaching the excellent classroom materials? By requiring middle schools to pair with high schools to write a single plan for participation in RITES, we ensure that middle and high schools not only communicate, but also come up with a logical sequence that best serves their students. This will include deciding what grade and in what school each standard will be taught. Once the standards are assigned, there will be only two possible modules for each standard. The short courses will ensure that the teachers gain the required understandings. Each short course will have in common a focus on deep understanding of science concepts and guiding scientific inquiry in classrooms. The adaptations created by teachers will be warranted by their own analysis of their students’ needs, and the effects of these adaptations on student asRhode Island Technology Enhanced Science Program

Answers to Questions

page 10

sessments will be discussed in follow-up meetings during the school year. By using student data to warrant the teachers’ efforts, evaluate their effects, and decide on next steps, teachers have an opportunity to engage in inquiry in the course of their own professional work. 4. Reviewers noted a lack of detail on the content of the short courses. Courses will carry graduate credit. Yet there is also a statement that “In addition, for each course completed each teacher will receive an allowance …related to classroom implementation.” Are the courses that teachers will take the same ones that they will deliver in their classrooms? Please provide details of the graduate level short courses intended for teachers. The short courses for teachers are distinct from the RITES modules that participating teachers will offer to students in their classrooms. However, in keeping with the ideas of "pedagogical content knowledge," the core PD courses will be devoted to having teachers review RITES student modules, learn the modules’ content and pedagogy, customize the modules, and contribute findings and observations to an online RITES community. Thus, teachers will be learning both content and pedagogy in the context of materials that they will use in instruction—the one PD strategy that repeatedly results in student gains. The allowance is credit that can be used by participants’ schools for the purchase of equipment needed to implement RITES materials. Typical purchases would include Sakai fees, probeware, a classroom digital projector, a wireless router, or lab equipment. By putting this on a perparticipant and per-course basis, the project provides strong incentives for enrollment and completion. Each teacher will undertake a project that involves selecting, implementing, revising, and sharing a customized module, and passing peer review of the customizations. Our hope is that this process of principled reflection, data based customization, and online sharing of high-quality, inquiry-based materials will become common practice among all Rhode Island science teachers. This has the potential of creating a sustainable community of practice that contributes to improved practice and materials. The short courses are designed to launch this process. There will be three kinds of short courses: an introductory course, 14 subject- and level-specific "core content" courses, and two independent study courses. The 14 core content courses will be further divided into seven courses that introduce the RITES material and approach in a subject area and seven paired follow-up courses that are to be taken after teachers have implemented one or more RITES modules. The first of the two independent study courses will be designed for the academic year when teachers first try RITES materials and is intended to support participants as they implement RITES content. The second course emphasizes analyzing and reflecting on the results of instruction. All short courses and modules will be developed by the four Resource Teams each of which will include faculty, teachers, and other educators. The modules will not need to be developed from scratch; the SAIL technology simplifies the assembly of modules from existing free resources developed by the partners, the National Science Digital Library, and other sources. There will be Resource Teams for biology, physical science, earth and space science, and applied mathematics. Short courses will be led by pairs of teachers and faculty who have completed the Leader’s Rhode Island Technology Enhanced Science Program

Answers to Questions

page 11

Workshop designed by the Design Team. This design ensures that expertise provided by science research faculty, expert teachers, and educational researchers informs the choice of student materials and the professional development program. All teachers will enroll in the introductory course. Teachers will also be required to share in a community space a reviewed report on the implementation of at least one RITES module. For most teachers this will require enrollment in the two independent study courses. To fulfill the requirement of six courses for a RITES certificate, most teachers will enroll in a pair of the core content courses in their primary teaching area and one more core course. This last course will usually be the first of the pair in a related discipline. The Introductory Course The goal of the Introductory Course is to introduce all teachers to the RITES project, its goals, resources, educational philosophy, and technology. Because all teachers will take this course, it will be offered frequently and in different formats. It will be developed by the Design Team as part of their role of providing project-wide consistency and fidelity to the core RITES ideas. The content of the course will include the following. 1) RITES goals, structure, and program. 2) RITES examples. 3) The central role of inquiry. 4) RITES resources. 5) Nuts and bolts. The Initial Seven Paired Content Courses The core content courses are designed to increase participants’ pedagogical content knowledge: their understanding of research-based instructional strategies in the context of their subject and standards-based modules that incorporate RITES design principles. The seven pairs of short courses will be designed for teachers of: 1) middle school physical science, 2) middle school biological science, 3) middle school earth system sciences (ESS), 4) applied mathematics, 5) high school physics, 6) high school chemistry, 7) high school biology. We do not plan a high school ESS pair because this subject is not taught in RI high schools as a core science course. Since ESS content is 30% of the NECAP, we will ensure that ESS standards and related modules are incorporated in appropriate biology, chemistry, and physics courses. The applied mathematics course is primarily middle school content and will also not be offered at the high school level. Naturally, this is our best estimate of the courses that schools will demand— the specifics may change on the basis of feedback from schools and teachers. The first of the seven paired core short courses will all follow the same basic framework. Each of the seven pairs addresses 7-10 standards and their associated RITES modules. The first course will generally be taken during the first summer of a participants’ involvement in the project. The focus of this first course is to increase teacher knowledge of student subject- and level-specific learning patterns and to see how the design of the related instructional modules addresses these patterns. In the process, it introduces the RITES modules and has participants plan for using one Rhode Island Technology Enhanced Science Program

Answers to Questions

page 12

or more with their students. The tentative content framework for the first of the paired courses follows. 1) Review of student performance. 2) Overview of the modules in the content area and level. 3) Detailed analysis of one module. 4) Module selection. 5) Module customization. The second of each of the seven pairs of courses is indented for teachers who have already implemented one RITES module. The course will provide time to reflect on the content and student performance and revise the materials accordingly. It will also give participants a chance to share their findings with colleagues and learn to use the online community. Finally, they will plan to implement other RITES modules and other online resources. The tentative content framework for the second of the paired courses follows. 1) Review of student performance. 2) Collaboration. 3) Module revisions. 4) Peer reviews. 5) Implementation planning. The Two Independent Study Courses These courses are designed to increase teacher ability to manage the RITES modules and to analyze student performance. The first provides implementation assistance during the academic year. The second focuses on scoring students and analyzing their performance and is offered in the spring and summer after teachers have implemented one or more modules. The courses are independent study in the sense that they are designed to help individual teachers with their projects. There will not be a curriculum per se, but a series of resources for teachers. Group facilitators will be responsible for guiding the conversations and organizing activities in response to teacher issues. Teachers will be enrolled in collaborative groups of 25 or fewer participants, led by a project-trained facilitator. These will meet online or face-to-face. Participants will be expected to participate regularly, share their concerns, and help others. Software will monitor participation in the online versions to ensure a minimum for graduate credit. Both courses are designed for use with participants from different fields by addressing pedagogical and implementation issues that are common across science courses. As a result, there will be multiple versions of these courses. This will give the project scheduling flexibility while providing cross-fertilization across disciplines and levels that is rare in education. The major topics covered in the first course will be: Experimenting in your teaching. Rhode Island Technology Enhanced Science Program

Answers to Questions

page 13

Managing diversity. Supporting student inquiry. The major topics covered in the second course will be: Knowledge Integration scoring. Accessing and analyzing NECAP and RITES benchmark data. Reviewing modules submitted to the RITES database. The following are possible additional topics that will be addressed as needed—others will be added to meet teacher interests. For each topic there will be resources, FAQs, and discussion areas that will grow with the project. Technological issues. Probeware—what to buy, what to make. Classroom management. Challenges with inquiry-based learning. Fostering modeling skills. Online resources. 5.

“Each [participant] will focus on student assessment data from the classroom trial identifying why parts of the materials were educationally unsuccessful for some students. Participants will then customize the materials to address these problems for later use in the classrooms.” Reviewers question the wisdom of allowing teachers to customize the materials. How will quality control be assured so that all students have the opportunity to take rigorous courses? Why is it assumed that if students do not do well it is the research-based materials that are the cause of the problem? What is the evidence/research base that supports the idea that teachers are knowledgeable enough to customize research-based curriculum materials?

Instruction is an engineering design process. Experts might produce a design for an abstract bridge based on the fundamental physics of materials, but that design must be customized for any particular bridge to fit the load requirements, physical constraints, and environment. In the same way, curricula materials, even if validated by research, represents an abstraction that must be adapted to each new situation. Customization is as old and established as teaching. Teachers regularly customize textbooks by selecting chapters and skipping others, adding some explanations, emphasizing certain concepts, changing some of the vocabulary, implementing appropriate labs, inventing problems, and adding ancillary materials. There is concern that this reduces the fidelity of the enacted curriculum, but every class is different and requires adjustments in instruction. In the hands of a thoughtful teacher, this kind of customization will increase the fit between the materials and students. Only recently has customization begun to be used with electronic media as part of teacher professional development (Brown & Edelson, 2001; Nibaghan, Slotta, & Cuthbert, 1999). The ease with which electronic media can be edited, used in classrooms, and archived makes customizaRhode Island Technology Enhanced Science Program

Answers to Questions

page 14

tion a more attractive strategy that is more easily shared with colleagues. Customization is far easier than authoring, although it uses the same technology. A teacher might customize an activity much the way he or she might customize a textbook, by selecting parts, adding some explanations, emphasizing certain concepts, and changing some of the vocabulary. For activities based on SAIL, customization is as easy as using a word processor—no programming is required. The existing research indicates that teachers who are inexperienced or unfamiliar with materials can easily decrease the value of the materials (Baumgartner, 2004). One common problem is that they yield to the pressures of limited class time and eliminate sections without regard the development of ideas and the underlying instructional design. This problem has been addressed by insisting that customization be informed by data or theory and providing rubrics for evaluating changes (Spitulnik & Linn, under review) This becomes an important part of TPD, engaging teachers in interacting with the curriculum and its instructional strategies, instead of passively enacting it. We are confident that every teacher has a better understanding of his or her students and school environment than the authors of the modules, and, with guidance provided by effect TPD, can translate that understanding into improved material. Challenging Courses and Curricula 1. A “comprehensive, rigorous science curriculum accessible to all students” is a high priority for the state. Enhancements of the current curriculum are proposed, generating materials aligned with state standards and associated assessments. The Concord Consortium has been funded for many years to develop instructional materials that incorporate the use of technology. RITES “will develop an exciting collection of free, highly interactive, online educational resources…” How will these be different from the materials Concord Consortium has already developed? Please provide sufficient evidence to convince reviewers that the resulting mix will result in a coherent, comprehensive and rigorous science curriculum for middle and high school students in all participating schools While the RITES materials will use the proven approach and tools developed by the Concord Consortium (CC) and many of the ideas that inform its classroom-tested materials, the RITES content will be unique. The primary difference stems from the different project goals. CC materials were developed as part of research projects designed to examine questions related to the effectiveness of particular software tools with certain audiences. For example, one project that developed learning activities looked at the applications of molecular dynamics in biology; another explored the use of probes with young students. RITES materials will turn this around and ask how a particular standard can be best addressed with any ICT-based materials, including any of the CC resources. It is important to realize that our approach to ICT-based materials involves the development of student learning activities that consist of two parts: sophisticated models and tools, and a pedagogical surround. The models and tools used in RITES have been developed by CC and others and incorporated into the SAIL platform. These will used as-is with little or no modification. Additional models and tools might be added, however, if authors require ones that are open source. The curriculum applications of the models and tools, however, depend entirely on the instructional goals and will be adjusted to the RI standards. Furthermore, the models and tools Rhode Island Technology Enhanced Science Program

Answers to Questions

page 15

always have a pedagogical surround that provides scaffolding, background, multi-media resources, instructions, help, assessments, and links and allthis will be new, although it may be based on similar materials at CC or elsewhere. The main source of coherence will be the standards. Each of the 52 standards described in the proposal will be addressed by two modules. Within one main standard, the modules will provide a coherent learning progression. Each module will include assessment items, including ones similar to items on the NECAP--either released items, variants of these, or ones the project generates from the standards. The modules will be tested in their target grade using their assessments, an analysis of student logs, and teacher observations. The modules will then be revised on the basis of the results. Additional coherence will be provided by the design of the modules. The modules will all conform to a single set of curriculum design criteria as described on pp. 9-10 of the proposal. In addition to being standards-based, they will all base learning on inquiry and meet the seven AAAS criteria for high quality. They will have a common format that supports proven instructional patterns such as predict-observe-explain. Another source of coherence will be the consistent use of the same modeling and tool software in different contexts. For instance, an atomic view of materials is important in every science topic. When an atomic-scale topic needs to be investigated, the Molecular Workbench can be used. If one of the hundreds of existing models do not fit exactly, an author can easily configure MW to support student inquiry at the appropriate level to match a particular standard. Similarly, investigations using real-time data acquisition and display will use the same software at all levels. . 2. Reviewers raised questions about equitable access to technology. The online educational courses “will be suitable for use in computer labs or classrooms with from one computer per class to one per student.” How will the merits of each of these situations be evaluated in terms of its efficacy in improving student learning and understanding of science? How will adjustments be made it is determined that one computer per student is the most effective method? Because the software will log its use and users, we will know whether it is being used by a teacher in a whole-classroom mode, by groups of students, or by individual students. We will convert these into measures at the class level that will be used as independent variables in the analysis of student performance and gains. There is considerable research that indicates the value of small groups of students working together around a single computer. This will be the primary implementation model for RITES materials. But because we will support students working alone at home or in a library, the registration and assessment systems will support one-on-one use. This also ensures that when, inevitably, there will be ubiquitous access to computers, the RITES materials will continue to be useful. However, our design does not assume 1:1 classroom implementation and will not require it, and will not study such implementations. We are confident that there will be equitable access by RITES students to computers at a ratio of 3:1. There are state funds for ICT that are administered through RIDE. When schools apply for Rhode Island Technology Enhanced Science Program

Answers to Questions

page 16

these funds, a major criterion for funding will be whether the requested technology is needed to support the RITES program. At the same time, RIEDC will make a major effort to tap its corporate partners for technology to ensure equity. 3. The online materials/resources will be free and therefore some students who have access to home computers will be able to download software on their own computers and work outside of the classroom. How will this differential access to resources by both students and teachers be monitored and/or evaluated for its effect on student learning? All software will have built-in logging capacity that will report, at a minimum, total time used and progress for each student. These data will be time-stamped, so we can infer from the time of day whether the use was in school or not. Even if students use computers that are not on the network, they can access the materials using USB flash drives; their use data will be written to the drives and uploaded next time the drive is refreshed from a networked computer. As part of our analysis of student performance and gains, we will examine the amount of use during school hours and the amount of use outside school hours. If this analysis shows difference related to performance we will try to understand the causes and enlist teachers and all RITES partners in helping find solutions. 4. “RITES enthusiastically supports Physics First, but will not require all participating schools to adopt it.” How will it be assured that the same professional development program that prepares teachers who teach in the traditional sequence will also be suitable for teachers who teach Physics First? How will the success of the Physics First curriculum as compared to the traditional curriculum be evaluated in terms of student proficiency on NECAP? By focusing on materials that address standards that must be achieved in both the Physics First (PF) and the traditional curriculum, we serve both sequences of courses. We plan to develop two modules for each standard. For modules addressing physics and biology standards, which might be at the 9th or 11th grade depending on whether PF is used, one of the two modules will be tailored to PF implementations. There are already several efforts underway statewide related to Physics First. While RITES partners are convinced that reversing the sequence can be effective when the course content is also changed and the teachers get related professional development, RITES is more likely going to be effective statewide by not requiring all schools adopt Physics First. As part of our analysis of student performance using NECAP and gains as measured by RITES assessments, we will examine the impact of participation in Physics First. If, as expected, good implementations of PF accounts for student gains, RITES will definitely make those data available and even more strongly urge RITES schools to include PF in their RITES plans. Institutional Change and Sustainability 1. Given this project is a statewide effort supported by the Governor and state legislature of Rhode Island, the panel was impressed by the project’s potential to become a model for PK-16 statewide education. What will be the structure of this model for statewide Rhode Island Technology Enhanced Science Program

Answers to Questions

page 17

improvement in science education? How will it be assured that the project’s research and evaluation feed into the structure so that the model becomes sufficiently robust to be transferable? How does the evaluation plan help to determine if the model is indeed transferable? The RITES project grew out of state government and will be closely integrated with it. In addition to the strong political support of a Governor who is intensely interested in STEM education and technology, the state department of education, the state supported universities, and other state agencies are all partners. In addition the Rhode Island Economic Development Agency will be contributing a 50% FTE slot to bridge between the Governor, legislature, other state projects, and business. An important part of the project evaluation involves examining the structural systems in place and their changes. This will include examining the partnerships and institutions and tracking their changes over time. Please see the evaluation section for additional detail on the plan for developing data and narratives that should be helpful to other states and larger districts. Evidence-based Design and Outcomes 1. In what ways is your proposed project innovative? In what ways will the proposed project add to the knowledge base? What is the potential for dissemination of the proposed model and why? The program is innovative by doing the following: a) Using data and state standards (Grade Span Expectation – GSE) to take into account needs at the school level and collaboration with teachers to design program; b) using online curriculum resources in Professional Development to supplement school instructional resources for specific GSE’s; c) using a multi-state assessment (New England Common Assessment Program – NECAP) item analysis to gauge progress; d) using technology to catalogue patterns of software usage, thus supporting the study of learning through inquiry; e) utilizing computer technology to provide a support platform to a community that learns about education dynamically, shares resources, and gathers data jointly though powerful software built into the activities themselves; f) using instruction teams composed of both K-12 and higher education faculty, thereby utilizing the expertise from both of these groups, and influencing the pedagogy used in both communities; h) uniquely engaging schools and teachers in through a mix of top-down and bottom-up models, thus increasing collaboration and increasing the likelihood of continuation; g) articulating instructional efforts across the middle school/high school and high school/college boundaries. Materials, assessments, and findings will be very public. Materials, reports, minutes, and the technical tools will all be free and online. RITES will assist other programs to adopt and adapt parts of the resources that fit their needs. 2. Core Partners in MSP Targeted proposals must agree to submit data in support of the MSP Program Evaluation. Please provide evidence that your core partners are committed to providing this data which will include: a) K-12 student achievement scores, b) teacher data including number of hours of Professional development, disaggregated by demographic factors, educational background and courses taught; c) higher education Rhode Island Technology Enhanced Science Program

Answers to Questions

page 18

faculty, disaggregated by demographic characteristics and research/teaching commitments, time offering pre- and in-service teacher PD, and their own professional development as teachers and learners. In addition to a narrative response here, you may include additional letters from appropriate individuals from your core partners indicating their commitment. The RITES partners agree to provide the data described, limited only by the need for confidentiality of student records and other sensitive personal data. Student achievement data will be provided by RIDE, the state department of education in the form of NECAP data and any other data the project requires. Participating schools will agree to provide the teacher data annually as part of their application process. The three higher-education partners, URI, CCRI, and RIC, agree to provide the required faculty data. We would be pleased to submit letters from appropriate individuals indicating their commitment, if that would be helpful.

Project Evaluation 1. In what ways does your evaluation inform all aspects of your proposed project, including (a) teacher quality, especially growth in teachers’ content knowledge and classroom practice; (b) institutional change in both K-12 and higher education; and (c) the Partnership itself? The Education Alliance designed an external evaluation to examine both the implementation and impact of the RITES program. The evaluation design includes quasi-experimental approaches as well as qualitative methods to continuously inform RITES staff and partners on all aspects of program implementation and improvement efforts. Specifically, the overarching goal of the evaluation plan is to support stakeholders in understanding (a) how to enhance program implementation to meet MSP objectives, and (b) how to leverage program activities to ultimately contribute to gains in student achievement. The Alliance proposes to organize and analyze data, and disseminate findings through a formal feedback structure established via Design Team and Partnership Leadership Team meetings. Evaluators will present findings at the close of each school year in a timely fashion for program improvements to be put in place. Additionally, evaluators will have informal, ongoing feedback meetings with PIs for updates and pressing programmatic changes. In Years 3 and 5, The Alliance will prepare a formal evaluation report examining RITES implementation and impact to date, with Year 5 including a summative evaluation of the entire initiative. The evaluation design proposes multiple measures of the RITES program implementation and impact and is consistent with a scientifically rigorous approach to evaluation. The plan allows for triangulation of findings, identification of consistencies and inconsistencies across data, and considers alternative explanations for the findings identified. The five-year evaluation plan centers on four components of the RITES program: professional development (PD), classroom instruction, student achievement, and structural systems (including partnerships and institutional systems). Each of these program components warrants ongoing external investigation in order to offer program improvements toward sustainability. Based on these key areas of evaluation focus, five questions are posed by the evaluation plan: Rhode Island Technology Enhanced Science Program

Answers to Questions

page 19

1. How and in what ways are the professional development opportunities offered by RITES contributing to teacher knowledge and practice? 2. To what extent are participating teachers implementing reformed instructional strategies around science in the classroom? 3. To what extent is the RITES Program linked to improved student academic achievement in science? If an effect is observed, what is the magnitude of the effect and how does this effect impact minority subgroups (i.e., ethnic and gender minority groups)? 4. To what extent are the planned RITES program components building partnerships between and among higher education and K-12 partners? What are the factors contributing to, or hindering, successful partnerships? 5. To what extent does RITES contribute to institutional change for both higher education and K-12 agencies? What are the factors contributing to, or hindering, change agents? Evaluation of each program component requires a specific methodological strategy to adequately respond to these questions. The methods to be used to provide measures for each component in the external evaluation and a timeline are presented in Table 2 (note that this table is reorganized from the original proposal to be more responsive to question 1) and further described thereafter. Additionally, using data collected by RITES to assess program benchmarks, the evaluators will integrate these resources into analyses for each component—for example, including a secondary analysis of student enrollment in STEM or advanced placement courses to inform examination of changes in student achievement, or assessing teacher attendance at professional development to inform contributions to teacher knowledge and practice. Table 2. Evaluation Methods by Program Component Program Component

Method

Year 1

Professional Development

Expert Panel Review

X

Web-based Teacher PD Logs

X

Classroom Instruction

Student Achievement

Structural Systems

Year 2

Year 3

Year 4

X X

X

Year 5 X

X

Classroom Observation

X

X

Teacher Interview

X

X

X

NECAP

X

Student Beliefs Survey

X

Stakeholder Interviews

X

X

X

X

X

Stakeholder Focus Groups

X

X

X

X

X

Document & Artifact Review

X

X

X

X

X

Online Professional Learning Community Dialogue X

X

X

X

X

Rhode Island Technology Enhanced Science Program

X

X

X

X

X X

Answers to Questions

page 20

Professional Development Opportunities. RITES will deliver short courses and online materials for customization and use by teachers. Alliance evaluators will employ a panel of experts, inclusive RITES National Advisory Committee members, to review and score modules/curriculum and materials provided by RITES for indicators of challenging content, quality pedagogical practices, and relevance to classroom instructional needs. The Alliance has utilized this approach in examining online PD materials for PBS’s TeacherLine program. Using a web-based teacher log modified from existing MSP PD instruments (Council of Chief State School Officers, 2006), The Alliance will measure aspects of teachers’ PD opportunities as well as teachers’ perceptions of their pedagogical and content knowledge and attitudes toward inquiry-based instruction in their classroom. Research indicates that teachers’ self-efficacy in instruction and content interacts with their attitudes toward inquiry-based strategies, impacting classroom instruction (Harwood, Hansen, & Lotter, 2006; Hudson & Lydon Brooks, 2005). All participating teachers will be asked to complete the logs once monthly, recording the types, frequency, quality and content of RITES PD. Log data will provide measures of teachers’ satisfaction with content, experiences, facilitators, and interrelation of content with instruction, as well as perceived student impact. Finally, the evaluation plan will utilize secondary data collected by the RITES program for ongoing program monitoring. Secondary analysis of, for example, data on PD enrollment and retention, teacher content knowledge before and after participation in RITES, and student enrollment patterns, will be integrated with primary evaluation data analyses to provide a comprehensive understanding of RITES implementation, materials, and impact. Moreover, triangulation of these data resources provides a formative understanding of program implementation for benchmarking purposes while also affording measures of change. Classroom Instruction. The Alliance will conduct classroom observations to examine the extent to which teachers are implementing reformed science teaching strategies in the classroom. In Years 2 and 4, evaluators will randomly select 10% of the total sample of participating RITES teachers to conduct classroom observations using the Reformed Teaching Observation Protocol (Arizona State University, 2000). Observations at the high school level will take place in science classes, with middle school observations scheduled to capture an entire science lesson. Similarly, comparison group observations will also be scheduled with a randomly selected sample of teachers who have yet to participate in the RITES program. All observations will be followed by a brief teacher interview to provide lesson context and student population information. This quasiexperimental approach allows evaluators to link the PD opportunities to classroom practice, informing the extent to which RITES PD impacts teacher instruction. Student Achievement. To examine the extent to which the RITES program is impacting student achievement, The Alliance has identified three potential quasi-experimental designs: (1) matched sample, (2) staggered comparison, and (3) linear modeling. While each design includes an objective and moderately rigorous analysis of changes in student achievement, each has limitations. The final design will be determined during the initial design stage of the RITES program on the basis of its power to detect an effect and assure reliability of results. A matched sample design will examine science achievement test scores of 8th and 11th grade students in Rhode Island, as compared to that for similar students in non-RI schools. New HampRhode Island Technology Enhanced Science Program

Answers to Questions

page 21

shire, Vermont, and Rhode Island administer the New England Comprehensive Assessment Program (NECAP), providing multi-state student populations from which to select a matchedcomparison sample of non-RITES students. Similar to a matched sample, a staggered comparison design allows for those students in schools where RITES has yet to be implemented to serve as a comparison group. These designs, as opposed to pre-/post designs, are appropriate given that recent changes to state achievement tests have not placed scores for all tests on a common scale, precluding the comparison of current scores to previous achievement scores. Finally, a hierarchical linear model (HLM), where the student is nested within teachers and teachers within districts, will take into account the contextual implications of RITES at each of these levels. While evaluators will work toward an HLM analysis to examine RITES impact on student achievement, the number of participating districts proposes a potential barrier to this analysis, limiting the variability and, in turn, the reliability of a nested analysis. Also of note is the focus of the evaluation on formative approaches to assist in building sustainability of the RITES program. To that end, the HLM analysis is a summative approach to be utilized in the final year of program implementation where both the sample size and potential impact on student achievement are maximized. Ongoing descriptive and inferential statistical techniques will be otherwise used to provide timely feedback on all aspects of the RITES program. The Alliance will also examine the extent to which the RITES program is impacting student intent to pursue STEM careers as a benchmark for student achievement in science. Two tools--the Career Decision-Making System-Revised (Harrington & O’Shea, 1992) and the Students’ Motivation Toward Learning scale (Tuan, Chin, & Shieh, 2005)--will be integrated and adapted into a single survey instrument to examine students’ interest and motivation in STEM as well as STEM careers. A 10% random sample of RITES’ students will be selected to participate in the survey at selected points in time. In Year 1, survey data and student enrollments in advanced science and/or applied math courses will serve as baseline measures of students’ interest and motivation in STEM and STEM careers, while data from Years 3 and 5 will allow researchers to investigate changes in student interest. Where possible, evaluators will examine program impacts by student subgroups, including ethnic minority and female students separately. Secondary data will be made available to evaluators to integrate with primary evaluation data collection. Student enrollment in STEM courses and advanced placement courses as well as graduation rates will supplement evaluators’ quasi-experimental analyses of RITES impact on student achievement. Data collected through Concord Consortium’s online modules on the frequency and duration of student interaction with these materials will also provide a secondary source for triangulation of student impact analysis. Structural Systems. Established research approaches will be leveraged to examine how partnerships and institutions evolve as a result of the RITES program. Each is described below. Partnership Building. Utilizing the Partnership for Reform in Science and Mathematics (PRISM) rubric, The Alliance will develop focus group and interview protocols to examine the following indicators of successful partnerships: (1) vision and goals, (2) communication, (3) decisionmaking, (4) responsibility and accountability, and (5) change and sustainability (Huges & Gilbert, 2007). Interviews and focus groups will be conducted with key stakeholders from higher education (deans, department heads/chairs, faculty) and public education (teachers and school, Rhode Island Technology Enhanced Science Program

Answers to Questions

page 22

district, state administrators). Qualitative data collected through interviews and focus groups will be analyzed to examine indicators across higher education and public education partners. To the extent possible, evaluators will also conduct discourse analysis of online communications to further examine partnerships. Leveraging the PRISM rubric as a framework, evaluators will examine the level of partnerships--beginning, emerging, developing, and accomplished levels of partnership building--evidenced from this program. Institutional Changes. The Education Alliance proposes to measure institutional change both with and across higher education and public education RITES partners. To do this, the Alliance will adapt the concept of the balanced score card (BSC, Kaplan & Norton, 1992). The BSC is an integrated performance management framework that assists organizations in translating strategic objectives into relevant performance measures and has more recently been applied in higher education (Bremser & White, 2000; Beng Keok & Kean Thong, 2007). In this instance, the BSC has been used to measure financial commitment to institutional change, awareness of change among costumer/stakeholders, implementation of innovation, and internal structures to support change and action. Research indicates that partnerships evolve from relationships and connection to innovation and action. The corresponding theory of action suggests that strongly forged partnerships can be leveraged for institutional change. Consequently, interview and focus group data on partnerships will also be used to examine institutional changes. Findings will be organized in terms of the BSC to provide program recommendations toward institutional change. Additionally, program milestones (e.g., hired staff devoted to RITES) and artifacts (e.g., school improvement plans’ incorporation of RITES) will be collected to examine the extent to which RITES program components have become embedded in the culture of core partner entities. Finally, evaluators will look for indications that RITES has impacted preservice courses. This is expected to happen by faculty offering STEM preservice courses participating in RITES activities and appropriating RITES materials and approaches in their own teaching. We will interview teachers at URI and RIC who offer STEM preservice courses and look for evidence of project impact. We will also use the module electronic logs to determine whether their students use and customize any RITES modules and to quantify that use. 2. Does the evaluation plan provide for an objective analysis – either via an external evaluator or via an unbiased analysis by an evaluator within a partnering organization who is clearly separate and distinct from the partnership participants – of project effectiveness and “what works” and “what does not” to inform mid-course project corrections and/or modifications? The Education Alliance at Brown University will conduct an external evaluation of the RITES MSP Program. Although The Education Alliance is a department at a RITES partnering university, The Alliance is a research and development department distinctly separate administratively and functionally from partnering departments at Brown. With a thirty-year history of research, evaluation, development, technical assistance and consultative services, the Alliance has actively pursued development of its work in STEM content areas. With seven active STEM external evaluation projects and several completed and newly emerging projects, The Education Alliance has demonstrated its capability of integrating evaluation activities and findings into program implementation through creative technologies and professional collaboration. The Alliance’s curRhode Island Technology Enhanced Science Program

Answers to Questions

page 23

rent STEM projects range from implementation to impact studies, from primary to secondary education settings, and share a local investment in STEM education as well as building capacity and partnerships through program improvement. Knowledge and experience working on external evaluation projects in Rhode Island provides the essential track record for rapport and access to data to carry out an evaluation of state level initiative. 3. The unit of analysis is to be the student. Please indicate why this is the appropriate unit for this project? The ultimate goal of RITES is to improve student learning in science through partnerships, professional development, reformed instruction, and institutional change. This larger goal suggests that the student as the unit of analysis is appropriate for assessing the impact of RITES. In terms of providing formative feedback for program improvements, the unit of analysis varies to include the partnership, professional development, classroom instruction, the student, and the institutions depending on the evaluation question to be answered.

Rhode Island Technology Enhanced Science Program

Answers to Questions

page 24

References Arizona State University. (2000). Reformed Teaching Observation Protocol (RTOP) Training Guide. ACEPT Technical Report No. IN00-2. Baumgartner, E. (2004). Synergy Research and Knowledge Integration: Customizing Activities Around Stream Ecology. In M. Linn, E. A. Davis & P. Bell (Eds.), Internet Environments for Science Education. Mahwah, NJ: Lawrence Erlbaum Associates. Beng Keok, C., & Kean Thong, L. (2007). Intergrating bos, swot analysis, balanced scorecard and outcome-based framework for strategy formulation of engineering school. Proceedings of 37th Annual Frontiers in Education Conference. Bremser, W. G., & White, L. F. (2000). An experiential approach to learning about the balanced scorecard. Journal of Accounting Education, 18(3), pp. 241-255. Brown, M., & Edelson, D. (2001). Teaching by design: Curriculum design as a lens on instructional practice. Paper presented at the Annual meeting of the AERA. from http://www.inquirium.net/people/matt/BrownAERA2001.pdf. Harrington, T. R, & O'Shea, A. J. (2000). The Harrington-O'Shea career decision-making system-revised. Circle Pines, MN: American Guidance Service. Kaplan, R.S., & Norton, D.P. (1992). The balanced scorecard: Measures that drive performance. Harvard Business Review, 71(1), 71-79. Nibaghan, J., Slotta, J., & Cuthbert, A. J. (1999). A professional development model for effective Internet use: The Web-based Integrated Science Environment (WISE). Paper presented at the National Association for Resarch in Science Teaching, Boston, MA. Spitulnik, M. W., & Linn, M. C. (under review). Professional Development and Teachers’ Curriculum Customizations: Supporting Science in Diverse Middle Schools. Berkeley, CA: University of California. Tatar, D., Roschelle, J., Knudsen, J., Shechtman, N., Kaput, J., and Hopkins, B. (2008) Scaling up innovative technology-based mathematics. Journal of the Learning Sciences, 17:2, p. 248-286. Tuan, H., Chin, C., & Shieh, S. (May, 2005) The development of a questionnaire to measure students' motivation towards science learning. International Journal of Science Education, v27 n6 p639-654.

Rhode Island Technology Enhanced Science Program

Answers to Questions

page 25