Evaluation Measures, Purposes, And Schedule

Practice:

Evaluate outcomes to show your program is making a difference

Key Action:

Identify outcome measures and instruments

SAMPLE MATERIAL: Evaluation Measures, Purposes, and Schedule

Purpose:

An important aspect of planning an effective evaluation is to set a clear purpose for each data collection instrument you use. In this sample table, a district outlines the purpose for each instrument—achievement tests; and surveys of administrators, teachers, students, and families— and aligns those purposes with grant objectives and district strategic goals. They also propose a schedule for administering tests and surveys, which supports data collection logistics.



Note: While this proposal included experimental and quasi-experimental evaluation designs, it is a useful model for any type of evaluation.

Source:

From 2007 Magnet Schools Assistance Program (MSAP) grant application, developed by St Paul Public Schools (SPPS), in collaboration with their external evaluator, Mid-continent Research for Education and Learning (McREL), 2007.

1

Practice:

Evaluate outcomes to show your program is making a difference

Key Action:

Identify outcome measures and instruments

Table 2. Evaluation Measures, Purposes, and Schedule

Measure

Respondent

MCA-II

Treatment and comparison group students

SAT-10

Treatment and comparison group students

Student survey (existing survey or McRELdeveloped) During Year 1 (2007-08), McREL will locate or develop a survey aligned with the Magnet grant objectives (and aligned with SPPS district strategic goals). Teacher survey (existing survey or McRELdeveloped) During Year 1 (2007-08), McREL will locate or develop a teacher survey aligned with the Magnet

Treatment and comparison group students

Washington and Arlington teachers

Purpose

Proposed Schedule

To determine whether achievement of students in magnet schools exceeds that of students in comparison group; to determine the relationship between exposure to the intervention and achievement gains. To determine whether achievement of students in magnet schools exceeds that of students in comparison group; to determine the relationship between exposure to the intervention and achievement gains. Additional SAT-10 administration may take place to meet needs of grant evaluation. Measure: extent to which students perceive the instructional capacity of teachers to be of high quality; student engagement; extent to which students perceive decreased racial isolation; extent to which students participate in heterogeneous group-based activities inand out-of-school (see also attendance data); extent to which students feel “prepared for the future.”

Spring 2008 Spring 2009 Spring 2010

Extent to which teachers perceive themselves as being involved in joint training and planning and curriculum alignment; perception of recruitment efforts for hiring diverse staff; extent to which teachers believe that progress related to full implementation of the Critical Components of NCHSTE has been made each year; extent to which teachers perceive they are supported in

Spring 2008 Spring 2009 Spring 2010

Spring 2008 Fall 2008 Spring 2009 Spring 2010 Spring 2008 Spring 2009 Spring 2010

2

Measure grant objectives (and aligned with SPPS district strategic goals). Administrator survey (existing survey or McREL-developed) During Year 1 (2007-08), McREL will locate or develop an administrator survey aligned with the Magnet grant objectives (and aligned with SPPS district strategic goals). Parent survey (existing survey or McRELdeveloped) During Year 1 (2007-08), McREL will locate or develop a parent survey aligned with the Magnet grant objectives (and aligned with SPPS district strategic goals). Attendance data

Respondent

Washington and Arlington administrators; other SPPS administrators

Parents of middle and high schoolaged students eligible for Washington or Arlington

SPPS

Practice:

Evaluate outcomes to show your program is making a difference

Key Action:

Identify outcome measures and instruments

Purpose their implementation of the project; extent to which adequate professional development activities have been provided. Extent to which administrators perceive that the marketing plan is viable; administrator perceptions of teacher involvement in joint training and planning and curriculum alignment; perception of recruitment efforts for hiring diverse staff; perception of whether partnerships have been developed and the nature and type of such partnerships; extent to which administrators perceive they have supported teachers in the implementation of the project; extent to which adequate professional development activities have been provided for teachers. Parental perception of the impact of the marketing plan on their enrollment choices; extent to which parent and community participation increased.

Proposed Schedule

Spring 2008 Spring 2009 Spring 2010

Spring 2008 Spring 2009 Spring 2010

To ascertain attendance at magnet schools (Washington Fall 2008 and Arlington) as well as the matched comparison Spring 2009 group; to determine participation in in-school and outSpring 2010 of-school group-based activities (student demographic data will also be collected to determine the extent of interaction with heterogeneous groups).

3