Anticipating Data Collection Issues For Rigorous Evaluation

Practice:

Get quality data into your evaluator’s hands

Key Action:

Troubleshoot data issues as they emerge

TOOL: Anticipating Data Collection Issues for Rigorous Evaluation

Purpose:

As you plan and begin your data collection, it is important to anticipate issues common to rigorous evaluations of magnet programs. Use this tool to think about and address specific issues that may negatively impact your evaluation findings.

Instructions:

1. Review the list to learn about common data collection issues that may negatively impact your evaluation findings. 2. Based on the context of your particular district and your understanding of the data collection issues, identify the issues that are most likely to affect your evaluation that need proactive problem solving. 3. Determine how your evaluation team will take measures to address these issues and/or build in checkpoints to routinely monitor these issues throughout the process of data collection.

1

Practice:

Get quality data into your evaluator’s hands

Key Action:

Troubleshoot data issues as they emerge

Anticipating Data Collection Issues for Rigorous Evaluation Rigorous evaluation issue

How this may impact our evaluation

Ways to address the issue

Lack of treatment fidelity: To accurately assess the impact of the magnet program, you must determine the extent to which key elements (or “treatment”) of a magnet program were implemented as originally intended. Before you begin data collection, figure out how you will measure “treatment fidelity,” or the degree to which planned program activities are conducted and participants are reached. In other words, be clear about how you will know if you are doing what you said you would do.

Type here

Treatment “cross-over” in control or comparison group: The control or comparison group may engage in some or all of the elements of the magnet program being evaluated. Indeed, few schools or classrooms are intervention-free, and it is difficult to anticipate all the ways best educational practices may reach students. It is also difficult to predict ahead of time which schools will adopt what programs within the time frame of the evaluation study. For this reason, it is important to be clear up front with control and comparison schools about the purpose of the evaluation and the need to limit treatment crossover. You will also need to keep regular close contact with all groups to document what they are doing. Selection bias: In a quasi-experimental design, where students are not selected through random assignment, statistical techniques may account for some differences between the treatment and comparison groups. Evaluators will try to make the best matches between schools and individual students to control for variables. However, when students choose to participate in magnet programs, it can be difficult to eliminate the effect of this selection bias on student outcomes. You need to minimize the effect of difficult-to-measure differences between your treatment and comparison group (e.g., level of motivation, parent involvement). Adapted from: U.S. Department of Education, Office of Safe and Drug-Free Schools. (2007). Mobilizing for evidence-based character education (p. 36). Washington, DC: Author. The entire guide can be downloaded at www.ed.gov/programs/charactered/mobilizing.pdf (last accessed December 10, 2008).

2

Practice:

Get quality data into your evaluator’s hands

Key Action:

Troubleshoot data issues as they emerge

Rigorous evaluation issue

How this may impact our evaluation

Ways to address the issue

Attrition: The loss of students, parents, and teachers, as well as entire schools, can threaten an experimental or quasi-experimental design by leaving evaluators with inadequate numbers of participants to make statistically significant analyses. It may be difficult to make predictions about participation of students and schools for the length of time an evaluation may need data. Attrition of students is common for many magnet programs, which tend to serve urban areas that traditionally experience high rates of student mobility. Urban districts also face school closures because of program improvement policies or declining enrollment, which may mean loss of original comparison or control schools. It is important to “over-recruit” (or “over-sample”) students and schools at the beginning of the study to anticipate attrition. Design breakdown: Execution problems—such as the failure to collect data appropriately or within the set time frame—will interfere with even the best-laid plans. Well-researched planning will help ensure your data collection plan is feasible and frequent check-ins with the evaluation team will make sure you stay on schedule and troubleshoot any issues that emerge. Consent bias: Some people will decline to participate in a study, and those who do not participate may have different characteristics from the people who consent to take part. For this reason, consent forms should include the option to decline and an accompanying request for minimal background information relevant to the study’s objectives. You will then be able to document differences between those who decline and those who participate. The best way to reduce this bias is to encourage everyone’s participation in the study and to conduct random assignment after a sufficient percentage of consents has been obtained.

Adapted from: U.S. Department of Education, Office of Safe and Drug-Free Schools. (2007). Mobilizing for evidence-based character education (p. 36). Washington, DC: Author. The entire guide can be downloaded at www.ed.gov/programs/charactered/mobilizing.pdf (last accessed December 10, 2008).

3