Making Sure Data Are Valid And Reliable

Practice:

Get quality data into your evaluator’s hands

Key Action:

Use technique to ensure valid and reliable data

TOOL: Making Sure Data Are Valid and Reliable

Purpose:

Before you can confidently interpret and analyze your evaluation data, you must ensure that the data you collect are valid and reliable. Otherwise, they won’t adequately support your outcomes. Use the questions and suggestions in this table to ensure the data you collect are valid and reliable.

Instructions:

1. Review the “Questions to Consider” in the table to ensure that the data you collect for your evaluation are valid and reliable, and that your sample size is adequate. 2. Consider the suggestions or proactive measures you might take to ensure valid and reliable data. 3. Based on what you learn, note specific actions you might use in your district to ensure that your evaluation generates valid and reliable data.

1

Practice:

Get quality data into your evaluator’s hands

Key Action:

Use technique to ensure valid and reliable data

Making Sure Data Are Valid and Reliable Questions to consider Do the data represent the outcomes that the instrument is supposed to measure? Are the data valid? A valid measure assesses what it is designed to measure, which allows for comparison of results across studies. Have we ensured that our measures are reliable? A reliable measure produces stable responses regardless of the data collector. An unreliable measure will yield varied responses depending on differences between interviewers or data collectors.

Suggestions

Actions we’ll consider

First make sure you select instruments that measure the kinds of outcomes your magnet program is expected to produce. You can increase measurement validity by using field-tested instruments that have demonstrated reliability and validity. Note: State assessments have been recalibrated or revised during the years of your study, so you may not be able to compare data from year to year. Discuss with your evaluator how you will test for reliability of your instruments. You may want to borrow from existing instruments, have an expert panel review and react to new instruments, or pilot test the instruments in real settings and among members of your target audience. Build in time and resources to test for reliability and calculate reliability coefficients so that you can assure stakeholders that you have strong instruments. Note that qualitative data collection (e.g., observations, open-ended interviews) poses different validity challenges. Ideally, the instruments you create for these purposes will require low levels of inference. Provide time and resources for researcher training in use of the instruments to minimize differences in participant responses across data collectors.

2

Questions to consider Do we have the right data? Do we have enough data? To be valid for decision-making, data must answer your questions about program outcomes and include a sufficient number of participants to be representative of your target population and its various subgroups. Validity of data is improved when data collection is “triangulated,” for example, when various methodologies are used to measure the same phenomenon, or multiple researchers conduct a structured observation of the same phenomenon.

Practice:

Get quality data into your evaluator’s hands

Key Action:

Use technique to ensure valid and reliable data

Suggestions

Actions we’ll consider

You will need an adequate sample size to ensure your data are valid. Make sure you begin with large enough numbers of students and schools in your evaluation study, taking projected attrition into account. Then, as data collection begins, check your data to make sure that subgroup data are appropriately coded and that sufficient numbers of students in subgroups have taken the tests as planned. Otherwise you may not have enough students in particular subgroups to make those data valid.

3