Judging Whether A Survey Is Valid And Reliable

Practice:

Get quality data into your evaluator’s hands

Key Action:

Use technique to ensure valid and reliable data

TOOL: Judging the Validity and Reliability of a Survey

Purpose:

Surveys are commonly used to measure program outcomes. However, to yield accurate information, surveys must be both reliable and valid. The following narrative and checklist can help you determine whether surveys proposed for use in your evaluation have been tested for reliability and validity. Note: Reliability and validity are sometimes termed “psychometric properties.” Conducting the statistical tests required to establish survey reliability and validity can be labor intensive. Individual magnet school evaluations generally do not have the time or financial resources to do thorough psychometric testing of an instrument. The most efficient alternative is to use existing evaluation instruments that provide appropriate data documenting their reliability and validity.

Instructions:

1. Read the brief definitions and examples of “validity” and “reliability.” 2. Review the checklist with your evaluator as a way to begin a discussion about the validity and reliability of survey instruments being used in your evaluation.

1

Practice:

Get quality data into your evaluator’s hands

Key Action:

Use technique to ensure valid and reliable data

Judging the Validity and Reliability of a Survey Magnet schools are designed to produce both cognitive (improved learning in reading and mathematics) and noncognitive outcomes (increased student and staff engagement with academic content; decreased racial isolation). Commonly, some outcomes are measured through surveys, and it’s important that such surveys are both valid and reliable, so that they yield credible information. Validity: Instrument validity means that the survey or test measures what it says it’s measuring, and not some other concept. For example, if a survey of students claims to measure their engagement in school activities, then it should ask questions that address the extent to which they voluntarily commit time and effort to school-related endeavors. The keys here are the words “voluntarily” and “effort.” For example, we might ask how much time they devote to extracurricular activities (i.e., Do they spend more time in school than is required?) or what books they read for pleasure (i.e., Is the content of the Type here books related to the content of the curriculum?). We shouldn’t ask if they “like” school—we can like something (“I like having the TV on when I’m home”) without being engaged with it (“the TV is a nice background while I’m cooking dinner”). Reliability: Instrument reliability means that the survey or test yields the same results on repeated trials. For example, a thermometer should read 212 degrees Fahrenheit every time it is placed in boiling water. Without research tools and procedures whose reliability is documented, it’s not possible to draw credible conclusions from your evaluation or make data-based decisions about how to improve your program. Checklist for Validity and Reliability 1. Does the survey instrument include accompanying data on its validity and reliability? 2. If so, which of the following methods were used to establish these psychometric properties? a. Test-retest reliability: Administering the same test to the same groups multiple times, with similar results b. Factor analysis: Determining whether items that are conceptually linked show similar linkages in the answers c. Split-half reliability: Using two versions of questions and comparing the results d. Other statistical techniques, such as structural equation modeling, an analytic technique that assesses the “fit” between the questions and the underlying constructs

2