Thursday Validity, Reliability And Generalizability

Validity, Reliability and Generalizability

Validity HOW MIGHT YOU BE WRONG? Internal Validity External Validity Construct Validity

Validity Are your findings really about what they appear to be about or did something else cause them?

Internal Validity Did your treatment actually cause the outcomes? Did anything else potentially cause the outcome instead of your treatment?

INTERNAL VALIDITY Internal validity addresses the "true" causes of the outcomes that you observed in your study. Strong internal validity means that you not only have reliable measures of your independent and dependent variables BUT a strong justification that causally links your independent variables to your dependent variables. At the same time, you are able to rule out extraneous variables, or alternative, often unanticipated, causes for your dependent variables. Thus strong internal validity refers to the unambiguous assignment of causes to effects. Internal validity is about causal control.

EXTERNAL VALIDITY External validity addresses the ability to generalize your study to other people and other situations. To have strong external validity (ideally), you need a probability sample of subjects or respondents drawn using "chance methods" from a clearly defined population

External Validity When you have strong external validity, you can generalize to other people and situations with confidence. – External validity refers to the population we use and the setting. If I do a study with fifth graders, then the results can be applied to a similar set of fifth graders.

Construct Validity Does your test measure what you think it really measures. What are you measuring or observing? Keep it simple and account for it.

CONSTRUCT VALIDITY

Construct validity is about the correspondence between your concepts (constructs) and the actual measurements that you use. A measure with high concept validity accurately reflects the abstract concept that you are trying to study. Since we can only know about our concepts through the concrete measures that we use, you can see that construct validity is extremely important. It also becomes clear why it is so important to have very clear conceptual definitions of our variables.

Construct Validity Construct validity is often established through the use of a multi-trait, multi-method matrix. At least two constructs are measured. Each construct is measured at least two different ways, and the type of measures is repeated across constructs. For example, each construct first might be measured using a questionnaire, then each construct would be measured using a similar set of behavioral observation categories.

RELIABILITY In order to make any kind of causal assessments in your research situation, you must first have reliable measures, i.e., stable and/or repeatable measures. If the random error variation in your measurements is so large that there is almost no stability in your measures, you can't explain anything!

Reliability Researcher Bias-How might you have effected the study? Account for it!

RELIABILITY Picture an intelligence test where an individual's scores ranged from moronic to genius level. No one would place any faith in the results of such a "test" because the person's scores were so unstable or unreliable.

Reliability Reliability is required to make statements about validity. However, reliable measures could be biased and hence "untrue" measures of a phenomenon) or confounded with other factors such as acquiescence response set. Picture a scale that always weighs five pounds too light. The results are reliable, but inaccurate or biased. Or, picture an intelligence test on which women or people of color always score lower (even if this doesn't occur on other tests). Again, the measure may be reliable but biased.

Reliability Identifying and controlling threats to reliability and validity in qualitative research is critical. Asking the question of whether another researcher, going into the field and asking the same interview questions would get the same answers is the overarching premise of reliability.

Generalizability (Be Honest) The study is generalizable to other school heads from international schools in Asia. However, one limit to this study was that the study will be conducted with schools only in Asia. Therefore, the findings will not apply universally. A second limit to the study is that the entire study was funded by the researcher, with limited funds for air travel and miscellaneous expenses. Therefore, schools were selected within a four hour flight range of the researcher’s home town. Telephone interviews were deemed less reliable as they did allow for interpretation of body language and visual clues.