Research Notes

Research Notes Characteristics of Good Research Design 1. Appropriateness to the research question • Whether the design does the possible job of providing trustworthy answers to the research questions. • A given research question can be addressed with a number of different designs. • Many designs are completely unsuitable for dealing with certain research problems • There are many research questions of interest to nurses for which highly structured designs are unsuitable. 2. Lack of Bias (The lack of unwanted influences that can produce a

distortion in the results of a study.) • The goals of experimental research can be summarized by four Major questions: 1. What is the strength of the evidence that a relationship exists between two variables? 2. If a relationship exists, what is the strength of the evidence that the independent variable of interest (e.g. an intervention), rather than extraneous variables, caused the outcome? 3. If the relationship is plausibly causal, what are the theoretical constructs underlying the related variables? 4. If the relationship is plausibly causal, what is the strength of evidence

that the relationship is generalizable across people, settings, and time?

2

These four questions correspond to the four types of design validity that form a framework for evaluating experiments. 1. Statistical Conclusion validity: This concerns the potential inappropriate use

of statistical procedures for analyzing data, leading to invalid conclusions about the relationship. - Statistical Power- refers to the ability of the design to detect true relationships among variables -

Is concerned with the question: “Is there a relationship between the independent and dependent variables?”

- Some specific threats to statistical conclusion validity include: a.

Low Statistical power – the power of a statistical test concerns its ability to reject the null hypothesis, that is, to document a real relationship between the IV and the DV. - Significant effects may be missed because of inadequate sample size or failure to control extraneous

b.

sources of variation.

Violated assumptions of statistical tests - Most statistical procedures are based on a variety of assumptions about the experimental data and the sample from which they are collected. If these assumptions are not met, statistical outcomes may lead to erroneous inferences.

c.

Error Rate – With certain tests, the probability of drawing incorrect conclusions increases as the number of repeated tests increases. Statistical procedures are generally available to control for this threat.

3 d.

Reliability and Variance – Statistical conclusions are threatened by any extraneous factors that increase variability within the data, such as unreliable measurement, failure to standardize the protocol, environmental interferences, or heterogeneity of subjects. These threats contribute to statistical error variance, which is a function of all variance in the data that cannot be explained by treatment effects.

2. Internal Validity - this refers to the degree to which it can be inferred that the experimental treatment (IV), rather than uncontrolled extraneous variables, is responsible for observed effects. - This addresses the question: “Given a statistical relationship between the IV and the DV, is there evidence that one causes the other?” - Extraneous variables present threats to internal validity because they offer competing explanations for the observed relationships between the IV and the DV; that is, they interfere with cause-and-effect inferences. -

True experiments have a high degree of internal validity because of the controlling properties of randomization and control groups. (refer to table for the list of types of threats to internal validity)

- Types include: 1. History – refers to the occurrence of external events that

take place concurrently with the IV that can affect the DV. Example: If we study the differential effect of two forms of exercise on knee extensor strength, history

4

effects may include some subjects’ participation in other athletic activities or other therapies that affect knee extensor strength. 2. Maturation – This refers to processes occurring within

the subjects during the course of the study as a result of the passage of time rather than as a result of a treatment or IV. - Maturation effects may cause subjects to respond differently on a second measurement because they have grown older, stronger, healthier, tired or bored since the first measurement. - Maturation is a relevant consideration in many areas of nursing research. Maturation does not refer to aging or development exclusively but rather to any change that occurs as a function of time. Thus wound healing, postop recovery, and many bodily changes that can occur with little or no nursing or medical intervention must be considered as an explanation based on the effects of the IV. 3. on

Testing – This refers to the effects of taking a pre-test subjects’ performance on a post-test.

- The mere act of collecting data changes the response that is opinions

being measured particularly in those that deal with and attitudes. - In true experiments, testing may not be a problem because

its groups.

effects would be expected to be about equal in all

5

- The Solomon four-group design could be used if researchers

wanted to isolate intervention effects from pre-

test effects. 4. Selection – This encompasses biases resulting from pre-existing differences between groups. - When individuals are not assigned randomly to groups. There is always a possibility that the groups are non-equivalent. - They may differ in ways that are subtle and difficult to detect. - If the groups are non-equivalent, differences on outcomes may result from initial differences rather than from the effect of the IV. 5. Mortality – Also called attrition, refers to the differential loss of subjects from comparison groups; that is dropouts

occur for specific reasons related to the experimental situation. - The loss of subjects during the course of a study may differ from one group to another because of a priori differences in interest, motivation, health, etc. - The risk of attrition is especially great when the length of time between points of data collection is long. - If attrition is random (i.e. those dropping out of a study are similar to those remaining in it with respect to extraneous characteristics), then there would not be bias. - In general, the higher the rate of attrition, the greater the likelihood of bias.

6

6. Instrumentation – This is concerned with the reliability of measurement. - This bias reflects changes in measuring instruments or methods of measurement between two points of data collection. Ruling out threats to internal validity Many threats such as history, maturation, selection, statistical regression, testing, instrumentation and selection interactions, can be ruled out by the use of random assignment and control groups. Random assignment cannot rule out the effects of attrition, imitating treatments, or compensatory reactions. Blinding subjects and investigators will control many of these effects. 3. Construct Validity - Construct validity of causes and effects concerns the

theoretical conceptualizations of the intervention and response variables and whether these have been developed sufficiently to allow reasonable interpretation and generalization of their relationship (Portney & Watkins, 2000). - This addresses the question: “Given that a cause-and-effect relationship is probable, to what theoretical constructs can the results be generalized?” a. Operational definition of the variables – threats to construct

validity are related to how variables are operationally defined within a study and to potential biases introduced into a study by subjects or experimenters.

7

- These threats were originally defined by Campbell and Stanley under the category of External Validity. These are now subdivided by Cook and Campbell into construct validity and external validity. - When studies incorporate only one form of measurement or examine only one form of treatment, the results will apply only to a limited aspect of the construct. Therefore, if a study addresses only one form of treatment op one form of measurement, generalization of the results of that study is limited. Example: Construct of pain which is multidimensional Suppose pain is treated with relaxation exercises or transcutaneous electrical nerve stimulation (TENS), measures of success may vary depending on whether we assess pain by using a visual analogue scale (VAS), by measuring range of motion of involved joints, or by observing he efficiency of functional tasks. VAS – reflects the patient’s subjective and relative feelings of pain intensity ROM test reflects physiological concomitants of pain Functional evaluation is influenced by personality, attitude, motivation, and lifestyle. Each of these assessments measures a different aspect of pain that reflects components of the total construct of pain.

8 b. Time frame within operational definitions

The element of time cannot be ignored in defining the construct of treatment, and testing may need to be done at various intervals to determine the range necessary to achieve maximal effectiveness. For example: If we study the effect of TENS over a 2-week period, we cannot generalize outcomes to events that might occur over a longer period of treatment. If treatment shows no effect within this time frame, we would be inaccurate to conclude that TENS does not work.. c. Hawthorne Effect – subjects nay behave in a particular

manner largely because they are aware of their participation of a study. Subjects often try their best to fulfil the researcher’s expectations or to present themselves in the best way possible, so that responses are no longer representative of the natural behavior. d. Experimenter Effects – Subjects’ behavior ay be affected by

characteristics of the researchers. The investigators may react more positively to subjects in the experimental group or give less attention to those in the control group, because of an emotional or intellectual investment in their hypothesis. If this is the case, the results in the original study might be difficult to replicate in a more neutral situation.

9

This threat to construct validity can be avoided by employing testers who are blinded to subject assignment and the research hypothesis. 4. External Validity – this refers to the extent to which the results of a study

can be generalized beyond the internal specifications of the study sample. - Addresses the question: “Can the results be generalized to persons,

settings, and times that are different from those

employed in the a.

experimental situation?”

Expectancy Effects- includes placebo effect and nocebo effect Placebo effect – this occurs when subjects administered a pesudointervention

improvements. That

show

changes

or

same placebo might not

have any benefits when not

administered in the

context of a study. Nocebo effect - this involves adverse side effects experienced by those receiving the placebo. b.

Novelty Effects – When treatment is new, subjects and researchers alike might alter their behavior in various

ways. - Results mat reflect reactions to the novelty rather

than

to the intrinsic nature of an intervention; once

the

treatment is more familiar, results mat be

different. c.

Interaction of history and treatment effects – this concerns the ability to generalize results to different periods of time in

the past or future.

10

Example: If we look at the results of nutritional studies

for reducing cholesterol in the diet,

results may be quite

different today from results

obtained 20 years ago, when

knowledge about the

effect of diet and exercise on CV

fitness was less

developed, and when society and the

media were less

involved in promoting fitness and health. d.

Measurement Effects – Researchers collect a considerable amt of data

in

most

studies,

information,

such

as

pretest

background data, and

so forth. The results may not apply

to

group of people who are not also exposed to

another the

same data collection (attention-giving) procedures. Other threats to external validity: 1.

Interaction of selection and treatment – When samples are confined to certain types of subjects it is not reasonable to generalize results to those who do not have these characteristics.

2.

Interaction of setting and treatment – If we demonstrate a causal relationship between an exercise program and functional improvement using patients in a rehabilitation hospital, can we generalize these findings to a nursing home or to home care? This question can only be answered by replicating effects in different settings.

11

Interaction of selection and treatment Interaction of setting and treatment Interaction of history and treatment -

Maybe it is just these people. Maybe it is just these places. Maybe it is just these times.

12

13

How Can We Improve External Validity? Population

Random Sampling Sample

Our Study

Tim e

Places

Replicate, Replicate, Replicate

People

Settings Places

Use theory

Our Study People

Times