Experimentation

Dr.V.Veera Balaji Kumar BHMS., M Sc., M Phil

ISSUES IN EXPERIMENTATION Randomization Persons in the control group as well as the

experimental group must be equally likely candidates, and, to be really correct, the persons who participate in your experiment should be drawn randomly from all persons who could conceivably be candidates for the new treatment or procedure. The next step is to randomly assign each one to either experimental or control group. This insures statistical equivalence of the experimental and control groups. What this means is that, in a broad sense, the groups are identical with the one exception of the

Establishing a CauseEffect Relationship What criteria do we have to meet? there are three criteria that you must

meet before you can say that you have evidence for a causal relationship: Temporal Precedence Covariation of the Cause and Effect No Plausible Alternative Explanations

Temporal Precedence you have to be able to show that your

cause happened before your effect consider a classic example from economics: does inflation cause unemployment? as inflation increases, more employers find that in order to meet costs they have to lay off employees. So it seems that inflation could, at least partially, be a cause for unemployment.

Is it possible that fluctuations in employment can affect inflation? But both inflation and employment rates are

occurring together on an ongoing basis. If we have an increase in the work force (i.e., lower unemployment) we may have more demand for goods, which would tend to drive up the prices (i.e., inflate them) at least until supply can catch up. So which is the cause and which the effect, inflation or unemployment?

EXPERIMENTAL VALIDITY

INTERNAL VALIDITY Cook T., and D.Campbell refer internal validity

as “ the approximate validity with which we infer that a relationship between two variable is causal or that the absence of a relationship implies the absence of cause.” Any research findings drawn from experimentation in the absence of internal validity will be superficial and deceptive.

Key question in internal validity is whether observed changes can be attributed to your program or intervention (i.e., the cause) and not to other possible causes (sometimes described as "alternative explanations" for the outcome).

Internal validity is only relevant in studies that try to establish a causal

THREAT TO INTERNAL VALIDITY (OR) EXTRANEOUS VARIABLES Six major types: 2.History threat 3.Maturation threat 4.Testing threat 5.Instrumentation threat 6.Selection bias threat 7.Statistical regression threat 8.Mortality threat

Threats to internal validity

• Let's imagine that we are studying the effects

of a compensatory education program in mathematics for first grade students on a measure of math performance such as a standardized math achievement test.

History threat

• It's not your math program that

caused the outcome, it's something else, some historical event that occurred. For instance, we know that lot's of first graders undergo abacus training . • And, we know that in every abacus training gives tips to quick solve difficult math problems. Perhaps these abacus training cause the outcome and not your math program.

Maturation Threat • The children would have

had the exact same outcome even if they had never had your special math training program. • All you are doing is measuring normal maturation or growth in understanding that occurs as part of growing up -- your math program has no effect.

Testing Threat • only occurs in the pre-post

design • taking the pretest made some of the children more aware of that kind of math problem • it “prepared" them for the program so that when you began the math training they were ready for it

Selection Bias Improper assignment of

respondents to the treatment conditions. Two ways 1. Wrong selection of test units in

experimental group. As a result sample does not represent the population from which units are drawn. 2. Test units are assigned to the experimental group differ from those assigned to the control group.

Instrumentation Threat • Only operates in the pretest-posttest situation. • The change from pretest to posttest is due not

to your math program but rather to a change in the test that was used. • Instrumentation threats are especially likely when the "instrument" is a human observer. The observers may get tired over time or bored with the observations. • Conversely, they might get better at making the observations as they practice more

Mortality Threat Means that people are "dying" with respect to

your study. Usually, it means that they are dropping out of the study. When mortality is a threat, the researcher can often gauge the degree of the threat by comparing the dropout group against the nondropout group on pretest measures.

Statistical Regression Threat • A statistical

phenomenon that occurs whenever you have a nonrandom sample from a population and two measures that are imperfectly correlated. • Also known as a "regression artifact" or "regression to the mean" • you can only go up ( or down) from here" phenomenon.

EXTERNAL VALIDITY

The validity with which we can infer that the

presumed causal relationship can be generalized to population from which the samples were taken.

EXTERNAL VALIDITY Experiments conducted in

natural settings can offer greater external validity compared to those conducted in controlled environments.

EXPERIMENTAL ENVIRONMENT Laboratory environment – conducted in controlled / artificial environments. Extraneous variable can be controlled. Effective in eliminating history effect. Field environment – conducted in the natural settings. High degree of external validity.  No control over extraneous variables  low

internal validity.  Require greater time and effort. Expensive

TYPES OF EXPERIMENTAL DESIGNS Pre-experimental One-shot design One group pre-test post-test design Static group design

True experimental design Pre-test post test control group Post test only control group Solomon four group Quasi experimental design Statistical design

Pre-experimental design Lack proper control mechanisms to deal with

influence of extraneous variables. When there is lack of resources for a detailed research. Three types: One group One-shot design ( after only design) One group pre-test post-test design Static group design

One-shot design Involves exposing the experimental group to

treatment (X) after which the measurement (O1) of the dependant variable is taken. This can be symbolically shown as follows: X

O1

Drawbacks: Test units are not selected randomly No comparative measure for O1 No control of extraneous variables.

One group pre-test post-test design Involves exposing an experimental group to

the treatment (X). Measurements are taken before (O1) and after (O2) the treatment. Symbolic expression : The difference between O1 and O2 will be the impact of treatment on the dependant variable. An HR manager may conduct a training program (X) of his employees to increase their productivity (O). To know the impact the program, he will measure the productivity before (O1) and after (O2) the program.

Statistical group design Two groups of test units the experimental

group and the control group are involved. The experimental group is exposed to the treatment and the control group is not exposed to the treatment. The measurements are taken for the groups and compared. Symbolic expression : O1 – measure of DV of the experimental Group (EG) O2 – measure of the DV of the control group (CG)

Drawbacks of Statistical group design Selection bias – non-random selection units.

Result in differences between experimental and control groups.

Mortality effect – the treatment may be

strenuous and as a result the participants may drop out.

True experimental design Post test only control (randomized) group Pre-test post test control (randomized) group Solomon four group

Posttest-Only Control Group Design  "Population" is ALL persons to whom this treatment might possibly

be applied.  The arrows and the letter "R" to "Sample" indicates that a sample is randomly selected is drawn from the pool of all potential patients.  The arrows and the next letters "R" indicate that the "Sample" is randomly assigned (ditto) to either experimental or control group.  The Arrow to the letter "E" to the letter "O" is the experimental group. The "E" represents the experimental treatment, and the "O" represents the observation (measurement) of the effect of the treatment.  The other branch, with a "C, represents the control group. You can simply do nothing to them. Either way, they are the control group for your hot new idea.  It is extremely important, however, that the only difference in the

Posttest-Only Control Group Design

 The posttest-only randomized

experiment is strong against the single-group threats to internal validity because it's not a single group design!  It's strong against the all of the multiple-group threats except for selection-mortality.  Selection-mortality threat is especially salient if there are differential rates of dropouts in the two groups. This could result if the treatment or program is a noxious or negative one (e.g., a painful medical procedure like chemotherapy) or if the control

Pre-test post test control group Test units are assigned randomly to experimental

and control group. Pretest measures are taken fro both the EG and CG The EG is exposed to the treatment. CG is not exposed to the treatment. EG: R O1 X O2 CG: R O3 X O4 The treatment effect is calculated as follows: TE = ( O2-O1 ) – ( O4-O3 )

The Solomon FourGroup Design  The Solomon Four-Group Design is a very tight

experimental design, controlling well for both internal and external potential sources of error or ambiguity.  Here an experiment is conducted with four groups – 2 experimental and 2 control groups.  6 measures are taken – 2 pre test and 2 post test.  This study is also known as four group or six study design.  Symbolic expression

QUASI EXPERIMENTAL DESIGN A quasi-experimental design is one that looks

a bit like an experimental design but lacks the key ingredient -- random assignment. It is better than pre-experimental designs. Many types : Time-series designs Control Group Time Series Design Equivalent Time Samples Design Equivalent Materials Design Non-equivalent Control Group Design Counterbalanced Designs

Time-series designs • You make a few observations to establish a baseline, do the intervention, and then make a few more measurements. • The major threat to the internal validity of the time series is history -- a charge that the results obtained would have occurred with or without the experimental intervention. • Longitudinal studies done to identify temporary and permanent trends. • History and instrumentation threats for the internal validity. • E.g. series of measures taken before and after a

Time-series designs 'A', the outcome measure is stable at a certain value until your intervention, then it increases and stabilizes at a higher level. 'B', the outcome measure is again stable at a certain value until the intervention, then it decreases and stabilizes at a lower level. 'C', the outcome measure is stable at a certain level until the intervention, then it increases immediately after the intervention; however, it then returns to the preintervention baseline. 'D', the outcome measure is - well basically all over the place. In a situation like this it will be tough to "tease out" the effects of your

STATISTICAL DESIGNS Aid in measuring the effect of more than one

independent variable, instead of conducting a series of experiments fro each independent variable. Helpful is isolating the effects of extraneous variables. Four prominent types: Completely randomized design Randomized block design Latin square design Factorial design

Completely randomized design Used when the researcher has to evaluate the

effect of a single variable. The effects of the extraneous variables are controlled using the randomization technique. Random assignment of test units to treatment groups. Later the post treatment measurements are evaluated. EG1,EG2,EG3 are experimental groups. X1,X2 & X3 are different treatments. O1, O2 & O3

E.G. CBD A pharmaceutical researcher plans to

evaluate the efficacy of a weight loss drug made his company. He want to compare it with existing drugs for weight loss. So uses the CBD : he randomly assigns test units to three experimental groups. For one group he administers his company’s drug. And for other groups he administer competitors drug and measure weight after a 3 month period. 

Graphical portrayal

Randomized block design Used when there is

one major extraneous variable that will influence experimental results. The test units ore grouped or blocked based on the extraneous variable. So the effect of the extraneous variable does not affect the depending variable. It is equivalent to the stratified random sampling

A new Health drink Example of a Randomized block design

Store type Price change

Drug stores

Super market

Hyper market

Rs. 10

Store 1

Store 5

Store 2

Rs. 12

Store 3

Store 8

Store 6

Rs. 14

Store 4

Store 9

Store 7

The effect of store type can intervene the effect of the new health drink on the sales. So the sample is blocked or grouped into subgroups based on the store type.

Latin square design Used when two interactive external variables

has to be controlled. Blocking technique is used. The blocking or the EV are divided into an equal number of levels and so is the independent variable. The table is then developed with levels of one E.V representing the rows and levels of another representing the columns. The levels of the IV (or treatments) are exposed to each cell on a random basis, so that there is only one treatment in each row and column

A new advertising campaign A 3 * 3 Square table

Income levels Pricing levels Low income

Middle income

High income

Ad-A

Ad-C

Rs 10,000

Ad-B

Rs 12,000

Ad-C

Ad-B

Ad-A

Rs 14,000

Ad-A

Ad-C

Ad-B

• We cannot examine the inter-relationships between pricing income levels and adverting programs. • In situations where any of the variables does not have same number of levels as that of other two variables this design is not valid

Factorial design