Power Analysis

Power

The Four Components to a Statistical Conclusion Sample size The number of units (e.g., people) accessible to study Effect size The salience of the program relative to the noise Alpha level The odds the observed result is due to chance Power The odds you’ll observe a treatment effect when it occurs

The Four Components to a Statistical Conclusion Sample size Amount of information Effect size Salience of program Alpha level Willingness to risk Power Ability to see effect that’s there

The Effect Size Is a ratio of...

The Effect Size Is a ratio of...

Signal Noise

The Effect Size Is a ratio of...

Signal Noise

Difference between groups Standard error of the difference

Given Values for Any Three, You Can Compute the Fourth. ● ● ● ●

n = f(effect size, a, power) effect size = f(n, a, power) a = f(n, effect size, power) power = f(n, effect size, a)

The Decision Matrix In reality What we conclude

The Decision Matrix In reality What we conclude

Null true Alternative false In reality... • • •

There is no real program effect There is no difference, gain Our theory is wrong

The Decision Matrix In reality What we conclude Accept null Reject alternative We say... • • •

There is no real program effect There is no difference, gain Our theory is wrong

Null true Alternative false In reality... • • •

There is no real program effect There is no difference, gain Our theory is wrong

The Decision Matrix In reality What we conclude Accept null Reject alternative We say... • • •

There is no real program effect There is no difference, gain Our theory is wrong

Null true Alternative false In reality... • • •

There is no real program effect There is no difference, gain Our theory is wrong

1-α THE CONFIDENCE LEVEL The odds of saying there is no effect or gain when in fact there is none # of times out of 100 when there is no effect, we’ll say there is none

The Decision Matrix In reality What we conclude

Reject null Accept alternative We say... • • •

There is a real program effect There is a difference, gain Our theory is correct

Null true Alternative false In reality... • • •

There is no real program effect There is no difference, gain Our theory is wrong

The Decision Matrix In reality What we conclude

Reject null Accept alternative We say... • • •

There is a real program effect There is a difference, gain Our theory is correct

Null true Alternative false In reality... • • •

There is no real program effect There is no difference, gain Our theory is wrong

α TYPE I ERROR The odds of saying there is an effect or gain when in fact there is none # of times out of 100 when there is no effect, we’ll say there is one

The Decision Matrix In reality What we conclude

Null false Alternative true In reality... • • •

There is a real program effect There is a difference, gain Our theory is correct

The Decision Matrix In reality What we conclude Accept null Reject alternative We say... • • •

There is no real program effect There is no difference, gain Our theory is wrong

Null false Alternative true • • •

In reality...

There is a real program effect There is a difference, gain Our theory is correct

The Decision Matrix In reality What we conclude Accept null Reject alternative We say... • • •

There is no real program effect There is no difference, gain Our theory is wrong

Null false Alternative true • • •

In reality...

There is a real program effect There is a difference, gain Our theory is correct

β TYPE II ERROR The odds of saying there is no effect or gain when in fact there is one # of times out of 100 when there is an effect, we’ll say there is none

The Decision Matrix In reality What we conclude

Reject null Accept alternative We say... • • •

There is a real program effect There is a difference, gain Our theory is correct

Null false Alternative true • • •

In reality...

There is a real program effect There is a difference, gain Our theory is correct

The Decision Matrix In reality What we conclude

Reject null Accept alternative We say... • • •

There is a real program effect There is a difference, gain Our theory is correct

Null false Alternative true • • •

In reality...

There is a real program effect There is a difference, gain Our theory is correct

1-β POWER The odds of saying there is an effect or gain when in fact there is one # of times out of 100 when there is an effect, we’ll say there is one

The Decision Matrix In reality What we conclude Accept null Reject alternative We say... • • •

There is no real program effect There is no difference, gain Our theory is wrong

Reject null Accept alternative We say... • • •

There is a real program effect There is a difference, gain Our theory is correct

Null false Alternative true

Null true Alternative false In reality... • • •

There is no real program effect There is no difference, gain Our theory is wrong

• • •

In reality...

There is a real program effect There is a difference, gain Our theory is correct

1-α

β

THE CONFIDENCE LEVEL

TYPE II ERROR

The odds of saying there is no effect or gain when in fact there is none

The odds of saying there is no effect or gain when in fact there is one

# of times out of 100 when there is no effect, we’ll say there is none

# of times out of 100 when there is an effect, we’ll say there is none

α

1-β

TYPE I ERROR The odds of saying there is an effect or gain when in fact there is none

POWER The odds of saying there is an effect or gain when in fact there is one

# of times out of 100 when there is no effect, we’ll say there is one

# of times out of 100 when there is an effect, we’ll say there is one

The Decision Matrix In reality What we conclude Accept null Reject alternative We say... • • •

Accept alternative We say...

• •

• • •

There is no real program effect There is no difference, gain Our theory is wrong

• • •

In reality...

There is a real program effect There is a difference, gain Our theory is correct

1-α

β

THE CONFIDENCE LEVEL

TYPE II ERROR

There is no real program effect There is no difference, gain Our theory is wrong

Reject null

•

Null false Alternative true

Null true Alternative false In reality...

There is a real program effect There is a difference, gain Our theory is correct

α

1-β

TYPE I ERROR

POWER

The Decision Matrix In reality What we conclude Accept null Reject alternative We say...

Null false Alternative true

Null true Alternative false In reality... • • •

There is no real program effect There is no difference, gain Our theory is wrong

• • •

In reality...

There is a real program effect There is a difference, gain Our theory is correct

1-α

β

THE CONFIDENCE LEVEL

TYPE II ERROR

CORRECT • • •

There is no real program effect There is no difference, gain Our theory is wrong

Reject null Accept alternative We say... • • •

There is a real program effect There is a difference, gain Our theory is correct

α

1-β

TYPE I ERROR

POWER CORRECT

The Decision Matrix In reality What we conclude Accept null Reject alternative We say... • • •

Accept alternative We say...

• •

• • •

There is no real program effect There is no difference, gain Our theory is wrong

• • •

In reality...

There is a real program effect There is a difference, gain Our theory is correct

1-α

β

THE CONFIDENCE LEVEL

TYPE II ERROR

The odds of saying there is no effect or gain when in fact there is none

The odds of saying there is no effect or gain when in fact there is one

α

1-β

TYPE I ERROR The odds of saying there is an effect or gain when in fact there is none

POWER The odds of saying there is an effect or gain when in fact there is one

If you try to increase power, you increase the chance of winding up in the bottom # of times out of 100 when # of times out of 100 when is noof effect, we’ll sayI error. there is an effect, we’ll say rowthere and Type there is none there is none

There is no real program effect There is no difference, gain Our theory is wrong

Reject null

•

Null false Alternative true

Null true Alternative false In reality...

There is a real program effect There is a difference, gain Our theory is correct

# of times out of 100 when there is no effect, we’ll say there is one

# of times out of 100 when there is an effect, we’ll say there is one

If you try to decrease Type I The errors, In you reality increase the chance Whatof winding up we in the top row conclude and of Type II Accept null Reject alternative error. We say... • • •

There is no real program effect There is no difference, gain Our theory is wrong

Reject null Accept alternative We say... • • •

There is a real program effect There is a difference, gain Our theory is correct

Decision Matrix Null false Alternative true

Null true Alternative false In reality... • • •

There is no real program effect There is no difference, gain Our theory is wrong

• • •

In reality...

There is a real program effect There is a difference, gain Our theory is correct

1-α

β

THE CONFIDENCE LEVEL

TYPE II ERROR

The odds of saying there is no effect or gain when in fact there is none

The odds of saying there is no effect or gain when in fact there is one

# of times out of 100 when there is no effect, we’ll say there is none

# of times out of 100 when there is an effect, we’ll say there is none

α

1-β

TYPE I ERROR The odds of saying there is an effect or gain when in fact there is none

POWER The odds of saying there is an effect or gain when in fact there is one

# of times out of 100 when there is no effect, we’ll say there is one

# of times out of 100 when there is an effect, we’ll say there is one