Prof Kuhns Repot Final

Item and test analysis report

By J.S.Ditshego Bed Hons 2008

Report on test and item analysis Bed Hons 2008 Computer Integration Josiah Seraga Ditshego

ii

Acknowledgements I wish to express my sincere appreciation to Prof Kϋhn for his professional support and constructive comments. I would also like thank a class of 2008 for the moral support. A special thank you to my family and in particular my wife Aletta.

iii

Descriptive abstract This report represents a working tool to problems faced by High school educators in their quest to improve the quality of item and test analysis. The higher failure rate in matric requires a closer look into the learner’s responses to questions/items.

iv

Table of content No

Content

Page

List of tables List of figures

vi Vii

1 2 3 4 4.1 4.2 4.2.1 4.2.2 4.2.3 4.2.4 4.2.5 4.3 5 5.1

Introduction Purpose of the report Procedure Test analysis Descriptive statistics Frequency graphs Grouped frequency table Cumulative frequency Frequency histogram Frequency polygon Cumulative frequency graph Reliability coefficient of a test Item analysis Difficulty and discrimination of a set of

6 7 8 8.1 8.2 8.3

test scores Conclusion References Appendices Appendix A Appendix B Appendix C

1 1 1 2 2 3 3 3 4 5 6 6 8 8 12 13 14 14 15 16

List of Tables Table

Description

Page

Table 1

Descriptive statistics

2

Table 2

Grouped frequency table

3

Table 3

Cumulative frequency

4

distribution table v

Table 4

Coefficient of reliability

7

Table 5

Difficulty index (p)

8

Table 6

Interpretation of difficulty

9

Table 7

level of questions Number of students in the

11

Table 8

upper and the lower group Discrimination index (D)

12

List of figures Figure

Description

Figure 1

A normal distribution curve

Figure 2

Frequency histogram

Figure 3

Frequency Polygon graph

Figure 4

Cumulative frequency graph

vi

Page

Terminology List Term

Definition

Mean

Define as the average of scores

Mode

It is the score that occurs most frequently

Median

It is the score that divide the distribution

Reliability coefficient

into two halves Defined as the extent to which the test is likely to give consistent scores (Borich & Kubiszyn ). vii

STEDV

It s the value that include all the scores in a distribution. It as an estimate of variability, and includes the mean in the

Descriptive statistics

distribution. Describe basic features of data

Item analysis

Described as statistical analysis which shows the effectiveness of individual test items (Borich & Kubiszyn).

1

Introduction

The Office of Educational Assessment (2005) defines item analysis as a process, which examine student responses to individual test items/questions for the purposes of assessing their quality and the quality of a test a whole. This item analysis report is based on twenty multiple-choice test question administered to twenty-five students

viii

(Refer to Appendix B).The quality of individual test items/ question is assessed by comparing student’s item responses to their total scores. The summary of the performance of the test is shown in (Appendix A).

2

Purpose of the report

The purpose of this report is to disseminate information to parents; School management team and institutional support co-ordinators.This report is based on the descriptive statistics about twenty multiple choice test items administered to twenty students.

3

Procedures

The spread sheet was used to collate from twenty students. Twentyfive multiple choice test items were administered to twenty students. Microsoft Excel was used to calculate scores. The scores were used to calculate mean, median mode and standard deviation. These values were used to get a normal distribution graph. Three graphs were drawn using Microsoft Excel. Reliability coefficient was calculated using kuder-Richardson equation (i.e. K20).Difficulty index (p) and discrimination index were also calculated using questions and results from the test.

4 Test analysis 4.1 Descriptive statistics The descriptive statistics describe basic features of data. They describe summaries about the sample, measurements and quantities in a manageable form. A set of test scores were used to calculate the mean, median, mode and the standard deviation as shown in Table 1. ix

Table 1:

Descriptive statistics Mean Mode Median STEDV2 STEDV

65 65 65 479.57 21.90

The table shows that the mean, median and mode are the same hence it is a normal distribution curve. The rational here is that if the distribution is truly normal, the mean, median, and mode are all equal to each other i.e. (bell-shaped) as referenced in Figure 1

0.0

0.1

0.2

0.3

0.4

Figure 1: A normal distribution curve

The values on the x-axis represent standard deviation from the mean. The area where there is a change in curvature is one standard on either side of the mean, then follows two, three and four respectively. The curve is symmetrical, i.e it can be divided into two equal halves. It is a heterogeneous distribution. -4SD

-3SD

-2SD

-1SD

65

4.2 Frequency graphs 4.2.1

Grouped frequency table

x

+1SD +2SD +3SD

+4SD

The highest score and the lowest score are used to find the range. The number of intervals is a choice. The size of the interval is from eighty (range) divided by ten (number of intervals). Table 2: Grouped frequency H (Highest value)

100

L (Lowest value)

20

Range

80

Number of intervals

10

Size of interval

8

4.2.2

Cumulative frequency

Cumulative frequency is calculated by adding the frequency of the class interval and the frequencies of the preceding interval until the class interval as indicated in Table 3.

Table 3:

Cumulative frequency distribution

Lower

Upper

Interva

Middle

Values

Values

l

Values

16

23

16-23

19.5

1

1

24

31

24-31

27.5

2

3

32

39

32-39

35.5

0

3

xi

Frequency Cumulative Frequency

40

47

40-47

43.5

2

5

48

55

48-55

51.5

3

8

56

63

56-63

59.5

2

10

64

71

64-71

67.5

6

16

72

79

72-79

75.5

1

17

80

87

80-87

83.5

3

20

88

95

88-95

91.5

3

23

96

103

96-103

99.5

2

25

4.2.3

Frequency histogram

The frequency histogram is drawn from the class interval and frequency. The histogram is shown in Figure 2:

Figure 2

Frequency histogram

xii

Frequency histogram 7 6 5 4 3 2 1 0

Fr e q u e n cy 4.2.4

23

31

39

47

55

63

71

79

87

95

103

16

24

32

40

48 56 64 Interval

72

80

88

96

Frequency polygon

The frequency polygon is drawn from cumulative frequency and the upper values. The frequency values and the upper values are shown in Table 3: The frequency polygon graph is illustrated in Figure 3: Figure 3

Frequency Polygon graph

Frequency Polygon 7 6 5 4 3 2 1 0

Frequency 1 Middle values 4.2.5

Cumulative frequency graph

xiii

Cumulative frequency is drawn by using cumulative frequency values and upper values. Cumulative frequency graph is shown in Figure 5. Figure 4

Cumulative frequency graph

Cumulative frequency graph (An Ogive)

30 25 20 15 10 5 0 3 10

87

71

55

39

23

Cumulative Frequency

Upper values

4.3 Reliability coefficient of a test The reliability coefficient of a test is defined as the extent to which the test is likely to give consistent scores (Borich & Kubiszyn).The values ranges from zero to one where zero is no reliability and one is perfect reliability. The kR20 is used to measure test reliability of interitem consistency. The high value indicates a strong relationship between items on the test.

The formulae for calculating KR20 is:

xiv

KR20 = (k/k-1) (1-∑pq/∂2) Where: K=No of items p=Proportion that passed q=Proportion that failed ∂2=Variance The values used to calculate KR20 are referenced in Table 4. Table 4:

Coefficient of reliability

K

20

K-1

19

Total pq

3.83

SD (standard deviation)

21.9

(SD)2

479.6

KR20

1.03

It is evident from the table that the value of KR20 is 1.04; this implies that the test has a perfect reliability.

5

Item analysis

xv

Item analysis is described as statistical analysis which shows the effectiveness of individual test items (Borich & Kubiszyn).

5.1 Difficultly and discrimination indices of a set test items The questions and the results from the test were used to describe the difficulty of each question and the corresponding discrimination index as referenced in Table 5 Table 5:

Difficulty index (p) Difficulty index (p)

#Questions

#Correct

#Answered

p

Q1

21

25

0.84

Q2

22

25

0.88

Q3

17

25

.68

Q4

12

25

0.48

Q5

21

25

0.84

Q6

17

25

0.68

Q7

11

25

0.44

Q8

12

23

0.52

Q9

13

25

0.52

Q10

8

24

0.33

Q11

23

25

0.92

Q12

19

25

0.76

Q13

15

25

0.60

Table 5:

Difficulty index (p) (continued)

xvi

Difficulty index #Questions

#Correct

#Answered

p

Q14

21

25

0.84

Q15

20

25

0.80

Q16

22

24

0.92

Q17

15

24

0.63

Q18

8

24

0.33

Q19

13

25

0.52

Q20

16

25

0.64

The percentages of students who answered the item correctly is reflected in column (p).It measures the level at which the question was difficult to answer. If the difficulty index is higher then the question was too easy. The value 1.00 implies that all students answered this correct response and the question was too easy. If the p value is .75, the item is acceptable but if the p value is .25, then the item is too difficult. The interpretation of difficulty level of questions is listed in Table 6. Table 6:

Interpretation of difficulty level of questions

Questions

Proportions

Interpretation Reason

Q1

0.84

Unacceptable

Too easy

Q2

0.88

Unacceptable

Too easy

Q3

0.68

Acceptable

Fine

Q4

0.48

Acceptable

Fine

Q5

0.84

Unacceptable

Too easy

Table 6:

Interpretation of difficulty level of questions (continued)

xvii

Questions

Proportions

Interpretation Reason

Q6

0.68

Acceptable

Fine

Q7

0.44

Acceptable

Fine

Q8

0.52

Acceptable

Fine

Q9

0.52

Acceptable

Fine

Q10

0.33

Acceptable

Fine

Q11

0.92

Unacceptable

Too easy

Q12

0.76

Unacceptable

Too easy

Q13

0.60

Acceptable

Fine

Q14

0.84

Unacceptable

Too easy

Q15

0.80

Acceptable

Fine

Q16

0.92

Unacceptable

Too easy

Q17

0.63

Acceptable

Fine

Q18

0.33

Acceptable

Fine

Q19

0.52

Acceptable

Fine

Q20

0.64

Acceptable

Fine

It is evident from the Table 6.that 35% of the questions i.e. (question 1, 2, 5,11,12,14 and 16) are unacceptable and the implication is that they were too easy.65% of the questions i.e. (3, 4, 6, 7, 8, 9, 10, 14, 15, 17, 18, 19, and 20) are acceptable and fine as referenced in Table 6. The interpretation of the difficulty level of questions is referenced in Table 5.

xviii

The discrimination index is used to ability of test items to distinguish between the lower and the upper group of students taking the test as shown in Table 7 Table 7:

Number of students in the upper and lower group

Level

Value

Upper

15

Lower

10

These measures the ability of an item to discriminate of differentiate among students who got higher score to those who got lower scores. It is actually the difference between percentages of correct response in the upper group and the correct response in the lower group. Refer to Table 8 for calculations on the discrimination index. Table 8:

Discrimination index (D) Discrimination index

#U

#L

D

15

6

0.60

15

7

0.53

14

3

0.73

8

4

0.26

15

6

0.60

12

5

0.46

9

2

0.46

10

2

0.53

xix

Table 8:

Discrimination index (D) (continued) Discrimination index 3 0.46 0 0.53 9 0.33 5 0.60 3 0.60 6 0.60 6 0.53 7 0.53 3 0.60 3 0.13 1 0.92 5 0.40

10 8 14 14 12 15 14 15 12 5 12 11

All the values obtained are positive therefore it is a positive discrimination index. The item discrimination index ability is adequate. A higher index implies higher scoring students tended to select the response more often as seen in Table 8.

5 Conclusion The value of KR20 is 1.04; this implies that the test is reliable. Students who answered a gives question correctly is likely to answer other questions correctly. The other implication is that should another test be administered using similar items, the relative scores of students would show very little change.

7.

References

xx

1

A Guide to interpreting the item Analysis Report.(2004).Retrieved April 01 2008, from http://www.asu.edu/uts/InterplAS.pdf

2

Image: Standard deviation diagram.svg [Image] (n.d.).Retrieved April 01 2008, from http://en.wikipedia.org/wiki/Image:Standard deviation_diagram.svg#file.

3

Kubiszyn & Borich, G, (2007).Education testing and Measurement. Classroom Application and Practice (8th Ed).John Wiley & sons, inc.United States of America.

xxi

8 8.1

Appendices Appendix A

#Questi

#Corre

#Incorrec

Prop

Prop

ons

ct

t

Correct

Incorrec

0.84 0.88 0.68 0.48 0.68 0.44 0.52 0.52 0.33 0.92 0.76 0.60 0.84 0.80 0.92 0.63 0.33 0.52 0.64 0.68

t 0.16 0.12 0.32 0.52 0.16 0.32 0.56 0.48 0.48 0.67 0.08 0.24 0.4 0.16 0.2 0.08 0.38 0.67 0.48 0.36

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20

21 22 17 12 21 17 11 12 13 8 23 19 15 21 20 22 15 8 13 16

4 3 8 13 4 8 14 11 12 16 2 6 10 4 5 2 9 16 12 9 Total

xxii

pq

0.13 0.11 0.22 0.25 0.13 0.22 0.25 0.25 0.25 0.22 0.07 0.18 0.24 0.13 0.16 0.08 0.23 0.22 0.25 0.23 3.83

Key St No

C

B

D

D

B

C

D

A

C

B

A

C

B

D

A

A

C

D

B

C

Q1

Q2

Q3

Q4

Q5

Q6

Q7

Q8

Q9

Q10

Q11

Q12

Q13

Q14

Q15

Q16

Q17

Q18

Q19

Q20

1 2 3 4 5 6 7 8 9 10 11 12 13

C C C C C C B C C C C C C

B B B B B A B B B B B B B

B D D D D D A D D B D D D

A D D B C D B B A A D D A

C B B B B C B B B B B B B

D D C C C C C C C C C C C

A A D B B A B B D D D D D

A A A A D B D D C A D A

D C C C C C D B B D C D C

D B B B D D D C D C B A B

A A A A A A A A A A A A A

D C C C C C C C C B C C C

A B B A B A B B B A B A B

A D D D D D D D D D D D D

A A A C A A C A A D A A A

A A A A A A A A A A A A A

C C C C A A A C C C C C A

B D B B B B D A B D D B B

D B D C B D D B D B B B B

B C C C C C C A A C C D C

14 15 16

C C C

B B B

D D D

A D D

B B B

C B C

D A D

A A A

C B C

B D B

A A A

C C C

B D B

D A D

A A A

A C A

A B C

B D

B D B

C D C

17 18 19 20 21 22 23

B C D C C B C

B B C B A B B

C B A D D A D

C D D D D B B

B B B B C B B

A A A C C C C

D D B D A B B

D D A A D B D

C D D C C D B

D C A D D C

A A C C A A A

D C D D C C C

B A A B A B B

D D A D D D D

A A D A A C A

A B A A A A

C B C A A C

C B B D B D A

A B A B D D B

D C B C C C A

24 25

C C

B B

B D

A D

C B

D D

A A

A

D C

D B

A A

D C

A B

A D

A A

A A

C C

B D

D B

B C

8.3

Appendix C Key St No 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

C

B

D

D

B

C

D

A

C

B

A

C

B

D

A

A

C

D

B

C

Q1 1 1 1 0 1 1 0 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1

Q2 1 1 1 1 0 1 0 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1

Q3 1 0 1 1 1 0 1 1 0 0 0 1 1 1 1 0 1 1 0 1 1 1 1 1 0

Q4 1 0 0 0 1 1 1 1 0 1 1 0 0 1 1 0 0 0 0 1 0 0 1 1 0

Q5 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 0

Q6 0 0 1 1 1 1 0 0 0 1 0 1 1 1 1 1 1 1 1 1 1 1 1 0 0

Q7 0 0 0 0 0 1 0 1 1 1 1 1 0 1 1 1 1 0 0 0 0 0 1 0 0

Q8 1

Q9 1 0 0 0 0 1 1 0 1 1 0 1 1 0 1 0 0 0 0 1 1 1 1 1 0

Q10 1 0 0 0 0 0 0 0

Q11 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Q12 1 0 1 1 1 0 0 1 0 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 0

Q13 1 0 1 1 0 1 0 0 1 1 0 1 1 0 1 0 1 1 1 0 1 0 1 1 0

Q14 1 0 1 1 1 1 0 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 0

Q15 1 1 1 0 1 1 0 1 1 1 1 1 1 1 1 0 1 1 0 1 1 0 1 1 1

Q16 1 1 1 1 1 1 0 1

Q17 1 1 1 0 0 1 0 1

Q19

1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 0 0 0 1 1 1 1 1 0 0 0 1 1 1 1

Q18 1 0 0 1 0 1 0 0 0 1 0

Q20 1 0 1 0 1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 1 1 1 1 1 0

0 0 0 1 0 1 1 1 0 1 1 1 0 0 0 0 0 1 1 1 1 1 1

1 0 1 1 0 1 0 0 0 0 0 0 1 1 1 0

0 0 1 1 0 0 1 0 0 0 0 1 0

0 1 0 0 1 0 1 0 1 0 1 1 1 1 1 0 1 0 0 1 0 0 1 0

Total 18 6 13 10 11 17 4 13 8 20 9 17 17 14 20 12 13 13 10 11 15 14 18 18 6

% 100 100 90 90 90 85 85 85 75 70 70 65 65 65 65 60 55 55 50 50 45 40 30 30 20