Item Analysis Report

ITEM ANALYSIS REPORT

COMPILED BY

:

PHALADI ML

STUDENT NUMBER

:

26509360

INSTITUTION

:

UNIVERSITY OF PRETORIA

COURSE

:

B Ed [Hons] COMPUTER INTEGRATED EDUCATION

TABLE OF CONTENT

INDEX 1. Introduction 2. Purpose of the report 3. Test Analysis 3.1 Descriptive Statistics 3.2 Frequency Graphs 3.3 Test Reliability 4. Item Analysis 4.1 Difficulty Index 4.2 Discrimination Index 5. Conclusion REFERENCES APPENDICES

LIST OF TABLES i. Simple Frequency Table ii. Group Frequency Table iii. Frequencies Specifications iv. Frequencies Variation v. Key Points vi. Results Analysis vii. Difficulty Index viii. Discrimination Index: Interpretation ix. Discrimination Index

PAGE NUMBER 5 5 5 5 8 10 12 12 13 14 15 16

PAGE NUMBER 6 6 7 7 10 11 12 12 13

LIST OF FIGURES a) b) c) d)

Normal Distribution Graph Data analysis: Frequency Histogram Graph Data analysis: Frequency Polygon Graph Data analysis: Cumulative Frequency Graph

PAGE NUMBER 8 8 9 9

1. INTRODUCTION Competence-based approaches are frequently adopted as the key paradigm in both formal or non-formal education and training. To support the provision of competence-based learning services, it is necessary to be able to maintain a record of an individual's competences in a persistent and standard way. In this paper, we investigate potential issues related with the definition of an item analysis for competencies description. Many approaches to standard-setting use item calibration and student score estimation results to structure panellists' tasks. However, this requires collecting standard-setting judgments after the item analysis results are available. The Scoring Guide Alignment approach collects standardsetting judgments during the scoring sessions from teachers participating in the scoring and only later combines these judgments with item analysis results to set the scores. The computational details of this approach and its application in a largescale testing program are described as the report becomes detailed. These reports present the results of the student’s test conducted in a school. The item analysis contains the statistical information for each individual question within the test and a summary for the test as a whole. There were twenty questions in the test, which was administered to twenty five students given the same time. The questions on the test were multiple choices. It describes the statistical analyses which allow measurement of the effectiveness of test items. The test creates an evaluation measures including understanding of the factors which govern effectiveness, means of measuring the level of difficultness which can enable the teacher to create more effective test questions and also regulate and standardise existing test.

2. PURPOSE OF THE REPORT To disseminate the information on student’s data and to give an analysis report based on the learners performance.

3. TEST ANALYSIS

Test analysis is regarded as the process of breaking a concept down into more simple parts, so that its logical structure is displayed. 3.1

DESCRIPTIVE STATISTICS

As depicted in the following table, student’s statistical information on the item analysis report divided into eight columns: Students, Data (Student’s Scores), Array, Interval (Lower and Upper limit), Middle Value, Frequency and Cumulative frequency. Illustrated below is a data matrix comprised of 20 questions and 25 students (labelled 1 through to 25). Items are represented in the matrix columns from left to right and students represented in rows. A value of “1” in the grid signifies that the students got the item correct; “0” indicates the student got the question wrong and “blank” indicates the student did not answer the questions. Looking at student “1” and “24” did not answer question number “8”, thus appear “blank” and getting six questions correct and the rest wrong. Refer to Appendix 1.

i)

Table 1: Simple Frequency table

Student

Data

Array

11 16 2 3 25 14 13 20 5 4 12 8 9 18 23 10 6 21 7 22 17 15 1 24 19

100.00 100.00 90.00 90.00 90.00 89.47 85.00 85.00 75.00 70.00 70.00 65.00 65.00 65.00 65.00 60.00 55.00 55.00 50.00 50.00 47.06 45.00 31.58 31.58 15.00

15.00 31.58 45.00 47.06 50.00 55.00 60.00 65.00 70.00 75.00 85.00 89.47 90.00 100.00

Interval Lower Upper Limit Limit 15 24 25 34 35 44 45 54 55 64 65 74 75 84 85 94 95 104

Middle Value 19.5 29.5 39.5 49.5 59.5 69.5 79.5 89.5 99.5

Frequency 1 2 0 4 3 6 1 6 2

Cumulative Frequency 1 3 3 7 10 16 17 23 25

As depicted in the table 1, student’s statistical information on the item analysis report divided into eight columns: Students, Data (Student’s Scores), Array, Interval (Lower and Upper limit), Middle Value, Frequency and Cumulative frequency. The statistical summary of student’s percentage scores of the results of the written test administered to a group of students. This summary is used to evaluate the effectiveness of the test. ii)

Table 2: Group frequency table

H Value L Value Range No. of Scores No. of Int. Size of Interval

100.00 15.00 85 25 10 9

The item analysis splits the students into two groups based on their scores on the test they written. The student’s scores on the test were arrayed in descending order,

where the student with The student’s scores are arranged in descending order with the highest score of 100% to all questions answered correct and the lowest score of 15% with few questions answered correct. iii) Table 3: Frequencies specifications Student

Data

11 16 2 3 25 14 13 20 5 4 12 8 9 18 23 10 6 21 7 22 17 15 1 24 19

100.00 100.00 90.00 90.00 90.00 89.47 85.00 85.00 75.00 70.00 70.00 65.00 65.00 65.00 65.00 60.00 55.00 55.00 50.00 50.00 47.06 45.00 31.58 31.58 15.00

X-MEAN 34.21 34.21 24.21 24.21 24.21 23.69 19.21 19.21 9.21 4.21 4.21 -0.79 -0.79 -0.79 -0.79 -5.79 -10.79 -10.79 -15.79 -15.79 -18.73 -20.79 -34.21 -34.21 -50.79

(X-MEAN)2 1170.49 1170.49 586.24 586.24 586.24 561.03 369.12 369.12 84.87 17.74 17.74 0.62 0.62 0.62 0.62 33.50 116.37 116.37 249.25 249.25 350.77 432.12 1170.23 1170.23 2579.38

TOTAL

11989.27

The above table illustrate the student’s scores with the calculation of the mean with progression towards getting the standard deviation and calculation using KuderRichardson-20 [Kr20] formula of reliability for the test, which is actually estimated from the consistency of all questions in the test. iv) Table 4: Frequency Variation Mean Mode Median STDEV # Upper # Lower STDEV2 K K–1 Total PQ KR20

65.79 65.00 65.00 21.90 15 10 479.57 20 19 3.83 1.044224204

From the above information in table 4 it is a summary of calculations on mean, mode, median, which helps in calculating the standard deviation that will be used to calculate the KR20. The information on table 4, reflect that mean is equal to mode and equal to median, then the skewness of the graph will result in normal.

3.2 FREQUENCY GRAPH a) Figure 1: Data analysis: Normal Distribution Graph Normal Distribution Graph 60 49.9

40 34.13

30

34.13

20 13.59

10

13.59 0.13

86 .9 10 8. 8 13 0. 7 15 2. 6

2.14 65

2.14 21 .2

-0 .7

0.13

-2 2. 6

0

43 .1

Frequency

50

Std Deviavtion

The method that was used to determine the test reliability was designed by Kuder and Richardson.

b) Figure 2: Data Analysis: Frequency Histogram Graph

Frequency

Frequency Histogram 7 6 5 4 3 2 1 0

Students Scores

24

34

44

54

64 Intervals

74

84

94

104

Figure 1 show the distributed output data on students’ scores, bases on the test written being projected in a frequency histogram graph.

b) Figure 2: Data Analysis: Frequency Polygon Graph. Frequency Polygon 7 Frequency

6 5 4

Student's Scores

3 2 1 0 19.5 29.5 39.5 49.5 59.5 69.5 79.5 89.5 99.5 Middle Point

Figure 2 illustrates the output data on students’ scores, bases on the test written being projected in a frequency polygon graph.

c) Figure 3: Data Analysis: Cumulative Frequency Graph. Cummulative Frequency 30 Frequency

25 20 15

Students Scores

10 5 0 24

34

44

54

64

74

84

94

104

Intervals

Figure 3 illustrates the output data on students’ scores, bases on the test written being projected in a cumulative frequency graph.

3.3 TEST RELIABILITY Table 6: Key Points Table 6 below illustrate the key information on what the key symbols mean.

Symbol Difficulty index (p) Discrimination index (D) UPPER (#U) LOWER (#L)

Explanation Proportion of students who answered the questions correctly. Measure of the extent to which a test questions differentiate between students who do well on the overall test and those who do not do well on the overall test. 15 Students who scored between 65 and 100% on their test. 10 Students who scored between 15 and 60% on their test.

The table 7 below illustrate the result output based on the questions which the students answered based on the test given to them in analysing the correctness, validity and the consistency level of difficulty pertaining to the questions answered by the twenty five students.

Table 7: Results Analysis Questions #Correct #Incorrect Answered Proportion Reliability Reason 1 21 4 25 0.84 Unacceptable Too Easy 2 22 3 25 0.88 Unacceptable Too Easy 3 17 8 25 0.68 Acceptable Well Structured 4 12 13 25 0.48 Acceptable Well Structured 5 21 4 25 0.84 Unacceptable Too Easy 6 17 8 25 0.68 Acceptable Well Structured 7 11 14 25 0.44 Acceptable Well Structured 8 12 11 23 0.52 Acceptable Well Structured 9 13 12 25 0.52 Acceptable Well Structured 10 8 16 24 0.33 Acceptable Well Structured 11 23 2 25 0.92 Unacceptable Too Easy 12 19 6 25 0.76 Unacceptable Too Easy 13 15 10 25 0.60 Acceptable Well Structured 14 21 4 25 0.84 Unacceptable Too Easy 15 20 5 25 0.80 Unacceptable Too Easy 16 22 2 24 0.92 Unacceptable Too Easy 17 15 9 24 0.63 Acceptable Well Structured 18 8 16 24 0.33 Acceptable Well Structured 19 13 12 25 0.52 Acceptable Well Structured 20 16 9 25 0.64 Acceptable Well Structured

#U 15 15 14 8 15 12 9 10 10 8 14 14 12 15 14 15 12 5 12 11

#L 6 7 3 4 6 5 2 2 3 0 9 5 3 6 6 7 3 3 1 5

D 0.60 0.53 0.73 0.27 0.60 0.47 0.47 0.53 0.47 0.53 0.33 0.60 0.60 0.60 0.53 0.53 0.60 0.13 0.73 0.40

Q 0.16 0.12 0.32 0.52 0.16 0.32 0.56 0.48 0.48 0.67 0.08 0.24 0.40 0.16 0.20 0.08 0.38 0.67 0.48 0.36

PQ 0.13 0.11 0.22 0.25 0.13 0.22 0.25 0.25 0.25 0.22 0.07 0.18 0.24 0.13 0.6 0.08 0.23 0.22 0.25 0.23

4. ITEM ANALYSIS The Item Analysis output consists of four parts: A summary of test statistics, a test frequency distribution, an item quintile table, and item statistics. It is a measure of the percentage of students answering a question correctly, which is a precise measurement of a hypothesized process which poses challenges. The higher the difficulty, the easier the question is.

4.1.Difficulty Index Table 9: Analysis: Proportion Questions #Correct Answered Proportion 1 21 25 0.84 2 22 25 0.88 3 17 25 0.68 4 12 25 0.48 5 21 25 0.84 6 17 25 0.68 7 11 25 0.44 8 12 23 0.52 9 13 25 0.52 10 8 24 0.33 11 23 25 0.92 12 19 25 0.76 13 15 25 0.60 14 21 25 0.84 15 20 25 0.80 16 22 24 0.92 17 15 24 0.63 18 8 24 0.33 19 13 25 0.52 20 16 25 0.64

Table 9: Result: Interpretation. The difficulty index can be used to alert the educator about potential problems within the questions. The assessment measurement is reliable if it reflects mostly correct score based on questions answered correctly by the majority of the students should be examined for grammatical or other irrelevant clues.

Questions 1 2 3 4 5

Difficulty Index (p) 0.84 0.88 0.68 0.48 0.84

Reliability Unacceptable Unacceptable Acceptable Acceptable Unacceptable

Reason Too easy Too easy Well Structured Well Structured Too easy

6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

0.68 0.44 0.52 0.52 0.33 0.92 0.76 0.60 0.84 0.80 0.92 0.63 0.33 0.52 0.64

Acceptable Acceptable Acceptable Acceptable Acceptable Unacceptable Unacceptable Acceptable Unacceptable Unacceptable Unacceptable Acceptable Acceptable Acceptable Acceptable

Well Structured Well Structured Well Structured Well Structured Well Structured Too easy Too easy Well Structured Too easy Too easy Too easy Well Structured Well Structured Well Structured Well Structured

The index is used to construct the measurement scale, to improve and evaluate the reliability of the questions used. Values for the difficulty index range from 33% (very difficult) to 92% (very easy). The reliability calculations developed primarily for use with norm-referenced tests, and the same is true for item analysis. In spite of that, item analysis can be used to advantage in evaluating the quality of tests in criterion-referenced tests as well, particularly the typical class test. Therefore, most of the questions show that questions were well structured, except where few questions show that students scored low. However, if you follow Bloom’s hierarchy of objectives, it is impossible not to have items of varying difficulty, where the student has to gain knowledge before coming up with the solution to the question asked. According to Seema Varma, the p-value provides the proportion of students that got the item correct. That mean, the difficulty of the questions with reference to student number 10 and 18 both got the lowest p-value (0.33). The highest p-value of 0.92 by student 11 and 16 is associated with the ability of gained knowledge. The large concentration is within the centre where most students scores lies around 0.50.

4.2.Discrimination Index Table 8: Discrimination Indices Questions 1 2 3 4 5 6 7 8 9

#L

#U

15 15 14 8 15 12 9 10 10

6 7 3 4 6 5 2 2 3

Discrimination Index (D) 0.60 0.53 0.73 0.27 0.60 0.47 0.47 0.53 0.47

Discrimination Decision Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively.

10 11 12 13 14 15 16 17 18 19 20

8 14 14 12 15 14 15 12 5 12 11

0 9 5 3 6 6 7 3 3 1 5

0.53 0.33 0.60 0.60 0.60 0.53 0.53 0.60 0.13 0.73 0.40

Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively. Distracters discriminate positively.

The test provides important feedback to both students and educators. Therefore, discrimination index provides correlation that describes the relationship between responses to a question and the total score on the test. For most, it provides an opportunity to students to demonstrate what they know and can do. The purpose of this test is to differentiate among students in terms of their ability and understanding. As a correlation, question discrimination is between 0.25 and 0.75. Easy questions cannot distinguish among students in terms of their performance. Questions having low discrimination values need to be reviewed carefully for confusing language or incorrect key used. If the language is not confusing, then the questions need to be critically reviewed. There are three components of item analysis: item difficulty index, item discrimination index and distracters effectiveness. The inter-correlation among questions has a greater relative number of positive relationships. Therefore question discrimination indices and the test‘s reliability coefficient are related in this regard. High reliability of questions tends to move in the same direction. As compared to low reliability of questions, tends to be unrelated to each other in terms of who answered the questions correctly. The format of this report is similar to the format of the Category Analysis Report. This report, however, presents the results on the original scale that was used for each multiple choice question (i.e. A, B, C and D). The majority of test questions, when students agreed with a question, by providing a correct response.

5. CONCLUSION In finding out how well students meet the curricular objectives. The discovery is that students who got the questions incorrect also scored high o the test overall while students who got questions correct scored low on the overall test. Therefore the test needs a further review. This report will help in the future improvement on the teacher’s side in generating the test questions to test his/her students. The educator can improve the quality of the test developed. Generally the test is used as a diagnostic tool that measure knowledge and understanding of the student’s gained knowledge. The diagnostics used for classroom instruction, curriculum development and educator development which provide reliability and meaningful. The test analyses were conducted to provide information about the properties of existing measures in what the teacher tested the students’ ability. In analysing the

test, whether questions were putatively measuring the same thing were sufficiently dimensional for test analyses and estimating item parameters such as difficulty and discrimination. These analyses were intended to provide preliminary information about item characteristics and performance that could help guide decisions for the improvement in the test. This analysis is a process which examines the student’s responses to individual test items (questions) in order to assess the quality of those items and of the whole test. It is a valuable method which may improve items which will be used to eliminate ambiguous and misleading items in this test. The item analysis is a good assessment method for increasing the teacher’s skills in test construction and to identify the specific areas of the content which need greater emphasis or clarity or need for improvement. The test report is extremely helpful for identifying concrete areas where improvements can be made. While the report provides a view of the big picture, the test report brings the larger issues into sharper focus. The test report provides a structured view of item results in relation to the scale to which each item belongs. The responses are of fairly similar with respect to the difficulty of the questions. This analysis allows the educator to analyse the performance of the test. Students’ results have strong implications for students’ motivation, morale, and performance rate.

REFERENCE 1. Payne DA. (1994). The Assessment of Learning — Cognitive and Affective. Lexington, Ky: D.C. Heath and Company 2. Seema, Varma. (n.d) Preliminary item statistics using Point=Biserial Correlation and P-values. Morgan Hill, CA. 3. Kubiszyn, T.,Borich, T. (2007), Educational Testing and Measurements: Classroom Application and Practice. (8th ed.), NC: John Wiley & Sons, Inc 4. Philosophy Dictionary definition of analysis The Oxford Dictionary of Philosophy. Copyright © 1994, 1996, 2005 by Oxford University Press.

APPENDIX 1 1. Students answers from the questions Key St No

C Q1

B Q2

D Q3

D Q4

B Q5

C Q6

D Q7

A Q8

C Q9

B Q10

A Q11

C Q12

B Q13

D Q14

A Q15

A Q16

C Q17

D Q18

B Q19

C Q20

1 2 3 4 5 6 7 8 9 10 11 12 13

C C C C C C B C C C C C C

B B B B B A B B B B B B B

B D D D D D A D D B D D D

A D D B C D B B A A D D A

C B B B B C B B B B B B B

D D C C C C C C C C C C C

A A D B B A B B D D D D D

A A A A D B D D C A D A

D C C C C C D B B D C D C

D B B B D D D C D C B A B

A A A A A A A A A A A A A

D C C C C C C C C B C C C

A B B A B A B B B A B A B

A D D D D D D D D D D D D

A A A C A A C A A D A A A

A A A A A A A A A A A A A

C C C C A A A C C C C C A

B D B B B B D A B D D B B

D B D C B D D B D B B B B

B C C C C C C A A C C D C

14 15 16

C C C

B B B

D D D

A D D

B B B

C B C

D A D

A A A

C B C

B D B

A A A

C C C

B D B

D A D

A A A

A C A

A B C

B D

B D B

C D C

17 18 19 20 21 22 23

B C D C C B C

B B C B A B B

C B A D D A D

C D D D D B B

B B B B C B B

A A A C C C C

D D B D A B B

D D A A D B D

C D D C C D B

D C A D D C

A A C C A A A

D C D D C C C

B A A B A B B

D D A D D D D

A A D A A C A

A B A A A A

C B C A A C

C B B D B D A

A B A B D D B

D C B C C C A

24 25

C C

B B

B D

A D

C B

D D

A A

A

D C

D B

A A

D C

A B

A D

A A

A A

C C

B D

D B

B C

APPENDIX 2

2. Student’s Data Matrix Key Student No 11 16 2 3 25 14 13 20 5 4 12 8 9 18 23 10 6 21 7 22 17 15 1 24 19

C

B

D

D

B

C

D

A

C

B

A

C

B

D

A

A

C

D

B

C

Q1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 1 1 1 0

Q2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 0

Q3 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 0 1 1 0 0 0 1 0 0 0

Q4 1 1 1 1 1 0 0 1 0 0 1 0 0 1 0 0 1 1 0 0 0 1 0 0 1

Q5 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 1 0 0 1

Q6 1 1 0 1 0 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 0 0 0 0 0

Q7 1 1 0 1 0 1 1 1 0 0 1 0 1 1 0 1 0 0 0 0 1 0 0 0 0

Q8 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1

Q9 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 1 1 0 0 1 0 0 0 0

Q10 1 1 1 1 1 1 1 0 0 1 0 0 0 0 0 0 0 0 0 0

Q11 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0

Q12 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0 1 1 1 1 0 1 0 0 0

Q13 1 1 1 1 1 1 1 1 1 0 0 1 1 0 1 0 0 0 1 1 1 0 0 0 0

Q14 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0

Q15 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 0 1 1 0 0 1 1 1 1 0

Q16 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Q17 1 1 1 1 1 0 0 1 0 1 1 1 1 1 1 1 0 0 0 0

Q18 1 1 1 0 1

0 1 1 0

0 1 1 0

Q19 1 1 1 0 1 1 1 1 1 0 1 1 0 1 1 1 0 0 0 0 0 0 0 0 0

Q20 1 1 1 1 1 1 1 1 1 1 0 0 0 1 0 1 1 1 1 1 0 0 0 0 0

1

0 0 0 0

0 1 0 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0

#Correct 20 20 18 18 18 17 17 17 15 14 14 13 13 13 13 12 11 11 10 10 8 9 6 6 3

#Answered 20 20 20 20 20 19 20 20 20 20 20 20 20 20 20 20 20 20 20 20 17 20 19 19 20