Lesson 5e Essay Questions Construction

EVALUATION Test Item Construction Essay Questions Sylvester Saimon Simin Keningau Teachers Training College

Characteristics of Essay Questions • Freedom of response – As with short-answer items, students produce their own answers. However, they are free to decide how to approach the problem, what factual information to use, how to organize the answer, and what degree of emphasis to give each aspect of the response. Thus it is useful for measuring the ability to organize, integrate, and express ideas.

Comparison of Selection-Type Items and Essay Selection-Type Items Learning outcomes measured

• Good for measuring outcomes at the knowledge, comprehension, and application levels • Inadequate for organizing and expressing ideas

The use of a large number of Sampling of items results in broad coverage, which makes representative content sampling of content feasible

Essay Questions • Inefficient for measuring knowledge outcomes • Best for ability to organize, integrate and express ideas. The use of a small number of items limits coverage, which makes representative sampling of content infeasible.

Preparation of good items is difficult and time consuming.

Preparation of good items is difficult but easier than selectiontype items.

Objective, simple, and highly reliable.

Subjective, difficult and less reliable.

Factors distorting scores

Reading ability and guessing.

Writing ability and bluffing.

Probable effect on learning

Encourages students to remember, interpret, and use the ideas of others.

Encourages students to organize, integrate, and express their own ideas.

Preparation of Items Scoring

Types of Essay Questions Restricted-Response Questions – Places strict limits on the answer; the subject matter are narrowly defined by the problem, the specific form of the answer is commonly indicated (eg: ‘list’, ‘define’, ‘give reasons’), use of introductory material or special directions • Eg: Describe the relative merits of selection-type tests items and essay questions for measuring learning outcomes at the comprehension level. Confine your answer to one page. • Eg: Mr. Roger, a Form 1 science teacher, wants to measure his students’ ‘ability to interpret scientific data’ with a paper-and-pencil test. (a) Describe the steps that Mr. Roger should follow. (b) Give reasons to justify each step.

– Advantages: • Prepared more easily, related more directly to specific learning outcomes, scored more easily

– Disadvantages • Little opportunities for students to demonstrate their abilities to organize, to integrate and develop new pattern of response

– Useful for comprehension, application and analysis levels of learning

Types of Essay Questions Extended-Response Questions – Almost unlimited freedom to determine the form and scope of their responses. – In some instances practical limits are imposed such as time or page limits, but restrictions on the material to be included in the answer and on the form of response are held to a minimum. • Eg: For a course that you are teaching or expect to teach, prepare a complete plan for assessing student acheivement. Be sure to include the procedures you would follow, the instruments you would use, and the reasons for your choices. (synthesis outcome) • Eg: (The student is given a complete acheivement test that includes errors or flaws in the directions, in the test items, and in the arrangement of the items.) Write a critical evaluation of this test using as evaluative criteria the rules and standards for test construction described in your textbook. Include a detailed analysis of the test’s strengths and weaknesses and an evaluation of its overall quality and probable effectiveness. (Evaluation outcome)

– Advantages: • Creative integration of ideas, overall evaluation of materials, and a broad approach to problem solving.

– Disadvantages • To evaluate the answers with sufficient reliability to provide a useful measure of learning

– Useful for synthesis and evaluation levels of learning

Guidelines for Writing Essay Questions 1. Use essay questions to measure complex learning outcomes only. – Essay test are suitable for situations that require students to give reasons, explain relationships, describe data, formulate conclusions. – Where supplying the answer is vital, a properly constructed restricted-response question is more appropriate – At the synthesis & evaluation level, such as the production of a complete work (eg. a plan of operation) or an overall evaluation of a work (eg. evaluation of a novel or an experiment) requires the use of extended-response questions.

Guidelines for Writing Essay Questions 2. Relate the questions as directly as possible to the learning outcomes being measured. – Each question should be specifically designed to measure one or more well-defined outcomes. Thus, as with objective items, a precise description of the performance to be measured will help determine the content and form of the item. – The extended-response item requires greater freedom of response and involves a number of learning outcomes. If the task is prescribed too rigidly in the question, the student’s freedom is apt to be infringed upon. One solution is to indicate the criteria used in evaluating the answer eg. ‘your answer will be evaluated in terms of its comprehensiveness, the relevance of its arguments, the appropriateness of its examples, and the skill with which it is organized’.

Guidelines for Writing Essay Questions 3. Formulate questions that present a clear task to be performed. Learning Outcomes

Sample Terms

Comparing

compare, classify, describe, distinguish between, explain, outline, summarize

Interpreting

convert, draw, estimates, illustrate, interpret, restate, summarize, translate

Inferring

derive, draw, estimate, extend, extrapolate, predict, propose, relate

Applying

arrange, compute, describe, demonstrate, illustrate, rearrange, relate summarize

Analyzing

break down, describe, diagram, differentiate, divide, list, outline, separate

Creating

compose, design, devise, draw, formulate, make up, present, propose

Synthesizing

arrange, combine, construct, design, rearrange, regroup, relate, write

Generalizing

construct, develop, explain, formulate, generate, make, propose, state

Evaluating

appraise, criticize, defend, describe, evaluate, explain, judge, write

Types of Complex Outcomes and Related Terms for Writing Essay Questions

Guidelines for Writing Essay Questions

4. Do not permit a choice of questions unless the learning outcome requires it.

Guidelines for Writing Essay Questions

5. Provide ample time for answering and suggest a time limit on each question.

Rules for Scoring Essay Questions 1. 2. 3. 4. 5. 6.

Evaluate answers to essay questions in terms of the learning outcomes being measured. Score restricted-response answers by the point method, using a model answer as a guide. Grade extended-response answers by the rating method, using defined criteria as a guide. Evaluate all of the students’ answers to one question before proceeding to the next question. Evaluate answers to essay questions without knowing the identity of the writer. Whenever possible, have two or more persons grade each answer.

Checklist for Evaluating Essay Questions 1. Is this type of item appropriate for measuring the intended learning outcome? 2. Does the item task match the learning task to be measured? 3. Is the question designed to measure complex learning outcomes? 4. Does the question make clear what is being measured and how the answer will be evaluated? 5. Has terminology been used that clarifies and limits the task (eg: ‘describe’, not ‘discuss’) 6. Are all students required to answer the same questions? 7. Has an ample time limit been indicated for each question? 8. Have adequate provisions been made for scoring answers (eg: model answers or criteria for evaluating)?

Common Types of Bluffing in Answering Essay Questions 1. Student repeat the question in statement form (slightly paraphrased) and tells how important the topic is (e.g., “The role of assessment I teaching is extremely important. It is hard to imagine effective instruction witout it, etc.”). 2. Student writes on a well-known topic and fits it to the question (e.g., a student who knows testing well but knows little about performance assessment and is asked to compare the two might describe testing in considerable detail and frequently state that performance assessment is much superior for evaluating the type of learning measured by the test). 3. Student liberally sprinkles the answer with basic concepts whether they are understood or not (e.g., asked to write about any assessment techniques the importance of ‘validity’ and ‘reliability’ is mentioned frequently). 4. Student includes the teacher’s basic beliefs wherever possible (e.g., “The intended learning outcomes must be stated in performance terms before this type of test is constructed or selected”). * Bluffing is most effective where plans have not been made for careful scoring of the answers.