Asl [compatibility Mode]

Day One Session Four Topic: Performance-Based Assessment and Student Portfolios as Alternative Assessment Techniques I. Learning Targets: At the end of this session the participants should be able to: a. define the term performance-based assessment; b. identify uses of performance-based assessment; c. construct performance tests for one’s students; d. define the terms portfolio and portfolio assessment; e. build a system for portfolio assessment; f. appreciate performance-based and portfolio assessment.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

II. Session Notes A. Performance-based assessment  Definition: performance-tests : use direct measure of learning rather than indicators of these : teacher observes and evaluates student abilities to carry out complex activities that are used and valued outside the immediate confines of the classroom  examples: field of athletics : diving and gymnastics are performances that judges rate directly field of music : band contest where judges directly see and hear the competence of the violin or trombone players

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

 Uses of performance tests -- can assess processes and products or both  examples: -- process: teacher assesses the reading process of each student by 1. noting the percentage of words read accurately during oral reading 2. noting the number of sentences read by the learner that are meaningful within the context of the story 3. noting the percentage of story elements that the learner can talk about in his/her own words after reading.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

--product: the fourth grade learners assemble a portfolio of their writing products, both rough and final drafts. --process and product: Social studies learning processes and products are assessed by having learners engage in a variety of projects built around a question answered in a variety of ways involving individual and groups writing assignments, oral presentations, and exhibits

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

-- can be embedded in lessons  a performance activity and assessment that is embedded in a unit on electricity in a general science class -- can assess affective and social skills using rating scales 

self-direction

 ability to work with others  social awareness

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

 Developing Performance Tests

Step 1. Deciding what to test -- knowledge, intellectual skills, habits of mind Step 2. Designing the assessment context -- create a task, simulation or situation that will allow learners to demonstrate the knowledge, skills and attitudes that they have acquired

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Step 3. Specifying the scoring rubrics -- rubrics: a carefully constructed scoring system. : may be used for products (poems, essays, drawings, etc.); complex cognitive processes; observable performance; and habits of mind and social skills. Step 4. Specifying Testing Constraints. -- time -- equipment -- reference materials -- prior knowledge of the tasks -- other people -- scoring criteria

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

B. Portfolio Assessment  Definition: tells a story of the learner’s growth in proficiency, longterm achievement and significant accomplishments in a given academic area.  What is a portfolio? It is a planned collection of learner achievement that documents what a student has accomplished and the steps taken to get there. The collection represents a collaborative effort among teacher and learner, to decide on portfolio purpose, content and evaluation criteria.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Developing a system for portfolio assessment Step 1. Deciding on the purposes for a portfolio  monitoring student progress  communicating what has been learned to parents  assigning a course grade Step 2. Identifying cognitive skills and disposition. Step 3. Deciding who will plan the portfolio. Step 4. Deciding which products to put in the portfolio and how many sample of each product to include.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Step 5. Building the portfolio rubrics. Step 6. Developing a procedure to aggregate all portfolio ratings. Step 7. Determining the logistics.  timelines  how products are turned in and retrieved  where final products are kept  who has access to the portfolio  plan a final conference with individual learners, and parents

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Essay Portfolio Rating Form ___________ First Draft ___________ Second Draft ___________ Final Draft To be completed by student: 1. Date submitted: ________________________________________________________ 2. Briefly explain what this essay says about you. _______________________________ _____________________________________________________________________ 3. What do you like best about this piece of writing? _____________________________ ______________________________________________________________________ 4. What do you want to improve on the next draft? _______________________________ ______________________________________________________________________ 5. If this is your final draft, will you include this in your portfolio and why? ___________ ______________________________________________________________________ To be completed by teacher: 1. Quality Reflection Rating Description 5 States very clearly what he/she likes most and least about essay. Goes into much detail about how to improve the work. 4 States clearly what he/she likes and dislikes about the essay. Gives detail about how to improve the work. 3 States his/her likes and dislikes but could be clearer. Gives some detail about how the work will be improved. 2 Is vague about likes and dislikes. Gives few details about how essay will be improved. 1 No evidence of any reflection on the work. 2. Writing Conventions Rating Description 5 The use of writing conventions is very effective. No errors evident. These conventions are fluid and complex: punctuation, grammar usage, sentence structure, spelling. 4 The use of writing conventions is effective. Only minor errors evident. These conventions are nearly all effective: punctuation, grammar usage, sentence structure, spelling. 3 The use of writing conventions is somewhat effective. Errors don’t interfere with meaning. These conventions are somewhat effective: punctuation, grammar usage, sentence structure, spelling. 2 Errors in the use of writing conventions interfere with meaning. These conventions are limited and uneven: punctuation, grammar usage, sentence structure, spelling. 1 Major errors in the use of writing conventions obscure meaning. Lacks understanding of punctuation, grammar usage, sentence structure, spelling.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

3. Organization Rating 5 4 3 2 1

Description Clearly makes sense. Makes sense. Makes sense for the most part. Attempted but does not make sense. Does not make sense.

4. Planning (1st draft only) Rating Description 5 Follows up on all suggestions for revision. revisions are definite improvement. 4 Follows up on most suggestion for revision. Revisions on the previous draft. 3 Addresses some but not all suggested revisions. Revisions are a slight improvement over earlier draft. 2 Ignores most suggestions for revision. Revisions made do not improve the earlier draft. 1 Made only a minimal attempt to revise, if at all. Sum of ratings: _______________ Average of ratings: ________________ Comments: _______________________________________________________ _____________________________________________

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

(Source: Kubiszyn and Borich, pages 179-184)

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

(Source: Kubiszyn and Borich, pages 179-184)

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

(Source: Kubiszyn and Borich, pages 179-184)

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Day Two

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Day Two Session One Topic: Determining the Quality of a Test through Validity I. Learning Targets At the end of this session the participants should be able to: a. define the term validity b. describe the types of validity c. identify types of criterion-related validation d. identify types of evidence used in construct validation e. describe the methods used in the validity of the interpretations from the test scores f.

identify construct validation; and

g.

appreciate the validity of test.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

II. Session Notes

A. Definition and other features of validity  Validity refers to the appropriateness of the interpretation of the results of a test or evaluation instrument for a given group of individuals, and not to the instrument itself.  It is more correct to speak of the validity of the interpretation to be made of the results.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

 Validity is a matter of degree.  Best considered in terms of categories that specify degree, such as high validity, moderate validity and low validity.  Is always specific to some particular use interpretation.  Is a unitary concept based on various kinds of evidence.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

B. Approaches to Test Validation TABLE 3.1 Approaches to Test Validation

(Source: Gronlund and Linn, p.51)

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

C. Types of criterion-related validation (based on time difference only)

(Source: Gronlund and Linn, p.57)

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

D. Some types of evidence in construct validation Math Reasoning Test  Compare the test tasks to the task domain (content-related evidence).  Have test takers “Think Aloud” as they answer (to check on mental process).  Compare scores of math majors with scores of non-majors.  Correlate scores of other math reasoning tests.  Correlate scores with scores of reading tests (to rule out reading ability).  Correlate scores with math grades (criterion-related evidence).

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

E. Methods used in Construct validation •

Defining the domain of tasks to be measured.

 Analyzing the mental process required for the test items.  Comparing the scores of known groups.  Comparing scores before and after some particular treatment.  Correlating the scores with other measures.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

F. Factors that lower the validity of the interpretations from the test scores  Unclear direction  Reading vocabulary and sentence structure too difficult  Appropriate level of difficulty of the test items  Poorly constructed test items  Ambiguity

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

F. Factors that lower the validity of the interpretations from the test scores  Test items inappropriate for the outcomes being measured  Inadequate time limits  Test too short  Improper arrangement of items  Identical pattern of answers

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Exercise # 1 Read each of the following situations. Identify the type of reliability or validity each illustrates. For numbers, 1, 4, 5, 6, 8 and 9, interpret the correlation coefficient. Write your answer in your answer sheet (15 pts.) 1.

2.

3.

4.

A researcher constructed an achievement test in Math. He administered it to all first year students in a public high school. After a couple of weeks he administered it to the same students. He then correlated the scores on the two tests and obtained an r = 67. A graduate student made an anxiety scale for college students. She administered it to a random sample of 50 senior students. She also administered a standardized anxiety scale to the same students. If she applies the PPM formula to the scores in both instruments, what test characteristic does she establish for her anxiety scale? The end of the grading period was a month away. One of the general science teachers was assigned to construct the periodic test in the aforementioned subject. A couple of weeks prior to the administration of the test, it was shown to the teachers teaching the same subject in the same grade level. These teachers were requested to go over the items to determine if they fit the instructional objectives and the content area covered for that particular grading period. All of them agreed that the test items fit the conditions given. A researcher constructed two forms (Form A and Form B) of an achievement test in Filipino. He administered Form A in the morning of one school day and then administered Form B the following day. The correlation coefficient was 0.33.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

5. A teacher in a business school made a test to measure the clerical ability of graduating students from the two-year secretarial program. After three years, he traced the whereabouts of these same students and obtained their performance ratings as secretaries. The correlation coefficient was 0.83. 6. A researcher administered an achievement test in Science to second year high school students in March. Due to time constraints, he could not administer it a second time. However, he was still able to use a procedure that enabled him to obtain an r =0.75. 7. A Metrobank Outstanding teacher awardee prepared an honesty scale. He administered it to some government employees. After conducting an experiment, he found out that the honest employees obtained a high score in his scale while the dishonest ones did not. What did he determine, reliability or validity? Indicate which type. 8. So you prepared two forms of an achievement test in Chemistry. You administered Form A to fourth year BSE Chemistry students. After three weeks, you administered Form B to the same students. You correlated the scores on the two forms of the test and obtained an r = 0.25. 9. A teacher in an ICT school constructed a test to measure the fourth year engineering student’s ability in information technology. He administered it for the first time on October 15, 2007. He administered it for a second time to the same students in November 20, 2007. The correlation coefficient was r = -0.20.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Day Two Session Two Topic: Determining the Quality of a Test through Reliability I. Learning Targets At the end of this session the participants should be able to: a.

define the term reliability;

b.

define basic terminology: coefficient, validity coefficient and reliability coefficient;

c.

explain the procedure for determining reliability by correlation methods;

d.

discuss the methods of estimating reliability;

e.

explain reliability demands and nature of the decision; and

f.

appreciate the reliability of tests.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

II. Session Notes A. Definition and other features of reliability 

Reliability refers to the results obtained with an assessment instrument and not to the instrument itself.



An estimate of reliability always refers to a particular type of consistency.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

 Test scores are reliable (or generalizable) over  different periods of time  different scores of questions  different raters  Test scores are not reliable in general  The appropriate type of consistency in a particular case is dictated by the use to be made of the results.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

B. Determining reliability by correlation methods Table 3.1 Methods of estimating Reliability

(Source: Gronlund and Linn, p. 80) Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Test-Retest September 25

October 15

Form A

Form A

1. hosidos 2. hodis osid 3. kodsosicos 4. kdosodiso

1. hosidos 2. hodis osid 3. kodsosicos 4. kdosodiso

Score = 82

Score = 86

Figure 4.1 Test-retest method (using same test form)

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Figure 4.2 Equivalent-forms method (without and with a time interval)

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Figure 4.3 Odd-even scoring to use of split-half method (odd and even scores are then correlated Sum number Of odd items Correct

Sum number Of odd items Correct

Item 1 3 5 . . . Odd Score = 40

Item 2 4 6 . . . Even Score = 42

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

C. Methods of estimating reliability When examining the reliability coefficient of standardized tests, it is important to consider the methods that were used to obtain the reliability estimates. In general, the size of the reliability coefficient is related to the method of estimating reliability.

(Source: Gronlund and Linn, p. 97) Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

D. Some basic terminology

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

E. Reliability demands and nature of the decision  High reliability is demanded when the  Decision is important.  Decision is final.  Decision concerns individuals. Example:  Select or reject college applicant.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

 Low reliability is tolerable when the  Decision is minor importance  Decisions concerns groups  Decision making is in early stage  Decision is reversible Example: Whether to review a classroom lesson.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Day Two Session Three Workshop II. Exercises on Validity and Reliability I. Learning Targets At the end of this session the participants should be able to: a. answer exercises on validity and reliability; b. calculate Pearson r using both the deviation score formula and the raw score formula; c. interpret the Pearson r; and d. appreciate correlation coefficient. II. Workshop II 1. Discussion on exercise on reliability and validity. 2. Discussion on the Pearson product-moment correlation and the two formulae. 3. Calculation of Pearson r. 4. Interpretation of Pearson r.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

What is the Pearson product-moment correlation?  The Pearson product-moment correlation is a parametric statistical technique that indicates the relationship between two or more variables.  It gives an index of correlation, r, appropriate when the data represent either interval or ratio scales.  It takes into account each pair of scores and produces a coefficient between 0.00 and either +1.00. Why do we apply the Pearson product-moment correlation?  It is applied to determine the magnitude and direction of the relationship between two or more variables.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Exercise # 2 For the following pairs of variables, would you expect the correlation coefficient to be positive, negative or zero? a. A person’s total family income and his neighbor’s total family income. b. Number of registered voters in a district and the number voting on election day. c. Number of cigarettes smoked and kung function. d. Calories consumed and weight. e. Altitude and mean temperature. f. Gross national product and infant mortality rate. g. Number of cars in a household and total family income. h. Number of hours you study and your grade on an exam. i. Size of the head and intelligence test scores.

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Exercise # 3 Pearson’s product-moment correlation coefficient Calculation of the correlation coefficient (r) The raw-score formula N ∑XY - ∑X ∑Y

rxy = __________________________

√[N ∑X2 – (∑X)2][N∑Y2 – (∑Y)2]

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Scores of Ten Students on Two Tests Along with the Calculations and Totals Necessary for Computations of the Raw Score Product-Moment Correlation Coefficient Formula

Student

TEST X

TEST Y

X2

Y2

XY

a

8

3

64

9

24

b

2

1

4

1

2

c

8

6

64

36

48

d

5

3

25

9

15

e

15

14

225

196

210

f

11

12

121

144

132

g

13

9

169

81

117

h

6

4

36

16

24

i

4

4

16

16

16

j

6

5

36

25

30

∑

78

61

760

533

618

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

REFERENCES: 1. Gronlund, Norman E. and Linn, Robert L. (1990). Measurement and Evaluation in Teaching (6th ed.) New York: MacMillan Publsihing House 2. Kubiszyn, Tom and Borich, Gary (2004). Educational Testing and Measurement – Classroom Application and Practice. (7th ed.) Singapore: John Wiley and Sons, Inc. 3. McMillan, James H. (1997). Classroom Assessment: Principles and Practice for Effective Instruction. Boston: Allyn and Bacon 4. Ferguson, George A. and Takane, Yoshio (1989). Statistical Analysis in Psychology and Education. (6th ed.). Singapore: McGraw-Hill Book Co. 5. Tanner, David E. (2001). Assessing Academic Achievement. Boston: Allyn and Bacon Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)

Thank you 

Create PDF files without this message by purchasing novaPDF printer (http://www.novapdf.com)