Approach 7 Evaluasi.docx

Approach 7: Objectives-Based Studies The objectives-based study is the classic example of a quasi-evaluation approach that focuses on a narrow set of questions. Madaus and Stufflebeam (1988) provided a comprehensive look at this approach in an edited volume of the classical writings of Ralph W. Tyler.

Advance Organizers In this approach, some statement of a program’s objectives constitutes the advance organizer. The defined objectives provide the basis for determining what information should be collected. The specified information needs provide direction for identifying pertinent information sources and developing or selecting tools to measure program recipients’ performance in relation to each objective. When program objectives are defined in great detail, they include specification of cut scores above which a program is judged to have met its objectives.

Purposes The usual purposes of an objectives-based study are to specify and define clearly what a program is intended to accomplish and to determine the extent to which the program achieved its objectives. Program administrators use the results of objectives-based studies to report on the extent to which their program delivered the promised outcomes. Objectives-based program reports also allow clients and program recipients to reach their own evidence-based judgments of a program’s level of success. Program staffs may use judgments of which objectives were not achieved as diagnostic feedback for use in rethinking and improving a program’s design and execution. Funders may use results of objectives-based studies to sustain funding, increase or decrease funding, or terminate a program.

Sources of Questions This approach’s general question is, Did a program achieve its objectives? The objectives may be defined by a program’s staff, defined and mandated by the program’s funder, or formulated and explicated by the evaluator in consultation with the program’s staff. The validity of the objectives resides mainly in their acceptability to the program’s staff, beneficiaries, and funder.

Questions The specific question addressed by objectives-based studies is, To what extent did the program achieve each of its stated objectives? The program’s objectives are expected to be defined in very clear terms. In some studies, each program objective is defined in such detail that it specifies what is to be achieved, the conditions under which the achievement is to be produced, how the achievement is to be measured, and the level on the pertinent measurement scale that is to be counted as successful.

Methods Typically, but not always, an objectives-based evaluation is an internal study done by a developer or other program leader or, less often, by a program service provider. The methods used in objectives-based studies essentially involve specifying operational objectives and collecting and analyzing pertinent information to determine how well each objective was achieved. R. W. Tyler (for example, 1932, 1942, 1950, 1966, 1967) stressed that a wide range of objective and performance assessment procedures usually should be employed. This sets his approach apart from studies that focus on a particular method, such as an experimental design or a single

standardized test. Criterion-referenced tests and students’ work samples are especially relevant to the objectives-based approach.

Pioneers Tyler is generally acknowledged to be the pioneer of the objectives-based type of study, although Percy Bridgman and Edward Thorndike should also be credited. Several people have developed variations of Tyler’s model. They include Bloom, Englehart, Furst, Hill, and Krathwohl (1956); Hammond (1972); Metfessel and Michael (1967); Popham (1969); Provus (1971); and Steinmetz (1983). Although Tyler developed the objectives-based approach for use in evaluating educational programs, this approach’s influence has spread far beyond the confines of education. Objectives-based evaluations can be found in virtually all fields of service, and it is common to see government requirements specifying that evaluations be conducted to determine the extent to which each funded program achieved its objectives.

Use Considerations The objectives-based approach is especially applicable in assessing tightly focused programs that have clear, supportable objectives. Even then, such studies can be strengthened by judging program objectives against intended recipients’ assessed needs, searching for side effects, and studying the process as well as outcomes. In practice, it is rare for evaluators to question program objectives (Scriven, 1974, 1991).

Strengths Objectives-based investigation has been the most prevalent approach to evaluating programs. Perhaps this is due to the approach’s ease of application. It has commonsense appeal; program administrators have had a great deal of experience with it; and it makes use of published rules for writing operational or behavioral objectives, both norm-referenced and criterion-referenced testing, and performance assessments.

Weaknesses Common criticisms are that objectives-based studies report findings only at the end of a program; that such information is neither timely nor pertinent to improving a program’s implementation; that the information often is far too narrow to constitute a sufficient basis for judging a program’s level of success, especially for the full range of beneficiaries; that objectives-based studies do not uncover positive and negative side effects; that they may credit unworthy objectives; and that they fall short of assessing a program’s significance (see Scriven, 1974).