Using The Learning Delta To Manage Learning__white Paper__greg Luther - High Delta Learning Llc

Data-Driven Decision-Making in K-12 Education: Using the Learning Delta to Manage for Results A High Delta Learning  White Paper Gregory D. Luther Founder and Principal High Delta Learning, LLC May, 2003

Page 1 "…When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of Science, whatever the matter may be."  Lord Kelvin, 1884

Introduction The emerging market for K-12 education services is placing cities, school districts, and parents into the position of education consumers who have a growing need for accurate, understandable, and timely information on which to base their choices. Similarly, education philanthropies have choices to make with regard to which reforms or innovations to back with their financial resources. In the words of the Thomas B. Fordham Foundation: Private individuals and organizations… can spend their dollars exactly where they seek to do the most good. If they direct their money, energy, and influence toward the right targets. . . their leverage will help move the system itself. The philanthropic sector-America's unique blend of private organizations with public-minded goals-has the freedom of action to push on the right pressure points, and it has clout that i

most parents lack .

But a critical difficulty facing education philanthropiesas well as cities, school districts, and parentsis that schools do not adequately account for their primary product: learning. Measuring and analyzing the impact of reforms and innovations on students' learning progress is a problem because schools do not employ performance measures and management processes like enterprises outside of the education

sector. Typically, instead of employing a "generally accepted accounting principle" for learning measurement, there is a call for outside research to assess or validate a program's salutary effects, to be performed using multivariate statistical analysis and experimental designs taken from social science and clinical drug trials. In short, because schools do not measure their own performance, somebody else has tousually at considerable expense and with substantial delays between the point of data capture and the reporting of results. In this paper I propose the Learning Delta as a top-level performance metric for schools. Learning Delta is a measure of learning progress that captures each student's change in mastery for a given period of time. In addition to improving the manageability of individual student progress, Learning Delta data can be aggregated to arrive at progress measures for classrooms, entire schools, or school systems. Learning Deltas can be used to minimize the time students spend waiting for learning to happen. Perhaps most importantly, continuously accumulating and analyzing longitudinal Learning Delta data for all students represents an embedded research process that can inform and help to improve instruction in every classroom, school, and school system that uses it.

In this paper I make a three-part argument: 1. More can be done to improve education by managing the effectiveness of teaching than by any other single factor. 2. Using the Learning Delta to measure learning progress is fundamental to managing and improving the effectiveness of education reform programs, schools, and teaching generally. 3. In order to implement Learning Delta Management, classroom workflows must be digitized in order to enable the instrumentation of teaching and learning in schools.

Part 1: Value-Added Testing and Teacher Effectiveness There is already a substantial body of empirical work from the state of Tennessee that substantiates the importance and utility of collecting data on students' learning progress. Together with the Tennessee ValueAdded Assessment System (TVAAS), Dr. William Sanders, at the University of Tennessee ValueAdded Research and Assessment Center, found that of all the factors influencing the observed change in student test scores from year to year, teacher effectiveness was by far the most powerful.

Page 2 The TVAAS has been in use since 1991. Every student in grades 2-8 is tested each year in math, reading, language arts, science, and social studies. Testing of high school students began in 1995. Each student’s test results are compared to his or her scores from the previous year, and the average score increases of all the students taught by each teacher are reported to administrators for use in evaluations and personnel decisions. The headlines from Sanders' findings in Tennessee include the followingii: 1. The top quintile of teachers-ranked according to their students' average increase in test scores--raised their students’ achievement test scores 39 percentile points more than teachers from the bottom quintile. 2. Teachers performing in the two lowest quintiles in Tennessee failed to produce any achievement gains with most of their students. 3. The performance difference between students having three consecutive teachers in the top one-fifth of the ranking and three in the bottom one-fifth was 50 percentile points(!!). 4. The teacher quality effect on student performance was found to be highly significant and larger in size than any other factor-including class size and "heterogeneity." In Dr. Sanders view, then, "…more can be done to improve education by improving the effectiveness of teachers than by any other single factor." And the key to that effort is collecting data on the learning

progress being made in the classroom.

material satisfactorily by the end of the term.

Part 2: Using Learning Delta (and An Illustration)

Data Discipline

Learning Delta is given as the change in a student's mastery of an academic subject divided by the change in time. Where Sanders' "value-added" is a measure of academic distance, Learning Delta is a measure of speed, i.e. distance Figure 2 Learning Delta: the Rate of Learning Progress

∆ Mastery

Success Rate ∆ Time

divided by time, as shown in Figure 2. The "distance" term can be thought of as a scale score that is used in the same way as Sanders uses it. Time is measured in school terms. So if a course lasting one term is given a value of 100 points, and a student masters half of the units making up that course in one half of the academic term, then the student is proceeding at a pace of 100 points per term and has a Learning Delta of 100 (i.e. 50 points ÷ 0.5 terms = 100 points/term). If on the other hand, the student has only mastered curriculum units worth 30 points by the mid-term, that student has a Learning Delta of 60 (i.e. 30 points ÷ 0.5 terms = 60 points/term), and unless she/he increase their Learning Delta to 140, she/he will not complete the course

The "data discipline" associated with Learning Delta is quite simple. Step 1 - A "pre-test" is administered on or near the first day of class to establish the baseline learning level of each student. • Pre-test questions are drawn from the same set of test questions that would appear on a final exam. A pre-test may also include questions that represent prerequisite knowledge and knowledge that is advanced beyond the scope of the course. • A student's pre-test score is a measure of the subject knowledge they brought into the class with them. • Pre-test resultstagged to each student, to the specific subject units and skills being examined, and to the teacher and classare recorded and/or stored. • The difference between a student's pre-test score and the score required to demonstrate mastery in a subject. is their initial Learning Gap, which also serves as the Planned Learning Delta for that student. Step 2 - As instruction progresses, additional tests are administered on a weekly, bi-weekly, or monthly basis. Once again, each test is effectively a final exam, and students' scores reflect their new level of subject knowledge attained since the beginning of the term. Step 3 - At the time of each assessment Learning Delta figures are calculated for each student and the average is calculated for the class as a whole.

Page 3 • As interim Learning Deltas are measured, the students, teachers, and parents can see if adequate progress is being made to the goal of mastery, and if not, they can adjust course while there is still time. • Students who are progressing too slowly will have low Learning Deltas that identify them for special assistance or motivating. • Students who are moving much faster than the rest of the group will have high Learning Deltas that identify them for additional "enrichment" work or acceleration into a more advanced class. A key difference between the Learning Delta and "value-added" measures is that Learning Delta is designed to be a frequent measure that enables the teacher and school leaders to make immediate-term course corrections in every classroom. The "value-added" measure as used by TVAAS is a once-a-year measure that, as a result, is retrospective. Although it can give a reading on how a school performed in the prior year, as any particular school year progresses it provides less and less help with how to improve learning performance in specific classrooms next week or next month. Appendix A shows what Learning Delta data would look like over a term and how that allows classrooms and teachers to be grouped into performance quintiles.

Thought Experiment

Figure 3 Out of 7,000 Students in Reform Program

What follows is a "thought Group 1 - Reading at Grade Level (at 50th %ile or above) experiment" Group 2 - About One Year and illustraBehind (assume they score tion of why around 20%ile) data on Group 3 - About Two Years students' Behind (assume they score around 10%ile) learning progress Total should be used to drive decision-making in education philanthropy. The example I will use is based on an actual philanthropyfunded reading reform program (with some of the details disguised) in a state where roughly one out of every five students in the target population fail to achieve "proficient" status on the state's reading tests, and about one in ten demonstrate only "minimal" reading ability. For the purposes of this example, the program covers 7,000 students across 20 schools, and it costs about $2 million per year to administer. Figure 1 shows a simple cost-persuccess calculation for this programwhich may involve expenditures on professional development, specialized curricular materials, technology innovations, or all these elements.

Having made this calculation, the question remaining for the donor to answer is, "Do you think $4000 per student is a satisfactory number?" In the event that the Figure 1 - Program Cost Per Success donor is a venture-philanthropist Number of Students in Program 7,000 with an especially keen interest Total Program Cost for One Year $2,000,000 in results, the further question Number of Students Meeting 500 may be, "Shall we continue Reading Standards Who Wouldwith this program or do n't Have Otherwise something else where the Program Cost per Student$4,000 impact may be greater?" Success

% of Population

Number of Students

Students Below Grade Level

78.9%

5,523

11.7%

819

819

9.4%

658

658

7,000

1,577

The experiment involves applying some of the findings from the work of Dr. William Sanders and TVAAS to this sample education philanthropy program and posing the following questions: • What if we measured the Learning Deltas being produced by teachers participating in a philanthropy-funded education reform program by capturing data from the students on a monthly or quarterly basis? • If Sanders' findings about teacher performance in Tennessee were duplicated elsewhere, how would this information enable us to increase the "return" of an education reform program or other innovation that is underway?

The Base Case

Measures of learning progress are important because, as Sanders has shown, highly effective teachers move their students along at a much greater pace than average or belowaverage teachers. And the situation in any particular school is further complicated because the curricular "distance" that must be covered in order to meet standards of competency can vary greatly from one student to another.

Page 4 school year. Quintile 3 teachers Base Case accomplish some improvement, but Q 1+2 Q3 Q4 Q5 Total not enough to bring $3,000,000 1500 their students up to standard (see table $2,000,000 1000 in Appendix B for detail). As a result Write-off Yield = 4.1 Yield = 7.4 Yield = 2.3 of these combina$1,000,000 500 tions of teacher 132 effectiveness and 132 328 0 164 164 student deficit, $0 0 0 0 then, only 40% of Q 1+2 Q3 Q4 Q5 Total ($400,000) ($400,000) ($400,000) Group 2 students will be brought up ($800,000) ($1,000,000) -500 to grade level, and only 20% of the ($2,000,000) -1000 Group 3 students. ($2,000,000) Overall, based on Sanders' findings in ($3,000,000) -1500 Tennessee, only Teacher Performance Quintiles 7% of the 7,000 Program Cost Group 2 Students Group 3 Students students in this program are likely to be counted as Case graphic in Figure 4 shows the successes at the end of the year As shown in Figure 3, because 20% effect of the differing levels of (i.e. students who met the statewide of students in the reform program teacher effectiveness predicted from reading standard who would not we are looking at score below Sanders' findings for students who have otherwise). The cost per "proficient" status on the state's are one and two years behind in student-success will, therefore, come reading tests and about 10% demontheir reading skills. Specifically, in at about $4,400 and the return can strate only "minimal" reading ability, teachers in the bottom two quintiles be thought of as 2.3 successes-perwe get roughly 1,500 students out of do not produce any improvement, $10,000 "invested" by the philanthe 7,000 who are functioning below and there is a 39 percentile point thropy. grade-level. difference in results between quintiles 3 through 5, which is assumed For the purposes of this illustration However, as shown in the figure, to be distributed evenly across these we then assume that the 800+ analyzing results according to quintiles. So Quintile 5 teachers Group 2 students that scored performance quintile makes certain produce gains of 39%ile points in "BASIC" on the statewide compeprogram improvement actions their students; Quintile 4 teachers tency test are one year behind, and immediately obvious. For example, produce gains of 26%ile points; and the remaining Group 3 students who the yield achieved by Quintile 5 Quintile 3 teachers produce gains of scored "MINIMAL" are two years teachers is 7.4 successes- per13%ile points in their students. behind. To represent a typical $10,000, and the Quintile 4 yield is heterogeneous classroom these 4.1. But over half of the $2 million student groups are distributed Under these conditions, when program cost was spent on classuniformly across all the teacher combined with the children's varying rooms that produced no sucquintiles. levels of skill deficit, only Quintile 4 cesses. What if actions were taken and 5 teachers are likely to sucThe implications of the Sanders to either a) increase the effectiveceed in bringing students up to findings for these 1,500 students is ness of lower quintile teachers, or b) grade level by the end of the what is most interesting. The Base move more Group 2 and 3 students Number of Successes

Program Cost

Figure 4

Page 5 into quintile 4 and 5 classrooms? In addition, there are borderline cases (the yellow cells in the Base Case table in Appendix B) where some improvement was seen, but not enough to meet the proficiency standard. What if additional funds were spent to extend the program and put those kids across the finish line? What-If Scenarios

The second scenario in this experiment moves Group 2 and 3 students out of the bottom quintile classes and into the top two quintiles. Even assuming that program money follows the kids and overall spending remains at $2 million the program's success rate almost doubles to 13%; the cost per student-success has dropped by 50%; and the program's yield has doubled from 2.3 to 4.6 student-successes per $10,000. (See table in Appendix C for detail.) In the third scenario we extend the

the program's impact and yield. In the absence of such data, though, proactive management of this type is much harder because decisions depend on a chain of personal observations, opinions, and judgements, which without hard data to back them up, more often get translated into politics rather than action.

program's reading interventions into the next grade in order to finish the job that was started with the borderline students. This timeassuming that spending increases proportionally by 4% to $2.08 millionthe program's success rate increases by 64%; the cost per student-success drops by 37%; and the program's yield has increased from 2.3 to 3.6 successes-per-$10,000. (See table in Appendix D for detail.)

Managing with the Learning Delta

The point of management information, like the Learning Delta, is to identify and target the areas that need to be better understood and that are candidates for scrutiny and change. Once exceptions have been identified, though, the purpose of management information is not to answer the question "what really happened?"but rather, "who am I going to call?" As was illustrated in the above thought experiment, the critical first step is to identify exceptions either based upon specific criteria or using a normative approachas in the examples above.

As shown in Figure 5 (and in Appendix E), if scenarios two and three are both put into effect, the program's success rate would increase by 193%; the cost per studentsuccess would drop by 64%; and the program's yield would increase from 2.3 to 6.3 successes- per$10,000, a jump of 176%! If, then, we had in our possession Learning Delta data that allowed us to rank classrooms/teachers in order of average actual Learning Delta per class, we could modify the allocation of students to teachers and/or enhance the skills of the low performing teachers in order to increase

Figure 5 Summary of Scenarios 4500

70.00

4000

50.00 3000 40.00

2500

2000

30.00

1500 20.00 1000 10.00 500

0

0.00 Base Case Student Successes

Scenario 2

Scenario 3

Students Successes per $10,000

Scenarios 2 + 3 Cost per Student Success

Successes per $10,000

Number of Successes / $ Cost per Success

60.00 3500

Unlike impact studies, the purpose of performance management information is not to produce scientifically precise analyses of efficient causes. Instead, performance measures and management information, to be of value, must support real-time navigation in the sense of providing near immediate feedback on and control over speed, course, and resource consumption. So because performance measurement is very much like a steering wheel in a car, it needs to be attached to and a part of a school's everyday processes. Otherwise, teachers and school leaders cannot steer around obstacles; they can only drive into them, and then file an accident report later.

Page 6 Learning Delta Supports a Managerial Approach to Education Improvement

A managerial approach to education improvement asks different questions about schools than a curriculum-based reform or one based upon community outreach, etc. But the answers to the managerial questions can be critical to the success of these other reforms. For example, which attributes of a classroom's, a school's, or a school system's performance are what we expect (under a particular reform

model), which need to be improved, and which represent performance beyond what we thought we could produce? At a minimum, a managerial approach to education improvement requires answers to five key questions: 1. What are the standards that represent how we expect educational processes to perform? 2. How do we measure actual performance against these standards?

Figure 6 Key Performance Management Questions 1.

What are the standards

Selected Specifics •

Processes include student grouping, curriculum planning, lesson planning, instruction delivery, classwork/homework assignments, tutoring, testing, student performance planning, parent communication, etc.

•

What are the reform-related or other standards that correspond to each of the educational processes above? Are they widely publicized and/or formally adopted?

•

Are there data capture mechanisms in place that support performance measures?

•

Are they fact-based, objective, and timely?

•

Are they efficient to use and non-disruptive to teaching?

•

Are measures captured and reported frequently enough and with adequate turn-around to enable corrective action?

•

Are measures specific enough to enable positive identification of the people and/or processes needing improvement?

•

Are measures specific enough to enable positive identification of people and/or processes demonstrating excellence?

•

Is there a store of historical measurement data allowing for trend analysis?

•

What are the performance trends versus key educational processes and standards?

•

Are there clearly articulated improvement goals and objectives?

•

Are they being or have they been met?

that represent how we expect educational processes to perform?

2.

How do we measure actual performance against these standards?

3.

Do our measures allow us to improve performance?

4.

What is our track record vis a vis these standards?

5.

Are we improving?

3. Do our measures lead us to actions that improve performance? 4. What is our track record vis a vis these standards? 5. Are we improving? The key elements of a managerial model for education improvement is the subject of another paper, but some of the important considerations are listed in Figure 6. Learning Delta, as one top-level performance measure, is a first-level response to these questions no matter what other type of reform is being pursued. We know, for example, that a rising fifth grader who reads at a third grade level must progress more quickly than other fifth graders in order to meet the standard for reading competence by the end of fifth grade. We can express that requirement as a Learning Delta like "150 points per term." At an interim point in the school year we can identify students or classrooms who are progressing too slowly, and we may be able to apply additional interventions to accelerate them to a successful conclusion. Or we may find that the original interventions prescribed by the reform model in use were never implemented in certain classrooms. In either case, by capturing and analyzing Learning Delta data to identify exceptions, school leaders can do two things: a) reduce the cost of observing (i.e. inspecting) classes that are already meeting performance standards, and b) build their fact-base regarding what works and what doesn't in practice. Using this fact-base, schools will be able to make sustainable, repeatable improvements to classroom and school-wide processes that raise

Page 7 performance for the future. In other words, we can improve the state of education if we move beyond the "random acts of progress" too often seen in K-12 schools to data-driven decision-making that focuses on actual learning progress and its underlying drivers. Maximizing Learning Progress by Reducing Student Wait-States

Another important reason to adopt the Learning Delta as an interim performance measure is that it provides a better means than annual standardized tests for ensuring that all students will make adequate learning progress.iii Imagine a school that has as its standard that every student will achieve a Learning Delta of 60 points per term. Such a school would be different from conventional schools in several ways: • If an advanced student were to score 80 out of 100 on a pre-test, that studentand that student's teacher would not be able to meet the school's standard of 60 points per term of learning progress. Because the student's maximum gain in that class is only 20 points, it is up to the teacher and the school to make a change to that student's placement or curriculum, or adequate learning progress will not be achieved. • Similarly, if a student scored 10 out of 100 on a pre-test, it is unlikely that student would be able to achieve learning progress of 90 points in one term. In fact, that student's low score shows that she/he is not in the right classroom to begin with. Something needs to change if she/he is to meet the school's learning progress standard. And it is up to the teacher and the school to change it.

These examples show how the problem of student "wait-states" in conventional classrooms retard learning progress. Students who are substantially above or below the class average in attainment are either waiting for the class to catch up to them, or waiting for someone to help them understand what is going on. As shown in Figure 7, student wait-states are the mirror image of the distribution of attainments in a classroom that was grouped by age. Simply by using pre-tests and Learning Delta measures to proactively manage learning progress, student wait-states will be reduced and learning progress improved during a school year rather than one or two years later. Limitations

Learning Delta, though, is not and cannot be the single measure of success of a classroom, school, or education reform program. Learning Delta as a measure behaves a lot like an earnings number for a commercial company. Earnings is a key indicator of a company's health, but

it is almost useless if used in isolation. Publicly traded companies get compared on ratios like Return On Equity (ROE) and Price-Earnings ratio (P/E), both of which require an earnings number. But the key drivers of ROE for a nationwide clothing retailer which are themselves performance measuresare very different from those for financial services companies or car manufacturers. If as investors all we had to compare the performance of these disparate companies were raw earnings numbers, we'd be in trouble (…more trouble). However, we've still got to have accurate earnings numbers! They are critical, but they are also just the beginning. The case for Learning Delta is the same: don’t mistake Learning Delta for the only measure of a school's instructional success. Schools, like companies, are more complicated than that, and there are underlying drivers of Learning Delta that should be measured and managed. And there are also other important top-level measures, like graduation rates, that should be attended to.

Page 8 But, as with company earnings, we need Learning Delta as a measure of the learning progress produced by teachers and schools because learning progress is the very thing that schools are supposed to produce. Without such a measure, it is hard to imagine how teachers and schools can be systematically managed and improved. On the other hand, perhaps it is because we have been muddling through without such a measure that schools have remained essentially unimproved over the last forty yearsin spite of the tripling of per pupil expendituresiv.

Part 3: Putting the Learning Delta into Practice We can identify high performance schools because they are the ones that produce greater than average Learning Deltas. And as with other human enterprises, high performance schooling is largely a matter of good leaders providing sound management using effective techniques and systems. For example, KIPP Academy in Houston enrolls fifth graders who for the most part function at a third grade level. In the space of four years KIPP turns out ninth graders operating at a ninth grade level--as defined by exclusive preparatory schools. On average, KIPP's students make 6 year's academic progress in 4 year's time. KIPPand other high performance schools like it employ methods, practices, tools and techniques that are different from those found in ordinary schools. And they are well managed by leaders who are empowered to do so. For example: •

They explicitly define common roles and standards for teachers, students, and parents to play within well-defined processes covering attendance, classwork, homework, and testing, as well

as teamwork, culture, and behavior. •

They create new information flows that provide same-day feedback on key performance attributes. These feedback loops support teachers, students, and parents in their roles by quickly identifying exception conditions and bringing responses to bear so that the exceptions are brought into compliance with the school's predefined, published, and acknowledged standards.

But high performanceas researched in Samuel Casey Carter's the book No Excusesvis based largely on "management-by-walkingaround". And it is as a result that high performance schooling is negatively correlated with school size. Critics of the "No Excuses" research often claim that the schools studied are statistical outliers that are irrelevant to the problems of large schools because of the special conditions involved, small size among them. They ask, "What is the lesson of high performance schooling for the management of tens of thousands of students rather than hundreds?" And they conclude that there is none. There are distinct diseconomies of scale at work and classic "locus of control" problems that stand in the way of scaling up "No Excuses" practices. Simply put, attempting to duplicate "No Excuses" results using "No Excuses" techniques in large school settings would require many more principals and instructional leaders "managing-by-walkingaround" than school systems are able or willing to pay for. But a well understood lesson from outside the education sector is that even very effective manual processes break down under high volumes.

Conventional schools and classrooms deliver instruction, post assignments, collect and correct homework, and administer tests in an entirely manual fashion. Managing the effectiveness of large numbers of such classrooms eventually becomes infeasible as a simple result of increasing numbers of students and classes. By failing to separate the negative effects of size from the positive effects of good management, the critics miss the opportunity we are presented with: high-performance school management can be scaledup if schools would only do what the rest of the U.S. economy did starting 20 years ago as a response to growth: get digitized. Digitize what?

Four or five years ago the excitement about "cyber-schools" and the "virtual classroom" was mostly focused on distance learning and how it enabled "anywhere, anytime learning," i.e. the elimination of time and space as barriers to taking courses or earning degrees from offcampus locations. Subsequently, educational institutions began to discover the benefits to on-campus students of on-line courses, and so the "Web-enhanced campus" became an object of enthusiasm in higher education. Accordingly, the major distributed-learning platforms like Blackboard, WebCT, and Lotus LearningSpace tried to provide online analogues to the blackboard, the lectern, the classroom discussion, the student's raised hand, and so forth. As a result, the surrounding debate about "e-Learning" has mostly focused on whether an online class discussion is as good as the face-to-face variety, or whether listening to a lecture is as rich an experience on-line as it would be actually sitting in a lecture hall.

Page 9 The point I wish to make here about "digitizing" the classroom is different from the "virtual classroom" discussion described above. Classrooms, schools, and school systems are enterprises, which–like others– are comprised of methods, practices, tools, techniques, processes, knowledge and people. What needs to be digitized in K-12 education are the classroom workflowswhich connect the teacher to the students, parents, and the schoolso that basic data on what is getting done and what isn't can be used to manage classroom performance and improve results. Capturing data like Learning Gap, planned vs. actual Learning Delta, and the like is both necessary and desirable because it enables instrumentation of the classroom rather than virtualization. Using digitized performance information, school and district leaders can exercise anytime, anywhere, "line-of-sight" management of bricks-and-mortar classrooms in much the same way as with virtual classrooms, but without the disruption of trying to deliver instruction in a new way. Digitizing the classroom to accomplish instrumentation would enable broad-based performance improvement through the regular use of on-line reporting facilities and "drill-down" analysis to: a)

identify the Master Teachers who are the exemplars of high performance that should be emulated,

b) identify the classrooms and teachers needing help to improve their performance, and c)

bring the two groups together so that expertise and experience can be shared.

That is a fitting role for school leaders and managers, and digitizing the classroom is the critical

Figure 7 Key Attributes of a Learning Delta Management System •

Curriculum-Neutral  Accommodates whatever curriculum is already in use by a school. Does not impose a new one.

•

Pedagogy-Neutral  Does not require that classroom-based teaching be replaced with computer-based instruction.

•

Ease-of-Use  Fits well with and supports classroom routines. Data capture and feedback mechanisms are userfriendly.

2. Least costly

•

Allows use of hardware with lowest total cost of ownership: handheld computers like Palm OS or Pocket PC devices.

•

Minimizes requirement for network and/or centralized application support though use of application-service-provider (ASP) model.

3. Most efficient

•

Should automate the testing process from start-to-finish in order to relieve teachers of manual tasks and free up new time for teaching. Should not simply impose a new clerical process.

•

Should provide teachers with detailed, useful performance analyses on demand in order to increase teacher and classroom effectiveness.

1. Least intrusive

first step to making that type of analysis-driven facilitation possible. Digitize How?

With digital mechanisms for measuring the actual learning progress of each student, schools could begin to correlate learning results to teaching performance, rank classrooms accordingly, and then scrutinize classroom methods, tools and techniques to identify improvement actions. The current state in our schools, though, is that very little information comes out of classrooms once the teachers and students have gone in. As a result, instructional leaders, administrators, and parents cannot tell on a daily, weekly, or monthly basis which classrooms are on schedule and which are not, which curriculum elements--new or old--work very well or very poorly, or which students are functioning at grade-level, who have pulled ahead, and who have fallen behind. Eventually, these questions may get answered

when high-stakes tests are administered, but by then it is too late to improve outcomes for the students being tested. To put into practice data-driven performance management based on the Learning Delta alone, three key components are required: •

A testing process producing scales (scores) that have a defined (i.e. mapped) relationship to the curriculum, and which produces measurements that extend above and below grade level;

•

A database and data capture tools that track individual student scores and maintain links to classes, teachers and schools; and

•

A reporting and analysis facility that supports the type of datadriven decision-making illustrated above.

Page 10 These components comprise a minimum infrastructure for datadriven performance improvement in schools that would relatively simple to develop and implement. And for many schools having a system with just these components would represent a logical, high-impact starting point for "getting digital" and starting to practice data-driven decision-making. But for a school to obtain performance tracking and management functionality in the educational software market often requires that an entire instructional management system be purchasedalong with substantial amounts of "course-ware" and embedded curriculum that may not be appropriate or useable by classroom teachers. In addition, such systems require amounts of computer hardware that many schools can ill afford. Taken together, these attributes of vendor offerings represent substantial barriers to adoption by schools that compound the usual issues of cost, staff training, and curriculum management. The broad-based application of Learning Delta measures in schools requires that a more focused and easier-to-implement Learning Delta Management System be made available. To be broadly applicable in divers types of schools and communities it should: • Accommodate rather than disrupt the established instructional methods and materials, • Be designed for very low total cost of ownership, and • Provide teachers, students, and parents with the usual and customary benefits of automation rather than creating additional work (see Figure 7).

Change Management and People Factors

Most importantly, though, a Learning Delta Management System can only enable performance improvement if the real solution is already in place: a concerted

the data fairly or constructively, the environment needs to change to address those fears and establish common goals.

• Willingness - Are the users of a system willing to change how they go about their jobs in order to use the system successfully?

In the case of schools, control of the environment is shared mostly between two groups: the teachers' unions and school administration. If either one of these groups is opposed to adopting a new system, it will most probably fail. Similarly, if it's not in the interest of these groups for learning performance management practices to take root and spread, then they will not. So unlike smaller class size and increased professional developmentwhich probably benefit kids in some ways and certainly benefit the adults performance measurement as a means of managing schools is problematic because it has the clear potential to create winners and losers. It increases the riskiness of being a teacher or an administrator in a school that is not performing well, and so it is harder to win votes on the "willingness" dimension, both among a school's teaching staff and the groups in charge of the environment.

• Environment - Do system users believe that doing what it takes to make the system successful will be recognized and rewarded as a positive achievement by the groups that control the environment?

Of course, parents and communities have long wanted tools to measure our children's absolute and relative educational progress, but they are not the ones being asked to change. And they do not control the environment.

and serious demand for meaningful performance data from schools. That has to come first, or anyone attempting to implement such a system will likely fail. As experienced system implementers can all attest, "people factors" are often more decisive than a system's design or features in determining the success or failure of a project. K-12 schools are no exception. Three dimensions always need to be addressed: • Ability - Do the intended users of a system have the training, time, and resources required to use it successfully?

It turns out that the first two dimensions, ability and willingness, are heavily conditioned by the third: environment. For example, if a school's teaching staff are unable to implement a new learning management system because they don't have the training or the release time to get the training, the environment needs to change to accommodate those needs. If a school's teaching staff are opposed to a learning management system because they don't trust administrators/management to use

Page 11

Conclusion There are signs of positive change, however. Because of the No Child Left Behind Act and other factors, the secular climate for learning performance measurement in classrooms is probably better than it has ever been. But ideology and labor relations remain very high barriers to the success of data-driven learning management within K-12 education. One bellwether of this positive change may be the New Schools Venture Funda leading venture philanthropywho in their recent press release point to a shift away from schools being compliancedriven organizations to performance-based cultures focused on achievement. Venture philanthropy-based education reforms are in an excellent position to demonstrate the power for positive change that performance measures like Learning Delta represent. Education management organizations (EMOs) like Edison Schools and Chancellor Beacon Academies are also in a position to make a positive impact. In fact, because they are for-profit and so get criticized for diverting money "away from the children," they should be strongly motivated to use Learning Delta measures to ensure—and then later prove—that they are doing a better job than traditional schools. And so paying profits to shareholders is in some sense justified if it makes possible a demonstrably better education for our kids.

PRESS RELEASE (Excerpt) PALO ALTO, Calif., May 12 /PRNewswire/ -- NewSchools Venture Fund announced the launch of its new Performance Accelerator Fund, targeted at $20 million and designed to invest in ventures that enhance the capacity of school systems to produce high levels of student achievement. . . "As districts work to increase student achievement, we must ensure that they have the tools they need to be successful," said Kim Smith, co-founder and CEO of NewSchools Venture Fund. . . State and federal education policies are increasingly pushing for school systems to shift from compliance-driven organizations -- governed by rules, regulations and court orders -- to performance-based cultures focused on high student achievement. . . . . .The ventures in the Performance Accelerator Fund will focus on developing people who know how to lead and teach in a performance-based environment; designing tools that give those people the information they need to make good instructional decisions; and supporting practices that promote and reinforce success. The Fund will identify, fund and build ventures that address human capital -- including the recruitment, preparation and support of high-quality teachers and leaders -- and performance tools -- including information systems and assessment tools that enable teachers and leaders to make data-based decisions about their students' instruction. (emphasis added)

One particularly good chance to demonstrate the value of Learning Delta management is, then, is for the "managerial model" of school improvement to be incubated inside a substantial education reform program or EMO while the market for Learning Delta management software and services develops. To be convincing, a demonstration of the managerial model needs to span multiple classrooms, schools, and even school systems. It needs to have shown its effectiveness on a substantial scale in order to be a candidate solution for the systemic improvement of education in this country (something that No Excuses schools cannot do).

*

*

systemic improvement in K-12 schools is to transform the classroom into the atomic unit of datadriven performance management. The first step is to digitize the classroom and to start using measures like the Learning Delta to manage and improve school performance as a regular part of a school's weekly routine.

*

Because improving the effectiveness of teachers can improve education more than by any other single factor, our greatest opportunity to achieve

For more information contact: Gregory D. Luther, Founder and Principal ! High Delta Learning, LLC ! P.O. Box 398 ! Litchfield, CT 06759 tel: 860.567.8242 ! e-mail: [email protected]

Page 11 Appendix A EXAMPLE OF LEARNING DELTA DATA - Classroom A Teacher Name: Mastery Score: Learning Delta Standard:

Mr. Smith 80 40 points per term

Timeframe:

Aug

Sep

Oct

Plan vs. Actual Learning Delta

Dec

Pre-Test

Learning Gap / Planned Learning Delta

Test 2

Interim Learning Delta

Test 3

Interim Learning Delta

Final Test

Final Learning Delta

Student 100

39

41

47

24

61

34

82

43

2

Student 101

21

59

28

21

33

19

55

34

-25

Student 102

24

56

36

37

55

47

80

56

0

Student 103

55

25

57

6

72

25

95

40

15

Student 104

35

45

53

56

58

35

67

33

-13

Class Average Test Score

35

Student

44

Class Average Learning Delta

45

Class needs to achieve this average "speed"

29

Going too slowly!

% of Students Meeting Learning Delta Standard

Teacher Inter-ranking Rank

Average Class Learning Delta

Hume Hobbes James Locke Descarte

1 2 3 4 5

72 68 66 60 58

Smith

6

41

Weber Berkeley Russell Machievelli Mosca Michels Rousseau Nozick Rawls

7 8 9 10 11 12 13 14 15

40 39 38 32 22 18 16 15 15

76

64

41

Finished a little under plan

Speeding to catch up

41 40%

Teacher's Learning Delta Result for Term

Name

56

Quintile

Q1

Q2

Q3

Q4

Q5

Aug - Dec

Page 12

Appendix B

Figure 4 Base Case Quintiles 1 & 2 Teachers

7000 Third Grade Children

% of Population

Quintile 3 Teachers

Quintile 4 Teachers

Quintile 5 Teachers

Total Students

Number of Students in BRI Program

GROUP 1 - Reading at Grade Level (at 50th %ile or above)

78.9%

2209

1105

1105

1105

5,523

GROUP 2 - One Year Behind (between 10th and 25th %ile)

11.7%

328

164

164

164

819

GROUP 3 - Two Years Behind (between 10th %ile and below)

9.4%

Total

Predicted Student Improvement Based on Sanders' Findings (%ile Points)

Program Impact

% of Population

% Impacted

263

132

132

132

658

2,800

1,400

1,400

1,400

7,000

0

13

26

39

Predicted Number of Additional Students Meeting Reading Standard

GROUP 1 - Reading at Grade Level (at 50th %ile or above)

78.9%

already @ std

already @ std

already @ std

already @ std

0

0%

GROUP 2 - One Year Behind (assume at 20th %ile)

11.7%

NONE (@20%ile)

NONE (@33%ile)

164

164

328

40%

GROUP 3 - Two Years Behind (assume at 10th %ile)

9.4%

NONE (@10%ile)

NONE (@23%ile)

NONE (@36%ile)

132

132

20%

0

0

164

295

459

7%

Quintiles 1 & 2 Teachers

Quintile 3 Teachers

Quintile 4 Teachers

Quintile 5 Teachers

Total

Total

Program ROI All-In Program Cost/Students Participating in Program Program Cost per Teacher Quintile Add'l Students Reading @ 3rd Grade Level by End of 3rd Grade All-In Program Cost/Student Meeting Standard -- by Teacher Quintile Students Successes per $10,000

$286

$

800,000

$

400,000

-

-

write-off

write-off

$

400,000

$

164

$

400,000

295

2,442

4.1

$

$2,000,000

459

1,354

7.4

$4,355

2.3

Page 13

Appendix C Figure 4.5 Case 2 - Shift Quintile 1 & 2 Upward Quintiles 1 & 2 Teachers

7000 Third Grade Children

% of Population

78.9%

2209

GROUP 2 - One Year Behind (between 10th and 25th %ile)

11.7%

---

GROUP 3 - Two Years Behind (between 10th %ile and below)

9.4%

Total

Predicted Student Improvement Based on Sanders' Findings (%ile Points) % of Population

Quintile 4 Teachers

Quintile 5 Teachers

Total Students % Impacted

Shift Quintile & 5 teachers. 5,523 1105 1 & 2 students 1105 to Quintile 41105 164

328

328

819

---

132

263

263

658

2,209

1,400

1,695

1,695

7,000

0

13

26

39

Predicted Number of Additional Students Meeting Reading Standard

GROUP 1 - Reading at Grade Level (at 50th %ile or above)

78.9%

already @ std

already @ std

already @ std

already @ std

0

0%

GROUP 2 - One Year Behind (assume at 20th %ile)

11.7%

---

NONE (@33%ile)

328

328

655

80%

9.4%

---

NONE (@23%ile)

NONE (@36%ile)

0

0

328

263 591

263 918

40% 13%

Quintiles 1 & 2 Teachers

Quintile 3 Teachers

Quintile 4 Teachers

Quintile 5 Teachers

Total

GROUP 3 - Two Years Behind (assume at 10th %ile) Total

Program ROI All-In Program Cost/Students Participating in Program Program Cost per Teacher Quintile

Students Successes per $10,000

100.0%

Change vs. Base Case

$286

$

-

Add'l Students Reading @ 3rd Grade Level by End of 3rd Grade All-In Program Cost/Student Meeting Standard -- by Teacher Quintile

Increase vs. Base Case

Number of Students in BRI Program

GROUP 1 - Reading at Grade Level (at 50th %ile or above)

Program Impact

Quintile 3 Teachers

-

---

$

400,000

$

-

write-off

800,000

$

328

$

800,000

591

2,442

4.1

$

1,354

7.4

$2,000,000

0%

918

100%

$2,178

-50%

4.6

Page 14

Appendix D Figure 4.6 Case 3 - Extend Program into 4th Grade Quintiles 1 & 2 Teachers

7000 Third Grade Children

% of Population

78.9%

2209

GROUP 2 - One Year Behind (between 10th and 25th %ile)

11.7%

328

GROUP 3 - Two Years Behind (between 10th %ile and below)

9.4%

Total

Predicted Student Improvement Based on Sanders' Findings (%ile Points) % of Population

Quintile 4 Teachers

Quintile 5 Teachers

Total Students

% Impacted

Extend the 1105 program into 4th1105 Grade for borderline 1105 students.5,523 164

164

164

819

263

132

132

132

658

2,800

1,400

1,400

1,400

7,000

0

13

26

39

Predicted Number of Additional Students Meeting Reading Standard

GROUP 1 - Reading at Grade Level (at 50th %ile or above)

78.9%

already @ std

already @ std

already @ std

already @ std

0

0%

GROUP 2 - One Year Behind (assume at 20th %ile)

11.7%

NONE (@20%ile)

164

164

164

491

60%

9.4%

NONE (@10%ile)

NONE (@23%ile)

132

0

164

295

132 295

263 755

40% 11%

Quintiles 1 & 2 Teachers

Quintile 3 Teachers

Quintile 4 Teachers

Quintile 5 Teachers

Total

GROUP 3 - Two Years Behind (assume at 10th %ile) Total

Program ROI All-In Program Cost/Students Participating in Program Incremental Program Cost in 4th Grade (Students * Cost per Student) Program Cost per Teacher Quintile

64.3%

Change vs. Base Case

$298

$46,800

$

800,000

Add'l Students Reading @ 3rd Grade Level by End of 3rd Grade

-

All-In Program Cost/Student Meeting Standard -- by Teacher Quintile

write-off

Students Successes per $10,000

Change vs. Base Case

Number of Students in BRI Program

GROUP 1 - Reading at Grade Level (at 50th %ile or above)

Program Impact

Quintile 3 Teachers

$

446,800

$37,600

$

437,600

$

400,000

$2,084,400

4%

164

295

295

755

64%

$2,728

$1,481

$1,354

$2,762

-37%

3.67

6.75

7.39

3.62

Page 15

Appendix E Figure 5 Case 4 - Scenario 2 -PLUS- Scenario 3 Quintiles 1 & 2 Teachers

7000 Third Grade Children

% of Population

Quintile 4 Teachers

Quintile 5 Teachers

Total Students

% Impacted

Change vs. Base Case

Number of Students in BRI Program

GROUP 1 - Reading at Grade Level (at 50th %ile or above)

78.9%

2209

GROUP 2 - One Year Behind (between 10th and 25th %ile)

11.7%

---

GROUP 3 - Two Years Behind (between 10th %ile and below)

9.4%

Total

Shift Quintile 1 & 2 students to Quintile 4 & 5 teachers. 1105

1105

1105

5,523

164

328

328

819

---

132

263

263

658

2,209

1,400

1,695

1,695

7,000

Extend the program into 4th Grade for borderline students.

Predicted Student Improvement Based on Sanders' Findings (%ile Points)

Program Impact

Quintile 3 Teachers

0

% of Population

13

26

39

Predicted Number of Additional Students Meeting Reading Standard

GROUP 1 - Reading at Grade Level (at 50th %ile or above)

78.9%

already @ std

already @ std

already @ std

already @ std

0

0%

GROUP 2 - One Year Behind (assume at 20th %ile)

11.7%

NONE (@20%ile)

164

328

328

819

100%

9.4%

NONE (@10%ile)

NONE (@23%ile)

263

0

164

591

263 591

526 1,345

80% 19%

Quintiles 1 & 2 Teachers

Quintile 3 Teachers

Quintile 4 Teachers

Quintile 5 Teachers

Total

GROUP 3 - Two Years Behind (assume at 10th %ile) Total

Program ROI All-In Program Cost/Students Participating in Program Incremental Program Cost in 4th Grade (Students * Cost per Student) Program Cost per Teacher Quintile

193.0%

Change vs. Base Case

$303

$46,800

$

-

$

446,800

$75,200

$

875,200

$

$

800,000

122,000

$2,122,000

6%

Add'l Students Reading @ 3rd Grade Level by End of 3rd Grade

---

164

591

591

1345

193%

All-In Program Cost/Student Meeting Standard -- by Teacher Quintile

---

$2,728

$1,481

$1,354

$1,577

-64%

3.67

6.75

7.39

6.34

176%

Students Successes per $10,000

Page 16

End Notes: i Finn, Chester E. and Kelly Amis, "Making it Count: A Guide to High-Impact Education Philanthropy," The Thomas B. Fordham Foundation, September 2001, p. 16

ii See the following:

Wright, S. Paul, Sandra P. Horn, and William L. Sanders, “Teacher and Classroom Context Effects on Student Achievement: Implications for Teacher Evaluation,” Journal of Personnel Evaluation in Education, Vol.11, 1997 Sanders, William L. and June C. Rivers, “Cumulative and Residual Effects of Teachers on Future Student Academic Achievement,” Research Progress Report, University of Tennessee Value-Added Research and Assessment Center, 1996. iii Standardized testing, a key component of public education and the current "accountability" discussion, does not by itself provide the information needed to manage school performance. As Sanders and the state of Tennessee have shown, test scores by themselves are not a sufficient management tool because they measure attainment levels rather than the extent to which a school produced a change in attainment levels.

A consequence of managing schools according to test scores appears to be that students who already meet or exceed test standards for their age group ten to get less attention than other students. Sanders reported that to a disproportionate degree "high-scoring students were found to make somewhat lower gains than average and lower-scoring students… This finding indicates that it cannot be assumed that higher-achieving students will 'make it on their own.'” See: Wright, S. Paul, Sandra P. Horn, and William L. Sanders, “Teacher and Classroom Context Effects on Student Achievement: Implications for Teacher Evaluation,” Journal of Personnel Evaluation in Education, Vol.11, 1997, p. 14 iv "From 1960 to 1999, per-pupil expenditures in U.S. public schools grew (in constant 1998-99 dollars) from $2,638 to $7,896. During the same

period, the student-to-teacher ratio dropped from 26:1 to 17:1." (Source: "MAKING IT COUNT: A Guide to High-Impact Education Philanthropy," Thomas B. Fordham Foundation) v Carter, Samuel Casey, No Excuses: Lessons from 21 High-Performing, High-Poverty Schools, Heritage Foundation, April 1, 2000.