Developing An Early Warning System: Identifying Factors That May Predict High School Completion By 6th Grade Students

Developing an Early Warning System: Identifying Factors that may Predict High School Completion by 6th Grade Students (Revised 3/25/2009 1:46 PM) Prepared by

Kenneth T. Wilburn, Ph.D. University of North Florida

Marcia Lamkin, Ed.D. University of North Florida

Dax M. Weaver, MPH President, Health-Tech Consultants, Inc.

Prepared For

The Community Foundation of Jacksonville

For additional information, contact Ms. Pam D. Paul, Vice President, Community Initiatives The Community Foundation in Jacksonville 121 West Forsyth St., Suite 900, Jacksonville, FL 32202 (904) 356-4483, www.jaxcf.org

Early Warning System  2  Executive Summary Purpose The primary purpose of this study was to identify specific academic and social factors that may be used as “flags” to identify Duval County Public Schools (DCPS) 6th grade students who are at high risk of not completing high school. A secondary purpose was to understand how these factors may or may not be aligned with those identified by Neal, Balfanz, and other researchers (Neal and Balfanz, 2006). Four questions guided our 6th grade analysis: (1) Are significant numbers of students showing the signs of disengagement previously identified in our examination of 9th grade students who fail to complete high school? (2) Can we identify a set of indicators that flag sixth graders who have high odds of failing to complete high school? (3) From this set of indicators, can we identify a practical set of indicators routinely collected and reported at the individual level that flag sixth graders who have odds of failing to continue in school until graduation? (4) Can we then identify a set of those indicators that may be influenced by school and community programs? Data The data used in this analysis were provided by the DCPS Department of Instructional Research and Accountability. The data were gathered from the district’s data warehouse and provided to the research team in the form of several Excel spreadsheets. The data included selected demographic, academic, attendance, and disciplinary information from each student enrolled in grades 6–12 during the 2001–2002 through the 2007–2008 school years. Approximately 60,000–65,000 students were enrolled in grades 6-12 during each school year with each grade’s enrollment averaging between 10,000–12,000 students. Since the focus of this analysis was 6th grade students, we began by identifying all 6th grade students for the 2001–2002 school year and then tracking that cohort of students through each school year through the 12th grade (2007-2008). Ten thousand ninety-three (10,093) students remained for inclusion in the analysis after the elimination of withdrawals, transfers, and incomplete data sets.

Early Warning System  3 

Methods and Procedures Once the data set had been cleaned of cases with missing data or incorrectly coded data, an initial factor analysis was conducted to determine whether variables fell into coherent categories (i.e., factors) as predictors of completion. No one piece of data, such as attendance or mathematics standardized test scores, has enough weight to serve as a valid predictor. Therefore, this process of combining various data sets was repeated until a consistent set of categories was identified. Once a coherent set of predictors was identified, linear regression (stepwise) and logistic regression analyses were conducted on each identified factor in order to estimate the strength of each factor. Results A significant number of 6th grade students displayed the signs of disengagement, such as poor academic achievement and excessive absences, previously identified in our study of 9th grade students and in studies conducted by other researchers. Five factors proved to be of the greatest value in identifying members of the 2001-2002 6th grade cohort who did not graduate on-time from high school in 2007-2008. Each flag was a statistically significant flag (p< .05) even after controlling for the other flags. The five flags listed below, individually and in combination, identified 81.52% of the non-graduates. ƒ

Standardized Math Test Scores: 69% of the 6th grade students who failed to graduate on-time scored below Level 3 on the Florida Comprehensive Assessment Test.

ƒ

Standardized Reading Test Scores: 56% of the 6th grade students who failed to graduate on-time scored below Level 3 on the Florida Comprehensive Assessment Test.

ƒ

Failing Course Grades in Mathematics: 30.2% of the 6th grade students who failed to graduate on-time failed a math course.

ƒ

Failing Course Grades in English/Language Arts: 28.1% of the 6th grade students who failed to graduate on-time failed an English/Language Arts course.

ƒ

Excessive Absences: 12.2% of the 6th grade students who failed to graduate on-time missed more the 20% of the scheduled school days.

The number of Class 3 and 4 Code of Conduct violations occurring at the 6th grade among the study population was not very large (3%). As a result, this flag only identified 6.7% of the 6th grader who failed to graduate on-time. However, our tracking of students indicated that the number of Class 3 and 4 violations increase significantly as students get older and as more

Early Warning System  4  students become over-age for their assigned grade. Consequently, the conduct violations from subsequent years have more predictive power than those from Grade 6. While the yield for this factor was relatively small as a single factor and no greater when considered in combination with test scores or grades, we nevertheless consider this to be an important flag to monitor as it becomes a key indicator in later grades. Results from our study indicated 1. Significant numbers of 6th grade students show signs of disengagement previously identified in our examination of 9th grade students who fail to complete high school. 2. We can, with a reasonable degree of validity, identify a set of indicators that flag sixth graders who have high odds of failing to complete high school. 3. From this set of flags, we can identify a practical set of indicators routinely collected and reported at the individual level that flag sixth graders who have odds of failing to continue in school until graduation. 4. These indicators are factors that may be impacted by school and community programs. In conclusion, as we have learned from our previous studies of first-time 9th grade students and focus groups with students who had previously dropped out of school, students do not graduate from high school for a multitude of reasons. Many of those reasons are not linked directly to academic achievement, attendance, and behavior at school. However, our research supports the work by Balfanz (2006) and others (Balfanz, Hertzog, Neild, and MacIver, (2007)) that there is ample, readily available data for early identification of students with the greatest risk of failing to graduate high school. 

Early Warning System  5 

Table of Contents Page INTRODUCTION ……………………………………………………………………………...… 7 BACKGROUND……………………………………………………………………………..…...

8

A Descriptive Study of First-time 9th Grade Students Focus Groups with Dropouts, Parents and Service Providers METHODS AND PROCEDURES………………………………………………………………

12

Guiding Questions Outcome Variable Data Source Sample and Measures Data Analysis Procedures FINDINGS………………………………………………………………………………………...

17

Identifying Warning Flags Attendance Standardized Test Scores Course Grades Discipline Status Variables Final Selection of Identification Flags Validity of the Flags Limitations of the Study SUMMARY……………………………………………………………………………..…………

26

References…….………………………………………………………………………………….

30

Appendix: Demographic Characteristics of the Study Population………………………….

31

Early Warning System  6  List of Tables

Page

Table 1. School Related Variables of the Study Population 

14 

Table 2. Graduate and Non-graduate Percentages Identified by Preliminary Flags 

19 

Table 3. Double Warning Flag Combinations and Counts for all Duval 6th Grade Students in 2001–2002 Who Failed to Graduate in 2007-2008 

19 

Early Warning System  7 

Introduction This study is the third in a series sponsored by The Community Foundation of Jacksonville Florida (CF) designed to examine issues related to Duval County Public School (DCPS) students who fail to graduate from high school. The first study, Where Have All the 9th Graders Gone? (Wilburn, Weaver, and Wilburn, 2008), was a descriptive study of a cohort of first-time 9th grade students. The study was designed to provide the local community with a more detailed understanding of the non-completion problem in Jacksonville. The second study, Kids Say the Darndest Things (Roush and Wilburn, 2008), was a qualitative analysis of focus group data which had been collected by The Community Foundation and their community partners in the Learning To Finish Project. These focus groups were conducted with 16–24 year old students who had been identified by the school district as being of high risk of dropping out, currently enrolled in a school or community sponsored dropout prevention program, or had dropped out of school and were incarcerated in the Duval County jail for a criminal offense. As with the descriptive study, the primary goal of the effort was to gain a richer understanding of students who have dropped out of the public school system. This study used the information learned in the first two studies to develop an early warning system that could be used by the community to identify the students, at the end of the 6th grade, who are the most at risk for failing to complete high school. The report will be organized into the following four sections: •

Background. This section provides a review and summary of the 9th grade descriptive and focus group studies and how the outcomes from those studies were used to inform the 6th grade predictive study.

Early Warning System  8  •

Methods and Procedures. This section describes the methods and procedures used to conduct the 6th grade predictive study.

•

Findings. This section reports the results from the predictive analysis and their application to the development of an early warning system as well as the limitations of the study.

•

Summary. The final section reviews the main points of the study. Background In this section we provide a summary of the two studies we conducted as precursors to

beginning the process of developing flags for identifying 6th grade students with a high probability of not completing high school. A Descriptive Study of First-time 9th Grade Students The first study was designed with the overall goal of developing a better understanding of issues surrounding the graduation crisis and the dropout problem found in DCPS. Simply stated, the design of the study was to follow three cohorts of students from the time they first entered the 9th grade through their 4th or 5th years of high school. Using data provided by the DCPS Department of Research and Accountability, we were able to select a study population that met our criteria for inclusion in the analysis. As recommended by Balfanz (2007), we sought to study cohorts of 9th grade students who had been enrolled in DCPS during the previous grades, were enrolled in the 9th grade for the first time, and were engaged in an academic program for which a standard high school diploma was a reasonable expectation. Additionally, we targeted a study population that was reflective of the overall 9th grade population in gender, race, and socioeconomic status.

Early Warning System  9  Graduation and Dropout Rates. The analysis indicated that by the end of the 3rd year of high school, approximately 488 students (2.5%) of the original first-time 9th grade cohort had received their diploma. By the end of the 4th year, the number of students who had graduated increased to 11,842 students (60.74%), and by the end of the 5th year, the students who had received a standard high school diploma was 11,901 (61.04%). While approximately 61% of the first-time 9th grade students eventually graduated, 95% of those did so in the customary 4th year of high school. Only a small percentage (4.1%) of the students graduated early, and an even smaller percentage (0.005%) graduated after a 5th year of high school. In addition to tracking graduates and non-graduates, we also identified the number and characteristics of students labeled “dropped out” (i.e., did not withdraw or transfer but just failed to show up the next school year). Our calculations indicate that the number of students who dropped out increased with each succeeding cohort. The 2002 cohort had an estimated dropout rate of 25.5%, followed by 29.5% for the 2003 cohort, and rising to a high of 31.3% with the 2004 cohort. When examined as a whole, the estimated dropout rate for the combined three student cohorts was approximately 28.2%. What this means in terms of students lost is that over the 4 years of high school almost 2,000 students left each cohort without receiving diplomas. Focus Groups with Dropouts, Parents and Service Providers In 2007-2008, under the direction of Dr. Shannon Perry, chair of the projects qualitative focus group, seven focus groups met to explore the dropout situation in Jacksonville. Information from these sessions were transcribed and provided for analysis to Dr. Connie Roush, a qualitative research specialist with the Brooks College of Health at the University of North Florida. In this report, qualitative themes were identified with highlighted responses from each focus group and quotes provided to illustrate some of the major points. The major themes identified were participants’ attitudes and beliefs about the dropout situation; the influence of the family, school, and community environments; specific issues related to peer pressure; the

Early Warning System  10  culture of violence; the need to know and care for students; and accountability for school attendance. Each group of participants approached the focus groups from a different point of view or attitude shaped partially by their own beliefs about what spurs school disengagement and dropout. Major Themes. Each group reinforced the belief that getting to know a student takes continuous contact over time and a lot of listening. This lays the groundwork for a trusting and caring relationship necessary to (as one student said) “…attract us to school and keep us there.” Listening was a topic most often addressed by the middle school and incarcerated youth while all groups offered wisdom about the caring relationship and the time that it takes to evolve. Another theme our analysis identified was the power of the need to “fit in” to school culture and/or the “wrong crowd.” Whether it is with other students in school or with groups in the neighborhood, peer pressure is extremely strong. A third theme identified was the “culture of violence” that is so much a part of students’ lives. Many students described their disengagement from school as being “sucked in” to using and selling drugs and the violent lifestyle that goes with it. While the family emerged as a critical theme, it was seen by many teens as equally as positive and negative as a factor. For many, the discussion centered around the barriers put up by family to be supportive. In a smaller but equally powerful part of the discussions, students spoke about the people who motivated and inspired them to stay in school. In the course of reports on all previous themes, the influence of the school environment on disengagement and dropout is evident. There was a great deal of concern by all participating groups regarding the family’s communication with the school, conditions that prompt children and teachers to stay at a school, and the need for a variety of educational options. While all emphasized that the lines of communication must be open for parents, students, and the school to work together, few could provide any positive examples. Most of the examples were negative in nature describing the problems and consequences of poor

Early Warning System  11  communication. In regard to perceptions of the community, little time in the focus groups examined this in particular. However, most comments conveyed the message that the community was not aware and did not understand why students are leaving school and was not doing anything to address the problem. During the focus groups, participants explored the issue of school attendance from multiple perspectives. Students described “skipping school” and being “kicked out” of class; parents were concerned about timely notification of absences; and all participants discussed the need for accountability. One other topic of concern was the enforcement of attendance policies and the potentially negative consequences for students with special circumstances. Taking Action. Students, parents, and teachers discussed the different programs that can lead to graduation, the need to “graduate on time,” and the pros and cons of graduation options (the usual high school diploma, the G.E.D. pathway, and the certificate of graduation). They also discussed the experience of transferring to an alternative program (specifically the Pathways program). All three groups agreed that the high school diploma was the most desirable of the three. According to one teacher, the high school certificate has a major impact on the community as well as the students because it limits their employment opportunities. Another teacher stated that it basically indicates, “I went to school for 12 years but I did not earn a high school diploma.” When comparing the G.E.D. to the diploma, one parent told her child, “Some places won’t take the G.E.D., so it is better for you to get your high school diploma. It will open more doors.” From the perspective of a student, programs to help students stay in school were described this way: “When we get funding we try to make the kid fit into the grant so we aren’t actually working on what needs to be worked on. The people who are giving the money or disbursing the money do not understand the population to which they are giving. Schools

Early Warning System  12  try to make the money fit in the programs but it doesn’t work” (Roush and Wilburn, 2008, p. 17). Methods and Procedures Guiding Questions Four questions guided our 6th grade analysis: 1. Are significant numbers of students showing the signs of disengagement previously identified in our examination of 9th grade students who fail to complete high school? 2. Can we identify a set of indicators that flag sixth graders who have high odds of failing to complete high school? 3. From this set of indicators, can we identify a practical set of indicators routinely collected and reported at the individual level that flag sixth graders who have odds of failing to continue in school until graduation? 4. Can we identify indicators within that set that may be influenced by school and community programs? Outcome Variable The outcome variable used was whether or not the sixth graders in the 2001–2002 cohort graduated from high school on time or after one extra year. We chose this outcome, in part, because of our desire to replicate the study reported by Balfanz, Herzog, Neild, and MacIver (2007) which previously identified a set of sixth grade predictor variables. In addition, we selected this variable because in the previous analysis of 9th grade students, the vast majority of students who earn a diploma did so on time or within one additional year. Therefore, extending the time frame would be of little significance. It is also important to note that we

Early Warning System  13  selected to use graduating from high school as opposed to dropped out of school since each student who receives a standard high school diploma is clearly identified with a high degree of accuracy in the district’s data system; however, students who fail to graduate can do so for a multitude of reasons (e.g., transfers, withdrawal, failing to report to school) that are not captured with a high degree of accuracy by the district’s data system. Data Source As with previous studies, the data used in this analysis were provided by the DCPS Department of Instructional Research and Accountability. The data were gathered from the district’s data warehouse by a district staff member and provided to the research team in the form of several Excel spreadsheets. The data included selected demographic, academic, attendance, and disciplinary information from each student enrolled in grades 6–12 during the 2001–2002 through the 2007–2008 school years. Sample and Measures Since the focus of this analysis was 6th grade students, we began by identifying all 6th grade students for the 2001–2002 school year and then tracking that cohort of students through each school year including 12th grade (2007–2008 school year). We created a longitudinal dataset designed to follow the performance of students enrolled in Grade 6 during the 20012002 school year. The dataset included attendance, demographic components, math and reading course grades, and math and reading standardized test scores per year. These students were traced through the 2007-2008 school year, the year during which students normally complete Grade 12 and graduate or finish a program. Those still enrolled as DCPS students at the end of the 2008 school year were categorized as either “Graduated with diploma,” “Received certification of completion,” or “Transferred/withdrew.” These categories were added to the dataset.

Early Warning System  14  In our data file 10,516 students were identified as being in Grade 6 for the 2001–2002 school year. From this initial population, 423 students were deleted due to incomplete data sets which resulted in a data set of 10,093 students for our study population. This sample of students in Grade 6 consisted of White (47.7%), African-American (44.3%), Hispanic (3.7%), Asian-American (2.8%), and Multiracial (1.3 %) students. Slightly more students were male (51.5%) than female (48.5%) and slightly more than eight percent (8.3%) of these sixth graders required services as English Language Learners. For the purpose of this study, students in the two most severe classifications of special education (those who were not expected to graduate or to earn a certificate of completion) were eliminated from the sample. The remaining ESE students (those who spent at least part of the school day in regular education classes) were handled as regular education students. No data about free or reduced lunch eligibility were available for about half the students in the Grade 6 population. Data for 5,015 of the students indicated that 94% qualified for free or reduced lunch (about half of the total Grade 6 sample). Of the complete study population (10,093), only about 1% was younger than expected in the Florida public schools, while 41.5% were of “normal” age for the grade level and 58.4% were over the expected age indicating that they had been retained at some earlier grade level. At the end of the 2008 school year, 66.7% (6,733) of the students graduated by 2008 while 33.3% (3,360) did not. Table 1 provides an overview of the study population in regard to the school related variables such as program participation, disciplinary infractions, course grades and standardized test scores. Table 1. School Related Variables of the Study Population Group

Drop-Out Program

Variable

% Enrolled in Dropout Prevention Program

Completers

12.4

Non-completers

28.2

Early Warning System  15  % Not Enrolled in a Drop-Out Prevention Program Discipline

87.6

71.8

6.4

11.3

93.6

88.7

Reading 2002 Mean Score

1721.67

1698.17

Math 2002 Mean Score

1724.63

1741.99

Reading 2003 Mean Score

1811.22

1784.85

Math 2003 Mean Score

1819.72

1826.28

Reading 2004 Mean Score

1880.48

1722.11

Math 2004 Mean Score

1907.80

1772.70

Reading 2005 Mean Score

1904.81

1698.39

Math 2005 Mean Score

1941.38

1793.39

Language Arts 2002 Mean Grade

2.57

1.98

Math 2002 Mean Grade

2.47

1.83

Language Arts 2003 Mean Grade

2.55

2.09

Math 2003 Mean Grade

2.42

1.86

Language Arts 2004 Mean Grade

2.48

2.11

Math 2004 Mean Grade

2.23

1.81

Language Arts 2005 Mean Grade

2.39

2.28

Math 2005 Mean Grade

2.10

1.87

% Serious Violation (Code 3/4) % No Serious Violation (Code 3/4)

Standardized Test Scores (FCAT Development Scale Score)

End of Course Grade GPA

As presented in Table 1, there are significant differences in academic related characteristics between those 6th grade students who graduated from high school and those who did not. For example, for those students who were enrolled in a dropout prevention program, only 12.4% graduated high school while 87.6% of those who were not in the program earned their diploma. Among students who committed a serious conduct violation, a much larger percentage did not complete high school. There were also significant differences between completers and non-completers on the Developmental Scale Scores of the Florida

Early Warning System  16  Comprehensive Assessment Test (FCAT). When compared over the four years students were tracked, those who completed high school had higher reading scores each of the four years and higher math scores two of the years. While the differences were not always great enough to be significant, the final course grade point averages (GPA) in language arts and mathematics were also higher for high school completers for each of the four years. Data Analysis Procedures Once the data set had been cleaned of cases with missing data or incorrectly coded data, an initial factor analysis was conducted on the 23 different variables available in the Duval dataset to determine whether variables fell into coherent categories (i.e., factors) as predictors of completion. One of the reasons for beginning with this data reduction procedure was to determine if one piece of data, such as attendance or mathematics standardized test scores, had enough weight to serve as a valid predictor or would combinations of variables serve as stronger predictors. The process of combining various data sets was repeated until a consistent set of categories was identified. Once a coherent set of predictors was identified, linear regression (stepwise) analyses were conducted on each identified factor in order to estimate the strength of each factor. The factor analysis yielded eight variables strong enough to be included in the regression analysis. These are as follows: 1. End-of-sixth-grade standardized test scores in reading and mathematics were used. Developmental scores on the Florida Comprehensive Achievement Test (FCAT) were converted to one of five individual levels in each student record, and the five levels were employed in statistical testing.

Early Warning System  17  2. Although the nature of the courses varied from student to student, end-of-sixth-grade course grades in English and math were included. Letter grades were converted to numeric equivalents for the purpose of statistical testing. 3. Students who had incurred serious infractions of the conduct code (major violations or violations of zero tolerance policies) were identified and labeled dichotomously from students who had not incurred such infractions. 4. Absences during Grade 6 were calculated and included. No distinctions were made among reasons for absence: a simple total of days missed was initially used. Based on indications from previous studies, in subsequent analyses, absences were divided into two categories: less than 20% of the school days in 2001-2002 and more than 20% of the school days that year. 5. Graduation status for each year up to and including the routine six years of high school were aggregated and used as a single variable. Students were designated as “Graduated with diploma,” “Received certification of completion,” or “Transferred/withdrew” by the end of the 2007-2008 academic year. 6. Participation in dropout prevention programs during Grade 6 only was included. Duval County maintains seven different prevention programs: educational alternatives, dropout retrieval, disciplinary, alternatives to expulsion, teen pregnancy, Department of Juvenile Justice, and neglected or delinquent. Students were labeled by the specific program in which they participated or as non-participants in any of these programs. 7. Demographic variables such as race, gender, and date of birth were included. Dates of birth were subsequently converted to age in months and then divided into three categories: under expected age, normal expected age, and over expected age. 8. The need for services as English Language Learners was identified and coded dichotomously (services/no services) into the dataset.

Early Warning System  18  Findings Identifying Warning Flags As we worked through the identification procedure, it became apparent that the most valuable warning flags from an educational standpoint would be those that could be impacted by the school and/ or community based educational programs. These are (1) measures of academic learning, (2) school attendance, (3) student behavior, and (4) program placement. Consequently, some demographic variables (e.g., race, gender, primary language) were dropped from further analysis. Table 2 shows the yield of each of the eight preliminary predictors. Seven of these independent flags met our test of being within the influence of the educational program and accounting for at least 10% of the students who failed to graduate, but one did not. (1) Over-age for 6th grade (2) Earned a low FCAT score (Level 1 or 2) in math (3) Earned a low FCAT score (Level 1 or 2) in reading (4) Failed a 6th grade math course (5) Failed a 6th grade English course (6) Was enrolled in a Dropout Prevention Program (7) Absence from school more than 20% of the time (8) Receiving a serious (Code 3 or 4) disciplinary referral (This flag only identified 6.7% of the non-graduates. It is possible that this flag has some value but failed to meet the two pronged test because of the low number of students in the overall population who met

Early Warning System  19 

Table 2. Graduate and Non-graduate Percentages Identified by Preliminary Flags DPP Failed 6th Absence VCode Low Low Over>20% FCAT FCAT age for grade th Math Reading 6 Grade math course Graduated 158 96 4099 3157 3470 341 1097 on time Did not 409 207 2313 1878 2423 517 1016 graduate 12.2 6.7 68.8 55.9 72.1 15.4 30.2 Yield: % of nongraduates flagged

Failed 6th grade English course 860 946 28.1

N = 10,093 6th grade students/3,360 Non-graduates

One of the lessons learned from our qualitative analysis of interviews with students who had dropped out of school was that often there was no single factor that prompted students to leave school. As Balfanz has reported, “examining the occurrence of multiple flags and their impact on students’ graduation chances provides additional insight into the process and impact of student disengagement at the start of the middle grades” (Balfanz, 2007, p. 229). With this in mind, in our next step we set out to determine how our predictors might be used in combinations to better flag potential 6th grade non-graduates. Using the three strongest flags (i.e., being over age, FCAT scores, and course grades) we reanalyzed the data to examine the power in identifying non-graduating students when using two factors. These results are presented in Table 3. Table 3. Double Warning Flag Combinations and Counts for all Duval 6th Grade Students in 2001–2002 Who Failed to Graduate in 2007-2008 Risk Category FCAT Scores

Did not graduate

n

Valid %

Early Warning System  20  Low FCAT Math + Overage

1753

52.17

Low FCAT Math + Low FCAT Reading

1676

49.88

Low FCAT Reading + Overage

1548

46.07

Failed Math Course + Low FCAT Math

1638

48.75

Failed Math Course + Overage

1618

48.15

Failed English Course + Overage

1349

40.15

Failed English Course + Low FCAT Math

1340

39.88

Failed Math Course + Low FCAT Reading

1313

39.08

Failed English Course + Low FCAT Reading

1115

33.18

Failed Math Course + DPP

415

12.35

Failed English Course + DPP

378

11.25

Absence>20%+Older

381

11.34

Absence>20% + Failed English Course

358

10.65

Course Grades

Absence

Note: Yield < 10% for all other combinations

Attendance Our analysis confirmed the information previously reported by Balfanz (2007) that attending school less than 80% of the time increases the chance that students will not complete high school. In our study population, approximately 11% of the 6th grade students who failed to graduate were absent from school for more than 36 days. However, as reported in our previous study of 9th grade students (Wilburn and Weaver, 2007), attendance was not as strongly associated with failing to graduate as academic factors. While useful, in combination with other factors, independently attendance does not provide a meaningful yield as a predictor variable.

Early Warning System  21  Standardized Test Scores Consistent with findings from other studies, academic achievement identified the highest number of 6th grade students who failed to graduate on time (Balfanz & Boccanfuso, 2007; Balfanz 2007). However, unlike other studies, in this case standardized test scores were the most reliable predictor of failure to graduate. When used independently, having a low (Level 1 and Level 2) Florida Comprehensive Assessment Test score in Reading or Math accounted for approximately 62% percent of the 6th grade students who failed to graduate. While being overage identified the highest number of future non-graduates among the 6th grade study population, in most cases the reason that a student is overage is because that have been retained for failing to make an acceptable score on the FCAT. Consequently, being over-age may simply be a proxy for poor FCAT performance over time. In addition, it takes more than one year to be “overage” while FCAT score are available on a year to yearly basis. Course Grades In our work with the data provided by the school district for our descriptive study of 9th grade graduation rates and this study, we found many inconsistencies in the course grade data sets. In our efforts to identify early warning flags for 6th grade non-completers, we found the data to be so inconsistent that we initially removed the course grade variable form our analysis. However, because it is information that is readily available, we have included it in this analysis. When taken independently, failing one 6th grade math or English course identified approximately 36.2% to 28.1%, respectively, of the students who eventually failed to complete high school. Because of inconsistency in course grades from year to year, we believe that standardized test scores provide the most valid and reliable predictor of high school completion for sixth grade students, and we strongly recommend caution in using of course grades as an independent warning flag for identifying students. This point may be illustrated by the fact that course grade

Early Warning System  22  is the only academic factor that is stronger when combined with other factors. For example when failing a math course is combined with the low FCAT math score factor, the number of students identified increases from a math course alone yield of 30.2% to combined yield of 48.7%. A similar gain is also realized when the two factors of a failed English course and a low FCAT reading score are combined. For this reason we do not recommend using course grades as an independent predictor. Discipline The discipline codes used in this study were those that represented the most serious (i.e., Class 3 and 4) offences that a student could receive. For example, students’ behaviors that involve physical assault, weapons and the use and/or possession of illegal drugs fall into these categories. In the vast majority of cases, when a student receives this type of disciplinary referral, he/she is automatically suspended from school and/or removed from their home school and assigned to one of the District’s alternative schools for a period of 45 to 90 days. Consequently, the number of Class 3 and 4 Code of Conduct violations occurring at the 6th grade was not very large. As a result, this flag did not identify a large number of the 6th graders who eventually failed to complete high school. From our work in tracking the 6th grade student as well as our previous analysis of 9th grade students who failed to graduate, we know that the number of Class 3 and 4 violations increase significantly as students get older and as more students become over-age for their assigned grade. Consequently, the conduct violations from subsequent years have more predictive power than those from Grade 6. While the yield for this factor was relatively small (6.7%) as a single factor and no greater when considered in combination with test scores or grades, we nevertheless consider this to be an important flag to monitor. Status Variables (Dropout Prevention Program/Over-age)

Early Warning System  23  Being either in the District’s Dropout Prevention Program (DPP) or over-age for the 6th grade significantly reduces the probability that a student will successfully complete high school. Our analysis indicates that of the 858 6th grade students enrolled in one or more of the District’s Drop Out Prevention Programs, 341 (40%) graduated from high school on time while 60% failed to complete high school. Obviously, for a student to be placed in one of these programs as early as the 6th grade, he/she exhibits a number of our warning flags such as poor attendance, low FCAT scores, and poor course grades. Consequently, the probability for completing high school is low (i.e., 4 out of 10) for this student group. While the value of this factor as an early warning flag for identifying large numbers of 6th grade students is limited, 15% of the 6th graders who failed to complete high school were in the DPP. We believe it would be useful to examine the program as to a better understanding of what initiatives may be successful in working with high risk students. On the other hand, being over-age for sixth grade appears to account for a large number on non-high school completers. In our study population, 72% of the 6th grade students who were over-age failed to complete high school. While this initially appears to be a strong flag for identifying those who will not graduate from high school, as indicated in Balfanz’s 2007 study, this is primarily because a high percentage of over-age students scored at Level 1 or 2 on the FCAT, failed a 6th grade math or English course, and attended less than 80% of the time. The few over-age students who did not exhibit any of the other flags tend to graduate at the same rate as other 6th grade students. Consequently, being over-age is not the issue. Final Selection of Identification Flags Our final action in the identification of variables (i.e., predictive flags) was to narrow the number of flags to those that independently or in combination with other flags provided the strongest practical tool for early identification of 6th grade students who had the highest

Early Warning System  24  probability of not graduating from high school. As discussed above, some of the variables, such as being over-age and/or enrolled in a dropout prevention program, were actually products of other variables such as poor academic achievement or attendance. While a case could be made for including the discipline code flag in the final selection as it seems to be such a reliable predictor in later grades, it also was very duplicative in that over 90% of the 6th grade students who had a Class 3 or 4 code of conduct violation also had one of the other flags such as poor attendance or failing course grades. Therefore, we limited our final selection of flags to five school related factors: 1. FCAT math scores 2. FCAT reading scores 3. Failing a 6th grade math class 4. Failing a 6th grade English class 5. Greater than 20% days absent from school By using these five factors and eliminating duplicate records, we were able to identify approximately 82% of the students who failed to graduate. Validity of the Flags In order to estimate the validity of our final five flags while controlling for the other flags, we conducted a multivariate logistic regression to estimate the predictive power of each flag. The analysis showed that, all else being equal, 6th grade students who missed more than 20% of school attendance days were 6.07 times more likely not to graduate than students who missed less than 20% of school attendance days (confidence interval (CI) = 5.03 to 7.32). Students who failed English/Language Arts in the 6th grade were 2.86 times more likely not to

Early Warning System  25  graduate than students who passed Language Arts (CI = 2.15 to 3.17). Sixth grade students who failed a math course were 2.38 times more likely not to graduate than those who passed their math courses (CI = 2.15 to 2.62). Those 6th grade students who scored below 3 on the FCAT reading test were 1.62 times more likely not to graduate than students who scored 3 or above (CI 1.49 to 1.77). Finally, those 6th grade students who scored below 3 on the FCAT math test were 1.71 times more likely not to graduate than those who scored 3 or above (CI = 1.56 – 1.88). Each flag was a statistically significant flag (p< .05) even after controlling for the other flags. Our five flags, individually and in combination, identified 81.52% of the nongraduates. Among the 10,093 total study population, 49% (4,975) graduated with none of the five flags while 17% (1,706) had no flags but still did not graduate. Limitations of the Study It is important to recognize that, as with all investigations of this type, these numbers provide a description of the 2001-2002 6th graders who had not graduated by the end of 2008. In the course of our data analysis, we conducted both linear and logistic regressions with the common wisdom being that a logistic regression is the better test to measure predictors on a dichotomous variable, resulting in the graduation variable being yes/no on graduation. Though none of the 4,975 students in our 10,093 member 6th grade cohort had any of the five flags and 82% of the non-graduates had one or more of the flags, it is not valid to state that the combination of factors has a predictive power. The regression calculates mathematical possibilities, not actual counts. This is evidenced by the fact that 1,706 of the 6th graders had no flag but had not graduated by the end of the 2008 school year. In generalizing our results to other populations, it is also important to note that we did not control for race in the regressions that we ran. Since our 6th grade population was divided so evenly between Black and White, there seemed no need to do so. It is common practice to only control for such a factor if there are large imbalances inside the group. Consequently, caution

Early Warning System  26  should be exercised when applying our results to populations that have racial imbalances within the study cohort.

Summary Purpose The primary purpose of this study was to identify specific academic and social factors that may be used as “flags” to identify DCPS 6th grade students who are at high risk of not completing high school. A secondary purpose was to understand how these factors may or may not be aligned with those identified by Neal, Balfanz, and other researchers (Neal and Balfanz, 2006).

Data The data used in this analysis were provided by the DCPS Department of Instructional Research and Accountability. The data were gathered from the district’s data warehouse and provided to the research team in the form of several Excel spreadsheets. The data included selected demographic, academic, attendance, and disciplinary information from each student enrolled in grades 6–12 during the 2001–2002 through the 2007–2008 school years. Approximately 60,000–65,000 students were enrolled in grades 6-12 during each school year with each grade’s enrollment averaging between 10,000–12,000 students. Since the focus of this analysis was 6th grade students, we began by identifying all 6th grade students for the 2001–2002 school year and then tracking that cohort of students through each school year through the 12th grade (2007-2008). Ten thousand ninety-three (10,093) students

Early Warning System  27  remained for inclusion in the analysis after the elimination of withdrawals, transfers, and incomplete data sets.

Early Warning System  28 

Methods and Procedures Once the data set had been cleaned of cases with missing data or incorrectly coded data, an initial factor analysis was conducted to determine whether variables fell into coherent categories (i.e., factors) as predictors of completion. No one piece of data, such as attendance or mathematics standardized test scores, has enough weight to serve as a valid predictor. Therefore, this process of combining various data sets was repeated until a consistent set of categories was identified. Once a coherent set of predictors was identified, linear regression (stepwise) and logistic regression analyses were conducted on each identified factor in order to estimate the strength of each factor. Four questions guided our 6th grade analysis: 1.

Are significant numbers of students showing the signs of disengagement previously identified in our examination of 9th grade students who fail to complete high school?

2.

Can we identify a set of indicators that flag sixth graders who have high odds of failing to complete high school?

3. From this set of indicators, can we identify a practical set of indicators routinely collected and reported at the individual level that flag sixth graders who have odds of failing to continue in school until graduation? 4. Can we identify indicators within that set that may be influenced by school and community programs?

Results A significant number of 6th grade students displayed the signs of disengagement, such as poor academic achievement and excessive absences, previously identified in our study of 9th

Early Warning System  29  grade students and in studies conducted by other researchers. Five factors proved to be of greatest value in identifying members of the 2001-2002 6th grade cohort who did not graduate on-time from high school in 2007-2008. These factors were: ƒ

Standardized Math Test Scores: 69% of the 6th grade students who failed to graduate on-time scored below Level 3 on the Florida Comprehensive Assessment Test.

ƒ

Standardized Reading Test Scores: 56% of the 6th grade students who failed to graduate on-time scored below Level 3 on the Florida Comprehensive Assessment Test.

ƒ

Failing Course Grades in Mathematics: 30.2% of the 6th grade students who failed to graduate on-time failed a math course.

ƒ

Failing Course Grades in English/Language Arts: 28.1% of the 6th grade students who failed to graduate on-time failed an English/Language Arts course.

ƒ

Excessive Absences: 12.2% of the 6th grade students who failed to graduate on-time missed more the 20% of the scheduled school days.

The number of Class 3 and 4 Code of Conduct violations occurring at the 6th grade among the study population was not very large (3%). As a result, this flag only identified 6.7% of the 6th grader who failed to graduate on-time. However, our tracking of students indicated that the number of Class 3 and 4 violations increase significantly as students get older and as more students become over-age for their assigned grade. Consequently, the conduct violations from subsequent years have more predictive power than those from Grade 6. While the yield for this factor was relatively small, as a single factor and no greater when considered in combination with test scores or grades, we nevertheless consider this to be an important flag to monitor. Results from our study indicate: 1. Significant numbers of 6th grade students show signs of disengagement previously identified in our examination of 9th grade students who fail to complete high school.

Early Warning System  30  2. We can, with a reasonable degree of validity, identify a set of indicators that flag sixth graders who have high odds of failing to complete high school 3. From this set of flags, we can identify a practical set of indicators routinely collected and reported at the individual level that flag sixth graders who have odds of failing to continue in school until graduation 4. These indicators are factors that may be impacted by school and community programs.

In conclusion, as we have learned from our previous studies of first-time 9th grade students and focus groups with students who had previously dropped out of school, students do not graduate from high school for a multitude of reasons; and many of those reasons are not linked directly to academic achievement, attendance, and behavior at school. Even though the issues is complex and no one single factor or group of factors can account for every child’s decision to leave school, we believe that it would be unconscionable to ignore the early warning flags readily available to even the most casual observer. 

Early Warning System  31  References Balfanz, R., Hertzog, L., Neild, R. C. and Mac Iver, D. J. (2007). Preventing student disengagement and keeping students on the graduation path in urban middle-grades schools: Early identification and effective interventions. Educational Psychologist. 42(4), 223-235. Balfanz, R. (2006). Unfulfilled promise: The dimensions and characteristics of Philadelphia’s dropout crisis, 2000 – 2005. Philadelphia, PA: Philadelphia Youth Transition’s Collaborative. Roush, C., Wilburn, K. T. and Weaver, D.M. (January, 2009). Learning to finish focus group data: qualitative analysis report. Jacksonville, FL: The Community Foundation. StatSoft, Inc. (2007). Electronic Statistics Textbook. Tulsa, OK: StatSoft. WEB: http://www.statsoft.com/textbook/stathome.html. Wilburn, K. T. and Weaver, D. M. (November, 2008) Where Have All the 9th Graders Gone? A Descriptive Study of Three First-Time 9th Grade Student Cohorts. Jacksonville, Florida: The Community Foundation.

Early Warning System  32  Appendix Demographic Characteristics of the Study Population Group

Gender

Variable

Completers (%)

Non-completers (%)

Female

51.8

46.8

Male

48.2

53.2

3.6

2.6

45.2

43.4

Hispanic

3.9

4.0

American Indian

0.1

0.2

Multiracial

1.0

1.1

46.2

48.8

9.2

7.0

English Speaking

90.8

93.0

Socioeconomic Status Free Lunch (School Lunch Public Assistance Program)

23.3

20.5

5.1

6.7

Reduced Lunch

9.3

6.7

Not Eligible

2.7

1.5

59.6

64.5

Race/Ethic Group

Asian American Black

White Primary Language

Non-English Speaking

No Information