Study
Heppen and colleagues (2017) used a 3-year randomized controlled study to evaluate the Check & Connect program in a sample of grade 9 students at risk of not graduating on time. The study was conducted in 10 comprehensive high schools within a large urban district. Students in the study schools were 48 percent Hispanic, 21 percent white, 15 percent Asian, and 14 percent African American. Of the total sample, 21 percent were English language learners, and 67 percent were eligible for free or reduced-price lunches. Potential participants were identified based on their probability of on-time graduation as determined by the following risk factors: 1) absence for 10 percent or more of enrolled days in grade 9; 2) failing grade in at least one course in grade 9; 3) failing grade in Algebra I in grades 8 or 9; and 4) at least one “Needs Improvement” or “Unsatisfactory” citizenship grade (indicating general behavior and participation in class) in grade 9 courses. One hundred to 150 of the students who entered grade 9 in 2010–2011 and had the lowest predicted probabilities of on-time graduation were identified in each of the participating schools for participation in the study.
Five-hundred and fifty-three eligible students agreed to participate in the study and were randomly assigned to the intervention (n = 276) and control (n = 277) groups. More than 30 percent of students in the sample were current English language learners, about 34 percent were previous English language learners (met state criteria to be reclassified as fluent English proficient prior to grade 9), and 72 percent were Hispanic. About 23 percent of students were absent 10 percent or more of enrolled days in grade 9, about 98 percent had a failing grade in at least one course in grade 9, about 80 percent had a failing grade in Algebra I, and about 90 percent had at least one “Needs Improvement” or “Unsatisfactory” citizenship grade in grade 9. The average predicted probability of on-time graduation was 0.55. There were no statistically significant differences between students in the intervention and control groups on any of these risk factors.
Outcome measures were sourced from district administrative data and student surveys in the spring of 2012, 2013, and 2014. District administrative data provided information on student demographic characteristics, academic backgrounds (e.g., prior attendance, behavior and course performance grades, achievement test scores), enrollment in each academic year, attendance, course-level transcript data, exit exam status (i.e., pass/fail), graduation date, and diploma type. Dropout, on-time graduation, and 5-year graduation information was also collected from the state data system to capture students who had left the district (but remained in a public high school within the state) prior to the end of the study. Student surveys measured student engagement in social and academic activities. The Student Engagement Instrument (SEI), a 33-item scale, was used to measure affective engagement (teacher–student relationships, peer support for learning, and family support for learning) and cognitive engagement (control and relevance of school work and future aspirations and goals). The 12-item Student Engagement Questionnaire (SEQ-C) was also used to measure student engagement in the school, reflecting perceived relevance of schoolwork, teacher–student relationships, teacher expectations, and satisfaction with school. In addition, students were asked to report on their involvement in targeted academic supports, including tutoring, online credit recovery programs, and college preparatory programs, as a measure of the extent to which students in the control group accessed interventions similar to those provided to students in the intervention group.
Effects of the program on outcomes were evaluated using regression analyses with an intent-to-treat (ITT) framework. Analyses controlled for baseline student demographic and academic background characteristics. Analyses also controlled for attrition and nonresponse by applying inverse probability weights, which are based on the probabilities of having missing data.
Study
Guryan and colleagues (2017) used a randomized controlled study to evaluate the Check & Connect program among two cohorts of Chicago Public Schools (CPS) students; each cohort was followed for 2 school years. Sixty-nine schools serving students in grades K–8 were selected for participation, based on their representativeness of the district in terms of student demographic and socioeconomic characteristics, and on the availability of large enough groups of students in each grade with 10 to 35 absences in the previous year. These schools were placed in three groups based on geographic location, student race and ethnic demographics, and school-level absence rates. One school from within each group of three was randomly selected to be part of the first study cohort. Within these 23 schools, five grades out of seven (grades 1 to 7) were randomly selected to be in the intervention; the remaining two grades formed the control group. Students within the five grades selected for participation were then placed in groups of three, were matched based on baseline absences, and one of the three students in each group was randomly selected to be in the intervention group. Eligible students within each school and grade were then sorted in descending order by baseline absences and were offered the chance to participate in the intervention. In cohort 2, nine schools with nine returning mentors were selected, and students within each school were placed in one of the following five randomization blocks: 1) students who were assigned to the intervention group in cohort 1 (those selected from this group would receive the intervention for 4 years); 2) students who were assigned to the control group in cohort 1; 3) students who were in grades 1 and 2 in the first year of cohort 2 (since no students from cohort 1 were in these grades at start of cohort 2); 4) students who were new to the school since the randomization for cohort 1; and 5) students who were nominated for the study by their principals. Students in the control group were 1) those from a school randomly selected to be a control group, in a grade randomly selected to be a control grade within an intervention group; or 2) randomly selected to be a control within an intervention grade in an intervention school.
In cohort 1, 933 students were assigned to the intervention group, and 2,958 were assigned to the control group. At baseline, the mean ages of cohort 1 students in the intervention and control groups were 9.0 and 8.7 years, respectively. Fifty-three percent of students in both the intervention and control groups were male. Additionally, 59 percent of students in the intervention group and 57 percent of students in the control group were Black, followed by Hispanic (38 percent and 40 percent, respectively). During the year prior to randomization, students in the intervention and control groups attended 151.0 and 150.3 days out of a possible 170 days of school; this difference was statistically significant. At baseline, cohort 1 students in the intervention and control groups did not differ significantly on a number of demographic and outcome measures, including age, gender, race/ethnicity, learning disability, course failures, GPA, or days absent. In cohort 2, 1,038 students were assigned to the intervention group, and 1,111 were assigned to the control group. At baseline, the mean ages of students in the intervention and control groups were 8.4 and 8.5 years, respectively., In terms of race/ethnicity, 47 percent of intervention group and 46 percent of the control group were Black, followed by Hispanic (47 percent and 49 percent, respectively). During the year prior to randomization, students in the intervention and control groups attended 159.9 and 159.2 days out of a possible 170 days of school. Cohort 2 students in the intervention and control groups did not differ significantly on any of the baseline demographic and outcome measures assessed. Fifty-two percent of students assigned to the intervention group in cohort 1 and 33 percent of students assigned to the intervention group in cohort 2 participated in the program.
Student demographic, attendance, and achievement data was collected from CPS administrative records. Demographic data included birth date, race/ethnicity, eligibility for free and reduced-price lunch, and presence of an Individualized Education Plan (as an indicator for having a learning disability). Measures of attendance and absences included data on days absent (the percentage of enrolled days a student was absent over the school year) and days present (the percentage of enrolled days a student was present over the school year). Measures of achievement were annual GPA and math and reading test scores. Since CPS administered multiple tests over the study period, the study used scores from the Illinois Standards Achievement Test (ISAT), which was administered among students in the third through eighth grades annually, and the Measure of Academic Progress (MAP), which was administered to students in third through eighth grades and to students in second grade in the last 2 study years.
Regression analyses were used to evaluate the effects of the intervention within an intent-to-treat framework, adjusting for baseline demographic and outcome measures, including gender, race/ethnicity, age, presence of a learning disability, days present, days absent, GPA, and course failures. Randomization block fixed effects analysis was used for cohort 2 data to adjust for the unequal probability of selection into the intervention group across the five blocks of students. Estimated effects of the program on outcomes were pooled across both program years and the two cohorts of students.
