Study 1
Merrill (2020) used a quasi-experimental design to investigate the effectiveness of iMentor’s College Ready Program in a sample of about 1,600 students from two consecutive cohorts of incoming ninth graders in the 2012–13 and 2013–14 school years at eight New York City high schools. To be eligible, schools needed to serve low-income students and have a principal who was committed to the program. Compared with other New York City high schools, iMentor schools were more likely to serve Latino students and were less likely to serve White or Asian students. They also had more English language learners and students from low-income families. In addition, students in the iMentor schools scored somewhat lower on the New York state English and language arts and mathematics assessments and were more likely to be chronically absent, compared with students in other New York City high schools.
All incoming ninth graders at schools where iMentor was implemented were eligible to participate. Treatment group participants participated in the iMentor program, while comparison group students attended school as usual and had access to alternative college planning resources.
Two sets of comparison groups were used to assess student outcomes. The first group served as a comparison to assess social and emotional learning (SEL) and future planning outcomes, using annual surveys. Students who began ninth grade in the year before iMentor’s implementation served as the comparison group. The initial sample included 1,678 students who were on each school’s ninth grade roster as of Oct. 20, 2012 (for comparison schools), and Oct. 20, 2013 (for treatment schools). Of these students, 1,438 responded to the posttest survey in the spring of their senior year. At pretest, the treatment group (n = 1,177) was 45.6 percent female, 3.0 percent Asian, 32.2 percent Black, 62.2 percent Latino, 1.3 percent White, and 1.3 percent Other. Of the treatment youths, 22.9 percent exhibited chronic absenteeism in eighth grade, 35.4 percent were overage as of ninth grade (i.e., 15 years or older), 88.9 percent received free/reduced lunch, 28.3 percent were English language learners, and 21 percent received special education in eighth grade. The comparison group (n = 501) was 52.3 percent female, 47.7 percent male, 2.8 percent Asian, 34.8 percent Black, 59.1 percent Latino, 1.1 percent White, and 2.0 percent Other; 27.2 percent exhibited chronic absenteeism in eighth grade, 36.4 percent were overage as of ninth grade, 85.2 percent received free/reduced lunch in eighth grade, 22.9 percent were English language learners in eighth grade, and 21.9 percent received special education in eighth grade. There were no statistically significant differences between iMentor treatment group students and comparison group students on demographics, nor on their average eighth grade New York state math test and English and language arts test scores.
The student survey contained more 100 items, including measures of SEL and background demographics. Task persistence, defined as a student’s ability to sustain effort despite setbacks, was measured using eight items on a four-point scale, from strongly disagree (1) to strongly agree (4) [Walker and Arbreton 2004]. Critical Thinking, which measured problem-solving abilities, consisted of five items rated on a five-point scale from never (1) to always (5) [D’Zurilla and Maydeu–Olivares 1995]. Growth mindset, meaning one’s belief that intelligence and performance improve with effort, was measured using three items on a four-point scale, from strongly disagree (1) to strongly agree (4) [Richards, Ellis, and Neill 2002]. Help seeking, referring to the extent to which students will ask for help in class, was assessed with five items developed for the study on a five-point scale, from never (1) to always (5). Finally, self-advocacy, or the degree to which students promote themselves and their competencies, was measured using four items on a five-point scale, from never (1) to always (5) [Bolino and Turnley 1999].
Program effects on outcomes were assessed using a student-level fixed-effects model that controlled for school-fixed effects, baseline scores on the outcome measure, demographics, pre–high school characteristics (including being overage in ninth grade), and baseline scores on a measure of self-efficacy.
A separate comparison group was used to assess graduating with any degree and attending any college using administrative records provided by the New York Department of Education in spring 2018, the treatment group’s senior year. Sixteen comparison high schools were chosen from a pool of 356 potential matches that had a single admission method (i.e., screened or limited unscreened, not both), and were not a special education, charter, or specialized high school. Students from these schools who entered ninth grade in 2013, the same year the iMentor program was implemented, served as the comparison group. The treatment group (n = 764) was 51.2 percent female, 2.8 percent Asian, 34.8 percent Black, 59.7 percent Latino, 1.1 percent White and 2.0 percent Other. Average middle school attendance was 87.7 percent, 36.4 percent were overage as of ninth grade, 92.44 percent received free/reduced lunch, 24.9 percent were English language learners, and 17.6 percent received special education in eighth grade. The comparison group (n = 1,571) was 57.3 percent female, 42.7 percent male, 6.3 percent Asian, 28.5 percent Black, 60.6 percent Latino, 1.3 percent White, and 1.5 percent Other. Average eighth grade attendance was 88.2 percent, 31.9 percent were overage as of ninth grade, 92.3 percent received free/reduced lunch in eighth grade, 18.1 percent were English language learners in eighth grade, and 18.4 percent received special education in eighth grade. There were no statistically significant differences between the iMentor treatment group students and comparison group students in demographics or in their average eighth grade New York state math test and English test scores.
A comparative interrupted time-series analysis was used to assess program impacts on graduating with any degree (i.e., a Regents Degree, or a Local Degree that requires a lower score on the New York state Regents Exam) and attending any college (i.e., a 2- or a 4-year college), controlling for school, school year, demographics, and pre–high school characteristics. The models estimated trends in these outcomes in the years leading up to the start of iMentor and deviations from those trends after iMentor implementation began. In the years before the intervention, treatment and comparison schools did not statistically significantly differ in their proportion of “on track” students (defined as passing at least one Regents Exam and earning at least 10 course credits). During the pre-intervention period, students across the two groups of schools also did not statistically significantly differ in weighted grade-point averages, chronic absenteeism, having passed one Regents Exam, credits earned, academic credits earned, or attendance.
The study conducted subgroup analyses to investigate whether students and schools who met program participation goals differed in their outcomes from those who did not and whether students who reported feeling very close to their mentor differed in their outcomes from students who rated their relationships as less close.
