Study 1
Morrow-Howell and colleagues (2009) evaluated the impact of the Experience Corps (EC) program on the reading outcomes of children who participated in the program in three cities: Boston, Mass.; New York, N.Y.; and Port Arthur, Texas. All eight schools offering the EC program in Port Arthur participated in the evaluation. In New York, there were 16 EC schools, of which 6 participated in the evaluation. In Boston, eight of the nine schools offering the EC program participated in the evaluation.
At the beginning of the school year, teachers in the participating schools referred students in grades 1–3 who needed assistance with reading. After parental consent, each student was randomly assigned by a lottery system to either the program (n = 434) or a control group (n = 454). The participating students were 51 percent male and 49 percent female. Most students were African American (58 percent), followed by students of Hispanic origin (36 percent), followed by students of other race/ethnicity (6 percent). Close to one quarter of the students had limited English proficiency, and 14 percent had special education needs. The proportion of students across the grades who participated in the program was fairly comparable: 41 percent were in 1st grade, 36 percent were in 2nd grade, and 23 percent were in 3rd grade. All study participants were pretested in the first 2 months of the school year. Posttest data was collected beginning at 1 month before the end of the school year.
The students in the program group were assigned tutors who engaged them in one-to-one tutoring sessions and provided mentoring relationships over the course of a school year. The control group did not participate in the Experience Corps program but were not limited in receiving other available support services.
Standardized reading tests were used to assess various reading skills of the participating children. The tests included the Woodcock-Johnson Word Attack Subscale (WJ-WA), Woodcock-Johnson Passage Comprehension Subscale (WJ-PC), and the Peabody Picture Vocabulary Test (PPVT-III). The standardized reading tests were administered individually during a 30-minute, face-to-face interview session with the children by research staff. In addition, 10 grade-specific reading skills were assessed by the students’ teachers using a task-specific measure developed for the evaluation.
Independent sample t-tests were conducted to test for baseline equivalency of the program and the control groups on the outcomes. The reading abilities of the two groups at baseline were equivalent. The two groups also did not differ significantly on demographics and other variables such as school absences and classroom behavior. The impact of the EC program on each outcome was estimated by comparing the posttest scores for the EC and the control groups, after controlling for pretest scores on the outcome and other covariates such as gender, ethnicity, grade, program site, and classroom behavior. Further, to account for clustering within classrooms and schools, the Generalized Estimating Equation method was used.
Thirty students from the program group and 29 from the control group dropped out of the study after pretests. This attrition was equally distributed between the groups, and those who completed the posttests did not differ from those who dropped out, in terms of major demographic characteristics. Missing data due to student attrition and incomplete surveys was imputed using a Markov Chain Monte Carlo multiple imputation method. Students whose dates of birth were incorrect or missing at pretest were excluded from the analyses after imputations. Data was analyzed for 430 students in the program group and 451 from the control group. Effect sizes were calculated using Hedges’ g.