Implementation Fidelity and Outcomes in School-Based Interventions

Background: Most children with autism in the United States receive the bulk of their treatment through publicly-funded schools. However, most evidence-based practices (EBPs) for children with autism were developed and tested in highly controlled laboratory settings, with very high rates of intervention fidelity. Interventions delivered in the community often are delivered with much lower fidelity. While there is general consensus that intervention fidelity is associated with student outcome, there is little research examining this association in community settings.

Objectives: Using data from a large randomized field trial in an urban public school district of several EBPs for elementary school-age children with autism, we will: 1) evaluate teachers’ mean and variation in treatment fidelity; 2) estimate the association between treatment fidelity and child outcomes; and 3) determine factors associated with treatment fidelity.

Methods: Direct observation fidelity measures were used to assess implementation fidelity to three EBPs (discrete trial training (DT), pivotal response training (PRT), and visual schedules (VS)) bi-monthly in 73 classrooms for an academic year (n = 73 teachers and 126 children). Implementation fidelity was measured along three dimensions: intensity (how often the teacher delivered the intervention), accuracy (whether the intervention was delivered as described in the manual), and a composite fidelity score that comprised the product of intensity and accuracy. Child outcome was measured as change in cognitive ability using the Differential Abilities Scales, 2^nd edition (DAS-II) and overall performance on the Bracken Basic Concepts Scales. Differences in child outcome were estimated using linear regression models with random effects for classroom. Separate models were used where the independent variables of interest in turn included intervention intensity, accuracy, and the composite fidelity score. Qualitative interviews were conducted with a sample of the teachers to identify barriers and facilitators to treatment fidelity.

Results: Data are collected. Analyses are ongoing. Teachers’ fidelity to all three EBPs on a scale ranging from 0-4 was low and varied substantially across teachers (DT accuracy- Mean: 2.3, SD: 1.3; PRT accuracy- Mean: 2.0, SD: 1.3; VS accuracy- Mean: 1.6, SD: 1.2). There was substantial variation in child outcomes for both the DAS (Mean change: 4.0, SD: 7.8) and the Bracken (Receptive Mean Change: 16.4, SD: 22.4; Expressive Mean Change: 11.5, SD: 10.2). The associations of fidelity to each of the intervention components and child outcome, as well as factors associated with treatment fidelity, are being estimated now.

Conclusions: Implementation fidelity to EBPs in public schools is significantly lower than that observed in university-based research settings, and varies substantially across teachers. Many challenges exist to implementing EBPs in public schools including limited resources, training and highly complicated interventions. Strategies to improve the implementation of EBPs in public schools are urgently needed in order to improve access to high quality care and will be discussed. Additionally, students demonstrated higher gains on a measure of overall academic performance than in cognitive ability; academic measures may offer a more sensitive alternative, especially for school-age children, to measures of cognitive ability traditionally used in autism treatment trials.