Examining the Fidelity of Implementation of Comprehensive Treatment Models for Preschoolers with ASD

Background: Measuring the implementation of comprehensive treatment model (CTM) components with a well-designed and valid tool is critical in efficacy research (Hume et al. 2011), yet few intervention studies collect and report on the psychometric properties of their implementation measures. Utilizing validated implementation measures in the context of intervention research may produce more robust associations between fidelity of implementation and participant outcomes.

Objectives:   (1) To describe fidelity of implementation of critical components of the CTMs, and (2) To examine the reliability and validity of the instruments with study participants.

Methods:   Research staff across four states were trained to reliably score three fidelity measures used within the context of the study [i,e., LEAP, TEACCH, and business-as-usual (BAU) measures]. Four observations were then conducted across the school year in each of the 75 participating classrooms (N=25 TEACCH, 22=LEAP, 28=BAU), along with two reliability observations. During each classroom observation, all 3 fidelity measures were completed. The reliability of the measures was confirmed by examining test-retest reliability, internal consistency, and interrater agreement. Discriminant analysis was conducted to examine the subscales of each measure that most contributed to the ability of the fidelity measure to discriminate model types.

Results:

Descriptives: Average implementation of model components across the four observations was M= 83% for TEACCH (83-85% across 4 observations), M=92% for LEAP (91.7-92.4%), and M=83% for BAU (82-83%). Further analysis related to the overlap of implementation scores across measures is ongoing and will be reported.

Test-retest reliability for the measures across four observations was M= .81 for TEACCH (.56-.85 across 9 subscales), M=.81 for LEAP (.45-.88 across 8 subscales), and M=.69 for BAU  (.59-.82 across 8 subscales).

Inter-rater reliability was measured as the proportion of agreement between observers. Inter-rater reliability was 98% (TEACCH, 88-98%), 95% (LEAP, 88-100%), and 88% (BAU, 87-99%).

Internal consistency was examined using Cronbach’s alpha. For the TEACCH measure, the alpha was .95 (.59-.97), .96 for LEAP (.71-.95), and .93 for BAU (.68-.93).

Discriminant analyses were performed to identify the subscales of the fidelity measures that best discriminated between classroom types. On the TEACCH measure, three subscales-- social, visual schedules, work systems primarily contributed to the ability of the measure to discriminate between the 3 classroom types. On the LEAP measure social interaction best contributed to the discrimination between model types, and on the BAU measure, two subscales-- social/peer relations and classroom environment discriminated between the models.

Conclusions: Results indicate that CTM and BAU classroom staff were able to maintain high levels of implementation across the study period. In addition, when used with study classrooms, the measures remain psychometrically robust and continue to clearly discriminate between models. This study is one of the first in the field to systematically monitor intervention fidelity with reliable and valid tools, and implications for further analysis of this valuable data will be discussed.