Discriminant Analysis and Reliability Evaluation of Fidelity Measures for Comprehensive Treatment Models Serving Young Children with Autism Spectrum Disorders

Background:

Few comprehensive treatment models serving preschool aged children with ASD have well-developed measures of fidelity of implementation, and for the measures that do exist we know very little about their psychometric properties (i.e. reliability and validity). This is of great concern as fidelity tools must adequately measure the quality / degree of implementation to ensure successful adoption and replication of a model. Effective implementation of a comprehensive model, as measured by empirically validated fidelity tools, should have a direct impact on child outcomes.

Objectives:

This study empirically-validated treatment fidelity measures for two existing comprehensive treatment models-- Division TEACCH and LEAP, and a measure for non-model specific, eclectic classrooms serving children with ASD. The goals were to determine if these three instruments were reliable and could discriminate between preschool classrooms using TEACCH, LEAP, or an eclectic treatment approach (termed “Business as Usual or BAU” for study purposes).

Methods:

Research staff across three states was trained in the use of each fidelity measure and met criterion for inter-rater reliability. Generally, 4 observations were then conducted in 11 TEACCH, 10 LEAP, and 13 BAU preschool classrooms serving students with ASD across study sites over a four month period. During each classroom observation all 3 fidelity measures were completed. The reliability of the measures was determined by examining test-retest reliability, internal consistency, and interrater agreement.  Discriminant analysis was conducted to examine the subscales of each measure that most contributed to the ability of the fidelity measure to discriminate model types.

Results:

Test-retest reliability for the four observations was M = .77 for the BAU measure (.53-.86 across 8 subscales), M = .61 for the LEAP measure (.45-.86 across 8 subscales), and M = .63 for the TEACCH measure (.43-.86 across 9 subscales).

Internal consistency was examined using Cronbach’s alpha. For the BAU measure, the alpha was.94 (.71-.95), .93 for LEAP (.55-.90), and .93 for TEACCH (.42-.93).

Inter-rater reliability was measured as the proportion of agreement between observers. Inter-rater reliability was 93% (BAU, 82-97%), 95% (LEAP, 82-100%), and 89% (TEACCH, 87-96%), respectively.

Discriminant analyses were performed to identify the subscales of the fidelity measures that best discriminated between classroom types. On the BAU measure, two subscales-- social/peer relations and curriculum and instruction primarily contributed to the ability of the measure to discriminate between the 3 treatment models. On the LEAP measure social interaction and teaching communication subscales best contributed to the discrimination between model types, and on the TEACCH measure four subscales, communication, assessment, visual schedules, and social discriminated between the models.

Conclusions:

Results indicate that the TEACCH, LEAP, and BAU fidelity measures are psychometrically robust and clearly discriminate between the intervention models. Both the development/adaptation of these measures and the analyses of their psychometric properties are significant contributions to the field. The strength and impact of the intervention relies on accurate and measurable implementation.