A Critical Review of Outcome Measures Used to Evaluate the Effectiveness of Comprehensive, Community-Based Treatments for Preschoolers with ASD

Background: Central to the effective evaluation of program outcomes are the tools chosen to measure those outcomes. Comprehensive treatment programs for preschoolers with ASD vary widely in their treatment targets, and as such, outcome measures used across studies vary considerably. This variability has implications for clinical uptake of this data and for choosing the “optimal” outcome measures for research. Additionally, tool usage modifications, justification, and adherence to clinical assessment and reporting requirements impact test score confidence and generalizability of the results. The overall aim of this review was to summarize and critically evaluate how standardized tests are used as outcome measures for comprehensive community-based interventions for preschool-aged children with ASD.

Objectives:

1. Summarize the most frequently targeted treatment outcomes;
2. Summarize the dominant tools being used to measure these treatment outcomes;
3. Critically evaluate the use of these tools in terms of their intended usage, construct representation, and reporting requirements.

Methods: A variety of medical, educational and allied health databases were searched for studies published between 2004 and 2012 using various combinations of the keywords “community”, “intervention”, “treatment”, “outcome”, “preschool” and “autism”. After reviewing 200 studies, a total of 24 met the inclusion criteria. Outcome instruments were (1) identified, (2) categorized (cognitive, adaptive, language/communication, behaviour/autism symptomology), and (3) compared to criteria identified in the Standards for Educational and Psychological Testing (The Standards; AREA, APA, & NCME, 1999), a best practice framework created to guide the development, validation, administration, reporting, and appropriate use of educational and psychological tests, including those used with with children of ASD.  

Results: A total of 32 different outcome measures were identified across the 24 studies. Cognitive (83%) and adaptive behaviour (96% of studies) were the primary tools of outcome-based measurement for comprehensive autism treatment. Specifically, the Vineland Adaptive Behavior Scales (VABS) was the most frequently reported outcome measure, used in 92% of the studies reviewed, raising a possibility of systemic measurement bias. Behavioral and autism-specific (67% of studies), and language (42% of studies) tools were under-represented, despite these being central constructs of autism. The following Standards criteria were relatively well met: including the use of multiple outcome tools (83%), clear sample description (100%), and using tools that represented each of the three constructs associated with ASD (75%). However, concerns included an over-reliance on a single adaptive measure, substitution of different pre and post cognitive measures despite different theoretical constructs (58% of studies), under-reporting of technical information and justification for tests selected (38%), lack of reported assessor blindness to treatment condition (only 13%), and an under-reporting of administration procedures (8%) and test modifications (0% of studies).   

Conclusions: Cognitive and adaptive measures continue to be primary outcome tools for ASD treatment, despite a lack of clear fit to core ASD constructs. Substitution of different cognitive tools and adaptive measurement bias is also a concern. However, most of the reviewed studies met much of the criteria outlined in the Standards, with some reporting exceptions on test modifications and assessor blindness. Clinical and research implications will be discussed.