Evaluation of Performance-Based Measures of Functional Skills

Background:  Valid outcome instruments that produce reliable scores and are capable of detecting change are an essential component of research, treatment and program evaluation. Given that interventions for children with Autism Spectrum Disorder (ASD) are often intended to develop functional skills, outcome instruments that are valid indicators of these skills, and that reliably evaluate treatment effects are important. Towards this end, parent-reported outcomes are often obtained, and although they provide important information about function, they may be subject to response bias. Thus, performance-based outcome instruments that objectively evaluate functional skills are needed to supplement parent-reported outcomes, and to support the rigorous and systematic evaluation of interventions for children with ASD.

Objectives:  The purpose of this study was to identify performance-based instruments of Activities of Daily Living (ADLs) and socialization, evaluate their psychometric properties, and determine feasibility for use in intervention trials for ethnically diverse children with ASD ages 6-9 years.

Methods:  This study used mixed methods to achieve its objectives. First, a systematic approach was used to review research and literature to identify performance-based assessments of ADLs and socialization. Assessments were included in the review if they: 1) evaluated ADL’s and/or socialization; 2) included normative data for children with ASD ages 6-9 years of age; and 3) were performance-based, with evidence of psychometric support. Next, a panel of experts used a 32-point quality indicator scale designed specifically to assess the quality of outcome instruments and to rate psychometric properties, utility, and appropriateness for ASD (Law& MacDermid, 2014; Portney & Watkins, 2009). Using a Modified Delphi Technique, the highest rated instruments were further assessed to identify the two strongest instruments. These two instruments were then administered to 20 ethnically diverse children with ASD to evaluate feasibility, utility and discriminative validity.

Results:  Seven performance-based outcome measures of ADL’s and Socialization were identified and rated. Modified Delphi Technique identified the top-rated instrument for ADL’s as The Assessment of Motor and Process Skills (AMPS – Fisher& Jones, 2012) and for socialization, The Evaluation of Social Interaction (ESI - Fisher and Griwold, 2010). Both instruments were successfully administered to 20 children with autism in one hour or less demonstrating feasibility and utility. Discriminant validity testing using the standardized canonical discriminant function coefficient showed that the AMPS Process Score predicted the ADOS Autism Severity score (r = -.63) such that a high AMPS Process score predicted low ADOS severity (as hypothesized). The AMPS Motor score was a moderate predictor (r =.445). The ESI was a strong predictor (r=.84) but in the opposite direction expected.

Conclusions:  The AMPS shows promise as a performance based outcome measure of ADL’s having strong psychometrics and feasibility for administration to children with ASD. Preliminary data suggests that the AMPS shows adequate discriminant validity. While the ESI shows strong psychometrics, utility and feasibility, further assessment of discriminant validity is needed. Implications of these findings for research and practice will be discussed.