Social Validation of Evidence-Based Practices in Autism: Investigating the Evidence of Social Validity for Empirically-Demonstrated Treatments Identified By the National Autism Center and National Professional Development Center on ASD

Background:  

Empirically-demonstrated autism treatments considered to be “established” and “confirmed” as evidence-based practices (EBPs) have been identified by the National Autism Center (NAC) and National Professional Development Center on ASD (NPDC).  An important factor in the effective and widespread use of these interventions is social validity, generally defined as consumer satisfaction with the goals, procedures, and outcomes of programs and interventions.  The extent to which EBPs are socially validated by service providers plays a significant role in whether or not those treatments are adopted and implemented.  Without evidence of social validity for empirically-demonstrated autism interventions, there is less chance these treatments will be used effectively in schools, clinics, and homes.  Thus, social validity should be a consideration in the selection and effective use of EBPs.

Objectives:  

The purpose of this study was to investigate the extent to which EBPs identified by the NAC and NPDC demonstrate evidence of social validation, and to determine the types of social validity used by researchers to provide evidence of social validation.

Methods:  

Two separate analyses were conducted.  First, researchers compared the correspondence of NAC and NPDC EBPs with 60 autism interventions previously determined to have social validity, as defined as perceived value for their use within a comprehensive autism treatment program (Callahan et al., 2008, 2010).  This process resulted in 31 socially valid intervention components determined to align directly with one or more NAC/NPDC evidence-based practices.  These interventions were validated by experts in the field of autism treatment to ensure that the socially validated treatments represented the same interventions as reported in the NAC/NPDC studies, with an overall agreement of 99.6%.  Next, researchers analyzed all 942 articles cited by the NAC and NPDC studies for the 31 interventions to determine: (a) if the articles contained direct evidence of social validity, and (b) to identify which of seven categories of social validation (Reichow et al., 2011) were included by the authors.  Inter-rater reliability ratings were conducted on 21.3% of the articles regarding the presence of social validation evidence, with an overall agreement of 97.4% across all EBPs.  Assignment of evidence into social validity categories had an overall agreement of 88.4%.

Results:  

Across all EBPs, 24.2% of the articles cited by the NAC and NPDC reported social validity.  While all 31 EBPs had evidence of social validation, the results were variable, ranging from only 2.3% of articles for the use of “Extinction” to 60.5% for “Pictorial Story Board Simulations/Social Stories”.  The categories of “Consumers satisfied with results,” “Clinically significant behavioral change,” and “Socially important dependent variables” were the most frequently reported social validity indicators.  Relatively few researchers addressed the time- and cost-effectiveness of interventions and the settings/service providers associated with the implementation of EBPs.

Conclusions:

The relatively low rates of reported social validity for autism EBPs, especially within behavioral journals, underscores the need for additional research about how best to define, conceptualize, measure, and report social validation in autism efficacy research.  Such efforts will significantly improve the effectiveness of applied interventions and outcomes in autism.