Inter-Rater Reliability of Multi-Disciplinary Autism Spectrum Disorder Diagnoses

Background: One possible benefit of using multi-disciplinary teams to evaluate children at-risk of autism spectrum disorder (ASD) is the ability to reach consensus. However, the reliability of inter-clinician consensus deserves further examination. Previous studies have suggested modest agreement among clinicians for specific DSM-IV-TR diagnoses - supporting use of a broad ASD diagnosis espoused in DSM-5. However, the level of inter-clinician agreement for the broad ASD diagnosis remains understudied in young, high-base rate samples referred for initial evaluation. 

Objectives: The primary aim of the present study was to evaluate diagnostic agreement for broad ASD diagnoses provided by independent clinicians from a multi-disciplinary team. Secondary objectives were to explore relationships between diagnostic agreement for broad ASD, autism severity, demographics, and other clinical factors.

Methods: The population included 229 children ages 13-86 months (78.6% male) referred for evaluation of possible ASD. Three experienced clinicians (psychologist, physician, and speech / language pathologist) provided independent DSM-IV-TR diagnoses as well as the broad ASD diagnosis prior to reaching a final consensus diagnosis. Autism diagnostic observation schedule (ADOS) and Autism Diagnostic Interview-Revised (ADI-R) scores were obtained by independent administrators and only considered for final consensus diagnoses. Child Behavior Checklist, Vineland Adaptive Behavior Scale, and Preschool Language Scale-4 scores were also collected. Kappa coefficients evaluated diagnostic agreement. Relationships between clinician diagnostic agreement, demographics, autism symptom severity, and other clinical measures were computed using Pearson correlation coefficients.

Results: In a young, high base rate sample (Median age =42.4 months, SD=15.6; prevalence of ASD=57.6%), specific DSM-IV-TR categories had poor to fair inter-rater agreement (Kappa=.36-.52). The broad ASD category had better, but not optimal, reliability (Kappa=.54-.68). Age and sex were not associated with agreement after accounting for clinical factors. Agreement was lower for individuals with lower ADOS social and total scores (r=.17, p=.030) and lower ADI-R social and non-verbal communication and total scores (r>.24, p=.007). Restricted and repetitive behavior symptom levels were not significantly associated with agreement on the ADOS or ADI-R (p>.05). Agreement was lower in children with higher expressive and total language scores on Preschool Language Scales.   

Conclusions: These findings replicated previous research indicating increased inter-rater reliability with the broad ASD diagnosis, further supporting the shift to DSM-5 from specific DSM-IV-TR categories, and extending previous findings to a high-base rate sample of young children receiving evaluation. However, the reliability of the broad ASD diagnosis - when only based on clinician judgment (not ADOS or ADI-R) - was not optimal, supporting the need for using gold-standard diagnostic instruments to supplement clinical judgment. The lack of relationship between agreement and demographics after accounting for clinical factors is comforting and suggests consensus can be reached even in young children. The observation of poorer agreement with lower social symptom severity and better language ability highlights the need for clinicians to use sensitive instruments, rather than relying on inter-clinician consensus, to identify less socially and language impaired patients. Future research is needed to clarify the value of inter-clinician consensus.