External Validation of Autism Spectrum Disorder Classification in the Utah Autism and Developmental Disabilities Monitoring (ADDM) Network Site

Background:  Prevalence of Autism Spectrum Disorders (ASD) is estimated in Utah as part of the Autism and Developmental Disabilities Monitoring (ADDM) Network.  The ADDM Network uses a multi-source, population-based approach to identify children with an ASD from existing education and health records.  Given recent increases in ASD prevalence across ADDM Network sites including Utah, evaluating the potential for misclassification is important.  

Objectives:  This study’s objectives were: 1) measure agreement in classifying children as an ASD case between Utah ADDM (UT ADDM) clinician review and independent expert reviewers for final case definition, and 2) identify factors responsible for disagreement in final case status between the UT ADDM clinician review and independent expert review.

Methods:  UT ADDM clinician reviewers used a coding guide based on the DSM-IV-TR criteria for autistic disorder and ASD-NOS (PDD, PDD-NOS, Asperger’s) to identify eight-year-old children with ASD in study year 2008 from developmental and behavioral information abstracted from existing records.  Thirty records from children classified and thirty records from children not classified as ASD cases according to ADDM protocol were randomly selected and given to three independent reviewers considered medical experts in ASD diagnosis. The independent reviewers, blinded to UT ADDM case status, evaluated the records and reported impressions including final case definition (ASD case vs. non-case) and ASD subtype classification (Autism vs. ASD-NOS), degree of case certainty, reasons for low certainty or non-case status, and quality of record.  Reliability was measured using Cohen’s kappa measure of agreement.  Multiple logistic regression models were formulated to investigate the factors responsible for disagreement in final case status between UT ADDM clinician review and independent expert review.

Results:  Strong two-way agreement was achieved between the UT ADDM clinician reviewers and each of the three independent expert reviewers for final case definition (case versus non-case; κ = 0.82, 0.92 and 0.82). However, there was fair two-way agreement for ASD classification subtype (Autism vs. ASD-NOS; κ = 0.33, 0.36, and 0.13). This was not surprising as ADDM subtyping is based on the number and pattern of behaviors in the records, rather than clinical judgment.  Despite strong agreement on two-way final case definition, disagreement occurred with UT ADDM for at least one of the expert reviewers in 9 out of 30 records classified as a case by UT ADDM.  The record–level factors found to influence discordance in final case definition included independent expert reviewer’s rating of record quality (p =0.01) and degree of certainty (p = 0.007). Reviewers most often cited that the child’s profile could possibly be accounted for by Intellectual Disability as the reason for not classifying a child as an ASD case.

Conclusions:  Overall, there was very good agreement between UT ADDM and independent expert reviewers on ASD case status providing support for ADDM ASD prevalence estimates.  Our findings are consistent with other studies that have found conflicting agreement among clinicians in the identification of ASD subtypes.