Development of an Automated Classification Algorithm for the Surveillance of Autism Spectrum Disorder

Background:  The Autism and Developmental Disabilities Monitoring (ADDM) Network conducts population-based surveillance of autism spectrum disorder (ASD) among 8-year old children in multiple US sites. Each ADDM site collects special education records and/or medical records from providers who evaluate children for developmental problems. Information is abstracted from developmental evaluations that contain descriptions of behaviors consistent with ASD.  Trained clinicians then review all abstracted evaluations to determine whether the child meets the ASD surveillance case criteria.  The number of abstracted evaluations has dramatically increased since the year 2000, affecting the resources and timeliness of the surveillance system.

Objectives:  To develop and evaluate a machine-learning classifier that predicts surveillance ASD case status using the words and phrases contained in children’s developmental evaluations. 

Methods:  We used data from the 2008 metropolitan Atlanta ADDM site to create an algorithm that predicts whether a child meets ADDM ASD case status using words and phrases contained in the child’s developmental evaluations.  The sample comprised 1,162 children (including 601 meeting ADDM ASD case status) with a total of 5,396 developmental evaluations.  Evaluations for each child were concatenated into a single file, and the text was processed to remove punctuation, remove the suffixes of words, and count the occurrence of all words and 2-3-word phrases. A random forest algorithm constructed 10,000 classification trees to identify the words and phrases that were informative for predicting ASD case status.  We trained a second, “final”, random forest classifier using only the informative words and phrases. We assessed the algorithm’s performance by having it predict ASD case status for the records collected by the metropolitan Atlanta ADDM site for the 2010 surveillance year (1,450 children with 9,811 evaluations; 754 children met ADDM ASD case status).  We compared the algorithm’s predictions to the clinician-assigned case classifications. We also estimated ASD prevalence based on the algorithm’s classifications.

Results:  The algorithm predicted ASD case statuses that were 86.6% concordant with the clinician-determined case statuses (84.1% sensitivity, 89.4% predictive value positive). The area under the resulting receiver-operating characteristic curve was 0.932.  The algorithm was more likely to disagree with the clinician ratings on records where the clinicians indicated greater uncertainty about the case classification. Algorithm-derived ASD “prevalence” for 2010 metropolitan Atlanta study area was 1.46% compared to the published (clinician-determined) estimate of 1.55%.

Conclusions:  A machine-learning algorithm was able to discriminate between children that do and do not meet ASD surveillance criteria by using the text contained in developmental evaluations. The 86.6% algorithm-clinician agreement is somewhat lower than the 90.7% inter-clinician agreement observed for the 2010 ADDM Network. While there are many logistical issues to explore (such as whether performance would be similar at other ADDM sites), this approach has the potential to improve the efficiency of ASD surveillance.  In addition, classification algorithms trained on the ADDM surveillance data may ultimately be useful to other studies or activities that seek to ascertain ASD from electronic information.