Simons Simplex Collection: ADOS and ADI-R Training and Reliability Maintenance in Multi-Site Phenotyping Research

Background: Specific standards of inter-rater reliability on the ADOS and ADI-R are now the expected norm in quality autism research and are critical given known variability in clinical diagnosis.  Yet little has been done to study the maintenance of reliability.  Some charge that it is an unachievable aim.  As ‘diagnostic gold-standard’ measures should consideration be given to applying more continuous reliability maintenance, especially for multi-site designs? In requiring unparalleled rigorous phenotyping and reliability standards, the Simons Simplex Collection (SSC) and Simons Foundation Autism Research Initiative (SFARI) have offered us a glimpse into just such an aim.

Objectives: To describe the effectiveness of a three-tiered ADOS and ADI-R research reliability training and maintenance program applied to examiners across the 12 sites as part of ongoing SSC data quality assurance.

Methods: ADOS and ADI-R administrations for 2663 simplex families were collected across 12 sites as part of a phenotyping protocol for the Simons Simplex Collection. Clinicians from across these sites (23 designated site supervisors and 28 site examiners) established research reliability on the ADOS/ADI-R prior to independent data collection for inclusion in the SSC repository.  Reliability is defined as 90% inter-rater agreement on ADI-R and 80% agreement on ADOS (both full protocol and algorithm).  Each supervisor and examiner attended formal training workshops at UMACC or through SSC.  Sites were assigned a UMACC ADOS/ADI-R trainer as SSC Consultant.  Site supervisors established reliability on the ADI and all Modules of the ADOS with UMACC trainers via live and videotape observation.  Site examiners had the option of establishing independent reliability with UMACC or on-site reliability with their site supervisor. 

Site supervisors managed within-site reliability maintenance.  They were required to directly train and supervise site examiner administrations face-to-face, ensure all administrations were videotaped, run in-house trainings, themselves regularly administer each measure, and co-code each examiner’s first quarterly ADI-R and ADOS and submit it to UMACC for review. 

Cross-site reliability maintenance was supervised by UMACC consultants. ADOS/ADI-R training DVD’s were provided for coding by sites every 6 months and percent agreement with trainer consensus was collected for each examiner and supervisor.   Monthly cross-site supervisor teleconferences were provided to discuss coding discrepancies.    

UMACC consultants monitored and ensured maintenance of site reliability through on-site training, regular teleconference with individual sites, and videotape review of site staff administrations during training, every 6 months for reliability maintenance checks and as needed during quarterly data quality validation.  Inter-rater agreements were calculated and tracked.  Targeted remediation of individual administrations and coding was triggered by findings of three consecutive deficient administrations. 

Results: In the first year of formal data tracking (April 09-March-10), the average inter-rater reliability for individuals on the project from DVDs submitted then co-coded by UMACC consultants was 85% Protocol and 83% Algorithm for the ADOS (n=41), and 90% Protocol and 89% Algorithm for the ADI-R (n=39). 

Conclusions: Inter-rater reliability equal to expected research standards (90% ADI-R and 80% ADOS) can be achieved and sustained across large-scale multi-site studies albeit with intensive training and high levels of infrastructure, monitoring and supervision.