Development of a Novel Functional Social Communication Classification Tool for Preschool Children with ASD: Preliminary Assessment of Intra- and Inter-Rater Agreement

Background:  The Autism Classification System of Functioning: Social Communication (ACSF:SC), a content-valid tool for preschoolers, describes functional ability in social communication, the hallmark challenge in ASD. Understanding the range of social communication abilities is necessary to create a common language, to characterize and to describe groups of children with similar abilities. The five levels of ACSF:SC describe social communication of preschool children with ASD (level 1=most functional to level 5=least functional). Familiar raters select the level that best matches the child’s usual (i.e., typical) as well as best (i.e., capacity) social communication.

Objectives: To examine intra- and inter-rater agreement of parents and professionals using the ACSF:SC.  

Methods: We recruited parents of preschool children with ASD (age 3.0 to <5.11 years), who nominated two professionals working with their child (who were then assigned to one of two rater groups [cohorts]). Participants completed the ACSF:SC on two occasions. Demographic data were collected about the child, the rater’s relationship to the child, and frequency of observations of the child’s social communication. Quantitative and qualitative data were collected through a thought process questionnaire (TPQ) that evaluated the rating process, utility and perceptions of using the ACSF:SC. Percent agreement scores described the results from the TPQ, intra-rater (from Time 1 to Time 2) and inter-rater agreement. 

Results: Eleven sites in Canada recruited parents. Characteristics of child sample (n=29): mean age = 4.18; 90% males. Nominated professionals included clinicians (n=19) and pre-school educators (n=18).

On several items from the TPQ, 88%-100% of raters reported following User Guide instructions with the ACSF:SC. When asked about the comprehensibility (e.g., ease of understanding level descriptions) and utility (e.g., ease of matching child descriptions to abilities) of the ACSF:SC guide, 77%-92% of  parents and 92-97% of professionals responded positively. All five ACSF:SC levels were represented although there were fewer children in levels 1 and 5.

Intra-rater agreement for capacity (C) and typical (T) performance from Time 1 to Time 2 were: parents (n=17) 65%(C) and 71%(T); professionals (cohort #1, n=20) 67%(C) and 50%(T); and professionals (cohort #2, n=7) 86%(C) and 71%(T).

Inter-rater agreement for capacity (C) and typical (T) performance at Time 1 showed: parent and professionals (cohort #1) 26%(C) and 37%(T); parent and professionals (cohort #2) 45% (for both C and T); and professionals (cohort #1 and cohort #2) 38.5%(C) and 46%(T). 

Conclusions: The raters’ high level of comprehension on the use and utility of the novel ACSF:SC is encouraging for the future acceptability and adoption of the tool. The context of where and with whom social communications occur is an important consideration for identifying a child's level of functional ability. This may explain differences in time 1 inter-rater agreement between professionals, as well as between parents and professionals. Intra-rater agreement ratings were good for parents and professionals. Once inter-rater agreement and validity are established, ACSF:SC can be used in research and clinical practice to examine abilities over time, whether interventions change these abilities, and to identify contextual factors that may influence different social communication abilities.