Researchers investigating the social communication behaviour of nonverbal children with autism often use observational measures in order to reflect children’s communicative skills in familiar everyday settings with their regular communication partners. Decisions relating to the number of sessions in which children should be observed are often made in relation to resource issues rather than via a systematic procedure. Generalisability theory provides an objective means of estimating the optimal study design, in terms of both the number of sessions observed and the number of raters required, in order to provide a stable measure of a variable of interest.
Objectives:
We conducted a Generalisability study to estimate the optimal study design for key variables from the Classroom Observation Schedule to Measure Intentional Communication (COSMIC). We investigated the stability of these variables in relation to changes in context and in relation to observations made at different time points within one context.
Methods:
Eight children with autism (5 boys, 3 girls, mean age 75 months) attending an autism-specific school were videoed in snack, free play and work-based sessions for 10 minutes each on one day. Each child was also videoed in a second snack session within the next few days. Each video was independently rated by two researchers according to the COSMIC protocol. The frequency, variance, distribution and skewness of the COSMIC variables initiation, correct response, speech and request for object were examined. Only correct response had sufficient statistical properties to merit further examination. The variance components relating to participants, participants x sessions, participants x sessions x raters and participants x raters were calculated via ANOVA and entered into a Generalisability calculator spreadsheet. Optimal study designs (where the value of g ≥ 0.70) were derived from this spreadsheet.
Results:
The mean nonverbal mental age was 23.4 (SD = 9.3). The mean frequency of each of the key variables was less than 2 per 10-minute session with the exception of correct response, for which the frequency was 5.4. Inter-rater reliability for correct response was very high (ICC = 0.97). The optimal study design relating to changes within snack sessions across time was 1 rater x 8 sessions and for changes in context was 3 raters x 16 sessions.
Conclusions:
Generalisability studies provide a means of determining optimal designs relating to the number of sessions and raters required in order to provide a stable estimate of a variable of interest. This study demonstrates that it may be necessary to observe children in more sessions than may typically be the case in studies involving observational measures. For example, within one context, the results suggest that each participant should be observed in 8 10-minute sessions. If the study aims to investigate children’s behaviour across a range of sessions, then as many as 16 10-minute sessions coded independently by 3 raters would be required. Furthermore, the relatively infrequent nature of the social communication of low-functioning children with autism means that certain variables may not be amenable to this method of calculating optimal study designs.