An Initial Investigation Using Augmented Toys and Statistical Models to Automatically Categorize Object Play Behaviors

Background: The way in which infants play with objects can be indicative of their developmental progress and may serve as an early indicator for autism spectrum disorders. To better quantify these indicators, retrospective video studies have been used to correlate differences in object play among infants with subsequent diagnoses. Such studies require laborious video annotation. While computers can assist in the coding process, it remains a bottleneck where researchers must repeatedly inspect video at speeds much slower than real-time to indicate points of interest. To help alleviate this bottleneck we are exploring automated methods for collecting and annotating developmental data during object play. In particular, we are using computer algorithms that leverage statistical models to process data collected from our toys containing embedded wireless sensors.

Objectives: Our goal is to use neurotypical adults to determine if wireless sensors embedded in toys can provide sufficient data for the creation of statistical models of play behaviors. We will also automatically provide quantitative measures of object play similar to those produced by coding schemes used in retrospective video studies.

Methods: Five neurotypical adults were recruited. Each subject participated in a minimum of two play sessions lasting 7 – 26 minutes (µ=16.32 minutes, σ=7.17 minutes). Five augmented toys collected data at each session as participants performed loosely defined scenarios that included exploratory, relational, and functional play tasks. A total of 12 sessions were completed resulting in 3.8 hours (228.49 minutes) of sensor data containing 3,999 instances of object play. The data was independently labeled by two people with 24 possible primary actions for each toy (120 distinct classes). Statistical models were constructed using aggregate features computed over short temporal windows. Models were trained for specific play behaviors, toys, and a general binary categorization of relational play, using the iterative AdaBoost framework.

Results: Models trained to identify the 24 different instances of object play (across all participants) for 5 distinct toys performed with an average frame-level accuracy of 41.7%, while models trained to identify the 24 play behaviors regardless of the toy's form factor performed with an average accuracy of 30.1%. These numbers improve by approximately 10% if the models are trained in a user-dependent fashion. The more generalized binary model had an average recall of 99.0% for instances of relational play (only missing 1.0%). However, a high number of false detections were also identified, causing very low precision.

Conclusions: While more research is required, early results demonstrate that augmented toys can capture sufficient information about object play allowing the generation of statistical models which automatically filter data for later review. Our models seldom omit play events, however extra events are detected. Researchers may easily dismiss these false detections as erroneous. Our results indicate that automatic processing methods have the potential to dramatically reduce the amount of time researchers spend coding data and allow them to more rapidly ask and answer relevant questions about object play and how it relates to a diagnosis of autism spectrum disorders.