Objectives: The aim is to build a video application, MEBOOK, to overlay features of the protagonist on the face of the reader and replace the scene background with animated video pertinent to the story.
Methods: MEBOOK runs on a computer equipped with a RGB-D camera. The depth data allows us to separate the reader from the background and track a 3D model of the face in real-time. We use the 3D model to insert synthetic objects that are geometrically aligned with the face independent of pose and motion. For example, the trunk of Dumbo can be visually attached to the reader’s face. Also, the 3D model allows us to adjust the viewpoint so as to create the illusion that the reader is looking directly at him or herself. The separation of foreground from the background is akin to the “green”-screen technology used in newscasts to substitute in a more suitable background, such as a real jungle video, while leaving the foreground objects intact. The user interface of MEBOOK has the rendered video next to the story text. Highlighted keywords allow the reader to change characters and background video.
Results: This study is to demonstrate the feasibility of MEBOOK, an application that utilizes a RGB-D camera to capture a 3D model of a child which is used to render self-video depicting the reader as part of a digital story. Using an appropriately chosen story, we will demonstrate the real-time response of the system, the overall quality and robustness of the rendering as well as the intuitive and engaging user interface of the overall application.
Conclusions: We have designed a software application, MEBOOK, to enhance digital story books with interactive visualization tools, making them suitable for children with ASD. The novelty is the use of the child’s face as a character in the story, a form of self-modeling, to engage the child in the story. A subsequent study to measure the effectiveness of MEBOOK in enhancing comprehension and self-regulation in reading among 5-7 years old children with ASD is underway.