The Relevance of "Realworld Evaluation" to Autism Intervention Research

Background:  

In some settings, a researcher may encounter an autism intervention that has already been implemented within the community. This often occurs when an administrative unit has implemented or scaled up an intervention across a broad population of children (e.g., Nicholson et al, 2010; Anderson et al, 2006). It is also extremely common in low and middle income countries, where interventions for autism typically emerge from within the community rather than a research laboratory, and where randomized controlled trials and even quasi-experimental designs are not feasible. Interventions in these settings often involve a blended program, combining elements of tested programs and local models.  For these situations in particular, there is value in examining the potential effectiveness of existing programs. Borrowing from the broader field of international evaluation, autism interventions can be examined through the lens of “RealWorld Evaluation” (RWE; Baumberger, Rugh & Mabry, 2006). This approach offers an opportunity to expand the knowledge base about available and effective interventions while working within the constraints typical of community settings.

Objectives:

The current study presents a systematic examination of a parent training model for autism following the RealWorld Evaluation approach.   

Methods:  

The first six steps of the RWE method were used to frame and conduct the evaluation, which was undertaken as part of a collaborative relationship between UCLA and an NGO in India called AFA.  First, the program theory was used to develop a logic model for the program. Budget, time, data, and political constraints were examined and incorporated into the design. The evaluation took place over three consecutive sessions of a 12-week program. After extensive testing of measures to ensure appropriateness, a total of 48 parent dyads received pretest and posttest assessments using tools tied directly to the program theory.  Assessing and addressing the strengths and weaknesses of the evaluation design has taken place in consultation with AFA.

Results:  

Using the RWE approach, this evaluation demonstrated significant gains across multiple measures of parent knowledge, empowerment, acceptance, and skills. The use of a post-test comparison group further strengthened the validity of these findings. The greatest challenges to the evaluation were 1) large work loads required of the evaluation team, and 2) accommodating and accounting for naturally occurring variations in the intervention.

Conclusions:  

The current study is the first use of the RWE approach for an autism intervention. Resulting in both formative and summative data, this study was successfully conducted with minimal changes to the existing structure, content and procedures of the program. RWE was developed for those working under budget, time, data and political constraints. As we seek to understand more about effective and culturally appropriate intervention approaches for children in diverse contexts, a RWE process has particular salience, particularly where there is little infrastructure and limited expertise, or simply no real possibility at present to support more rigorous research designs.