20067
Improving Fidelity Measurement of Pivotal Response Training to Increase Systematic Use
As demand for broader use of empirically supported behavioral treatments increases, accurate measures of fidelity of implementation (FI) of these practices has become a critical issue. In autism intervention research, only about 20% of autism intervention studies included FI measures (Dane & Schneider, 1998; Durlak & DuPre, 2008). The lack of reporting limits the conclusions that can be drawn regarding the relationship between patient outcomes and specific treatment provided. FI measurement is often time-consuming and costly due to the detailed review of video recordings of intervention. This may limit measurement in many research and community programs. In autism research, most validated measures of FI were developed in a research setting and have received minimal adaptation for use by community providers.
Objectives:
Assess the feasibility of adapting a simpler, five-point Likert scale FI measure of a naturalistic behavioral intervention, Pivotal Response Training (PRT), from a research-validated trial-by-trial (TBT) FI measure.
Methods:
36 video samples of research and community providers using PRT were rated by trained, reliable coders for FI in ten components using a TBT measure. A five-point Likert scale was adapted from the TBT measure. The same 36 video samples were rated for FI in the same PRT components using the adapted FI tool by trained, reliable coders. 30% of the video samples were second-coded and assessed for inter-rater reliability. Frequency percentages from the TBT measure were scaled to a corresponding Likert scale rating to compare the two FI measures: 1(0-29%), 2(30-49%), 3(50-79%), 4(80-99%), and 5(100%). Percent agreement across measures for the 10 components in PRT were examined.
Results:
Three components showed same-point agreement between the two FI measures in >80% of the videos: student attention, developmentally appropriate cues, and shared control. Four components showed same-point agreement between the two FI measures in 60-79% of the videos: clear cue, maintenance/acquisition tasks interspersed, direct reinforcement, and turn-taking. Three components showed same-point agreement between the two FI measures in 40-59% of the videos: contingency, reinforcement, and reinforcement of attempts. For nine components, both FI measures showed agreement in pass/fail criteria and agreement within 1 point for over 80% of the videos coded; the remaining component, reinforcement of attempts, showed agreement in pass/fail criteria and agreement within 1 point for 63% of the videos.
Conclusions:
The adaptation of a five-point Likert scale FI measure appears most feasible with antecedent components of PRT, in which same-point agreement occurred for over 80% of the videos. Agreement between the two FI measures appears lowest for consequence strategies (i.e., contingency, reinforcement for appropriate behaviors, and reinforcement of attempts). Operational definitions of child behavior may need to be refined to improve agreement between the two FI measures. Overall, these data suggest that a simpler, Likert scale FI measure can be adapted from a trial-by-trial FI measure, with 9 out of the 10 components showing agreement in pass/fail and agreement within 1 point on the Likert scale for over 80% of the videos coded.