Promising new education interventions are potentially being ‘unnecessarily scrapped’ because trials to test their effectiveness are insufficiently faithful to the original research, a study warns. Researchers ran a large-scale computer simulation to examine how much ‘fidelity’ compromises the results of school-based trials of new learning innovations and strategies. ‘Fidelity’ is the extent to which these evaluations adhere to the original research on which the educational intervention is based.

The cautionary note is being raised after researchers ran a large-scale computer simulation of more than 11,000 research trials to examine how much ‘fidelity’ influenced the results. In science and the social sciences, ‘fidelity’ is the extent to which tests evaluating a new innovation adhere to the design of the original experiment on which that innovation is based.

In much the same way that scientists will test a new drug before it is approved, new strategies for improving learning are often evaluated thoroughly in schools or other settings before being rolled out.

Many innovations are rejected at this stage because the trials indicate that they result in little or no learning progress. Academics have, however, for some time voiced concerns that in some cases fidelity losses could be compromising the trial. In many cases, fidelity is not consistently measured or reported.

The new study put this theory to the test. Researchers at the University of Cambridge and Carnegie Mellon University ran thousands of computer-modelled trials, featuring millions of simulated participants. They then examined how far changes in fidelity altered the ‘effect size’ of an intervention.

They found that even relatively subtle deviations in fidelity can have a significant impact. For every 5% of fidelity lost in the simulated follow-up tests, the effect size fell by a corresponding 5%.

Story Source:
Materials provided by University of Cambridge. Note: Content may be edited for style and length.