As digital education tools flood the higher education market, instructors are challenged to determine whether a resource will be effective in their educational context.This presentation discuss how to support instructors through implementation research that that focuses specifically on whether outcomes persist across face-to-face and online courses.
Research and efficacy around digital education tools is abundant in the literature.Yet, how relevant are the findings to instructors if the research was conducted in a context that does not resemble their own?In this presentation we discuss a novel methodology for researching and evaluating and effectiveness and impact of digital learning tools that begins with a concrete understanding of the various contexts it is used in, how it is used, and what the outcomes are in that particular context.The approach then builds an evolving portfolio of increasingly rigorous evidence so instructors can be confident that the results they observe in the research will be likely for their students. The presentation showcases this approach through an implementation study of Achieve Read & Practice, as it was implemented in both a set of online and face-to-face courses.We discuss the differences in contexts, implementation, and outcomes observed in both.
Achieve Read & Practice is a digital learning tool that includes an eBook with embedded adaptive quizzing.The tool was built on learning science in that retrieval practice improves learning and information retention (Agarwal, Bain, & Chamberlain, 2012; Butler, 2010). And, formative assessment with immediate feedback supports student engagement and learning by helping them self-assess their knowledge and respond to that feedback with additional practice (Nicol & Macfarlane-Dick, 2006). Self-regulation research has suggested that tools like the progress meter support student retention by allowing them to monitor the execution of their learning activities (Carneiro, Lefrere, Steffens, & Underwood, 2012), while the ability to access hints supports student motivation (Ferlazzo, 2015). The learning tool displays eBook content in brief sections based on segmentation and chunking literature which has indicated that spacing supports organization of material for comprehension and retention (Miller, 1956).
Prior to the tool being used at scale, a study was conducted to understand how Achieve Read & Practice was implemented in various educational settings to meticulously document how instructors chose to use it, understand what the perceptions were of students and instructors, and determine whether use of the product was related to important learner outcomes like retention and academic performance. We compared the data collected from completely online courses and those taught in a traditional face-to-face environments to determine whether there were differences in implementation patterns and outcomes.
Findings suggested that there were differences in how the tool was used based on mode of course delivery.For example, in online courses instructors tended to assign all of the chapters at the beginning of the semester and let students work through the tool at a self-paced rate while in face-to-face courses instructors tended to assign one chapter per week with a more directed walkthrough of the content.Though, when the outcomes between the instructors and students in both contexts were compared we found that users in online and face-to-face courses both realized a positive relationship between engagement and retention and academic performance.Important to note, however, that qualitative data collected to compliment the qualitative data suggest that instructors and students alike identified differing benefits when reflecting on their use.
This presentation presents both the methodology used to measure the effectiveness and efficacy within context and expands on the implementation and outcome findings of the study.