When Universal Design for Learning has not yet been fully implemented in course design and students still struggle, what might educators do? Introducing a prescriptive analytics approach, attendees will leave understanding how traditional analytics ideas can be extended to compare simulated worlds where support interventions did or did not occur.
Session Goals
Attendees will be able to describe prescriptive analytics and how this approach differs from other analytics flavors including descriptive analytics, predictive analytics, learning analytics, and educational analytics. This approach will be illustrated via a proof-of-concept example of implementing a prescriptive analytics approach with an undergraduate accounting course at a women only university. Attendees will be able to assess the approach’s potential for improving support recommendations to students as well as targeting course redesign recommendations to faculty and other course developers.
Study Motivation
Knowing that implementing the guidelines of the Universal Design for Learning (UDL) framework can take time, educators face the question of what to do when our UDL efforts have not yet been fully implemented and our students are not mastering the course learning objectives sufficiently. The present research aims at closing the performance gap for students across the full range of dis/ability, whether disclosed or not, even when efforts to anticipate and design for student learning requirements (such as through UDL) fall short and students struggle to learn course material. In doing so, this project utilizes an atypical analytics approach, using prescriptive analytics to project predicted potential outcomes for individual students in different simulated worlds, allowing comparison of hypothetical future scenarios in order to make recommendations to students in the present. Additional potential exists to inform course redesign efforts by identifying places in the course where students are struggling. This study provides an example of how prescriptive analytics could be implemented within higher education institutions.
Background and Perspectives
Adaptive learning systems are well positioned to assist in effectively teaching learners with a wide range of perceptual and processing variability, but their effectiveness remains tied to the material designed into them. Given that students with disabilities are increasingly part of the student mix (Kimball et al, 2016), an approach such as Universal Design for Learning (UDL) can guide making course material accessible in ways that are beneficial for all students (Tobin & Behling, 2018). However, implementing both UDL and adaptive features can be time consuming and often happens through iterative (re)design cycles over multiple semesters. Thus, educators face the issue of how to help struggling students when a course is not (yet) fully universally designed.
There is need to go beyond the traditional response of providing accommodations for students with disabilities in higher education given that only about 35% choose to tell their institution about their disability (Newman & Madaus, 2015). This study builds on recent research (Author, 2022) demonstrating the positive impact on student learning of providing content via different modalities (i.e., text, video, audio, interactive, or mixed content). It illustrates a learning analytics informed approach combining formative data traces from tutoring and adaptive activity to build a predictive model that identifies points during a course where recommending tutoring may be warranted to support struggling students.
Data and Method
The example analyzed in this study comes from a high enrollment, accelerated (6-week), introductory online undergraduate English course at a women’s institution utilizing both an adaptive learning system and online tutoring with human tutors. (Students at this institution are offered a limited amount of free online tutoring.) The data combines formative data traces from an adaptive learning system with usage data from an online tutoring service using human tutors, learning management system (LMS) data, and the institution’s student information system. After descriptively investigating visual patterns across the course, the analysis utilizes a Bayesian network to predict when during the course students would benefit most from receiving additional tutoring. The model is trained with data from five sessions taught during 2018-2019 (339 students in 33 course sections), and then tested with the subsequent session (54 students in 4 course sections). Example recommendations for students are generated and evaluated as an illustration of the technique.
Results
Results show patterns can usefully be observed in the data, informing predictive modeling. These patterns indicate that some struggling students elect to use multiple modalities in the adaptive system and others choose tutoring assistance. By modeling the activity sequence structure from adaptive and tutoring systems, this type of approach could be employed with other courses and at other institutions to indicate where it is unfortunately not currently clear how to make course content work smoothly for all students from a standalone universal design standpoint (or sufficient resources have not yet been applied) and thus where individualized tutoring is predicted to be beneficial. The project illustrates prescriptive analytics that could inform tutoring recommendations for faculty and/or students as well as points where course redesign might be usefully targeted.
Discussion and Implications
Prescriptive analytic applications have yet to be widely developed or deployed in higher education, presenting a gap this research addresses, particularly aiming to minimize disparities in outcomes for traditionally underserved students. Despite the promise of prescriptive analytics for comparing potential student outcomes in simulated worlds (Frazzetto et al., 2019), learning analytics applications to date have typically focused on descriptive or predictive analytics or evaluation (Dawson et al., 2014). As the present research demonstrates, student learning data aggregated from across multiple campus systems in data warehouses can now be effectively utilized to support analytic directions involving calculating potential outcomes that make use of Bayesian network modeling by investigating optimal decisions while accounting for uncertainty, particularly when combined with the knowledge mapping that accompanies adaptive learning system implementation.
The proof-of-concept prescriptive analytics approach presented here targeted aiding students across the full ability spectrum. In particular, the study demonstrated that this approach can indicate a) where tutoring recommendations may beneficially augment students’ use of multiple content representations in an adaptive learning system when students show signs of struggling, and hence b) where future course redesign may be warranted. The intent is to take the complexity involved with Bayesian networks of entire courses (and potentially entire programs) and utilize the data collected from multiple campus systems to create meaningful, straightforward comparisons of students’ potential outcomes in different simulated worlds that can be communicated to stakeholders in understandable ways. This can positively impact course design practice and benefit students who are typically underserved by higher education. Potential for significant impact on institutional practice exists for both the university studied here as well as other institutions supportive of data-driven decision making, analytics, adaptive learning, and course development practices utilizing UDL.
Interactivity Plan
Attendees will be engaged through a series of interactive questions inspired by the differences between the various types of analytics approaches. After an overview of the example prescriptive analytics application from the research, attendees will participate in a series of online questions they can respond to on their phones that are designed to help them think through the potential of the approach and contrast it with other analytics and analysis approaches. The questions and discussion will be geared toward making practical sense of the technical material. These activities will reinforce the value of simulating worlds in which to develop predictions and engage attendees in discussing possible applications of this approach, both for real-time implementation and for post-hoc analysis. Take-aways for possible future research will be discussed, helping the online research community expand the research techniques being considered and developed. Faculty, course developers, student support professionals, administrators and the research community will be challenged to consider how the illustration presented might be applied in future institutional research to address campus support.
References
Author. (2022).
Dawson, S., Gašević, D., Siemens, G., & Joksimovic, S. (2014). Current state and future trends: A citation network analysis of the learning analytics field. Proceedings of the 4th International Conference on Learning Analytics and Knowledge, 231–240. https://doi.org/10.1145/2567574.2567585
Frazzetto, D., Nielsen, T. D., Pedersen, T. B., & Šikšnys, L. (2019). Prescriptive analytics: A survey of emerging trends and technologies. The VLDB Journal, 28(4), 575–595. https://doi.org/10.1007/s00778-019-00539-y
Kimball, E. W., Wells, R. S., Ostiguy, B. J., Manly, C. A., & Lauterbach, A. (2016). Students with disabilities in higher education: A review of the literature and an agenda for future research. Higher Education: Handbook of Theory and Research, 31, 91–156.
Newman, L. A., & Madaus, J. W. (2015). Reported accommodations and supports provided to secondary and postsecondary students with disabilities: National perspective. Career Development and Transition for Exceptional Individuals, 38(3), 173–181.
Tobin, T. J., & Behling, K. (2018). Reach everyone, teach everyone: Universal Design for Learning in higher education. West Virginia University Press.