An evaluation of the effectiveness of adaptive learning at the University of piloting a number of adaptiveMaryland University College (UMUC).
Background
Innovation in higher education has become the cornerstone for student success and transformation in the learning environment. As the field of adult-centered education has evolved to include more next generation innovations such as digital courseware, the need for an institution to successfully implement and measure the effects of innovation is an essential tactic to improve the quality and depth of students' learning experience. Adaptive learning, "an approach to instruction and remediation that uses technology and accumulated data" (Waters, 2014) to personalize instruction, has emerged as one solution. To examine the effectiveness of adaptive learning, the University of Maryland University College (UMUC) is piloting a number of adaptive platforms in courses across the curriculum. While course completion rates and student achievement were greater in the adaptive courses, the differences did not reach statistical significance; thus, additional testing with larger samples is under way.
Introduction
As an open-access university with a diverse student body, UMUC must meet students where they are academically; thus, comprehensive support and a continuum of interventions, which we believe adaptive learning across the curriculum to be, are needed. Theoretically, when implemented at scale, the data-rich learning environments provided by adaptive learning platforms may enhance the learning experience by targeting opportunities for personalized instruction. This presentation will highlight preliminary findings from UMUC's piloting of adaptive learning, discuss use of the learning analytics provided by the adaptive learning platforms, and explore the challenges of implementing adaptive learning at scale.
Participants
This study consisted of 285 students enrolled in four courses, HIST 156: History of the United States to 1865, CMIS 141: Introductory Programming, FINC 330: Business Finance, and MATH 009: Introductory Algebra, during the Summer 2015 term. Of the 285 students, 138 were enrolled in adaptive learning sections, while 147 students were enrolled in control sections. The mean age of students in the sample was 32.
Methods
All of the courses, with the exception of MATH 009, used adaptive learning technologies for two weeks. In MATH 009, the adaptive learning platform was embedded in the full eight-week course. To examine the effect of adaptive learning on student achievement, chi-square tests were conducted to determine if there were relationships between successful course completion and final letter grades by condition. Student perceptions of adaptive learning were gathered through a locally-developed survey.
For MATH 009, which used the adaptive technology for the entire eight weeks of the course, correlations were conducted to determine if there was a relationship between study time and student achievement, and a regression was conducted to determine if achievement on specific course elements were predictive of course success.
Results
Course completion did not differ significantly by condition. Overall, course completion rates were higher in the adaptive sections, with 81% (n = 112) of students in the adaptive sections successfully completing the course compared to 74% (n = 109) in the control sections. For MATH 009, the course completion rate was nearly 93% (n = 26) for students in the adaptive condition.
Final letter grades did not differ significantly by condition; however, students in the adaptive condition earned more final letter grades of A and B than students in the control condition. Overall, nearly 44% (n = 60) of students in the adaptive condition earned final letter grades of A, compared to 35% (n = 52) in the control condition. In the adaptive condition, 31% (n = 43) of students earned final letter grades of B, compared to 27% (n = 39) in the control condition. In CMIS 141, students in the adaptive condition earned 19% more final grades of A and B than students in the control condition, with 40% (n = 10) earning final letter grades of A. In MATH 009, nearly 86% (n = 24) of students in the adaptive condition earned grades of A or B compared to 71% (n = 40) in the control condition.
Additional tests were run in MATH 009, as the entire course was embedded in the adaptive platform. These tests revealed a moderate correlation between study time and score gain in solving equations, r(26) = .458, p = .019, with study time explaining 21% of the variation in score gain. There was also a moderate correlation between study time and score gain in percents and real numbers, r(25) = .472, p = .017, with study time explaining 22% of the variation in score gain. There was a strong correlation between study time and score gain in whole numbers, fractions, and mixed numbers, r(26) = .501, p = .009, with study time explaining 25% of the variation in score gain. A multiple regression was run to predict final calculated grades from post-test scores on the MATH 009 goal assessments. All seven goals significantly predicted final calculated grade, F(6,16) = 3.579, p = .019, r2 = .413. Thus, post-test goal scores explained 41% of the variance in final calculated grades. A stepwise regression revealed that systems of equations and inequalities and formulas and solving inequalities were a significant predictor of final calculated grade F (2, 20) = 9.843, p = .001, r2 = .446. Systems of equations and inequalities alone was also a significant predictor of final calculated grade, F(1, 21) = 10.265, p = .004, r2 = .296.
Discussion
These findings align with previous research (Boersma, 2013) that suggests students in adaptive courses have higher course completion rates and earn more grades of A and B than students in comparison courses. Although the differences in course completion and student achievement did not reach statistical significance, the results suggest that adaptive learning may positively impact student persistence and engagement. Additional studies with larger sample sizes and more fully adaptive courses are planned. While adaptive learning platforms collect a plethora of data on both student and faculty use, making sense of that data is important and time-intensive.
References
Boersma, J. (2013, August 4). New research validates effectiveness of adaptive learning. Emerging EdTech. Retrieved from www.emergingedtech.com
Waters, J.K. (2014, April 16). The great adaptive learning experience. Campus Technology. Retrieved from http://www.campustechnology.com.