Quantitatively Analyzing Adaptive Learning Patterns: Results from a Pilot Professional Blended Learning Course

    Adaptive learning seeks to provide a personalized, efficient, and effective learning pathway through course content (ELI, 2017).  Adaptive learning systems track students’ mastery of content through frequent assessment mapped onto granularized nodes connected by pre-requisite knowledge requirements.  Students can flexibly map their pathway through these nodes so long as they demonstrate a minimum prerequisite knowledge level set by the designers. (Dziuban, Moskal, & Hartman, 2016; Feldman, 2013; Howlin & Lunch, 2013).  Adaptive learning was found on the Educause Horizon Report from 2015-2018 with an expected near term adoption but was not listed in 2019 (Alexander, B. et al., 2019). Challenges to scaling up appear to be related to cost and some resistance to the higher engagement with content expected when combined adaptive learning is combined with active learning in face-to-face environments (Lieberman, 2018).  This presentation analyzes use of adaptive elements in terms of student mastery.

Context

A blended course, “Teaching Science in the Elementary/Middle School”,  was redesigned to use adaptive learning. Face-to-face sessions involved application of the online content in local elementary schools.  The online content consisted of 50 nodes of content that were linked into 8 course objectives. The adaptive learning system used for this course was from Realizeit.  The data collected by the system at the student level includes the following:

1. Determine Knowledge (DK) is a factor of whether the student elected to complete a pre-assessment before starting a course objective.

2. NumRevise (NR) is a variable counting the number of times a student re-attempted a node’s assessment to improve their knowledge state.

3. Knowledge State (KS) is a measure of the learning that a student demonstrated in a node, objective, or the course overall.  It is a measure of the percentage of correct responses on the assessments.

4. Knowledge State Growth (KSG) is a measure of the change in the percentage of assessment questions answered correctly in each node for an objective (or the course) at the end of the course as compared to when answering questions in the DK pre-assessment.

5. Correct (C) is a measure of the percentage of a question type answered correctly.

6. Question Type (QT) is a factor for the type of assessment question:  True or False, Matching, Multiple Choice, Ordering, or Enter Answer.

Research Questions

1. How does the use of Determine Knowledge pre-assessment correlate to Knowledge State?

2. How does Knowledge State Growth for students who used Determine Knowledge differ across course objectives?

3. How did students use the Determine Knowledge function with different course objectives?

4. How does the number of re-assessments attempted relate to Knowledge State Growth?

5. How did the percentage of correct responses vary in regards to the Question Type asked?

Methods

Analysis of Variance (ANOVA) and multiple regression were used to investigate the research questions.   Assumptions for normality and homogeneity of variance were checked. Students self-enrolled in these sections, so randomization was not present and thus no causative claims will be made.  Normality of residuals was tested using the Shapiro Test. Levene’s Test of Homogeneity of variances was run since all datasets were not normal. ANOVA tests can be robust to normality violations, but for violations of homogeneity of variances, Welch’s test is used

For Question 1, the KS variable was highly positively skewed (non-normally distributed).  The Shapiro Test suggested rejecting the null hypothesis that the distribution was normally distributed (W=0.93394, p=7.3879x10-15).  Levene’s test of homogeneity of variances showed that the variances were not statistically different (F=.006,p=.9381).   

    For Questions 2 and 3, KSG is only calculated for students who completed the DK assessment, so only data from those students was included.  The Q-Q plot for KSG showed noticeable departures from normality. This was confirmed by the Shapiro-Wilk normality test (W=0.424, p = 2.2x10-16).  Levene’s test of homogeneity of variances showed that variances were statistically different (F=16.593, p = 2.2x10-16).  After conducting Welch’s test, a descriptive statistic was calculated for the mean KSG for those students who completed the DK assessment.  The Durban Watson Test failed to reject the null hypothesis that there is no autocorrelation (D-W: 2.096, p=.45).

    For Question 4, descriptive statistics were calculated with an Excel pivot table for each course objective.  Values were calculated for percentage of students who completed DK for each objective, KS disaggregated by whether a student used DK or not for each objective, and the average number of revisions made for each objective.

    For Question 5, the Q-Q plot for C showed departures from normality confirmed by the Shapiro-Wilk test (W=0.8525, p<2.2x10-16).  Levene’s test showed that variances were statistically different (F(4,4481) = 115.37, p<2.2x10-16).  Welch’s test was used.

Results

1. A statistically significant difference for each objective’s KS was found between students who did and did not complete DK (F(1,547)=9.698, p=.00194).  The average KS was 86.3% for those who did NOT complete DK before an objective and 83.6% for those who did.

2. The  mean value for KSG for those who completed DK was 3.2%.  ANOVA was not conducted to compare objectives since the data was both non-normally distributed and had heterogeneity of variances.

3. Statistically different results were found between KSG by objective (Welch’s F(7,67.98)=2.37, p=.0314).  Descriptive statistics were calculated for each of the course objectives to compare the percentage of students completing DK, the average KS for students who did not complete DK, the average KS for students who did complete DK, and the average number of revisions.  See Table 1.

4. Using regression, the number of revisions a student made significantly related to their KSG (coefficient = 0.004822, t = 2.88, p = .00427).  Therefore, for each revision, it can be predicted that the science methods students tended to be associated with an increase in their assessed KS by 0.5%.  See Table 2.

5. Correct and Question Type did show a statistically significant relationship (Welch F(4, 325.95) = 116.38, p = 1.6 x 10-61) . Tukey’s HSD showed adjusted significant differences between all question types except Ordering vs. Multiple Choice and True/False vs. Matching.  The rank order by question type follows. See Table 3.

Conclusions

    It was interesting that the use of Determine Knowledge was actually correlated with students having a Knowledge State rated as 3% lower than those students who did not complete the DK pre-assessment.  Additionally, for those who did complete DK, their KS improved by an average of 3%.

    When looking at the objectives that students chose to use DK in higher numbers, it is also interesting that most used it for the first objective on Foundations of Conceptual Change and the Scientific and Engineering Practices which are objectives that contain a large amount of new content.  On the other hand, the objectives that were about K-8 science content knowledge, and thus expected by the instructor to have the highest use of DK so that students wouldn’t need to watch videos or interact with content that they already knew, had the lowest rates of use of DK. It is possible that students also stopped using the DK as the course progressed since the objectives with the lowest DK rates were at the end of the course.

   The number of revisions a student made did correlate with a small impact on their KS (about 0.5% for each revision).  The objectives with the most revisions were Scientific and Engineering Practices (7.7), Crosscutting Concepts (4.4), Overview of Conceptual Change (4.2), and Instructional Models and Strategies (4.2).  It is possible that students made more revision attempts in these objectives because they were due earlier in the course. Students perhaps didn’t feel an appreciable change in their KS in relation to the time to “retake” an assessment.

    Finally, some speculative reasons for the ranking of question type by percentage correct can be produced.  Matching could be ranked highest because it allows students to use process of elimination to a greater extent than others.  True and false being second most frequently correct could have more to do with the ease of remembering a correct answer upon reassessment attempts.  Enter Answer had the lowest number of correct responses at 57% and the highest number of empty responses. It is speculated that students chose to “skip” these questions since they required greater effort to supply an answer and that they preferred objective questions.

Interpretation/Implications

    The usefulness of Determine Knowledge is somewhat questionable for students in a professional course similar to this teacher education course.  Students may rightfully feel that it is “not worth their time” to go through a pre-assessment if their ending Knowledge State is basically the same whether they take the assessment or not.  The reduction in the percentage of students completing this assessment as the course progressed also supports this inference. This function may possibly be more useful in a course with sequential content with more prerequisite knowledge requirements.  It also appears that the “impact” of re-assessing one's knowledge might be seen by students as not worth the additional effort.