How are students learning online? A detailed observation through the lens of Mastery Based Online Learning Modules.

How are today’s college students engaging with and learning from online learning systems? What fraction of students are actually benefiting from those systems that are becoming increasingly popular in higher education? How does the design of online learning resources affect students’ learning behavior and learning outcome?

Traditionally, the study of students’ learning engagement has been limited by the lack of effective observation methods.  On one hand, clinical observations and interviews provide detailed and accurate information, but are known to significantly alter students’ behavior and decision-making processes. On the other hand, survey responses, while being less invasive, typically only provide a general overview of students’ engagement over the course of an entire semester.

Analyzing students’ click-stream data produced by their interaction with online learning systems provides a promising alternative for observing student learning at scale. However, a challenge facing educational data mining research is how to determine the nature of the behavior behind each mouse click. For example, when a student quickly reads through a page of text, is he a disengaged learner or an efficiently learner?

Our earlier study, to appear in Proceedings of the 11th International Educational Data Mining Conference, suggested that the interpretability of click-stream data can be enhanced when measurements of students’ learning behaviors are coupled with measurements of learning outcomes. For example, when a student briefly studies a page of instructional text, then rapidly submits an incorrect answer to a follow-up problem, it is likely that the student is less engaged. On the other hand, if the student correctly solved the follow-up problem by spending a sufficient amount of time, he is more likely to be an efficient learner.

In this regard, mastery-based online homework has a unique advantage for producing highly interpretable student log data.  First proposed by both Bloom and Keller independently in the 1960s, mastery-based learning (MBL) breaks a topic down into multiple incremental units. Students are given enough time to reach mastery on each unit before moving on to the next one. In the online implementation of MBL, each unit combines instructional materials with formative assessment, allowing it to produce closely correlated learning behavior and learning outcome measurements at the same time.

In this presentation, we will describe our initial attempt at implementing online mastery-based homework by creating a sequence of 10 Online Learning Modules (OLM) in an introductory physics course at the University of Central Florida (UCF). Those modules serve as a “lens” for us to observe and understand students’ learning behavior at high resolution, and answer the following research questions:

1. What proportion of students are meaningfully engaged with each module in the sequence? Do students engage more with certain modules over others?
2. What are the potential factors in the design of the OLM sequence that may influence students decision to engage/disengage with learning?

OLM design and Data collection:

Each OLM module contains an instructional component (IC) and an assessment component (AC), as is shown in the figure below. The IC consists of both instructional text and ungraded practice problems, focused on teaching a single concept or a problem-solving skill. Students receive immediate feedback and have access to the problem solution after attempting any practice problem. Each IC typically takes about 10 minutes to an hour for a student to finish, which resembles a small unit in an online course. The AC consists of either 2-3 simple multiple-choice concept problems or 1 complex multiple-choice calculation problem, depending on the focus of the module.

Multiple OLMs are combined sequentially to form a learning unit on a given topic. A student passes a module by correctly answering all the questions in the AC, and can proceed onto the next module only after passing the current one. Each student can have multiple attempts on the AC. On each new attempt, a slightly different version of the assessment problem(s) drawn from a problem bank is presented to the student.  A key feature of the OLM is that students are required to make at least one attempt on the AC before being given access to the IC. After the initial attempt, students can either study the IC, or make additional attempts on the AC. On each attempt the student is presented with a slightly different problem until the problem bank in the assessment component is depleted. During an attempt the IC is temporarily locked from access.

The OLM design has three major advantages for data collection and analysis: First, students’ AC attempts before and after instruction serve as de-facto pre and post-tests, increasing the accuracy and frequency of learning measurement. Second, the length and types of learning resources in the IC allows for a richer variety of student learning behavior to be observed compared to many online homework systems. Finally, by combining instruction and assessment into one module, observations of learning behavior and learning outcomes can be interpreted in the context of each other. 

A sequence of 10 OLMs of increasing difficulty were created and hosted on UCF’s award-winning open source online learning objects platform, Obojobo, developed by the Learning System and Technology (LS&T) team at the Center for Distributed Learning. The modules were combined into a sequence and assigned to students via the Canvas learning management system, and assigned as homework to students in a traditional lecture-based, first semester college introductory physics course at UCF in the Fall 2017 semester.

The following types of data are being analyzed using a mixture-model clustering analysis to identify patterns in students’ learning: Outcome and time-on-task of students’ initial AC attempt before accessing the IC (pre-test); Duration of students’ interaction with the IC; Outcome and time-on-task of students’ AC attempts after they have finished interacting with the IC (post-test).

Results and Implications:

We observed a number of  student behavior patterns on both AC and IC of each module. On students’ initial AC attempts (pre-test), we can identify a group of students who spent very little (<40s) time solving the problems, and are most likely either randomly guessing or copying answers from friends. On the IC, we can also distinguish between students who only briefly interacted with the IC from those who normally or extensively interacted with the IC. Combining those two measurements we can categorize students’ learning behavior into four categories:

• Engaged: normally attempting the AC and engaging with the IC
• Brief-learning: normally attempting the AC, and briefly interacted with the IC.
• Low-confidence: Guessed on the AC but normally interacted with the IC.
• Disengaged: Guessed on the AC and briefly interacted with the IC.

            The sizes of those four categories are significantly different from one module to another. Notably, low-confidence and disengaged populations increase significantly on or after challenging modules, as well as on the last module in the sequence. On most modules, the low-confidence population was always larger than the “disengaged” population, and had better performance on their AC attempts after learning from the IC (post-test). On earlier and easier modules in the sequence, the Brief-learning group had significantly lower performance on post-test than the engaged group. However, on later and harder modules, the brief-learning group’s post-test performance is often indistinguishable from that of the engaged group. This suggests that on later modules, the brief-learning group consisted of students who have good understanding of the content which enabled them to pass the AC by only briefly looking at the IC.

            Another noteworthy observation is that on each of the last four modules (7-10) in the sequence, there is a distinct group of students who spent longer time (>3 min) on their post-learning AC attempt. Those students have significantly higher correct rates on the attempt than others, which indicates that they are serious problems solvers who are attempting the more complicated problems using the correct method. On module 7, those students predominately come from Engaged or Low-confidence learners, but on modules 8 and 9, those students can also come from brief learners, which seems to suggest that some students have managed to transfer their knowledge learned from module 7 to later modules.

            Overall, we observed that initially the majority of students are meaningfully engaged with the OLMs, as shown in the illustration below. Some students disengaged from learning on module 3, which involved a difficult problem in the AC, but most of them re-engaged on modules 4 through 7. However, after working through an exceptionally challenging module (module 7), roughly 1/3 of the students disengaged from learning and started to guess/answer copy on the following modules, whereas another 1/3 of the students persisted and most of them learned to solve the more complicated problems on the following modules. The persistent students have significantly higher exam scores and final course grades than the disengaged students.

             In summary, the results show that most students frequently make deliberate decisions about learning when completing the OLM sequence. Students’ levels of engagement are influenced not only by their self-motivation and aptitude, but also by the quality and design of online learning resources. The OLM design and data analysis serve as an effective method for evaluating and improving the quality of online learning resources.