We will present our ongoing study of the impact of design decisions for online courses on student performance and engagement. Specifically, we use learning analytics to learn how students navigate the course, perform on assessments, and participate in activities. The findings reveal the relationship between activity completion and learning performance.
Using learning analytics to study student performance and engagement in online courses
The Learning Science Lab (LSL) at NYU Stern team co-designs digital learning environments for business school students. Digital learning environments use technology to facilitate learning such as a simulation to teach sustainability or a video demonstration to illustrate computer programming concepts. In the design of those digital learning environments, the team makes a series of decisions around how learning content and artifacts are organized, displayed, and presented. The decisions around design are rooted in best practices and research from the field of learning sciences and educational technology.
We will present our ongoing study of the impact of design decisions for online courses on student performance and engagement. Specifically, we use learning analytics to learn how students navigate the course, perform on assessments, and participate in activities.
We measure student engagement in terms of participation in activities, communication around content, and interaction with artifacts to signal study and the development of new knowledge. Student performance is measured in terms grades, submissions, grades, participation and meaning of deadlines
Research questions
The questions that guide the study are as follows:
- Does student completion of lesson activities (in lesson questions, surveys, discussions) relate to learning performance (measured by the lesson/weekly deliverable, and final grade)?
- Does student attendance in meetups (online synchronous meetings) relate to learning performance (measured by the lesson/weekly deliverable, and final grade)?
- How does prior knowledge relate to engagement and performance (measured by student intake survey, completion, final grade)?
- How the frequency or number of assessments impact performance and completion (measured by gradebook and final grade)?
Research methods
The participant pool for this study are students enrolled in the NYU Stern Master of Science in Quantitative Management (MSQM) online degree program. The study focuses on the efficacy of the online learning design of 22 courses to promote student engagement and learning. The data sources required to gain this insight will observational data provided by the students as their natural participation in the courses.
The specific data sources include:
Course level grade book data from NYU Classes learning management system entered by the instructor or teaching fellow. Course level assignments, in-lesson questions, quizzes, activities, and survey data completed by the each student. These data are retrieved from the tools used to implement the learning design which may include Qualtrics, Google Apps, NYU Classes, Playposit, Zoom, McGrawHill Connect, and GoToTraining.
Research findings
The findings reveal the relationship between activity completion and learning performance, role of prior knowledge on performance, and frequency of activities on performance.