This presentation will focus on methods for using course-level data such as failure rates (DFW) and course GPA in research. The presenter will discuss the process of preparing course-level data and the methodology used in a research study comparing online and face-to-face courses taught by the same instructor.
In doing research on student outcomes in online education, access to student data is not always available or there may be barriers to accessing the data such as privacy regulations. In some cases, course-level data is more easily accessed. For example, you may not have access to individual students’ grades in a data set, but you may have the average grades for a set of courses. What can you learn from this course-level data? How can you use course-level data to answer your research questions? We designed a study around course-level data to answer questions about the comparative outcomes of students in online versus face-to-face courses. This presentation will describe the multi-step process that was developed to prepare course-level data for a study of the course outcomes for instructors who taught online and face-to-face within the same academic year.
There is a significant body of published literature that comparing outcomes in online course vs face-to-face courses. In the Online Learning Efficacy Research Database there are currently 248 of published comparative studies across 74 discrete disciplines from 178 academic journals. An examination of these studies reveals many small-scale studies comparing one or a few terms of online vs face-to-face courses. There are fewer large-scale studies comparing the outcomes from multiple terms or instructors. Due to the nature of these comparative research study designs, the results have limitations. Some studies do not control for factors such as the instructor, instructor experience, term taught, class size, and other factors. We set out to conduct a study that could reduce some of these factors that limit the value and rigor of these comparisons. The study was conducted at a large, comprehensive R1 institution with over 24,000 students taking online courses.
In this presentation will share the process of planning and conducting this study by answering the following questions:
- How did we get the data? Data access involved approval from our Institutional Review Board, Registrar’s Office and detailed discussions with our internal data analyst.
- What did we do with the data? This study included initial examination of the data files, manipulation, restructuring, and computing of the data.
- How did we build a one-instructor data-set? We developed a methodology for matching courses. This involved identifying courses that were taught by the same instructor teaching face-to-face and online. Courses were also matched by term taught and class-size.
- What did we analyze? This study outcomes examined were course-level GPA, and course-level DFW and DFWU rates (U=unsatisfactory score in pass/fail courses).
- What did we find? The results of statistical analysis will be shared based on the matching variables, and class size. The results suggest that instructor, term taught, and class size may not fully explain the differences in outcomes between online and face-to-face courses.
Reflection and Discussion: The study results suggest that instructor, term taught, and class size may not fully explain the differences in outcomes for the course types. Participants will be asked to brainstorm other factors that could contribute to differences in these modalities. They will be asked to discuss this in groups of 2-3 (5 minutes) and then they will be asked to share their ideas with the full group (5 minutes).
Session Objectives:
Participants will leave this session with:
- An awareness of data considerations for studies using course-level data
- Methodological strategies for analyzing course level data
- Results from a one-instructor comparison study