With the increase in online education a concern is raised with regard to student learning. The current empirical study examines three methods of delivery of a Master of Education foundation research course. It compares the relationship between student learning outcomes (scores) for face-to-face, hybrid, and online. The process, rationale and findings are shared.
Session Goals:
1. To share a case study that examines teaching a graduate research course by the same instructor in three delivery modes; face-to-face, hybrid & online, 2. Share empirical outcomes of the comparison between student learning outcomes (measured by scores) for the three deliveries, 3. Analyze SIRs reported results for the three deliveries, & 4. Engage the audience with small group discussion about questions.
Introduction:
The Master of Education foundation research course is considered to be out of the students’ comfort zone. The course aim is to teach how to read peer reviewed research studies with a critical eye and how to evaluate the research process. Therefore the course requires a different level of thinking, mostly higher levels in Bloom’s Taxonomy. Students start with learning the research terminology and various designs. Then they move to the specific research steps such as data collection, tools, analysis, etc. In the next step students are required to write critical evaluation, which is proven to be a challenging skill for many. The course content involves various new skills and cognitive complexity which in turn affects the emotional response of students and raises their level of anxiety. One student shared “it is like learning to read, speak, and write a new language in one semester.”
In the last three years in a mid-size college in the Northeast US, similar to other higher education institutions, the popularity and demand for online delivery has increased by both administrators and students. The increase was accompanied with some faculty resistance, questioning the efficiency of online delivery on students’ learning. Therefore, the author of this proposal decided to examine and compare different delivery models (face-to-face, hybrid, and online) for the same course.
The research question posed:
- Is there a significant difference between final course scores in face-to-face hybrid and online delivery?
- Is there a significant difference between the article evaluation scores for face-to-face, hybrid, and online delivery?
- Is there a significant difference between the signature assignment scores for face-to-face, hybrid, and online delivery?
- Is there a significant difference in the course evaluation (SIR) for face-to-face, hybrid, and online delivery?
Literature:
The Online Learning Consortium, in recently published data, reported a 263% increase in students who enrolled in online offerings in the last twelve years (2016 Higher Education Online Learning Landscape). Not surprisingly, with such data, more academic leaders and administrators promote the transition to online. Unfortunately, the process is mostly accompanied by budget constraints and urgent need to do the transition. This in turn, raises faculty concerns for the quality of learning in an online environment. Several studies in the last few years examined efficacy, student satisfaction, and other variables. Little consistency is found among the various research outcomes (Lim, Morris, & Kupritz, (2005); Means, Toyama, Murphy, Bakia, & Jones, 2009; Ary & Brune, 2011; Motii & Sanders, 2013), especially because of the large number of variables involved and the various research designs implemented.
Findings from an early study by Lim, et.al, (2005) reported no significant differences in learning outcome. A more recent study by Motti & Sanders (2013), who conducted an empirical analysis of student learning outcomes in an introductory Microeconomics course that was offered online and face-to- face and taught by the same instructor, suggest “no significant difference measuring student performance by overall semester average between online and in-class students.” (p.81) Still they raised the importance of further “research related to relative student performance in particular courses and disciplines to provide more nuanced understanding and comparison of the alternative delivery methods.” (p.82)
Methods
In an effort to control some variables, such as teacher, assignments etc. the researcher chose to conduct a case study of one course offered in three different delivery methods in spring 2016 and taught by the same instructor. The three sessions of the Master of Education foundation research course used the same syllabus and similar assignments to assess if students’ learning outcomes were achieved.
Participants: Forty two Master’s level students registered for the research course in spring 2016. Ten students chose to take the course online, 13 chose to take the course as a hybrid, and 19 chose to take the course in traditional face-to- face delivery. It is important to recognize that the face-to-face students were a cohort and this was their third course together.
The Course: A three credit, required, graduate core course. The course is offered in fall, spring, and summer. The researcher chose to use data only from spring 2016 to maintain consistency and control variables such as different instructors and different assignments. The majority of students in this program have full time jobs so the face-to-face course is offered in the evening hours, where students arrive after a teaching day. The idea behind offering the course online is to allow more flexibility for students and save them driving to campus after work.
The Learning Management System (LMS) used is Blackboard. Each course regardless of the mode of delivery has a shell on Blackboard and faculty are expected to upload syllabi and other basic documents. This course was offered for many years as face-to-face was then offered for two semesters as a hybrid before a decision was made to transfer it online. The decision was made by the faculty and supported by the department chair and dean. The course was prepared to transition online during spring 2015 and was piloted online for the first time in summer 2015. Data was collected and the fall 2015 semester was used to make some adjustments to the course design. In spring 2016 students had the option to enroll in the research course as an online, hybrid or face-to-face course. As mentioned earlier, the same instructor designed and taught the three courses that were equivalent in content and expected learning outcomes.
Students taking the face-to-face course met once a week and had to read the assigned chapters before arriving to class. Students in the hybrid course met once every other week at the start of the course and then once every two weeks. In the online weeks each student signed into the course on their own time and completed the weekly activities such as posting in the discussion forum, or taking a self-assessment quiz, as long as they met the weekly due dates. In the online course, the plan was to have one face-to-face meeting before the official start day of the course. Because of a snow storm the college was closed and the course started without the face-to-face meeting. Through Doodle a new date was rescheduled in the third week of the course. Later on, it turned out that missing that first face-to-face meeting had been a challenge for students who were not familiar with the college LMS. All course interactions in the online course occurred through Blackboard including office hours, uploading course assignments, and grading.
The course instructor was knowledgeable about online teaching and learning. She has been teaching online for the last six years. She is well familiar with the content and was the person who designed the course on Blackboard with support from ITR when needed.
Data Collection: The following are the data that was collected to address the four suggested research questions;
- A report of students’ scores in the article evaluations for the three deliveries.
- A report of students’ scores in the position paper for the three deliveries.
- A report of students’ final score in the course for the three deliveries.
- The course evaluation outcomes for the three deliveries. (The college use ETS SIRs)
Data Analysis:
Comparison of the scores was conducted empirically using SPSS. A qualitative analysis for the focus groups and students’ narrative were coded.
Results:
The semester just finished so the results shared in the proposal is partial using only students outcomes, since SIRs reports are not ready.
Meeting the course learning outcomes- The first assignment in which students evaluated two peer-reviewed research articles resulted in no significant difference in students’ scores between the various delivery methods. The second one is writing a position paper using the information learned in the course to help provide evidence to the arguments that students are presenting yielded a slightly significant difference in students’ scores between face-to-face and online. While no significant difference was found with regard to the final course scores in the three deliveries. An interesting observation that will be discussed in the presentation is the higher variation (STD) in online scores compared to face-to-face.
SIR Course Evaluations were not received yet by the time I have completed the proposal. Those will be added and discussed in the final presentation.
Conclusion:
Since not all data was analyzed at this point, the conclusion is limited. There was only one significant result indicating difference in students learning outcomes for one of the course assignments in the three different delivery modes. Future research should includ larger numbers of students and with different content. There is also a need for future research on the impact of the quality of online course design and online delivery on students’ performance.