NovoEd’s team of learning designers has conducted dozens of experiments on our courses, many in partnership with professors from Stanford, Columbia, and Iowa. These experiments include a variety of split tests on key variable, including lecture materials, assignment instructions, team formation, and pathways. In this session, we will share six different research projects - what we tested, what we observed, and what we learned.
NovoEd’s team of learning designers has conducted dozens of experiments on our courses, many in partnership with professors from Stanford, Columbia, and Iowa. These experiments include a variety of split tests on key variable, including lecture materials, assignment instructions, team formation, and pathways. In this session, we will share six different research projects - what we tested, what we observed, and what we learned.
Extended Abstract
Please keep extended abstract under 1500 words.
In the past year, NovoEd’s team of Learning Experience Designers have run over a dozen experiments in our MOOCs. These experiments include a variety of split tests (“A/B tests”) on almost every course variable, including lecture materials, assignment instructions, team formation, and course pathway. We’ve experimented with allowing learners to form their own teams or be assigned teams. We’ve experimented with providing multiple pathways to support differentiated learning. Not only do we have data from our experiments, we have also collaborated with researchers to identify the most effective online training strategies.
In this session, we will share six different research projects - what we tested, what we observed, and what we learned. Here is a short preview of each of some of the projects.
Team Formation A/B Test
This research was conducted by Chuck Eesley, Assistant Professor of Management Science & Engineering at Stanford University. The data is from 26,248 students in Technology Entrepreneurship, an eight-week free, open course ending in Dec. 2013. This analysis utilizes a multivariate regression format, with dependent variables of various engagement and satisfaction measures, independent variables including collaboration type, and control variables for demographics, engagement level, and more.
Eesley’s research shows that collaboration in online classes significantly improves learner engagement and course completion. Students who worked in teams were 16 times as likely to pass the course. As a baseline, of the 23,577 students working individually, only 2% (501) passed the course. However, 32% of all students on teams graduated - 1500% higher. Of this, 21% of students working in teams without mentors and 44% of students in teams with mentors passed.
Similarly, students in teams were more engaged in the community and contributed more to class discussions and peer evaluations. As one data point, for example, students on teams accessed the course five times as often. On average, learners working alone logged in once per week, but students in teams (no mentors) signed in 4.9 times per week, and students in teams with mentors signed in 5.5 times per week.
It is noteworthy that the completion rate of individuals resembles that of most free, open courses on other learning platforms where students work individually. It’s clear that basic “social features,” such as discussion boards, messaging, and social network sharing, are insufficient to drive higher engagement.
Felt Accountability
Robert Sutton and Huggy Rao are professors at Stanford University. They have written numerous journal articles and business publications, together authoring the bestselling Scaling Up Excellence: Getting to More without Settling for Less. In this book, they write about the concept of Felt Accountability.
The benefit of social learning comes when students feel part of a learning community, with peer responsibility. NovoEd achieves this with a combination of team-based assignments, mentorship, reputation systems, identity transparency, community moderation, and more. This “felt accountability” is a powerful intrinsic motivator that is effective at increasing course persistence and learning outcomes. NovoEd is built on a social fabric that engenders community, social relatedness, and connectedness” to intrinsically motivate learners. This drives NovoEd’s significantly higher engagement and completion rates.
We have experimented with a variety of techniques to drive felt accountability, including transparency, peer-to-peer feedback, organic team formation, and more.
Team Heterogeneity and Impact on Performance
Since the team places a core role in engendering “felt accountability,” we decided to further explore team composition preferences, and its impact on performance.
NovoEd hired data scientist Milad Eftekhar, a Computer Science PhD Candidate at the University of Toronto, to analyzed the differences between randomly-assigned and self-selected teams. Research discovered that users in self-selected teams tended to join teams that were more similar rather than different in nature. Furthermore, although there were mixed results about if teams with greater differences in most attributes were more successful, it was clear that teams with a greater diversity in skills were more successful.
In addition, Eftekhar analyzed signals to discern which activities had the greatest correlation with course completion. Overall, activity in community discussions were a strong indication of persisting in the course (remaining in the course past the halfway point), whereas team activity was a better signal of passing and completing a course. The strongest signal of course completion, however, was participation in peer review/evaluation activities.
From this, we explored the role of the instructor to scaffold and encourage teams. We also discovered various correlations between course activities and completion. From this, we now have a better idea how to guide team formation.
Presidio Institute A/B Tests
To further explore the role of teams, we wanted to experiment with different scaffolding for team collaboration. One of our partners, the Presidio Institute, is similarly experimental and decided to split test their pilot course this summer.
Learners were randomly divided learners into two cohorts, each with 110 enrolled learners and a steady-state of 80 active students. Although the two courses cover the same content, we tested two primary variables (so far): (1) one course had stronger scaffolding for team collaboration, including suggested agendas and team roles for team meetings, and the other one did not; and (2) one course included more informal opportunities for collaboration and discussion outside of the team, and the other did not.
The experiments are currently ongoing and will finish in early August. So far, we’ve seen a substantial difference in engagement between the two cohorts, which we suspect is primarily due to increasing accountability through better scaffolding.
Differentiated Learning and Learner Autonomy
Another important factor in felt accountability and intrinsic motivation more broadly is autonomy. Coined by Henri Holec in 1981, learner autonomy is the ability to take charge of one’s own learning. Although true learner autonomy is reflective of independent study and self-teaching (or autodidacticism), by giving learners the ability to choose their own learning path, we tested the role of learner autonomy.
In Princeton’s Making Government Work in Hard Places course, we analyzed offering multiple pathways for learners. During the course; 4 weeks had one set pathway for all students, and 3 weeks offered multiple options. That is, there were multiples sets of readings, each with a corresponding quiz. Unlike an A/B test, we allowed learners to choose their own pathway, thus allowing them to study material more relevant to their work or interests.
The hypothesis was that assignments that allowed for student choice would result in higher completion rates. At the conclusion of the class, we analyzed the results to determine if the weeks with multiple pathways had an effect on assignment completion rates for that week. Surprisingly, the results showed no significant difference.
We’ll share our conclusions, and what we recommend for the future.
Maximizing Autonomy with Self-Paced Courses
Perhaps that level of differentiation did not go far enough to create true learner autonomy. So, in partnership with +ACUMEN, we launched a series of self-paced on demand (asynchronous) “short form courses.” Each course could be completed in a couple of hours and required no teamwork or interaction. But as a result, learners could go at their own page, choose their own path, and select their own assignment topic.
Although we suspect these courses increased autonomy, the individual nature eroded the feelings of felt accountability. The results were not surprising - engagement and completion rates fell precipitously compared to their cohort-based MOOCs. We’ve since been experimenting with a variety of techniques to increase engagement in these self-paced classes, but still have found nothing as effective as collaboration and teamwork.