User Experience Testing in Course Design

The College created a Success Series of six courses in order increase student preparedness and readiness by:

• Acclimating students to the university community,
• Fostering intellectual engagement, and
• Developing academic and life skills.

It is expected that this series of newly created courses will better prepare students for the academic rigor of their degree program and have positive effects on student retention, student success, and learning outcomes.

Purpose

The purpose of this research was to collect feedback on the newly created course materials from current students in order to understand their impressions, workflow, and to refine instructions for tasks, learning activities, and assignments. The research was conducted in two phases so that initial data from phase one could be used to enhance/iterate course materials and retested in phase two. In the first phase, we collected feedback on three of the six new courses: Introduction to the university, psychology, and introductory English. In the second phase, we gathered data on introduction to the university, humanities, finance, and a second English course.

Participants

Since the Success Series was designed for students new to the university, we recruited 25 “newer” students who had between three and 18 credits at the university. Participants were recruited from a national pool of university students. We interviewed ten students in phase one and 15 students in phase two.

Procedure

In order to collect feedback of this nature, we conducted contextual inquiry (CI) research using semi-structured interviews. CI research enabled us to observe students’ behavior and to hear about their thoughts and feelings while they worked on the course materials. Interview and observational data were collected via individual remote online meeting sessions with students. Each student reviewed content from one course.

While interviews were in progress, the project team observed the interviews via the online screen sharing software and communicated with the researcher with via a chat feature. The chat tool enabled the team to ask the student questions via the researcher without disrupting the flow of the interview. The project team also collected observational and comment data from each student. Those data were shared during a 30 minute debrief meeting that followed each student feedback session.

The findings were compiled, prioritized, and implemented by the project team in follow-up debrief sessions that occurred over several weeks.  

Results and Implementation of Findings

The key findings are consistent with instructional design guidelines that support busy adult learners who sometimes experience adversity.

1. Our students have diverse busy lives and at times had to make trade-offs to complete their school work. They ranged in age, tech savviness, and reading abilities. In our interviews, we talked with students who were:
   • Inerrupted by their children and other family members,
   • Meeting with us at the library because their internet connection wasn’t working properly, and

   • Interested in using the classroom on a mobile device during their lunch hour at work; and one who was running home (literally) from work to participate.

2. Students approached course material in a variety of ways. Depending upon a student's immediate goal, sometimes our students needed to work in small chunks of time and locate resources, readings, and other assets quickly. To support students’ need for this kind of flexibility and reduce frustration, we strived to:
   • Ensure course structure consistency;
   • Provide specific guidance and step-by-step instruction; this was reinforced with word count, worksheets, and templates.
   • Accommodate students with various approaches--whether he/she goes straight to the assignment or wants to review the reading material first; and

   • Link materials needed to complete an assignment in the assignment instructions.

3. Text based syllabi can be challenging to engage students in content. To increase the opportunity to engage students and help them quickly relate to content, we attempted to:
   • Utilize friendly tone and language,
   • Remove unnecessary internal jargon,
   • Provide context for learning activities and assignments including weekly overviews, and
   • Re-order learning activities and readings with the most important/engaging activities listed near the top.