We intentionally chose to revise an inclusive OSCQR rubric to expand its impact on supporting diverse learners. Guided by the Universal Design for Learning framework, we modified and added 17 inclusive practices in OSCQR. 74 faculty and instructional designers evaluated the rubric and provided implications on course design and delivery.
The Project Topic and Goals
With the steady growth of online education at institutions of higher education (IHEs), many IHEs are developing and validating quality standards for online courses to ensure that all students receive quality online learning experiences. The existing course design rubrics primarily contain course design features, including course organization and presentation, learning objectives and assessments, interpersonal interaction, technology, and digital accessibility (Jaggars & Xu, 2016; Muller et al., 2020). Regarding updating course design features in IHEs, increasing efforts are made to design accessible courses in building inclusive learning environments to support the success of the increased enrollment of learners with disabilities and other marginalized groups of students in higher education (Linder et al., 2015; Author, 2021). However, critics have pointed out that the existing course design conceptual models (e.g. the ADDIE model, the Community of Inquiry theoretical framework), used to guide course design rubrics development are missing a crucial cultural component and knowledge of the students (Gamrat et al., 2022).
To address the call for modifying and evaluating inclusive course design rubrics to consider how students’ differences affect learning and to align course design and pedagogies, we conducted this study of inclusive course design rubric modification and evaluation. This inclusive course design rubric project is part of an ongoing effort to help educators engage with diverse learners and support student success in universities. The primary goal of this project was to help educators evaluate and improve inclusivity in their courses by using a rubric with best practices that make diversity, equity, and inclusion (DEI) related items measurable and actionable. In addition, this project aimed to investigate whether a noted resistance to instructional design processes from faculty members who were subject matter experts will score the elements of the revised rubric differently than instructional designers.
We purposefully chose to add inclusive practices in the SUNY’s Online Course Quality Review Rubric (OSCQR) (Online Learning Consortium, 2016) to expand the impact of OSCQR rubric on supporting diverse learners in course design and delivery. We chose to revise OSCQR rubric due to its wide adoption in the course design and evaluation process among IHEs in the United States (Lowenthal et al., 2022). We asked instructors and education specialists from IHEs to evaluate individual elements of the revised rubric on their relevance in making courses inclusive and accessible.
Our OSCQR rubric revision project attempted to respond to a growing need to address DEI in course design, based on the existing literature and extensive discussions with professionals and instructional designers. A unique perspective of our study is to amplify the voice of instructional support staff given the fact that they have first-hand knowledge and challenges in inclusive course design. In our presentation, we will share an inclusive OSCQR rubric we put together and the preliminary findings of this project. This revised OSCQR rubric includes 11 modified OSCQR items and 6 newly added inclusive practices. We received and analyzed 74 evaluation responses from 46 faculty members and 28 instructional designers. Findings from this project have implications for course design and delivery.
The Plan for Interactivity
We would like to invite all participants with diverse backgrounds to discuss this inclusive rubric project and its implications for course design and delivery in online education in the presentation. We plan to use interactive polling questions between presentation slides to engage you in data reports and conversations. Questions such as your current role in online course development, and your ratings on the controversial inclusive practices from our findings in comparison with the average scores from our faculty and ID groups will be used to prepare you to engage in the Q&A session. Specifically, we plan to get a closer look at participants’ rating of importance on the 17 DEI-focused rubric items. In our study, the 46 faculty participants rated 7 rubric items across 6 dimensions lower than the 28 IDs. We will display these 7 items on presentation slides and ask for participants’ ratings on them. In the Q& A session, we will provide three self-guided questions based on the project findings to guide the discussion. By the end of our interactive presentation, participants will be able to:
- Identify the similarities and differences between online instructors' and instructional designers’ perspectives on inclusive course design;
- Discuss the implications of such an inclusive rubric on online course design and delivery;
- Reflect on your own approach to addressing DEI issues in online teaching and learning.
The Importance of This Topic
It is important to have equity in mind when we design and allocate educational resources, opportunities, and support based on learner needs. Inclusion describes the active, intentional, and ongoing engagement with diversity – in people, the curriculum, and communities (Deramo, 2022). Our understanding of applying inclusive design and pedagogy in course design includes identifying diverse learning needs, exploring sociocultural contextual variables, including voices of multiple stakeholders (learners, faculty, and IDs), supporting the learning process, and guiding professional identity development.
The number of learners from diverse and ethnic backgrounds enrolled in IHEs is increasing. A 2015 study reported that 25% of full-time undergraduate students work full time and 40% work at least 30 hours a week (White et al., 2021). Learners who are from marginalized groups often face significant disadvantages throughout their lives, including in educational opportunities (Servaes et al., 2022). Learners are negatively impacted by the absence of inclusive practices which leads to marginalization, isolation, and racism in such environments (Sáenz et al., 2018). Existing literature shows that course design standards and peer review are helpful in promoting quality (Little, 2009; McGahan, 2015). Quality standards can streamline the course review process by providing clear guidelines. Even though many online learning quality rubrics exist and the awareness of inclusive course design from professionals and instructional designers grows, little empirical research has focused on inclusive course design rubric development and its validation. We modified and added inclusive practices to the OSCQR rubric with the purpose of expanding its impact on supporting students with diverse needs in online learning environments.
Our research group included two faculty members, one instructional designer, and one director of Universal Design for Learning and Accessibility Services, who have designed and delivered multiple online courses. We revised the OSCQR rubric from June 2022 to October 2022. Each of us revised the rubric individually and then we met three times to discuss the revision as a group until we reached an agreement about the changes. We used the Universal Design for Learning framework (CAST, 2018) to guide the revision process. Our suggested changes included being more explicit about using inclusive language, inviting tones, and technology tools, promoting belonging, developing a strong and salient sense of learner identity, and including a diversity and inclusion statement in the syllabus or the welcome message (Nortvig et al., 2018). In addition, we suggested that course materials should represent different perspectives and identities. Students' voices should be invited and used for the design of learning materials and for assessing their learning rather than course evaluations. In total, we modified the 11 items across the 6 OCSQR dimensions and added 6 inclusive practices to the OSCQR rubric.
Findings
46 faculty members and 28 instructional designers in IHEs rated the importance on each item in the revised OSCQR inclusive rubric. We found all 74 participants rated 49 rubric items as important with the highest average score on “Course includes a course information area and syllabus that make course expectations clear and findable” in the Overview & Information dimension (M = 4.72, N = 74). All participants rated the Technology & Tools dimension with the lowest rating out of the 6 OSCQR dimensions. We also found that IDs tended to rate rubric criteria higher, faculty tended to view the criteria as important, as well. To get a closer look at participants’ rating of importance on the 17 DEI-focused rubric items, the results showed that the 46 faculty participants tended to rate 7 rubric items across 6 dimensions lower than the 28 IDs. Among those 7 rubric items, we found that the 46 faculty participants tended to rate 3 rubric items as lower than important in designing inclusive online courses, i.e., course privacy policies for technology tools and data (M = 3.41), the use and format of tables (M = 3.63), and student autonomy in choosing assignment options (M = 3.93).
Discussion
Although there were some differences, the overall findings suggested that IDs and faculty members had common ground regarding their perspectives on this inclusive OSCQR rubric. Both faculty and IDs believed most rubric items to be important or very important in designing inclusive courses. In addition, our findings suggested years of experience in course design did not seem to matter regarding how course developer professionals evaluated rubric items, which aligned with existing literature (Martin et al., 2021). Our results showed the challenge of integrating technology tools into course design. We will also discuss the implications of related findings for professional learning programs with participants.