Student Authored Questions

Creating questions for assessments has long been considered a task for subject matter experts. When creating a large number of questions, this process can be tiresome and prone to error without sufficient self review. The process required to create questions and, furthermore, quality questions has been formalized by instructional designers. Guidelines for multiple choice questions are widely known and are impressed upon those creating questions. These guidelines form the basis for a process whereby students can also create high quality questions.

When a subject matter expert creates questions they are working towards an outcome. Typically that outcome is to assess knowledge of a particular concept or topic. A variety of questions are created that assess the student's cognitive abilities. These questions tend to align with Bloom's Taxonomy; with the number of questions at each level relating to the expected domain knowledge. When students are first tasked with creating questions they tend to create only questions at the first level: remembering. Students also find it faster to create these questions with the basic question creation guidelines and look to accomplish the task quickly.

In order to encourage creating questions for higher levels, two strategies are utilized. The first is to educate the student on the differences between question types. Many have never had to create questions, even ‘remembering’ questions, and they require instruction on what it means to ‘understand’, ‘apply’, or ‘analyze’. The second strategy is through gamification and virtual rewards, such as badges and tokens. Awarding greater incentives to higher level questions motivates students to create questions that are more challenging.

There is also great importance in providing simple interfaces for students to create questions in. Complex interfaces that lay out the full question and all the elements may work well for instructional designers and those that administer an LMS, but they are overwhelming for students. Breaking question creation into multiple steps with each step providing instruction and feedback provides a guided process for new question creators. The interface also provides feedback so that students can correct minor errors as well as provide sufficiently long questions.

After a student has created a question it must be evaluated both through automated means and by people. Automated systems check for inappropriate materials, that the question meets certain minimum standards, and utilizes natural language processing (NLP) to determine the Bloom's Taxonomy level. After automated checks, the question enters a peer review process where both students and instructors evaluate the question. Each question is viewed multiple times and peers are asked to answer the question both correctly and incorrectly. If during this process an issue is encountered then the reviewer may reject the question and feedback is provided to the student that created the question.

Through the repeated process of creation and review, students gain proficiency and understand how questions relate to the topics that they assess. While the students are not as efficient as subject matter experts in creating questions, the goal of improved understanding of the materials is achieved. This is measured through assessment of students on questions created by subject matter experts and comparing those that engaged in the creation process versus those that did not.

Attendees at this session will learn about the step by step process that students create questions with, the automated feedback systems, and the peer review process. Conversation will be encouraged, particularly around engaging across a variety of different courses along with various question types. Discussion on the development of upcoming advancements will be included. Opportunities for collaboration will also be explored in this session.