Building Variable Assessment Questions for Adaptive Learning Courses: Creating Variety and Challenge Efficiently

This workshop will provide (1) a conceptual overview of the types of questions that can be constructed in adaptive learning systems that provide a large number of random possibilities and (2) time to construct samples in the Realizeit adaptive learning system.  This is a non-commercial session just using Realizeit as a sample system because of the instructor’s experience with the system.  The skills learned in Realizeit will be able to be implemented with other systems.

 

Goals:

By the end of the session, participants will be able to

• Explain the need for large numbers of assessment questions in adaptive learning systems
• Create banks of Generated Variables
• Create Calculated Variables
• Create Multiple Choice questions with banks of correct and incorrect answers
• Create Free Response problems for math problems with randomized numbers
• Create Multiple Choice questions with calculated correct and incorrect answers
• Plan for machine scorable assessment questions that can have a wide variety of possibilities for integrity and learning purposes

 

Materials:  The supporting PowerPoint will be shared through the conference website and app.  Use of the Realizeit adaptive learning authoring tool will be available during the session.

 

Session Plan (45 minutes)

1. Quick overview of Adaptive Learning systems (2 minutes)
2. Rationale and strategies for planning for assessment questions that allow for greater reusability and variability (3 minutes)
3. Creating a Multiple Choice question with multiple combinations of correct and incorrect choices (5 minutes)
4. Creating Generated Variables and creating a Multiple Choice question with a bank of correct and incorrect choices (10 minutes)
5. Using Generated Variables with a Categorization Question (5 minutes)
6. Creating Free Response math questions with calculated and generated variables (5 minutes)
7. Creating a Question Stem with a random part (5 minutes)
8. Q & A (5 minutes)

 

Adaptive Learning systems provide for personalized pathways through the curriculum of a course.  Rather than a time period for a course with variable outcomes of learning, the goal is for a set level of learning to be produced by customizing the amount of time necessary for a student to learn the content based upon their prior knowledge and other abilities.  Content is broken into small chunks that students progress through as they are able to demonstrate mastery.  Adaptive systems use machine learning to predict both future success and provide recommendations and also adjust past “knowledge scores” as the system gains further knowledge about a students’ abilities.

Adaptive learning systems are able to personalize content and pathways based upon frequent assessment of student learning.  Each “chunk” of learning in an adaptive system often has about 10-15 minutes of learning via video, audio, or text.  This is associated with 3-8 questions.  Student learning is not measured as a percentage of correct answers, but rather through machine learning algorithms that estimate a student’s knowledge state (in a manner similar to item response theory).

It is not sufficient to have a question bank of only 3-8 static questions per learning chunks for several reasons.  First, for academic integrity, a small number of static questions may provide a strong temptation to cheat through communication with other students in the class.  Second, for students needing multiple exposures to the same learning content, they may give correct answers based upon memorizing the answers to this small set of questions.  In most cases this does not result in greater learning.  Instructors and Instructional Designers (ID) could choose to write large banks of assessment questions, but this workload can become unsustainable for an entire course.

Several strategies exist to maximize the multiple reusability of questions in an adaptive system.  This workshop will provide an opportunity for instructors and IDs to explore design opportunities in one adaptive learning system that provide for randomized banks of responses as well as randomization in the parts of a question.

 

Multiple Choice Questions with Variable Combinations of Correct or Incorrect Answers:  These questions can be created with multiple options for the correct answer, although only one will be shown at a time.  Additionally, multiple distractors/incorrect answers are also constructed.  The system only shows one correct and three incorrect answers each time the question is asked.  This provides a large combination of possible questions with the same stem.

Example:

Question: Which of the following is a gaseous planet?

Correct Responses:  Jupiter, Saturn, Uranus, Neptune

Incorrect Respones: Mercury, Venus, Earth, Mars

 

Displayed as:

Which of the following is a gaseous planet?

• Earth
• Venus
• Jupiter
• Mercury

 

Multiple Choice with Options from Random Sets: This question type is similar to the previous one, but banks of responses are created.  Not only are there multiple combinations of responses, but the banks can be easily reused with other questions providing ease of question construction.

Example:

Question:      Which of the following elements is an alkali metal?

Banks:        Alkali Metals: Li, Na, Rb, Cs, Fr

        Alkaline Earth Metals: Be, Mg, Ca, Sr

        Transition Metals: Fe, Au, Ag, Pt    

        Noble Gases: He, Ne, Ar, Kr, Xe

 

Displayed as:

Which of the following elements is an alkali metal?

• Be
• Li
• Ag
• Kr

 

Categorization Questions with Multiple Responses: A bank of responses are created from which a set number is pulled each time the question is asked.  Students categorize those random responses.

 

Example:

Categories: Metals    Nonmetals

Bank:        Ag-Metal        Ne-Nonmetal

        Au-Metal        C-Nonmetal

        Pt-Metal        P-Nonmetal

        Fe-Metal        Cl-Nonmetal

        

Displayed as:

Drag each element to the appropriate category.

Metals            Nonmetals

 

Ne        Ag        Pt        P

 

Free Response Math Problems with variables:  For problems requiring calculation, variables can be created so that every time a student receives a question, they get random numbers from which to perform their calculations.

 

Example:

Variable vi: a number between 1.0 and 9.0 with an interval of 0.1

Question:  A ball is tossed upward with an initial velocity of [vi] m/s.  How high will it go in the air?

Hint:  vf^2 = vi^2 + 2 g d where g = 10 m/s/s

Expected Answer: vi^2 / 20 meters (check answer to 2 significant figures)

 

Displayed as:

A ball is tossed upward with an initial velocity of 3.2 m/s.  How high will it go in the air?

 

______ m

 

(Expected Answer will be anything that rounds to 0.51, e.g. 0.512, 0.514, 0.509)

 

Creating a randomized math problem with multiple choice options (creating distractors based upon misconceptions)

 

Multiple Choice Math Problems with Calculated Options:  An instructor or designer may feel that providing multiple choice options for a calculation problem are helpful for students.  Distractors can still be constructed as to keep the problem well constructed.

 

Example:

Variables: A: integer from 1 to 10; B: integer from 1 to 10

Question: Solve the Equation [A] x + [B] = 13.

Answers:      (13 - B)/A (correct)

(13 - A)/B

13/(A+B)

13/(A-B)    

 

Displays as:
Solve the Equation 2 x + 3 = 13

• 2.6
• -13
• 3.67
• 5

 

Participant Engagement 

Participants will be working in a sandbox version of the Realizeit adaptive learning system.  They will be able to create their own questions using variables during the session.  The skills will actually be transferable to other adaptive learning systems as well.  This will not be a commercial session.