Most LMSs have tools that let students assess each other's work, but third-party providers may offer streamlined workflows and better visualizations of the reviewing process. We examine reasons for using peer assessment, and features for reviewing special kinds of work, like writing, case studies, and projects in the visual arts.
In almost any online course, students interact with other students. Very often, interaction takes place on some kind of discussion board, and usually grades are assigned by the instructor. An alternative is peer assessment.
Why peer assessment? In peer assessment, students are asked to give feedback to particular classmates on their work. This gives them more direction than simply being asked to comment in a discussion, where it is not clear to whom they should respond. This may make it easier for them to provide intelligent feedback, and it is easier to see if they have carried out the task correctly. Instead of the instructor unilaterally assigning the grade, student feedback can play a role. Often, reading peer reviews will give the instructor greater insight into the strengths and weaknesses of a piece of work. Thus, peer assessment can complement other assessment practices in an online course.
Why look beyond your LMS for a peer-assessment (PA) system? Virtually all LMSs come with peer-assessment modules, so one might ask, Why look elsewhere? The answer is that new peer-assessment applications offer more flexibility in workflow, and better visualizations of the rich data created in the process of peer assessment. Modules within LMSs often assign a fixed set of reviewers to each student's work, and display the reviews in tabular form to the instructor. Third-party tools frequently assign reviewers dynamically, so that no author loses out because their reviewers fail to do the reviews they were assigned, and they also have review visualizations that make it easy to see where reviewers agree or disagree on a particular piece of work.
Some systems offer support for training reviewers. Calibrated Peer Review, for example, has students complete three "calibration" reviews before going ahead to review other students' work. The student's assessment of the calibration work is compared with the instructor's assessment, and the student is shown how each of their responses compares with the instructor's response to the same question about the reviewed work.
Several systems enable multiple rounds of review, so that students can revise their work based on reviews they have received, and then be re-assessed by the same reviewers. Many support team projects, and some, like Spark Plus and Expertiza, allow students to rate contributions by their team members, as well as the work product of other teams. Many compute students' scores based on both their submitted work and their reviewing—though it should be noted that the instructor always has the option of assigning a final grade that is different from the one derived from the peer reviews.
Which PA systems are easiest to use? Many 3rd-party systems are LTI compliant, meaning that they can be treated as tools from within an LMS. Students need not sign in to the peer-assessment system separately after they sign into the LMS. Peer-reviewed assignments appear like any other assignment in the LMS's calendar, and grades assigned within the peer-assessment app show up in the LMS's gradebook. Some systems, such as Peergrade.io, allow students to create accounts and sign up via Clever, a platform that is used by more than 50% of K-12 schools in the US. A few systems, such as Mobius Slip, let reviewers write comments directly on the reviewed document, instead of just writing a prose review that describes how the work can be improved.
What are the main choices in selecting a PA system? Do you want to give students a detailed rubric to use in assessing their peers' work, one that includes several questions to be answered with either a numeric rating, a text comment, or a checkbox? This is called an analytic rubric, and is helpful in drawing the students' attention to salient aspects of the reviewed work. But some systems, for example, Critviz and Peerwise, use holistic rubrics, where students are asked to write a single piece of prose to assess the work as a whole.
Do you want your students to rate their classmates' work, or do you want them to rank students' work against each other? Advantages of rating include the ability to give two different works equal ratings, and the fact that rating makes it easier to use an analytic rubric that calls students' attention to all the important features of the reviewed work. However, ranking exhibits better inter-rater reliability (two people ranking the same artifacts are more likely to rank them in the same order than they would be to rate them at the same points on, say, a 7-point scale). Critviz is a purely ranking-based system, while systems like peerScholar and Eli Review support both rating and ranking. Most other systems are rating based.
Some systems are task based, whereas others are assignment based. whereas others are task based. In an assignment-based system, each assignment can consist of several phases, e.g., submission, review, resubmission, re-review, and teammate evaluation. In a task-based system, the instructor does not create an assignment, but rather a set of submission and review tasks that are linked together and follow each other in some order. It may be faster to set up a peer-reviewed assignment in an assignment-based system. But a task-based system has the advantage that each task can be treated as a separate activity in an LMS, so that all submission and reviewing deadlines can appear on the LMS's calendar. Assignment-based systems predominate, but Critviz, Eli Review, Emarking, and OSBLE are examples of task-based systems.
What systems are specialized for particular kinds of assignments? If you are teaching writing, then it makes sense to choose a system that has special features for evaluating prose. Examples are Eli Review, My Reviewers, Peerceptiv, and WEx (Writers' Exchange). If students are evaluating each other's software code, OSBLE has support for that. Mobius Slip grew out of case-study courses, where each team of students works on a separate case. Students working on a particular case will review others working on the same case. Critviz was designed to provide an electronic version of an art or design critique.
What are the costs? Systems may start out being free, but all software requires support, and support costs real money, so every large system charges for use. Three models are in use: student pays, department pays, and run-your-own. Of the systems that charge students, we have seen charges from $1 per month to $20 per semester. When the department pays, the cost is usually in the hundreds of dollars per semester. Severa; vendors allow schools to run their PA applications on their own servers. This helps reduce the cost, and it also helps schools that have restrictive policies on where FERPA data can be hosted.
Engagement. The questions asked in this abstract are amenable to audience interaction. For example, the audience can be asked, Name one advantage (or disadvantage) of peer assessment compared to message boards. Or, What are the advantages/disadvantages of using modules within your LMS compared with 3rd-party tools? When we get to discussing the features, the audience can be asked to estimate, for example, how many systems are ranking-oriented, or how many are task-based. Then the speaker can present the actual data.
Summary. This presentation will give attendees a good sense of the advantages of peer assessment and the capabilities of PA tools. It should be of interest to instructional-technology administrators, faculty developers, and individual faculty members. Even those who do not hear of a PA tool that can solve their needs, will know the right questions to ask in choosing a technology for peer assessment.