Improving peer assessment accuracy by incorporating relative peer grades

Massive Open Online Courses (MOOCs) have become more and more popular recently. These courses have attracted a large number of students world-wide. In a popular course, there may be thousands of students. Such a large number of students in one course makes it infeasible for the instructors to grade all the submissions. Peer assessment is thus an effective paradigm that can help grade the submissions at a large scale. However, due to the variance in the ability and standard of the student graders, peer grades may be noisy and biased. Aggregating peer grades to have an accurate and fair final grade for a submission is a challenging problem because the reliability and bias degrees of graders are usually unknown in practice. To address this issue, some probabilistic models considering the graders' reliability and bias are proposed. However, due to the sparsity of peer grade observations, it is difficult for these models to estimate the accurate reliability and bias of the graders as well as the true grades of the submissions. Compared with absolute peer grades, the relative peer grades, derived from the difference between the peer grades of two submissions graded by the same grader, are less sparse and more robust to the grader's bias. Thus relative peer grades are informative and helpful in cardinal peer grading estimation whose goal is to estimate the absolute numeric grades of submissions. In this paper, we propose two new probabilistic models to help improve the accuracy of cardinal peer grading estimation using the observed relative grades among submissions. In this way, the relation between the true grades among submissions is taken into consideration when deriving the final grades. Experimental results on real MOOC peer grading datasets show that the proposed models outperform baselines and the relation of true grades among submissions indeed contributes to the improvement in the grade estimation.