The Problems with Peer Grading on Coursera
We are witnessing "Plagiarism at Scale"
If you have enrolled in a MOOC on Coursera, chances are you’ve submitted a peer-graded assignment.
Peer-graded assignments - unlike auto-graded assignments - rely on the feedback of other students. On paper, this makes sense: several critics pointed out the limitations of auto-graded assignments. And, while, auto-graded assignments might work for certain coding exercises, it most certainly does not work well for descriptive/creative/exploratory projects. In peer-graded assignments, you receive descriptive feedback from your fellow studymates. And, unlike auto-graded assignments, peer-graded assignments are not binary: students understand that certain answers can vary, and the answer isn’t always a 1 or 0.
Coursera is one of the largest MOOC platforms today, helping millions of students worldwide. It’s also an experiment in progress for students and educators, who are witnessing online learning at scale. But, while creating and sharing content with millions of learners is achievable, Assessment, Feedback, and Communication have still not been achieved at scale. In this post, I’d to highlight some of the challenges with peer-reviewed assignments and suggest a few ways we can help improve them.
The Problems
Problem 1: Plagiarism/Cheating
On Coursera, just like in real educational institutions, plagiarism is quite prevalent. You don’t even need to copy answers from papers or textbooks. Just make a quick search on Google or YouTube, and you will find something like this:
Unfortunately, we are witnessing “Plagiarism at Scale”. Of course, this is not the MOOC platform’s intention, but ours.
A few keywords, often taken from the assignment prompt itself, are usually all that’s needed to find solutions online. Some popular courses even have complete walkthroughs on YouTube. While providers usually ask students to uphold an Honor Code, sometimes checking a box before submitting an assignment, doesn’t seem to do much to deter motivated cheaters.
Problem 2: Delays
Let’s say you are an honest student. You enroll in a MOOC, study the material, and submit your work on time. In order to pass an assignment (on Coursera), your work needs to be graded by at least 3 students. Plus, you need to grade the work of 3 other students too. Several students have reported a delay in getting their assignments graded - thus, the delay in completing the MOOC.
Delays occur when no one is available to review your assignment. Some course forums are full of pleas to “Please assess my assignment”. In fact, requesting reviews on the forum is one of Coursera’s own recommendations, but not a very useful one considering many course forums are deserted.
This problem is worsened when students enroll in a specialization and pay on a monthly subscription basis. Due to delays, you might end up paying for an extra month.
Since students are not studying synchronously, you might have to wait till another student completes the assignment and shares his feedback.
Problem 3:Lack of Quality Feedback (and the lack of feedback on feedback)
Essay-styled feedback sounds great on paper. But, in reality, most students do not give critical feedback on your work. There are two main reasons:
1. Creating critical feedback takes a lot of time and energy.
2. Peer-graded feedback assumes that students are qualified enough to rate each other’s work.
But, most students are enrolling in the MOOC for the first time. If you misunderstand a concept, you are going to give improper feedback as well.
In addition, there is no way for you to give feedback on a feedback. If a student provides low-quality feedback, or improper feedback (due to the student’s misunderstanding), there is no way you can correct your peer and try to clarify his error. Thus, erroneous feedback is considered feedback.
Problem 4: Anonymous Feedback
No one knows who’s assessed one’s work; no one knows who they’re assessing. While ostensibly meant to protect student privacy, this raises some serious when you can’t ask for clarification when you can’t gauge your feedback based on what you know about the author and their particular strengths and weaknesses, and unfortunately when anonymity brings out Internet trollishness in some students who feel they can leave nasty comments without any repercussions.
Admittedly, Coursera does ask for ID verification for certain courses. But, you will still find proxy/troll accounts giving improper feedback.
Possible Solutions
1. Cohorts
This is perhaps the best solution to tackle all the challenges of non-sync learning. Cohort learning means studying a MOOC live, in sync with a group of students. This ensures that all the students are studying the course at the same rate. You will receive feedback on time (or, minimum delay), since students have to meet deadlines and work week-by-week, the number of proxy/troll accounts will also decrease considerably.
Cohort-based courses (CBCs) are considered the next iteration of online learning. The only problem is that CBCs are not scalable - at least not to the extent of pre-recorded courses.
2. Less emphasis on certificate, more on learning
A reason you find rampant plagiarism and troll accounts is that some students enroll in MOOCs exclusively for the certificate. We need to inculcate this culture of learning for the sake of development and growth and keep the certificates secondary.
I like the fact that Coursera (even edX) provides the option of auditing courses. Thus, students can at least study the materials for free.
Lastly, there are also 100% free MOOCs on Coursera, that you can study for free - and you do not provide any certificate on completion.
3. Quality Feedback
Instead of providing essay-styled text submission, students can present their work through audio/video. This will obviously require more time and resources, but it is also a richer format. In video submissions, students can present their work as a presentation, showcasing their project/assignment, while explaining how they built it.
A video submission also helps the grader make a better judgment about the student’s work. If you are able to explain your work in simple words, that’s a good sign that you actually understood the concepts you used.
Lastly, for big projects, these video submissions can help you find your next internship/job. Again - you will spend a considerable time on filming and editing - but, the returns are high as well.
4. Points on Feedback
Apart from receiving points from someone’s feedback on your submission, students can also receive points if they provide critical feedback.
If you receive feedback from 3 students, you can select the best feedback from the list, and this user will receive a few points/marks for that.
Thus, MOOC platforms can incentivize users to provide high-quality feedback.
5. Penalty for missing deadlines
Just like incentivizing high-quality feedback, MOOC platforms can penalize students for late submissions. This could mean a points/marks deduction. Or, perhaps, if you keep missing the deadlines, you might be deterred from the MOOC until a certain date, or you won’t be able to access some features.
6. Limit the number of submissions
In the peer-graded assignments, once you submit your work, you can grade (and study) other students’ submissions. Unfortunately, some students misuse this system. They will submit a black file/document, look at other’s submissions, and copy-paste it. They will then make another submission, with the copied material.
Not only does this compromise the cheater’s work, but it also undermines the honest student’s work.
With unlimited submissions of auto-graded multiple-choice quizzes and peer-graded assignments, it’s easy to pass and receive a certificate without learning anything, and you don’t even have to cheat to do it.MOOC platforms can limit the number of submissions. Or, at least provide a time delay or mark deduction for re-submissions.
Conclusion
This analysis comes from a place of concern and respect. I truly appreciate Coursera’s achievements. It has changed hundreds of thousands (if not millions) of lives, all across the world.
In order to help the educators, students, and the online learning community, it’s important that we address the problems that we are facing today, so that can build a better tomorrow.
