How To Crowdsource Grading

Top-down grading by the prof turns learning (which should be a deep
pleasure, setting up for a lifetime of curiosity) into a crass
competition: how do I snag the highest grade for the least amount of
work? how do I give the prof what she wants so I can get the A that I
need for med school? That’s the opposite of learning and curiosity,
the opposite of everything I believe as a teacher, and is, quite
frankly, a waste of my time and the students’ time. There has to be a
better way . . .

So, this year, when I teach “This Is Your Brain on the Internet,”
I’m trying out a new point system supplemented, first, by peer review
and by my own constant commentary (written and oral) on student
progress, goals, ambitions, and contributions. Grading itself will be
by contract: Do all the work (and there is a lot of work), and you
get an A. Don’t need an A? Don’t have time to do all the work? No
problem. You can aim for and earn a B. There will be a chart. You do
the assignment satisfactorily, you get the points. Add up
the points, there’s your grade. Clearcut. No guesswork. No
second-guessing ‘what the prof wants.’ No gaming the system.
Clearcut. Student is responsible.

But what determines meeting the standard required in this point
system? What does it mean to do work “satisfactorily”? And how to
judge quality, you ask? Crowdsourcing. —Cathy Davidson, HASTAC

4 thoughts on “How To Crowdsource Grading”

Right, and I think that demonstrating knowledge of mathematical processes, or laboratory procedures, or dissection of cadavers and such like, are among the learning activities that wouldn’t translate well to this way of thinking. But there’s probably some learning in every subject that could be delivered in this manner. (Maybe, for instance, having the advanced students check the progress of the beginner students, all under the watchful eye of a reputation management system, so that if two students who usually get the “correct” answer both give a “no” to a student submission, the instructor could generally trust that answer — not for formal grading, but as a way of filtering the students who are “getting it” (and don’t need to do any more busywork on this level) and those students who are struggling (and therefore would benefit from more of the instructor’s personal time).

Dennis, I think that this is a very difficult process in my courses. I need to see that a student has mastered certain techniques. The successful completion of a problem (or a project or a proof) requires applying the rules correctly, and not making any arithmetic or algebra errors. I do not see how this process can be graded in this fashion. I think many “skill-mastery” courses would be in the same boat. Perhaps this is just my math-bias showing through, but I think this assessment method really could only work in certain courses…

Sean, if I understand you correctly, you seem to suggest that even if a student demonstrates understanding of the material, something is wrong if the course permits that student from doing work that won’t actually teach the student anything they don’t know. If the course is well-designed, the A students won’t have to do mindless busywork when they already know the material, they can jump ahead to the more challenging assignments.
So, what is more important — making sure students doing all the work (even when the student has demonstrated A-level mastery of the subject), or designing a course so that students who demonstrate A-level mastery of the introductory material can move ahead faster? I’ve never had a student work so far ahead that they submit their final paper weeks early and then just sit and twiddle their thumbs — if anything, working ahead is a strategy that lets the best students spend more time on the final projects that carry the most weight.
While I often quote passages simply because they are interesting and not necessarily because I agree 100% with them, I do see the value in this suggestion. Because people who are active on the internet are used to rating everything online, from books and movies to restaurant service to posts in forums, that it’s a logical step to have a group assess work in certain controlled instances.
But when I have students design websites, I as a 40-year-old with weaker eyes, and I as a textually-biased person would have a completely different reaction to a website that a 20-year-old designed to appeal to other 20-year-olds, and the only way I can get a fair sense of whether the website meets its goal (of appealing to 20-year-olds) is to get the website designer to ask other 20-year-olds.
Of course I still grade the student on meeting deadlines, on class participation, and various other assignments, but depending on the assignment — and note that Davidson is teaching a course called “This is Your Brain on the Internet,” so having students evaluate the success of their peer’s work makes a certain amount of sense.
I wouldn’t have a group evaluate something like the rules of grammar or Latin declensions or the laws of thermodynamics, since those are factual. But for these strictly factual details, I could already just give a multiple-choice test, graded by a computer.
Sean, I think one important factor that you’re missing is that students learn quite a lot when they evaluate each other’s work, and that the higher-order skill of evaluation (and defending an evaluation with evidence) is something a teacher can’t teach by forcing students to sit through lectures.

no no no no since when did everything in school change to this ” there are no losers” every thing is a valuable contribution , pass the buck on to the student if they want to it’s up to them …. people need to be pushed and challenged the bar needs to be SET this moving bar thing is the path to the mediocre ,and the big fantasy is that this ok…… well it is not you are not doing your job ! each will respond to there own talents …but when you start asking a 3 year to order when you are out to eat ? you are already not taking your job as parent /teacher to heart . you are the teacher , you must also be ready to make mistakes on your own ….no peer review no thinking by committee ……..do all the work you say ,get an A learn all the material understand it but do not do all the work ….get an C and so on ,you have now taught the ones who get things quickly to slake off ,and the ones who need to work a bit harder to settle for less .make sense?

Dennis G. Jerz says:

8 Sep 2009 at 1:48 pm

Right, and I think that demonstrating knowledge of mathematical processes, or laboratory procedures, or dissection of cadavers and such like, are among the learning activities that wouldn’t translate well to this way of thinking. But there’s probably some learning in every subject that could be delivered in this manner. (Maybe, for instance, having the advanced students check the progress of the beginner students, all under the watchful eye of a reputation management system, so that if two students who usually get the “correct” answer both give a “no” to a student submission, the instructor could generally trust that answer — not for formal grading, but as a way of filtering the students who are “getting it” (and don’t need to do any more busywork on this level) and those students who are struggling (and therefore would benefit from more of the instructor’s personal time).

Joshua Sasmor says:

8 Sep 2009 at 12:55 pm

Dennis, I think that this is a very difficult process in my courses. I need to see that a student has mastered certain techniques. The successful completion of a problem (or a project or a proof) requires applying the rules correctly, and not making any arithmetic or algebra errors. I do not see how this process can be graded in this fashion. I think many “skill-mastery” courses would be in the same boat. Perhaps this is just my math-bias showing through, but I think this assessment method really could only work in certain courses…

Dennis G. Jerz says:

8 Sep 2009 at 9:59 am

Sean, if I understand you correctly, you seem to suggest that even if a student demonstrates understanding of the material, something is wrong if the course permits that student from doing work that won’t actually teach the student anything they don’t know. If the course is well-designed, the A students won’t have to do mindless busywork when they already know the material, they can jump ahead to the more challenging assignments.
So, what is more important — making sure students doing all the work (even when the student has demonstrated A-level mastery of the subject), or designing a course so that students who demonstrate A-level mastery of the introductory material can move ahead faster? I’ve never had a student work so far ahead that they submit their final paper weeks early and then just sit and twiddle their thumbs — if anything, working ahead is a strategy that lets the best students spend more time on the final projects that carry the most weight.
While I often quote passages simply because they are interesting and not necessarily because I agree 100% with them, I do see the value in this suggestion. Because people who are active on the internet are used to rating everything online, from books and movies to restaurant service to posts in forums, that it’s a logical step to have a group assess work in certain controlled instances.
But when I have students design websites, I as a 40-year-old with weaker eyes, and I as a textually-biased person would have a completely different reaction to a website that a 20-year-old designed to appeal to other 20-year-olds, and the only way I can get a fair sense of whether the website meets its goal (of appealing to 20-year-olds) is to get the website designer to ask other 20-year-olds.
Of course I still grade the student on meeting deadlines, on class participation, and various other assignments, but depending on the assignment — and note that Davidson is teaching a course called “This is Your Brain on the Internet,” so having students evaluate the success of their peer’s work makes a certain amount of sense.
I wouldn’t have a group evaluate something like the rules of grammar or Latin declensions or the laws of thermodynamics, since those are factual. But for these strictly factual details, I could already just give a multiple-choice test, graded by a computer.
Sean, I think one important factor that you’re missing is that students learn quite a lot when they evaluate each other’s work, and that the higher-order skill of evaluation (and defending an evaluation with evidence) is something a teacher can’t teach by forcing students to sit through lectures.

sean France says:

8 Sep 2009 at 9:26 am

no no no no since when did everything in school change to this ” there are no losers” every thing is a valuable contribution , pass the buck on to the student if they want to it’s up to them …. people need to be pushed and challenged the bar needs to be SET this moving bar thing is the path to the mediocre ,and the big fantasy is that this ok…… well it is not you are not doing your job ! each will respond to there own talents …but when you start asking a 3 year to order when you are out to eat ? you are already not taking your job as parent /teacher to heart . you are the teacher , you must also be ready to make mistakes on your own ….no peer review no thinking by committee ……..do all the work you say ,get an A learn all the material understand it but do not do all the work ….get an C and so on ,you have now taught the ones who get things quickly to slake off ,and the ones who need to work a bit harder to settle for less .make sense?

Jerz's Literacy Weblog (est. 1999)

How To Crowdsource Grading

Related

4 thoughts on “How To Crowdsource Grading”

Leave a Reply Cancel reply