From Validity to Validation: How to Use Validation for Better Writing Assessment — CCCC 2009 — Session D09

• Michael Williamson, “Validity and Bias in Writing Assessment”
• Les Perelman, “The Five Paragraph Essay Makes People Stupid and Machines Smart”
• Brian Huot: “How I learned to Stop Worrying and Love the 5-Paragraph Essay”
• Nancy Glaser, “One of Many Myths: Does the Five-Paragraph Essay Sink or Swim in Large-Scale Writing Assessments.”
• Edward White, Respondent. (White recently published an amusing “satire” of a five-paragraph essay, which appeared in College Composition and Communication 59:3 (Feb 2008): 524-526

Another chapter in the love-hate relationship comp teachers have with formulaic assessments. (Mostly hate, with some codependency issues.)

Michael Williamson, IUP: Bias in Assessment

From PoV of rhetoric — systemic bias is at the heart of our work. James Berlin, theory of cultural studies based on examination of race, class and gender.

Assessment is a technology, it is research, any material artfifact of culture, technologies “have no essential nature” – their use defines their role in culture.

Presents Winton Manning, ETS researcher.  “What can happen when a smart person… gets involved with testing.” Testified for the teachers, in a Texas case in which testing for content led to a high rate of minority failures; the court found that the tests were inadmissible b/c there was no research to support them.  

Michael Bakke denied admission to Stanford because he was too old… “He successfully screwed… [much laughter] sued for discriminatory admissions.”  Bakke said that you set a minimum cut score on tests, above which the applicant is assumed to be able to succed, and select for diversity anywhere above that level. Bakke Measure of Academic Talent (MAT) to address bias in race, class, gender, to ensure social justice.  A bilingual poor student who scores above expectations should reward that student, while a student with a more privileged background should be rated lower.
Norbert Elliot, “Validity and the Role of Quantitative Data in Writing Assessment”

Elliot invoked the tattoos of Holocaust survivors to note the horrors that result when a student is reduced to number. Invoked a specific student who really needed “better observers” in order to help him progress in his writing.  Our decisions about what happens to students rest on a process of validation.

New Jersey Institute of Technology has a portfolio model constructed to produce evidence that the students are, or are not, meeting curricular goals.  This  model has good predictive value.  “Accuplacer Write Placer” has low reliability. 10%  

Looking at writing and critical reading sections of the SAT… 60% “Percent of Variance Explained”

Looked at course grade as the outcome, the grade as the predictor, there is very low correlation.

Locally-developed portfolio measures are the best way to assess student achievement.  What about placement? (Incoming class 2010 will use SAT, not WritePlacer)  Locally-developed portfolio system displaced a computer product.

Les Perelman, “The Five Paragraph Essay Makes People Stupid and Machines Smart”

The current interest in the 5-paragraph essay is related to the desire of testing companies to have essay assessments graded by machines.  Rumors that the SAT by 2015 will be machine-graded. Testing companies want machine scoring because it is cheap, much more reliable than humans, it’s fast, and the “construct validity can be manufactured”.

Testing companies emphasize the 5-paragraph-essay – written quickly, graded quickly, reliably graded, compatible with machine scoring – machines count very well.  Machines do not understand meaning very well.

E-rater and tired teachers of writing and, “Poor AP English Graders” can all achieve a high reliability by counting their way through the five-paragraph essay.  (“Does the essay as a whole contain a sufficient number of infrequently used words?”)  

Readers try to emulate the scores of the machines because readers don’t want their essays to be read by supervisors, so they start to grade like a machine.  

What’s wrong with the 5-paragraph essay? It privileges reliability over validity. (The arrows all stay together, but they miss the target.)

All  writing is unnatural, but the 5-par essay is “a really unnatural act” (showed image of a gymnast contorting her body, basically sitting on her own neck).

The 5-par essay prevents students from using a topic thesis statement as thesis.

Privileges form over content.  Used example of Bush’s rhetoric on invading Iraq.  Got an applause line saying of the 5-paragraph essay, “Just say No!”

Brian Huot: “How I learned to Stop Worrying and Love the 5-Paragraph Essay”

Rethinking our assumptions about “the presumed evil” of teaching and rewarding the learning presented in the 5-paragraph essay.  Brief comments will focus on pedagogy.  

Arguments of the pros and cons (and cons and cons) on comp list groups about formulaic and academic templates (“they say, I say”).   Any performance-based activity would be a good analogy.  In all case, the exercise and the measurement are a synecdoche. Is it unreasonable for a HS coach to ask a player to demonstrate a few layups as part of the assessment process.  The traditional dribbling and layup is hardly creative, it’s not a large part of basketball, but doesn’t it make sense?  Students who will eventually enter an environment where they will be penalized for double-dribbling need to be exposed to the rules.  

Not teaching basic structure and organization is like teaching future tennis players how to hit a ball without telling them that, when they get into the game, there will be a net and boundaries.  We teach fundamental, necessary building skills in other disciplines, without claiming that these skills prevent them from future creativity.  Teaching first-year journalism students how to write a five W lead does not exclude them from becoming future Tom Wolfes.  

Invoked the idea that the 5 paragraph essay can be taught badly or well; as an end, or a step.

We tend to forget how much we, as readers, tend to rely on formulae when we read. Those students who have never been taught to use structure cannot read it…   Being taught a layup will not prevent a player from doing a slam dunk, though the layup is not enough to make a good player.

Nancy Glaser, “One of Many Muths: Does the Five-Paragraph Essay Sink or Swim in Large-Scale Writing Assessments.”

Challenges…
…the myth that high-stakes essay prompts are designed to generate formulaic answers.
…the myth that a formulaic essay guarantees a high score, because essay readers are trained to score based on the formula.

Readers are trained neither to reward or penalize the five-paragraph essay; many readers would actually like to penalize the five-paragraph writing.  Readers are taught to ignore the formula. “Check your personal standards at the door,” and score on the basis of the quality of the writing.

• Clear position
• Insightful development
• Logical organization
• Well-chosen reasons and examples
• Facility with language
• Command of conventions of written English.

Showed examples of both low- and high-scoring papers that used the 5-paragraph formula, with the variance having nothing to do with the formula.  In the low-scoring essay, the supporting points did not cohere.

Edward White, Respondent.

Baldwin began by referring to the academic template, referred to White’s satire CCC Feb 2008.  After being an AP table reader… why do we have so many papers in which the entire submission, after taking an AP course, the student write “I don’t know how to do this sort of thing.”   Why did so many really good writers end up writing really poor essays?  He imagined himself as a cocky 18 year old who is a pretty good writer, defending the five-paragraph theme.  Realized how strange the form is, since the writing existed specifically to pass a test. The fact that he ended up doing pretty good writing was irrelevant, since he could only write three points (which prevents you from thinking too much, which is important for a timed writing situation).

The five-paragraph theme is like training wheels. It teaches a formula, and there are many writers who cannot produce anything until they have a formula. The important thing about training wheels is you eventually take them off when the kid learns how to write.  Many good writers are writing with the training wheels still on.  If writing is thinking made visible, then the training wheels really need to come off.  The 5paragraph theme is very restrictive to people who get past a certain point. But it’s useful for students who are still working on sentences.  

Mike Williams noted that we now have a new definition of validity, not an inherent property of a test (as reliability), but rather has to do with the use of test scores.  You can’t say a test is valid, you can say the use of test scores is valid, useful, consistent with the goals of the test.  

Consequential validity: (misuse of tests because the results were used improperly, dishonestly, in a biased fashion). The invalidity of students being trained to use the 5-paragraph theme.  We need to work with teachers to suggest other alternatives.

Large-scale placement testing is not a valid instrument for your institution. The only good placement test is going to be one tuned to your students, your program, your institution. We have to give up the dream that we can solve the placement issue by giving a large-scale test.  (The SAT “Writing Test” is basically a multiple choice test with a little impromptu writing thrown in.)

Placement is necessary, but we cannot rely on large-scale national tests to do our tests.  Directed self-placement is the wave of the future.