Serious Games Summit DC 2005, Day 2Sande Chen and David Michael: Roundtable — Beyond Q & A: Assessment Methods for the Next Generation of Serious Games (Jerz’s Literacy Weblog)
These are my very rough notes. Some of the speakers didn’t give their names, sometimes I couldn’t hear their names, and sometimes they gave their names while I was furiously typing something else.
The moderators are authors of Serious Games: Games that Educate, Train, and Inform
Presenters began this roundtable by noting an opposition between “Abigail McGillicuddy,” the traditional teacher, where the buck stops, and “HAL.”
Microsoft employee: emphasized the fuzziness of knowledge as the future, as opposed to a prescriptive system that records a single correct answer. In games, knowledge is treated as binary – right and wrong. Teachers, on the other hand, recognize few absolutes, emphasizing contextualization.
Human has to be there for the time being, but technology will
Richard Carey, Pearson Education. Pre-assessment to level a student into the gameworld; in an educational context you want to place the student right away into the proper level, rather than starting them out too easy. Getting the “picky teacher” to use the software.
Does “test” and “assessment” mean the same thing to everyone here?
John Fairfield, Rosetta Stone. Think of placement rather than assessment.
I noted that testing is just one method of assessing; that assessment can be done without any kind of test.
Eric Lauber, IUP – noted that the simulation “looks more real” but that we don’t have much evidence of transferability of what’s learned in one context to a different context.
Technical consultant – “What I care about is what is this information used to sell these games,” that is, convince people that they want to use it (even if it’s given away for free). A technical consultant needs better data, “more backend.” Won’t be able to convince medical clients to use those games unless he can provide longitudinal study data that shows improvement.
David Gibson – designing simulation that helps teachers deal with kids; wants to be able to represent the intellectual growth of simulated students. Noted that Vermont went to a portfolio method for assessment. Noted Ron Stevens’ work on transference in case-based learning, doing neural network analysis on problem solving, making early predictions of the strategies students are going to use. Also referred to research on the automated SAT, so that you don’t have to keep asking them the same kind of question once you’ve determined what the student knows.
Susan McLister, editor-in-chief of a technology magazine. Noted a major challenge in education is to convince teachers that games are not evil, that there is a direct tie-in between tames and what teachers are expected to do. Embedded in a game should be measurements for higher-order learning skills.
Military speaker – one of the longer histories of using games for training. Wants to know when to use a training game, now that they have games that they believe are training. Wants confirmation that they are doing what they think they are doing, what the ROI is versus traditional methods? Improved skill level, improved long-term retention?
Owen: Referred to the “happy path” and “adaptive thinking,” noted the need to create models around not just models and skills, but also confidence.
Completion assessment vs. process assessment.
Margaret Corbit, Cornell: Games are missing a piece in which students showcase and present what they learned in their own ways. Games don’t include a way for students to express what they learned.
Pat Youngblood, Stanford: [Was distracted… missed her point.]
Kids learning in cyberspace all the time… President of the U of Washington complained that media has not filled its mission of educating the masses. The spirit of games does not involve rigid assessment… (at least not about what educators value)
Designer: Can’t find off-the-shelf products that help him do what he needs to do, and assessment is typically one of the problems. Pressure from teachers who want games tied to state curricular expectations.
Mike Gibson: Beyond the learning within the games, how is it affecting the user’s ability in the real world. (A selling point.)
Another speaker – be sensitive to demographics of the audience. Want to be thinking about what you want to assess – a skill, knowledge, cognitive ability. Simulators typically do very badly in training people. Referred to data that says simulators have negative transfer training.
Microsoft: When it comes to assessment, be careful not to think that what we define as valuable data will be adopted by the education community. Embrace whatever the standars are that the education community accepts – pointed to “No Child Left Behind” (“love it or hate it”). While teachers prefer the qualitative model, there’s a need for quantitative methods. Instead of testing against content, assess the individual user’s strategic abilities – are the individual learners improving over time. Did they use the same strategy over and over in new situations that kept leading to failure, or did they adapt their strategies based on what they learned? Useful information for the teacher to determine where a particular student is falling behind.
Cynthia Phelps, School of Information Science in Houston. Are we missing the difference between assessing our learners, and assessing the learning environment (that is, the game). Instead of just assessing how it applies to the learner, how can the game design itself be iteratively improved.
Kip Carr, Lockheed Marketing. Asked how many in the audience play videogames (plenty of hands) more than three hours a week (about a quarter of the room). Noted that gamers are being assessed all the time without knowing it.
[I then asked how many don’t play – about a quarter raised their hands, but others looked uncomfortable. A quarter is probably a low number.]
Jake Troy, doing foreign language games. Games reach people who wouldn’t ordinarily do the training, when they have the choice of watching TV or doing something else.
Robert — Health care. About games assessing the player – what kinds of methodologies are game developers using.
Chen referred to a Gamasutra article on how games assess players, and noted that a history of cheat codes and walkthroughs.
Karen from Ideas dismissed the idea that cheating is a problem. Teachers want their students to learn, they aren’t really interested in what they do to learn.
I spoke out in favor of the idea that game that can be “cheated” – such that the learner can get around the learning task and get the goal without having learned — isn’t a well-designed game. A game that rewards unethical behavior is a simulation to train people how not to get caught, that’s bad, but if the game is designed with reference material, and the game gives the student a motivation to consult the reference material, that harnesses the student desire to “cheat” when bored with gameplay.
Game “cheat books” – the new textbook.
Microsoft noted that learners cheat when bored. [That info could be used to spruce up boring areas of a game, or to collect information on whether the student has really internalized the concept.]
Marine Corps: Game is a tool to be used by an instructor. In a training exercise where the instructor notices a leaner is using a cheat sheet, the instructor can “lovingly tell him” not to cheat during the exercise.
I noted that the whole concept of a game is a kind of cheating, in that you get to save and reload and try again. If the point system is closely aligned with the pedagogical goals, there’s little benefit to be gained from “cheating” (in the sense of bringing a cheat sheet that says what the “right” answer is).
But if a student in a game has to keep turning to a particular page, whether on a cheat sheet or in an in-game reference tool, in order to get information that the other students have internalized, then that “cheating” won’t help the leaner complete the game objectives.
A speaker from Finland noted that she designs collaborative games, in which she doesn’t consider working together to be cheating.
I noted that the whole concept of a game is a kind of cheating, in that you get to save and reload and try again. If the point system is closely aligned with the pedagogical goals, there’s little benefit to be gained from “cheating” (in the sense of bringing a cheat sheet that says what the “right” answer is). But if a student in a game has to keep turning to a particular page, whether on a cheat sheet or in an in-game reference tool, in order to get information that the other students have internalized, then that “cheating” won’t help the leaner complete the game objectives.
A series of comments on collecting data from multi-user online games. These games can generate so much data that it can be prohibitive.
Microsoft Howard: Designers should talk to teachers about what precisely they want to measure, munging the data stream and delivering information that is meaningful to teachers.
Owen: Find the “happy path” of the assessor, rather than focusing on what the learner learns. Build the assessor’s needs into the system.
Multiplayer games are good at collecting and storing data. That computer can do pre-analysis.
(Continuing…)