Matching Claims to Sources: Evaluating a Workflow Before You Rely on It
A system that reads sources and writes a study guide can fail in ways the final text hides. Evaluation starts by saying what the task is and what a bad result would look like.
Suppose an instructor is offered a system that reads three source excerpts on memory and returns a one-page study guide. The page looks finished. Evaluating it starts with questions a careful reviewer asks of any process whose product looks complete: what was it supposed to do, how would I know if it did that badly, and where could someone look along the way? Without answers, a good-looking guide is only a guide that looks good. Many failures show up only to someone who checks against the underlying material, which is why the person checking needs time and a standard.
Naming the failures first
Write the task in one sentence and list how it could go wrong. For a study guide the list might include a claim that appears in none of the excerpts, a claim stated more strongly than an excerpt supports, such as saying a study showed something that the excerpt only suggested, two sources blended into one, a definition that is subtly wrong, and a reference that does not say what it is credited with. The list is a hypothesis and will grow as errors turn up. Writing it first also fixes the standard in advance, which protects against admiring whatever the system produced.
Where a person can look
What an evaluator can inspect depends on the design. A fixed pipeline, with retrieve, summarize and format as separate steps, can be checked stage by stage: were the right passages retrieved, then was each summary faithful to them? A system that chooses its own steps raises a different question, which is whether it leaves a trail showing the sources it opened and the actions it took. If a system returns only the final text, an evaluator is reduced to reading the text closely against the sources. Whether a given system exposes its steps is a fact to establish for that system. This guide makes no claim about any.
Comparison helps. Have the system produce a study guide for one set of excerpts, write a second guide by hand from the same excerpts, and give both to a colleague who marks each claim as matched, overstated or unsupported without knowing which guide is which. This is a small, informal comparison and not a study. It can reveal errors in that example and help the instructor judge whether they are tolerable, and it records how long the checking takes. A single set of excerpts says little about other material, so the comparison should be repeated on a different topic before anyone generalizes.
Capability and benefit
Two questions hide in evaluation. One asks whether the system does the task, producing a faithful guide. The other asks whether using it helps students learn. Evidence for the first says little about the second, since a faithful guide can sit unused or be used in ways that add nothing, and a flawed one can prompt useful discussion. An instructor should say which claim is being tested and avoid treating evidence about one as evidence about the other. Finally, keep a short log of what was delegated, what was checked and what was found, so the next decision starts from evidence and not from impressions.
An original classroom example
Original, unevaluated example. An instructor gives a system three invented excerpts about memory (described, not quoted) and asks for a one-page study guide. Failure list written first: claim in no excerpt; claim stronger than its excerpt; two sources merged; wrong definition; reference that does not say what is credited. Guide A is system-made and Guide B hand-made from the same excerpts. A colleague marks each claim matched, overstated or unsupported without knowing the author. The instructor records checking time and repeats on a second topic.
Questions for your course
- What is the task in one sentence, and what would a bad result look like?
- What can a person inspect: separate stages, a trail of actions, or only the final text?
- What comparison would show whether the errors are tolerable?
- Are you testing whether the system does the task or whether students learn from it?