Does the Headline Match the Size of the Difference?
A fluent answer no longer says much alone about who understood what. A task built around a headline and a reported difference shows what a student can do with evidence and what a tool could do for them.
A fluent paragraph can be written by a student, by software or by both, in proportions no reader can see. That does not make written work useless. It means polish has to be read as one piece of evidence whose source is uncertain, and that an assessment needs more than polish to support a claim about what someone can do. If a tool can complete a task, the finished response alone cannot distinguish the routes by which it was produced. An instructor who wants evidence about the student has to design for that.
Start with the claim
Begin with what the course wants to be able to say: this student can judge whether a headline fits the size of a reported difference. A definition of effect size will not show it. Instead, give a short invented report in which two groups were compared and the standardized difference between them was small, under a headline calling the result dramatic. Here, a standardized mean difference expresses the group difference in units of variability; other effect sizes describe other relationships. A strong answer says what a small standardized difference means, notes that a headline word like dramatic needs support the numbers do not give, and adds that the size alone says nothing about whether the design justifies a causal claim or whether the difference matters in practice.
A different case
The second piece of evidence is a changed case, given later and completed independently. This time a report describes a large difference between groups, but the groups were self-selected, and the headline is modest. The student must apply the same ideas to new material and notice that a large difference from a self-selected sample raises a different problem. This gives additional evidence about whether the reasoning can be applied. It is still one sample from one occasion, and a student can handle one case and stumble on another. Evidence accumulates across tasks, occasions and formats, and the instructor decides how much to require given what the grade will be used for.
What a conversation adds
A short conversation about an answer can add information writing does not, because the instructor can ask why. It also has limits. Some students may speak less fluently under observation than they write, and what a conversation shows depends on the questions asked and the instructor's judgment. No single check, whether written, spoken or timed, proves that a student has learned something, and no single check reliably catches improper help. The better design question is how several modest pieces of evidence, each with a known weakness, fit together.
Two practical matters belong in the design. One is access. Tools, time, language and disability all affect what a task can fairly ask, and a format that suits one student can disadvantage another. The other is stakes. Low-stakes samples across a term can carry more of the evidence, which relieves pressure on any single task and gives the instructor a pattern to read. These are designs to try and judge in a particular course. Nothing here reports that any of them works better than the alternatives.
A detection score is a different kind of claim
A detector score is not the probability that an entire submission was generated by AI. TEQSA’s guidance says that a score alone is insufficient for an allegation. Interpret concerns through the institution’s established process and allow a student to respond. A learning check asks what someone can do; a misconduct decision asks a different question and needs its own evidence.
An original classroom example
Original, unevaluated assessment example. Task 1: an invented report compares two groups; the standardized difference is small and the headline says dramatic. Students write a sentence that matches claim to size and say what the size cannot show. Task 2, one week later and done alone: a new invented report with a large difference between self-selected groups and a modest headline. The instructor compares the two answers, adds a short conversation for a few students chosen by a fixed rule, and records what each piece of evidence could not show.
Questions for your course
- What claim do you want to be able to make about a student after this task?
- What could a strong answer on one task still hide?
- What does a conversation add, and what are its weaknesses?
- How would you combine several pieces of evidence without treating one as proof?