What does an AI marker miss?

I'd want to see the answers an AI marker gets wrong before I let it mark students' work. In particular, I'd want to see what happens when a student gives a good explanation that doesn't look like the answer we expected.

With automated marking, we need to understand the errors well enough to decide what we can use the system for, and how much checking it'll need.

A paper published this week gives us a useful example of why that matters.

A correct explanation can still be missed

Cheng-Wen He and colleagues studied automated scoring of biochemistry answers, using work from seven hundred and ninety-two undergraduates at one US university. Students answered two problems, one involving a more familiar application of their knowledge and another requiring them to apply it further from the original context.

The researchers trained separate BERT-based models for the two tasks. Overall test accuracy across the rubric categories was 0.91 for the near-transfer task and 0.87 for far transfer. This was a specialised scoring study, so those figures tell us nothing directly about a chatbot or a product a school might buy.

In one response, a student explained a prediction using a different kind of molecular interaction. The reasoning was valid. The human scorer gave that part of the answer full credit, but the model gave it zero. The researchers also found difficulties with ambiguity and information spread across sentences.

We don't need to teach biochemistry to recognise the assessment problem. A student can understand something and explain it in a way we hadn't anticipated. Before we use a marking system, I'd want to know what happens to that answer.

We need to know what the errors would change

The paper also shows why an accuracy figure needs context. For some criteria, almost all the human scores fell in the same category. With that pattern, I'd want to see how the model handled the less common scores before relying on its overall accuracy. For some criteria, it also couldn't give all the finer scores available to the human markers.

The human marking had mistakes too.

That last point matters. If a teacher and a model disagree, somebody still has to read the work and decide what the response deserves. We might find a model error, a marking mistake or a rubric that needs to be clearer.

Suppose a system marks down an explanation that a department agrees is correct. What happens next? If the teacher catches it before feedback goes to the student, we have something to investigate during the trial. If it reaches the student, we've told them a valid explanation was wrong.

Or suppose the system rewards an answer that uses all the expected vocabulary but never explains the relationship between the ideas. That could leave us planning the next lesson on the assumption that the class understands something it hasn't yet grasped.

I'd look for both kinds of error in a school trial. We'd need to find out whether they happen with the system and subject we're considering.

I'd also want the department to agree what the score is meant to help us do. How much error we can accept depends on that. Feedback on a practice answer gives us one decision to make. A grade that affects a student's progression needs much more scrutiny.

Try it with answers your department can discuss

I'd start with one department and one familiar formative task. Keep the first review small enough that colleagues can read every answer and discuss the disagreements properly.

Choose a task where the department can explain what counts as a good answer. Put together a small set of responses that includes a straightforward correct explanation, a valid alternative, a partly correct answer and a plausible misconception. A teacher can write these examples, so there's no need to put identifiable student work into a demonstration.

I'd include an answer that's awkwardly expressed but shows the subject knowledge, alongside one that sounds confident while getting an important relationship wrong. This small trial is my suggestion; the researchers didn't test it.

Have colleagues mark the responses before looking at the system's scores. Where colleagues disagree, resolve what the rubric means and record the reason. We need to agree why an answer deserves its mark before we compare it with the software's score.

Then compare the system's marking with that judgement. For each disagreement, keep the response, the relevant criterion, the system's score and the department's reason for accepting or changing it.

The discussion I'd want is quite specific. Did the system overlook a valid explanation? Did it give credit for a keyword without the reasoning? Could the teacher see enough to work out what happened?

An unusual answer might expose a weakness in our own criteria. That's useful too. A trial should give us room to improve the task as well as assess the software.

Ask the supplier what happens after a mistake

A demonstration should let us follow a disagreement all the way through to the feedback a student receives. I'd ask the supplier to show us how a teacher challenges a score and how the correction appears to the student.

Then I'd ask what happens to similar responses already marked. Does someone need to find them manually? Can the teacher see which criterion caused the problem? How would we know whether the same error keeps recurring?

Those questions tell us something about the work we're taking on. A system could save time on straightforward answers and still create a substantial checking job elsewhere. During a trial, record that checking time alongside any time saved.

Keep the next lesson in view

In Can students explain the work?, I suggested a short follow-up after AI-assisted work to help a teacher decide what to teach again. Automated marking needs to support that same decision.

If it helps us find a misconception sooner, we can respond to it. If we spend the time unpicking a score we can't explain, we need to include that in our judgement of the tool.

I'd leave the first department review with the marked examples and a written decision about where the system was useful, which responses needed a teacher and what remained unresolved. That gives us something concrete to discuss before we extend the trial.

If the trial shows that valid alternative explanations need a teacher's review, that review has to be part of the plan. We need to account for it before making promises about the time the tool will save.

For more on AI, teaching and school leadership, subscribe to DEEP Dispatch.