AutoCurricula GradeSync: grading handwritten exams until a human is needed

1Problem

The project README cites the OECD TALIS 2024 measurement of 4.6 hours a week spent grading by hand, out of a 41-hour working week. In the schools the authors work with, feedback typically arrives a week or two after the exam, and every grade is then retyped into the student information system. The README states these as the motivation, not as something I measured.

The hard part is not producing a grade. A model can do that on most pages. The hard part is deciding which grades should never be written without a person: pages that are barely legible, pages where the student wrote instructions aimed at the grader, and any case where the model is confident for the wrong reason. A system that grades everything and escalates nothing is a liability in a gradebook. So the design target was a fleet that works in the background and stops when a human judgement is needed.

2Approach

Scans are uploaded to a Cloud Storage bucket. A notification reaches a Cloud Run service through Pub/Sub, and a graph of agents built with the Google Agent Development Kit processes the job. Twelve components sit behind one governing harness: eight are model-backed agents on Gemini 3.5 Flash and Flash-Lite, and four are deterministic (risk scoring, schema repair and the two prompt optimizers).

Three rules shaped the design.

• Every grade carries an evidence span, a verbatim quote with a page number. A record without evidence is not written.
• Confidence is gated. A record is written to the gradebook only if its confidence is at or above 0.85 and it has cited evidence. Otherwise it is quarantined for one-click teacher review, never guessed and never silently synced.
• Facts a rule can settle are not left to the model. Legibility is a deterministic image metric (blur variance times contrast) that discounts the model's self-reported confidence. Dropout risk comes from z-scores and slopes over history. The permission gate ranks deny over quarantine over allow.

Every page is also screened for handwritten prompt injection, meaning text addressed to the grader instead of answers to the exam. A detection forces quarantine whatever the confidence. A second reader transcribes each page independently, so the quotes the grader cited are checked against a reading that never saw the grade.

Building this hit two defects in Google's libraries, both filed with minimal reproductions. In adk-python 6836, LlmAgent rejected temperature= with no discoverable route to generate_content_config; the README reports it fixed and closed on 28 August 2026, and that we checked the fix against four edge cases. In python-genai 2889, Vertex structured output returned empty dict-typed fields without any error; per the README the maintainers acknowledged it and filed an internal bug.

3Results

The production run used three sections of 36 synthetic exams each, sent one at a time to the deployed Cloud Run service. The count was read back from two surfaces that share no code path: the gradebook ledger and the review queue.

Each section held four papers: two stopped by the injection screen and two by the legibility floor. The 12 are the papers the synthetic roster was designed to hold, so this run shows the gate firing where expected; it does not show the system finding hard cases on its own. Wall clock from upload to completed was 135 s for section 10A and 255 s for section 10C.

The harness has 13 classes, 12 of them attacks and one an innocent control (direct imperatives, forged system authority, rubric substitution, micro-print, encoded text, language switch, metadata channel and others), two payloads per class, each hostile page paired with a clean twin so that a screener that quarantines everything scores zero. The screen was Gemini 3.5 Flash-Lite. Total cost was under one cent per campaign.

Measurement: the production run is one pass on the deployed service (revision 00058) with synthetic exams and a roster that fixes which papers should be held. Red-team numbers come from two campaigns of 24 scripted hostile payloads against our own screen; grading was not run in either, so the grade-move rate is not measured. Calibration is one run on 4 human-graded synthetic exams (12 criterion scores) with no held-out set; the README gives a 95% interval of 0.53 to 0.96 on that kappa. The run took 124.9 s and cost $0.63.

4What did not work

The prompt optimizer found nothing to promote

The meta-optimizer proposed three candidate prompts. One tied the baseline exactly and was rejected because acceptance requires a strict improvement. Two made things worse (MAE 0.250, QWK 0.800) by replacing the seed few-shot examples with a fabricated one. The loop stopped after one cycle. The baseline itself passes the MAE and bias gates but fails kappa by 0.005, so nothing was promoted. With only 12 criterion scores, one half-point change moves kappa by 0.045, nine times the gap to the gate, so the 0.85 threshold is close to a coin toss at this sample size. The report says a promotion would need at least 20 human-graded students. In the deployed demo this loop does not run at all, because the container ships without calibration data, and the README says so.

The first red-team campaign failed

83.3% was below the 95% bar, and the metadata class scored 0 of 2. The screener only looked at page images, while the grading prompt serialised the submission record, including the file name. A hostile file name therefore reached the model unscreened. File names were not on the list of input channels until the campaign showed it. The fix has three layers: reject hostile names at the API, redact the derived identity when a file is dropped straight into the bucket, and screen and sanitise every manifest-derived string before any model call. The same work added base64 decoding for the encoded-payload class. In an offline, model-free run the deterministic layer moved those two classes from 0 of 2 to 2 of 2 each, so these catches need no model call.

Leakage in the optimizer's own validator

Two candidates embedded a real calibration student, with the human score, inside a few-shot example. The anti-gaming validator inspects score distributions, not few-shot content, so it did not flag this. It did not help the candidates here, but with a larger few-shot budget a proposer could memorise the calibration set. The report lists a check for it as follow-up, and I have not verified that it was built.

Naming a screen after a product I do not use

The submitted video calls the injection screen "Model Armor". That is a Google Cloud product and the project does not use it; the screen is our own multimodal prompt plus a deterministic prescreen. The video was already submitted, so the README carries the correction.

Limits from the project's own audit

The repository holds an adversarial review of the engine dated 20 August. Among its findings from reading the code, the optimizer selects and evaluates on the same four samples, and records approved after rework reached the gradebook without provenance. I have not checked whether either was closed afterwards.

5Lessons

• Report what the gate did, not only what it passed. Twelve held papers is a result only because each one has a stated reason: six low-legibility and six injection, two of each kind per section.
• A red team that fails first is more useful than one that passes. The failed campaign found a channel, the file name, that no amount of tuning the image screen would have reached.
• Paired clean twins keep a screener honest. Without them, quarantining everything looks like perfect detection.
• Small calibration sets cannot carry a threshold. At 12 scores the gate is below the noise, and the right response was to refuse to promote, not to relax the bar.
• Upstream bugs are part of the work. Filing minimal reproductions led to a fix for one of the two, closed on 28 August 2026 per the README.