How accurate is our grading? Measured, not claimed

Run 4 — the latest run: high tier re-grade (Bands 8–9)

10 November 2026: the same 3 high-band Task 2 essays from runs 1–2 (published Bands 8–9), re-graded once each through the production high tier — the tier Sprint ($19) buyers get. Mean absolute error 1.33 · mean bias −1.33 · 1 of 3 within ±0.5. Same essays, low tier vs high tier: sports (Band 8) 6.0/6.5 → 6.5; part-time courses (Band 8) 7.0/7.0 → 7.5; social networking (Band 9) 6.0/6.5 → 7.0. The ceiling did not move: no Task 2 essay scored above 7.5, and the Band 9 essay came back 7.0. The high tier buys at most half a band — the next fix is prompt structure / rule-anchored scoring, not a higher tier.

2026-11-10 · Production high tier — FreeLLM auto, full reports, Sprint credits · Task 2 only (N=3)

Run 3 — mid-band essays (Bands 5.5–7)

8 November 2026: 5 published sample essays at Bands 5.5–7, graded once each through the same production low tier. Mean absolute error 0.40 · mean bias −0.40 · 4 of 5 within ±0.5. A sixth essay (Reading & Writing Skills, Band 6, 466 words) could not be submitted: the site caps Task 2 at 450 words. By reference band: 5.5 (n=2) MAE 0.25, bias −0.25; 6 (n=2) MAE 0.25, bias −0.25; 7 (n=1) MAE 1.00, bias −1.00. The same-prompt Discipline essays published at Band 6 and Band 7 both came back 6.0 — the grader did not discriminate the published 1-band gap.

2026-11-08 · Production low tier — FreeLLM auto, full reports, mid-band corpus · Task 2 only (N=5)

Runs 1–2 — high-band essays, after calibration (overall band, N = 7)

2026-11-06 · Production low tier — FreeLLM auto, full reports, after calibration · Task 2 MAE 1.67 (N=6) · Task 1 MAE 0.50 (N=1)

Run 1, before calibration (same day, same essays): mean absolute error 1.79 · mean bias −1.79 · Task 2 MAE 2.00 · Task 1 MAE 0.50.

Every essay, every score

Criteria order: Task Response / Task Achievement, Coherence & Cohesion, Lexical Resource, Grammatical Range & Accuracy.

What the numbers say

The error is not random, and calibration only partly fixed it. In run 1 all 7 essays came back low by 0.5 to 3.5 bands; the run-1 grader was instructed to be strict ('most self-taught candidates overestimate by 0.5–1.0'), and it over-corrected. For run 2 we removed that instruction and anchored every criterion to the official band descriptors instead. The bias shrank (−1.79 → −1.50) and Task 2 error fell from 2.00 to 1.67 — but every essay still came back low, and the two essays the publishers call Band 9 still lost the most (−2.5 each). No Task 2 essay scored above 7.0 in either run, whatever its published band: the remaining gap looks like a ceiling in how this tier judges Task 2, not just wording in a prompt. The Task 1 letter was off by only 0.5 in both runs. Run 3 (8 November, 5 mid-band essays) sharpens the picture: the error was 0.40 on average — 0.25 at Bands 5.5 and 6, but 1.00 at Band 7 — and the bias grew with the published band across all three runs (5.5: −0.25, 6: −0.25, 7: −1.00, 8: about −1.5, 9: −2.5). The grader is close where most candidates actually write, and compresses the top end. It also failed a direct discrimination test: two essays on the same prompt, published a full band apart (6 and 7), both came back 6.0. Run 4 (10 November) tested the one remaining simple explanation — that a higher tier would lift the ceiling. It did not: the same 3 high-band essays re-graded on the high tier came back 6.5, 7.5 and 7.0 against published 8, 8 and 9 (mean error 1.33, bias −1.33), at most half a band above their low-tier scores. The ceiling is not a tier problem. We publish all four runs in full so the next change can be judged against them.

How this was measured

Corpus: 7 published sample essays whose band the publisher states — four IELTS-Blog essays teacher-assessed at Band 7–8, and three model answers (IELTS Liz, IELTS Podcast) the publishers call Band 9 with examiner comments. Grader: the live production low tier (FreeLLM auto router, full reports) — the same tier Diagnosis ($7) and Prep ($14/month) buyers receive — except run 4, which used the live production high tier (FreeLLM auto router, full reports) that Sprint ($19) buyers receive. Each essay was submitted once per run through the normal /submit flow on 2026-11-06 (runs 1–2), 2026-11-08 (run 3) and 2026-11-10 (run 4) under a temporary test grant, expired immediately after each run; no essay was re-run within a run or cherry-picked. Between runs 1 and 2, only the grading prompts changed (calibration). Overall bands were re-checked against the grading records; criterion splits are as shown on each linked report.

Read this before comparing us with competitors

These reference bands are publisher-stated (teacher assessments and model answers), not official examiner scores — the official IELTS sample scripts are handwritten scans with no extractable text, so no official-band corpus was usable. Competitors who publish accuracy numbers (English Test Coach: 0.57 mean error on 27 essays; IELTS Writing Checker: 0.49 on 73) measured against officially scored essays. Our 1.50 (1.79 before calibration) at the low tier and 1.33 in run 4 at the high tier are therefore not directly comparable to their 0.49–0.57 — but they are directly comparable to our own other runs, which is the comparison that matters for whether we are getting better.