Results

The test set, read once

207 reports and 1,554 scored sections, read once with the chosen model. Each interval is a Wilson 95% interval: the range the true share is likely to lie in, given how many sections there were.

More findings go with worse scores

The model grades shortfalls without being asked to: the lower the score, the more findings. Each interval is a 95% interval from a bootstrap that resamples whole reports. Shuffling the scores among the sections is the control: with nothing to find, the measures should fall to chance, 0.5 and 0, and they do.

A section scored 1 or 2 has more findings than one scored 3 or 4 with a chance of 0.985 (0.979 to 0.991), ties counted half (the area under the ROC curve). With the scores shuffled: 0.494 (0.467 to 0.522).

Kendall's tau-b between the number of findings and severity (5 minus the score) is 0.803 (0.783 to 0.823). With the scores shuffled: −0.001 (−0.047 to 0.042).

By statement

Test set, chosen model. For each statement: the sections with shortfalls (scored 1 or 2) that were flagged; the sections scored 3 that were flagged; and, of the flagged shortfalls, those with a finding filed under the section's own statement.

• Learning culture

  Shortfalls flagged

  102/104

  Score 3 flagged

  2/79

  Own statement

  100/102

• Safe systems, pathways and transitions

  Shortfalls flagged

  70/70

  Score 3 flagged

  13/105

  Own statement

  70/70

• Safeguarding

  Shortfalls flagged

  88/89

  Score 3 flagged

  8/112

  Own statement

  75/88

• Involving people to manage risks

  Shortfalls flagged

  118/118

  Score 3 flagged

  4/75

  Own statement

  108/118

• Safe environments

  Shortfalls flagged

  66/67

  Score 3 flagged

  9/116

  Own statement

  66/66

• Safe and effective staffing

  Shortfalls flagged

  115/118

  Score 3 flagged

  9/77

  Own statement

  115/115

• Infection prevention and control

  Shortfalls flagged

  42/44

  Score 3 flagged

  12/137

  Own statement

  41/42

• Medicines optimisation

  Shortfalls flagged

  112/113

  Score 3 flagged

  10/81

  Own statement

  109/112

Of the 29 flagged shortfalls with no finding under their own statement, 23 are in Safeguarding or Involving people to manage risks: the model filed the shortfall under a neighbouring statement, such as staffing. The section is still flagged.

What a flag on a section scored 3 is

A seeded random sample of 20 of the 67 was read by hand:

• 13 are a real minor issue that the inspector noted and accepted;
• 5 are misreads, where a neutral or good statement was read as a shortfall, usually because of negative wording (a service that does not offer a task, or people who had no concerns to raise);
• 1 is an old problem quoted as if it were current;
• 1 is unclear.

The model choice, and what it cost

Two local models of about 4 billion parameters each were compared with the same prompt: Qwen3.5-4B and Gemma 4 E4B. The rule for choosing was written down before either ran, and used the development set only:

1. the higher share of sections scored 1 or 2 flagged;
2. if the two are within two sections of each other, the lower share of sections scored 3 flagged;
3. then the higher share of sections scored 1 flagged.

On the development set, Gemma 4 E4B flagged 364/367 sections scored 1 or 2 and Qwen3.5-4B 363/367, within two. The second rule decided: Qwen3.5-4B flagged 37/401 sections scored 3 (9.2%) against Gemma 4 E4B's 91/401 (22.7%).

Gemma 4 E4B was also run once on the test set, so the cost of the choice is measured, not guessed.

Section by section, Gemma 4 E4B caught 7 shortfall sections that Qwen3.5-4B missed, and Qwen3.5-4B caught 0 that Gemma 4 E4B missed (exact two-sided sign test, p = 0.016). Read by hand, each of the 7 states its shortfall in words, usually in one sentence inside a mostly positive section. In return, Qwen3.5-4B flagged 67 sections scored 3 against 136, and 0 scored 4 against 10. Both missed the same 3 shortfall sections; read by hand, their text describes no shortfall, so the score rests on evidence the page does not show.