Finding safety concerns in home care inspection reports
A small language model, run on one laptop, reads the Safe part of the Care Quality Commission's inspection reports on home care agencies in England. It lists each safety concern it finds with the exact words it rests on, and code checks every quote. Its findings are then compared with the inspectors' own scores.
It flagged 98.6% of sections with shortfalls, 8.6% of good ones and 0% of exceptional ones
The test set: 207 reports and 1,554 scored sections, read once, with nothing tuned on them. A section is flagged when the model gives at least one finding whose quote is really in that section. Flagging every section would also catch every shortfall, so the gap between the two groups is what counts.
Sections with shortfalls: should be flagged
- Score 1 · significant shortfalls
- Score 2 · some shortfalls
Sections at a good standard or better: should not be flagged
- Score 3 · a good standard
- Score 4 · an exceptional standard
Intervals and every other measure are on the results page.
What the words mean
- Section
- One of the eight Safe quality statements in one report, such as Safeguarding or Medicines optimisation.
- Score
- The inspectors' score for that statement: 1 significant shortfalls, 2 some shortfalls, 3 a good standard, 4 an exceptional standard.
- Finding
- A concern the model names: its category, the exact words it rests on and a one-sentence reason. Code checks that the words are in the section.
- Flagged
- A section with at least one finding, whatever statement the finding is filed under.
- Test set
- Reports kept aside and read once, at the end, after every choice had been made on a separate development set.
What it does not show
- Inspectors' reports are not daily care notes: they are written for the public, by trained people.
- A score rests on more evidence than the page shows, so some shortfalls cannot be found from the text.
- A flag is a screen, not a grade: it says nothing about how serious a finding is.
- The sample was drawn by rating, so its mix of scores is not England's. That is why results are given for each score.
- One prompt, and each model run once on the test set; run-to-run variation was not measured.
See the worked example: the model reads an invented report, and every quote is shown in place.