AI marking for lab quality assessment
EMQN · discovery sprint, multilingual benchmark and technical specification
EMQN’s board could see how far AI marking goes before commissioning anything. The benchmark ran against their own reports, in every language their schemes run in.
Overview
EMQN is a community interest company that works as an exam board for laboratories, running external quality assessment schemes for human genetic testing. Expert volunteers mark every report. A large part of that marking is a set of rule-based clerical criteria that assessors hand-check every time and describe as the dullest and least scientifically interesting part of the job.
EMQN asked OpenKit to find out whether AI could take the first pass at those criteria without weakening a process that laboratory accreditation depends on. OpenKit benchmarked six models against EMQN’s own reports in all six scheme languages, then wrote a strategy report for the board and a technical specification for a build team.
93-96%
Per-criterion accuracy across the benchmark as a whole
45%
Strict accuracy, meaning all 17 criteria correct on the same report
6 languages
Every language the schemes run in, each read separately
7 providers
Hosting providers assessed
Approach
What OpenKit produced
Before any model touched any data, OpenKit interviewed EMQN’s leadership and its assessment, scientific and IT teams.
EMQN’s members are diagnostic laboratories, so there was little room for a system that only looked promising in a demonstration. Three requirements were fixed before any technology conversation started: laboratory data stays inside UK or EU jurisdiction at every tier of hosting, a volunteer assessor signs every mark and the same criteria hold in all six languages.
Alongside the report and the specification, OpenKit produced:
- A benchmarking framework EMQN can rerun as models change.
- A hosting analysis weighing seven providers on US CLOUD Act exposure, UK GDPR, running cost and the self-hosted alternative. The options it recommends are hosted in the UK or EU by design, so they sit outside US CLOUD Act scope.
- A traffic-light confidence design, so the assessor sees how confident each draft is before signing it.
- A register of the risks a community interest company carries with any external AI provider.
- Interface mockups and an API questionnaire for the third-party platform the marking sits inside.
Findings
What the benchmark showed
Each model ran ten times against fifteen representative reports, so the benchmark shows how far its answers move between runs. Per-criterion accuracy came out at 93-96%, high enough to be useful. Strict accuracy, where every criterion on a report has to be right, dropped to 45%. That gap put numbers behind the assessor’s signature on every mark.
OpenKit read accuracy separately in each language and left the variance in, because a model that only performs in English would fail EMQN’s real workload. English and Spanish scored 99-100%. German, French, Italian and Portuguese, with their compound technical terms and diacritics, scored 93-95%.
OpenKit ruled out the most accurate model in the set because its licence does not permit deployment by an EU-based organisation. The recommendation went to the next best model EMQN can run.
Result
The recommended platform
The report and the specification describe a platform that clears the clerical criteria first, which buys assessors time and leaves the scientific judgement with them. EMQN’s IT team set the condition that a healthcare system cannot be benchmarked once and trusted thereafter, so the specification designs in re-verification against pre-assessed baselines as a normal operation.
Voice
Working with OpenKit has been a genuinely positive experience. Their team quickly understood the unique challenges of our business and the problem we were trying to solve, and delivered a thorough, evidence-based strategy for our AI-assisted marking platform. We were particularly impressed by their transparent approach, technical expertise, and commitment to long-term partnership and support.
Other engagements
What could your business do with AI?
Find out with experts who become part of your team. We uncover opportunities, get ideas working, and help your people build on the results. We reply within one working day.
Get in touch Governance & compliance