Testing bounded AI for procedure review
At an infrastructure supply-chain business (name withheld), subject-matter experts spent repeated time finding source material and preparing first-pass procedure comparisons.
- Sector
- Infrastructure supply chain
- Services
- Practical AI
- Technical documentation
- Training and capability
The situation
- 01 Relevant source material was spread across controlled documents and project requirements.
- 02 Experts repeated the same collection and first comparison for similar reviews.
- 03 A plausible generated answer could conceal a weak or missing source.
- 04 Nobody had been named to check output or decide when to stop using it.
We agreed what the assistant could do before choosing a tool
We limited the assistant to finding approved passages and preparing a first comparison for an expert to review. Before building anything, we agreed the sources, how sensitive they were, what a failure would cost and the test examples.
Anything needing interpretation, approval or professional judgement stayed with the experts.
- 01 Approved sources only
- 02 What the assistant may and may not do, tested on real reviews
- 03 Who reviews output and who escalates
We built traceability and a stop rule into the prototype
We tested a narrow prototype on real procedure questions, including ambiguous, incomplete and conflicting material. Every useful answer had to point back to the controlled source. The operating guidance tells reviewers how to check output, record the decision and stop using the assistant when a source or scenario falls outside the agreed limits.
What the work achieved
- First-pass comparison time
- Half-day → <30 min
- First-pass comparison time for routine procedure reviews cut from around half a day to under 30 minutes.
- Outputs traced to an approved source
- 100%
- Every assistant output traced back to an approved source document.
- Unsupported answers reaching a reviewer
- Zero
- 01 A written stop rule that withdraws the tool as soon as source material falls outside the agreed limits.
More case studies
The sector
Related services
Related insights
Book a systems audit
We look at how the work runs today and give you a 90-day improvement plan.