Introduction
Most AI projects don't fail because the model is wrong. They fail because the surrounding system — the data, the workflow, the evaluation, the rollout — wasn't designed. This playbook is the framework we use with our own clients when we move an AI initiative from a demo to a production workload.
It is not a research survey and it does not benchmark model providers. It is a sequence of decisions and artefacts that have to exist before a system is worth turning on for real users.
Phase 1 — Process selection
The single most common mistake is to start with a model. Start with a workflow. The right workflow for AI has three properties:
- High volume — automation pays back even at modest per-task accuracy gains.
- Tolerant of partial accuracy — a human can repair the output without redoing the task.
- Bounded inputs and outputs — the system can be evaluated against a ground truth.
Phase 2 — Architecture patterns
Four patterns cover most of what we build. They are not mutually exclusive — production systems usually combine several.
2.1 Retrieval-Augmented Generation (RAG)
The model answers questions grounded in documents you control — product manuals, policies, SOPs, historical tickets. Use RAG when accuracy and freshness matter more than open-ended generation.
2.2 Single-agent task automation
An agent receives a structured task, calls tools, and returns a result. Use it for back-office work — reconciliation, enrichment, ticket triage — where the task shape is fixed but the inputs vary.
2.3 Multi-agent orchestration
Specialised agents hand work to each other. A Lead Qualifier hands off to a Researcher, then to a Drafter. Use this when the workflow has distinct roles and the handoff itself carries information.
2.4 Human-in-the-loop
Every production AI system we ship has a place where a human takes over. Either because the task is too sensitive, the confidence too low, or the override is a regulatory requirement. Design this path before you design the model.
Phase 3 — Data governance
If the data layer is wrong, no amount of model quality recovers it. Three artefacts have to exist on day one:
- Consent capture and record — what data, for what purpose, for how long
- Retention and deletion policy — automatic, auditable, testable
- Access control — RBAC enforced at the data layer, not just the application
Phase 4 — Evaluation and guardrails
You cannot ship what you cannot measure. The minimum viable evaluation system has three components:
| Layer | What it measures | Example |
|---|---|---|
| Quality | Task-level accuracy against a held-out set | RAG answer matches reference in 87% of cases |
| Safety | PII leaks, prompt-injection resistance, refusal rate | 0 PII leaks across 10k eval queries |
| Operations | Latency, cost, error budget burn | p95 latency under 4s, $0.06 per task |
Run the eval on every prompt and model change. Block the release if a metric regresses more than an agreed threshold.
Phase 5 — Phased rollout
- Pilot — 5 to 20 users, manual fallback always available, weekly review
- Soft launch — one team or one region, human approval before any external action
- Canary — shadow traffic + sampled real traffic, automatic rollback on SLO breach
- General availability — full traffic, on-call rotation, monthly model and eval refresh
Build vs buy
A simple decision matrix covers most cases:
| Situation | Recommendation |
|---|---|
| Off-the-shelf product covers 80%+ of the workflow | Buy and configure |
| Differentiating capability, regulated data, or unique data moat | Build with a partner |
| Workload is bursty or experimental | Use managed APIs until volume is predictable |
Common failure modes
- Optimising the demo instead of the workflow
- Skipping the eval system because the demo looked good
- No rollback path — once the AI is live, no one knows how to turn it off
- Treating the model as the product instead of the system around the model
Where to start
If you are starting an AI initiative this quarter, the first deliverable is not a model. It is a one-page brief that names the workflow, the evaluation metric, the human override path, and the data governance posture. We can help you write that brief on a free 30-minute call.