The job the model is allowed to do
Write down whether the system drafts, classifies, retrieves, recommends or acts. A model that suggests text for a person to approve carries far less risk than one that sends messages or changes records on its own.
Guide · Artificial intelligence
Most AI features fail on data access, weak evaluation or missing human review rather than on the model itself. This guide walks a product lead through the choices that decide whether an AI feature earns trust once real users rely on it.
Evaluation planned before any prompt is written
Human review designed into the workflow
Inference cost treated as a product metric
Before you build
Write down whether the system drafts, classifies, retrieves, recommends or acts. A model that suggests text for a person to approve carries far less risk than one that sends messages or changes records on its own.
List every source the feature touches, who owns it, whether consent covers this new use and whether personal or confidential records would leave your infrastructure when calling a hosted model.
Hosted APIs are quick to start and improve without effort on your side; open-weight models give control over data residency and cost at volume but add serving, patching and monitoring work.
Agree a written definition of correct, acceptable and harmful output for your domain, with real examples, before anyone tunes a prompt. Without it, quality arguments become matters of taste.
Name the person or team who reviews flagged output, handles complaints and can switch the feature off. Oversight that belongs to nobody quietly disappears after launch.
Structure
Connectors, cleaning, chunking and an index (often a vector store alongside ordinary search) that supplies the model with current, permission-filtered context instead of relying on what it memorised in training.
Versioned prompts, model selection, tool definitions and structured output schemas, kept in source control so every change can be compared against the evaluation set.
Input checks for prompt injection and out-of-scope requests, output checks for policy, personal data and format, and fallbacks that return a safe response when a check fails.
A test set scored automatically on each change, tracing of production calls with inputs, retrieved context and outputs, and dashboards for latency, refusal rate and cost per task.
Review queues, confidence thresholds that route uncertain cases to people, feedback buttons that feed the test set, and an audit trail that shows who approved what.
Planning sequence
Gather a few hundred genuine inputs from the workflow you want to improve, with the answer an expert would give. This becomes the first evaluation set and exposes the hard cases early.
Try the simplest approach (a single prompt with retrieved context) against that set, and record where it fails. Many teams discover the problem is missing data rather than a weak model.
Decide what a person sees, what they can correct and how corrections are stored. Build this before widening access, because it is how quality improves later.
Ship to internal users or one customer segment behind a feature flag, compare outcomes with the old process and watch traces daily for surprising behaviour.
Set per-user rate limits, spending alerts and a model fallback order, then expand only when evaluation scores and review findings stay stable.
Mistakes
A handful of impressive answers in a meeting says nothing about the long tail. Without a scored test set, a prompt change can fix one case and silently break ten others.
An assistant that searches every document can show one customer or employee material they were never allowed to see. Filtering must happen before context reaches the model.
Giving an agent write access to email, payments or records without approval steps turns a wrong answer into a real-world incident that is hard to undo.
Long contexts, retries and chained calls multiply spend. Teams that never measure cost per completed task find the feature uneconomic only after it is popular.
The strongest AI products begin with a specific, repeated task that people already do badly or slowly: triaging support tickets, summarising long case files, matching invoices to orders. Naming the task tells you what inputs exist, what a good output looks like and who will notice when it goes wrong.
Starting from a model tends to produce a chat box looking for a purpose. Users try it, get a mixed experience and stop. A narrow feature that removes one painful step usually earns more trust than a general assistant, and it gives you a clear measure of success.
A model can only be as useful as the context it receives. Before design work, audit the sources: are they current, are they structured enough to search, and who is allowed to see which parts? Messy permissions are the most common reason enterprise assistants stall before launch.
Consent and purpose matter too. Records collected to run a service may not be usable for a new automated feature without telling people or updating your lawful basis. Settle this with whoever owns data protection early, because rebuilding a pipeline later is far harder than designing for it.
An evaluation set is a collection of real inputs with expected outcomes, scored automatically or by reviewers. It turns vague debate about whether the assistant feels better into a number you can track release by release, and it catches regressions when a provider updates a model.
Score more than accuracy. Track groundedness, refusal when information is missing, tone, format compliance, latency and cost. Keep adding production failures to the set, so the hardest cases you have seen are tested on every future change.
Oversight works when it is part of the interface rather than a policy document. Show reviewers the source passages behind each answer, make correction faster than rewriting from scratch and route low-confidence cases to them automatically.
Measure the reviewers’ experience as well. If they approve nearly everything without reading, the check has become theatre; if they rewrite most outputs, the feature is not ready. Both signals should feed decisions about where automation is allowed to expand.
Language models follow instructions found anywhere in their input, which means a malicious web page, email or uploaded file can try to redirect them. Treat retrieved content as untrusted, keep tool permissions minimal and require confirmation for any action with side effects.
Leakage runs the other way as well. System prompts, other users’ data and secrets in context can surface in answers. The OWASP list of risks for large language model applications is a practical checklist to review with your engineers before launch.
Effort concentrates in data preparation, evaluation tooling and review interfaces rather than in the model call itself. Integrations with older internal systems and permission models often take longer than the visible feature.
Running cost depends on context length, call volume, model tier and how many steps each task needs. Caching repeated context, routing simple requests to smaller models and capping retries are the usual levers, and each should be measured against your evaluation scores so savings do not quietly reduce quality.
FAQ
Rarely at the start. Most product teams get further by retrieving the right context and writing clear instructions for an existing model. Fine-tuning becomes worth considering when you have a stable task, a large set of labelled examples and evidence that prompting has plateaued.
You reduce it rather than eliminate it. Ground answers in retrieved sources, ask for citations, refuse when context is missing and measure unsupported claims in your evaluation set. For high-stakes output, a person should check before anything reaches a customer.
It depends on where you operate and what the feature decides. Data protection law such as the GDPR applies whenever personal data is processed, and the EU AI Act sets duties that scale with risk. Take legal advice for your jurisdiction; this guide is not legal advice.
Start from measured cost per completed task, not per call, then decide whether to absorb it, meter it or bundle it into a higher plan. Usage caps protect margins while you learn how heavily customers rely on it.
When the task genuinely needs several steps across tools and the cost of a mistake is low or reversible. Start with the agent proposing actions for approval, and remove the approval step only for actions your evaluation shows it handles reliably.
Next
How a delivery team scopes, builds and evaluates AI features for production use.
Open registerDetail on grounding answers in your own documents with permission-aware search.
Open registerA ready product model for teams that want an AI app platform rather than a bespoke build.
Open registerHow data-led businesses package models, datasets and insight as products.
Open registerA scorecard for judging AI developers on data, retrieval, evaluation, security and cost.
Open registerPrimary sources
Dated official documentation, standards, and research that support the factual claims on this page.
How to read this page
Scope, cost and timeline figures are planning guidance until confirmed in a written proposal. Diagrams and screens are illustrations, not client deliverables, and source-code rights and handover follow the signed agreement. See the delivery process and the full disclaimer.
Next decision
Define outcomes, constraints, evidence, rights and handover before delivery begins.
Commercial rights, repositories, environments, documentation, acceptance and handover remain contract-defined.