The model is not the product.

A model can produce an impressive answer in seconds. That does not mean a product exists.

The demonstration hides nearly everything that determines whether the answer will remain useful in the real world: where the context came from, what happens when it is wrong, who can correct it, how performance is measured, what it costs at volume and who is responsible on a bad day.

Kingsbury’s position is simple:

The model is a component. The product is the complete system that lets someone achieve an outcome safely, repeatedly and under accountable ownership.

Confusing the two produces prototypes that look finished and services that are nowhere near ready.

The model-shaped hole

Model selection can receive disproportionate attention because it is legible. Providers publish benchmarks. New releases create obvious comparison points. A prompt produces something a room can react to immediately.

The rest of the product is quieter. It consists of decisions about work.

The user job

Who is trying to do what, under which constraints? A product must improve a real task, not merely demonstrate a new capability. The person using it needs to know what the system can do, where it is uncertain and what they remain responsible for.

The workflow

Where does the system enter an existing service? What triggers it? Who receives its output? Which exceptions leave the automated path? If a useful answer has to be copied between five systems and reconciled by hand, the model may work while the product fails.

The context and data

What information is the system allowed to use? Is it current, relevant and lawfully available? Can its origin be traced? A capable model supplied with poor or inappropriate context will produce polished unreliability.

The controls

What must never happen? Which decisions require human review? What is recorded? Who can correct, pause or disable the system? Controls should be designed into the service, not added as a warning beneath the interface.

The evaluation

How will the team know whether the system is good enough for this specific task? General model benchmarks cannot establish service readiness on their own. The product needs representative cases, agreed quality criteria and a way to inspect failures as the surrounding work changes.

The operation

Who owns reliability, cost, latency, access, incidents, suppliers and change? What happens when a provider deprecates a model or alters a price? A product is not finished when it launches. It becomes an obligation the organisation must be able to operate.

The UK government’s Introduction to AI Assurance distinguishes the model from the wider system and the context in which decisions, outcomes and liabilities exist. NIST’s voluntary AI Risk Management Framework describes the same reality as socio-technical: benefits and risks emerge through the interaction of technology, people, processes and deployment conditions.

This is not a new production lesson. The 2015 paper Hidden Technical Debt in Machine Learning Systems documents the surrounding burden of data dependencies, glue code, configuration, consumers and changing external conditions. Its particulars predate today’s foundation models; its systems warning has aged well.

A contract-review example

Take a hypothetical assistant that identifies unusual clauses in supplier contracts.

The model call is the smallest visible loop: contract in, observations out. A product has to answer much more:

• which contract versions and jurisdictions are supported;
• how text is extracted and whether missing pages are detected;
• which internal policies and approved clauses provide context;
• how observations are linked back to the source wording;
• when the assistant must abstain;
• who reviews the result and records the final decision;
• how incorrect, missed and low-confidence findings are sampled;
• how sensitive documents are retained and accessed;
• what happens when the model, policy or law changes.

None of that is decorative infrastructure. It is the thing being delivered.

The Kingsbury product readiness test

Kingsbury does not call an AI capability ready for operational use until its team can answer these system questions concretely.

Boundaries

• Which inputs, users and situations are supported?
• When must the system refuse, defer or escalate?
• What happens at the boundary between model output and human authority?

Traceability

• Can an operator trace an output to its source context, model and configuration?
• Are prompt, policy, data and model changes recorded?
• Can a disputed result be reconstructed well enough to investigate it?

Controls

• Can an authorised person inspect, correct, pause and roll back the service?
• Which failures require escalation rather than another model attempt?
• Are security, privacy and access controls applied to the whole path?

Operation

• Is there a named owner after the launch team leaves?
• Are reliability, latency, cost, incidents and supplier changes monitored?
• Can the organisation maintain the service with the knowledge it has retained?

Replaceability

• Can the model be evaluated independently of its provider?
• Are source data, evaluation cases and decision records portable?
• Can the component be replaced without rebuilding the service around it?

If these answers do not exist, the honest label is prototype. That is not an insult. A prototype is valuable when it is allowed to reveal what the product still needs.

Build around change, not permanence

Models will move. The strongest model for a task may change between discovery and launch. Providers may alter interfaces, limits and commercial terms. An internal model may improve enough to replace an external one.

Design the product so the model can be evaluated and replaced without rebuilding the entire service. Keep task-specific evaluation outside the provider. Keep source data and decision records portable. Treat prompts and model configuration as versioned product material. Separate provider-specific behaviour behind a boundary the team can inspect.

This is not abstraction for its own sake. It prevents a temporary supplier choice from becoming the product architecture.

The UK government’s AI Playbook recommends testing outputs, recording provenance and design decisions, planning for monitoring and considering vendor lock-in. Those are product responsibilities because they persist after the first successful response.

What should survive the build

When Kingsbury says work has been delivered, the client should not be left with a mysterious endpoint and a dependency on the people who assembled it.

They should have:

• a working service tied to a named outcome;
• the evaluation cases and acceptance criteria;
• visible controls and escalation paths;
• an operating owner and a change process;
• a record of important design decisions;
• the ability to understand, challenge and improve the system.

Capability transfer is part of the product. If the people responsible for the service cannot explain how it fails, they do not yet control it.

The model matters. It can be extraordinary. But the serious work begins where the demo usually ends.

Sources and further reading

• NIST’s voluntary AI Risk Management Framework and AI RMF Playbook for governing, mapping, measuring and managing risks across an AI system. NIST is revising AI RMF 1.0 at the time of publication.
• DSIT’s Introduction to AI Assurance for the distinction between data, model, system, governance and wider operating context.
• The UK government’s AI Playbook on lifecycle management, evaluation, provenance, monitoring and supplier considerations.
• The NCSC’s guidelines for secure AI system development on security across design, development, deployment, operation and maintenance.
• Sculley et al., Hidden Technical Debt in Machine Learning Systems, on the production system surrounding learning code.