Adding AI to an Existing Product: Where the Hidden Costs Appear

The hidden cost of adding AI rarely sits in the first model call. It appears in data preparation, evaluation, fallbacks, usage controls, latency, privacy, monitoring and human review. A prototype can demonstrate promising behaviour quickly. A production feature must handle incomplete inputs, unavailable providers and incorrect outputs without creating unacceptable risk.

Before choosing a model, define the task, the consequence of a wrong answer and the method used to evaluate quality.

Model cost depends on the workflow

An API price is only one input. Actual usage depends on:

• system-prompt length;
• user-input size;
• retained conversation history;
• documents added to context;
• response length;
• retries;
• model selection;
• request frequency;
• additional moderation, classification or extraction calls.

A feature that sends a full document and complete conversation history with every request has a different cost profile from a short classification task.

Define the unit of use. What happens during one analysis, generated reply or completed workflow? How many model calls and external tools are involved?

Set sensible context limits, monitor usage by feature or organisation and cache results where repetition is safe. Provider prices can change, so avoid designing a product that works only under one permanent pricing assumption.

Retrieval requires its own data pipeline

When AI must answer from company documents, “upload the PDF” is not a complete system.

The workflow may need to:

• ingest several file types;
• extract useful text;
• remove repeated or irrelevant sections;
• divide documents into meaningful chunks;
• store metadata and permissions;
• retrieve appropriate passages;
• update or delete outdated sources;
• show supporting material with the answer.

Retrieval quality depends on document structure, metadata and access rules—not only on the language model.

If the product shows citations, each one should resolve to the real source passage. The model should not generate the appearance of evidence without a verified connection to the underlying material.

Evaluation must precede optimisation

A handful of successful demonstrations cannot establish reliability.

Build a set of realistic evaluation cases:

• a normal request;
• incomplete input;
• contradictory source material;
• a question not answered by the available data;
• an attempt to access restricted information;
• an unusually long or ambiguous request;
• a case that should be escalated to a person.

For each case, define acceptable behaviour. Sometimes that is a correct answer. In other cases it is a refusal, a request for clarification or an explicit statement that the evidence is insufficient.

Repeat evaluation when changing the model, prompt, retrieval method or data source. Otherwise, it is difficult to know whether the new version improved the system or merely changed its tone.

Design the failure path

AI providers can time out, return invalid output, reach usage limits or become temporarily unavailable.

Decide:

• how long the user should wait;
• whether a retry is safe;
• what happens with a partial result;
• whether a simpler fallback exists;
• when a person takes over;
• how duplicate actions are prevented.

A generated draft and an autonomous action have different failure consequences. The greater the consequence, the stronger the validation and approval requirements should be.

If the system writes to another service, separate suggestion from execution where practical. Give users enough information to understand what will happen before an irreversible action.

Define privacy and retention deliberately

Before sending data to a model provider, identify:

• which information is genuinely required;
• whether it contains personal or confidential data;
• where processing occurs;
• how long inputs, outputs and logs are retained;
• who can inspect them;
• how deletion works;
• whether sensitive fields can be removed or replaced.

Logs are valuable for diagnosis but can become a secondary store of sensitive information. Capture enough context to investigate problems without retaining everything by default.

Requirements vary by data type, provider, contract and market. Appropriate privacy and legal review belongs alongside the technical design; implementation alone cannot provide a legal guarantee.

Treat latency as product behaviour

One user action may involve retrieval, several model calls and external tools. Each step adds delay and another failure point.

Show meaningful progress, allow cancellation where appropriate and stream output only when partial results are genuinely useful. Longer tasks may be better handled in the background with a notification when complete.

The fastest model is not automatically the right model, and the most capable model is not necessary for every task. Match model choice to the required quality, latency and cost profile.

Make human review operational

“Someone will check it” is not a control unless the interface supports the review.

Define:

• who reviews which outputs;
• which sources and context they can see;
• how they correct the result;
• whether the proposed action can be reversed;
• how corrections feed future evaluation.

Human review should focus on decisions where the consequence justifies it. Routine approval of every harmless suggestion can create delay without improving safety.

For an early release, include only the AI workflow that tests the central SaaS MVP assumption. The same principles apply to AI voice-agent workflows, where handoff and monitoring are especially visible. For conversation design and system integration, see AI agents or use the contact section.

← Back to the blog