A production AI product is not a model wrapped in a chat box. It is an operating system for requests, knowledge, permissions, evaluation, recovery, and cost. Each part needs an owner and a failure boundary.
This guide is part of the NexisHub AI Infrastructure desk. It connects practical engineering, research, and product decisions to the wider NexisHub platform.
The operating idea
Consider an AI assistant answering questions about a product's documentation. The model may sound fluent even when a retrieval index is stale, the request is outside the user's permissions, or a source was never consulted. The surrounding infrastructure must prevent these failures from being mistaken for trustworthy answers.
The architecture should be based on the task. A question-answering tool may need permission-aware retrieval and citations; a tool-using agent needs separate execution authorization and audit logs. Neither should silently mutate authoritative billing, enrollment, or customer records.
NexisHub separates verified platform documentation, repeatable observation, and inference. No optimization can guarantee selection or citation by an external system.
Seven components with different responsibilities
First comes the user-facing API and authentication layer. Second is request validation, abuse limiting, and user or tenant context. Third is the model gateway with timeout, retries, budgets and provider failover policies. Fourth is source retrieval, including indexing, freshness, authorization and provenance. Fifth is controlled orchestration for tools and multi-step work. Sixth is evaluation with representative cases and regression tests. Seventh is operations: traces, incident response, cost measurement, secrets and retention.
These components need not be seven independently deployed services. A modest product can begin as one application with well-defined modules and database tables. Splitting everything into microservices before measuring the need introduces more failure paths and operational cost.
An example request from entry to answer
A signed-in user asks about the status of an owned project. The server derives the account and authorized project IDs from the session, not from an editable prompt string. It searches the permitted records, chooses the latest valid source, attaches source identifiers to model context, and instructs the model to say when evidence is missing. The output should identify what was observed and which part is an inference.
If a request needs a state-changing tool, a separate authorization step checks the requested action, target resource, and any required user confirmation. A confident model answer must not bypass that check. For high-impact changes, store an idempotency key and an auditable outcome so retries do not silently duplicate the operation.
Build an evaluation set before expanding capability
Create cases for ordinary queries, nonexistent documents, conflicting versions, malicious embedded instructions, forbidden cross-tenant sources, unusually large context, and tool timeouts. Record expected behavior at the level of citations, refusal to make unsupported claims, and protection of permissions. A single fluent example is not meaningful coverage.
Version prompts, retrieval configuration, data snapshots, and model choices. Measure answer quality separately from tool correctness and latency. When a provider changes behavior, rerun the same set and classify regressions instead of assuming that a newer model improves every task.
Operational reality: costs and failure recovery
Track token and retrieval costs per meaningful completed task, not just provider invoices. Bound maximum context size and retry counts; an unbounded loop can turn one failed request into many paid calls. Log enough metadata for diagnosis but avoid raw sensitive content unless the storage purpose, authorization and retention plan justify it.
A failed model call should produce a clear user-facing error or safe partial result, not an invented success. A failed database mutation must preserve transactional integrity. Workers should support idempotent resume or compensation when a timeout happens between the external tool call and the local state update.
Core principles
- Derive permissions server-sideNever trust user-supplied prompt text as an authorization decision.
- Keep evidence connectedMake retrieved material versioned, permission-aware and attributable.
- Test adversarial casesInclude stale, conflicting, missing and malicious data in evaluation.
- Bound costs and retriesCreate budgets and explicit timeout/recovery policies.
A practical implementation workflow
Apply the work in a controlled sequence. Keep a baseline, name an owner, and define the evidence that will show whether each step was completed.
- 1. Write the task contractDefine user intent, permitted data and exact success conditions.
- 2. Map the trust boundaryIdentify each account, model, tool, index and record authority.
- 3. Implement constrained retrievalBind tenant and owner filters before ranking candidate context.
- 4. Add tools only when neededAuthorize every action independent of the model's stated confidence.
- 5. Run a versioned evaluationScore correctness, evidence, authorization, latency and failure responses.
- 6. Operate with evidenceReview traces, error rates, privacy controls, costs and recovery procedures.
Common mistakes
Treating the model response as a record
Generated text is not a substitute for verified business state.
Vector similarity as authorization
A relevant search hit is still forbidden when outside the user's scope.
Unlimited autonomous retries
Repeated failures can duplicate operations and inflate spending.
How to measure it responsibly
Report success against defined tasks: evidence coverage, unauthorized source retrieval rate, stale-content errors, controlled tool action success, latency distribution and cost per resolved request.
Preserve a red-team regression suite and an explicit incident review process. Numbers should reflect actual samples with dates and versions, not invented platform marketing metrics.
Keep observed outputs, diagnostic scores, inferred causes, and business outcomes in separate fields. A modelled score is not a citation, and correlation is not proof of cause.
What comes next
Agent protocols and model providers will change. Clear permission boundaries, versioned sources, bounded actions and accountable operations are more durable than a single framework.
The durable response is to build pages that are accessible, semantically explicit, useful outside their original layout, and backed by evidence a reader can inspect.
Key takeaways
01Start with a task contract and trust model.
02Do not confuse retrieval relevance with access permission.
03Evaluate tool operations separately from response fluency.
04Keep recovery, monitoring and cost in the initial design.
Frequently asked questions
Does an AI product need microservices?
Not automatically. Module boundaries and tested operations matter more than the number of deployed services.
Can an LLM decide whether a user is authorized?
No. Application code and authoritative identity systems must enforce permissions.
What should a first evaluation include?
Ordinary cases plus missing evidence, outdated sources, cross-tenant requests, prompt injection, tool failure and cost limits.
References and further reading
Turn a useful idea into a maintainable product.
Nexis Studio helps organisations move from problem definition to product design, engineering, launch, and long-term support with clear technical and commercial boundaries.
Explore Nexis Studio