An AI trace is useful when it explains what happened between a user's request and the result they saw. A stack of model tokens and latency numbers is not enough when retrieval, tool permissions and downstream actions are involved.
This guide is part of the NexisHub AI Infrastructure desk. It connects practical engineering, research, and product decisions to the wider NexisHub platform.
The operating idea
Consider an assistant answering a question about a company's published policy. It may retrieve four passages, choose two, call a model, and return a confident summary. If the answer misstates the policy, the team needs to know which document version was retrieved, whether the wrong passage was selected, and what the model was allowed to infer.
Observability should connect a request identifier to retrieval, model execution, tool actions, policy decisions and the final response. It must also minimize exposure of private prompts, credentials and tenant data in log storage.
NexisHub separates verified platform documentation, repeatable observation, and inference. No optimization can guarantee selection or citation by an external system.
A trace through retrieval and generation
Start with a root span holding a request ID, route, environment and approved tenant-safe correlation identifiers. Add spans for input validation, retrieval, reranking or passage selection, context preparation, model invocation and response processing. Each span should capture the operation, outcome, latency and a bounded error category.
For retrieval, record source IDs, versions, collection identifiers and selection reasons where possible. If document text cannot be stored for privacy reasons, keep a safe source reference or hash rather than logging raw sensitive excerpts. The aim is to reconstruct decisions, not duplicate confidential data in telemetry.
Separate quality evidence from runtime health
A trace can show that a model call returned status 200, used a certain number of tokens, and completed in two seconds. It cannot prove the answer was correct. Quality evaluation needs a labeled test question, expected evidence, acceptance criteria and a review of the output.
Use a small regression dataset containing contradictory documents, absent answers, stale document versions and irrelevant retrieval candidates. Record whether the assistant cites the correct version, declines unsupported assertions and handles retrieval failure cleanly. A latency dashboard and a factuality assessment belong in different panels.
Tool actions need stronger evidence
An AI tool call that requests a side effect should record the authenticated actor, authorized scope, validated arguments category, idempotency key and final business acknowledgement. A 'tool succeeded' event may only mean a request was queued; it must not be represented as a completed customer outcome without downstream confirmation.
For an illustrative appointment assistant, a tool invocation might return a provisional hold identifier. Only the booking system's confirmed reservation should be shown as a completed booking. Similar boundaries apply to payment, email delivery, publishing and account changes.
A bounded NexisHub engineering example
NexisHub's documented Vercel incident illustrates a broader observability distinction. Its remote build completed compilation yet failed during output deployment. Later, a READY Preview still returned an admin 500 because a signing secret was absent. These were separate lifecycle events requiring separate logs and acceptance checks, not proof of a problem inside an AI model.
The same principle should apply to NexisHub's future AI features: a generated recommendation, a queued workflow, a deployed service and a verified customer outcome should not share one unqualified success flag. This is an architectural recommendation based on the observed deployment workflow, not a claim that a completed AI tracing system is already deployed.
Core principles
- Trace boundaries, not just model callsInclude retrieval, context assembly, guard decisions, tools and outcome acknowledgement.
- Protect data by defaultAvoid storing raw private conversations, API keys, third-party tokens or cross-tenant identifiers in shared logs.
- Keep technical success distinct from answer qualityA completed request is not proof of a truthful or useful answer.
- Correlate events with stable identifiersMake it possible to follow one request across workers without confusing it with another user's session.
A practical implementation workflow
Apply the work in a controlled sequence. Keep a baseline, name an owner, and define the evidence that will show whether each step was completed.
- 1. Map a single real requestDocument each data, model and tool boundary before choosing a telemetry library.
- 2. Define safe span fieldsUse operation name, timestamp, duration, outcome and non-sensitive source/version identifiers.
- 3. Instrument failure pathsCapture timeouts, unavailable documents, rejected tool permissions and downstream failures as explicit events.
- 4. Connect evaluationJoin each representative test case to its trace and human-reviewed outcome.
- 5. Set retention and access rulesSpecify tenant-safe retrieval, redaction, retention, deletion and incident access controls.
Common mistakes
Logging every prompt verbatim
Raw prompts can contain personal, financial or proprietary information unrelated to the telemetry purpose.
Equating tokens with useful work
Token counts are cost inputs, not proof of answer accuracy.
Calling a queued action complete
Async workflows need independently verified final-state acknowledgement.
How to measure it responsibly
Measure latency by stage, error rates by category, model and retrieval cost, source-version coverage, and the proportion of tool actions with verified outcomes.
Add quality measures from reviewed test cases: unsupported-answer rate, stale-source use, incorrect citation rate and boundary violations. State test-sample limits instead of presenting them as universal percentages.
Audit access to traces and use redaction tests to confirm sensitive fields remain unavailable to unauthorized viewers.
Keep observed outputs, diagnostic scores, inferred causes, and business outcomes in separate fields. A modelled score is not a citation, and correlation is not proof of cause.
What comes next
As AI products gain more tools and memory, the most valuable traces will identify where authority came from, which state actually changed and what evidence permits the result to be trusted.
The durable response is to build pages that are accessible, semantically explicit, useful outside their original layout, and backed by evidence a reader can inspect.
Key takeaways
01Track retrieval and tool boundaries in addition to model calls.
02Preserve source versions without leaking private text.
03Separate model execution from verified customer outcomes.
04Treat trace storage as a security and privacy system.
Frequently asked questions
Does successful model execution prove a reliable answer?
No. Runtime success and substantive correctness require different evidence.
Should traces include the full user conversation?
Not by default. Use purpose-limited, redacted fields and carefully reviewed retention controls.
How do I debug a wrong citation?
Follow the source version, retrieval candidates, selection step, context assembly and response generation for that request.
Can traces prove that a payment or booking finished?
Only if they include a trustworthy acknowledgement from the authoritative downstream system.
References and further reading
Turn a useful idea into a maintainable product.
Nexis Studio helps organisations move from problem definition to product design, engineering, launch, and long-term support with clear technical and commercial boundaries.
Explore Nexis Studio