Loading…
AI & Agents
How to design retrieval-backed assistants that show their work, respect access controls, and stay honest when the corpus does not contain an answer.
Enterprise teams often ask for an assistant that "knows our documents." That phrase hides several design choices. The useful product is rarely a free-form chatbot with a vector index bolted on. It is a retrieval-backed system with clear boundaries: which sources it may use, how it proves that an answer came from those sources, and what it should do when evidence is missing.
Before picking a chunk size or an embedding model, collect the questions people already ask—and the documents a careful analyst would open to answer them. Those pairs become your first evaluation set. If you cannot point to a source for a high-value question, retrieval will not invent a trustworthy answer later. It will only make an ungrounded reply sound more confident.
A practical intake worksheet asks: Who is the user? What system of record should win if two documents disagree? Is a partial answer acceptable? Must the assistant refuse when confidence is low? Those decisions belong in product requirements, not in prompt folklore.
When an assistant fails, teams often rewrite the system prompt. Sometimes that helps. More often the model never saw the right passage. Split your debugging:
Improving (1) looks like better parsing, metadata filters, hybrid search, and access-aware indexing. Improving (2) looks like tighter instructions, citation requirements, and answer schemas. Improving (3) looks like linking quotes to document titles, sections, and timestamps—not burying sources in a footer nobody reads.
In many enterprises, the hard part is not cosine similarity. It is whether the requester is allowed to see a paragraph at all. If your index flattens every PDF into a shared namespace, you have created a new disclosure path. Design for ACL-aware retrieval from day one: propagate document permissions into the index, filter before the model sees context, and log which objects were eligible for each answer.
Multi-tenant and partner scenarios need the same discipline. A "helpful" cross-tenant match is a security incident with friendly UX.
Users forgive a clear "I do not have an approved source for that." They do not forgive a polished answer that invents a policy clause. Prompt patterns that help:
None of these patterns eliminate risk. They make failure modes observable and reviewable.
Golden questions should include easy lookups, ambiguous phrasing, outdated document versions, and questions whose correct behavior is refusal. Score retrieval hit rate separately from answer faithfulness. When you change chunking, embedding models, or prompts, re-run the suite before calling the change an improvement.
Also track operational signals: empty retrieval rate, citation click-through, user corrections, and escalations to experts. Those metrics describe whether the assistant is useful—not whether a demo looked impressive.
Grounded assistants need owners. Someone must approve new corpora, retire stale content, and decide what happens when a source system changes. Treat the knowledge index like a product surface with release notes. Pair engineering changes with content stewardship. Without that pairing, even a well-built RAG pipeline slowly becomes a museum of last quarter's PDFs.
The goal is not omniscience. The goal is a dependable colleague that retrieves the right evidence, shows its work, and stays quiet when the archive does not support an answer.
More approved articles related to this topic.
A working checklist security and delivery teams can use before an LLM-powered feature moves from prototype to shared environments.
Read article →Tool calling lets models take action through APIs. These patterns keep side effects explicit, authenticated, and reviewable inside existing integration landscapes.
Read article →Let’s turn your next technology priority into a clear plan.
Tell us what you are trying to modernize, connect, secure, or automate. We’ll review your goals and recommend a practical next step.
