Loading…
Identity & Security
A working checklist security and delivery teams can use before an LLM-powered feature moves from prototype to shared environments.
Threat modeling for generative AI features does not require a new methodology cult. It requires the same questions security teams already ask—updated for prompts, retrieved context, and tool side effects. Use this checklist as a conversation guide in design reviews, not as a compliance sticker.
List what the feature touches: user prompts, system prompts, retrieved documents, tool credentials, logs, evaluation sets, and downstream records. Then list who can influence those assets: end users, internal operators, document authors, administrators, model vendors, and anyone who can plant content that might later be retrieved.
Indirect actors matter. A supplier PDF in a knowledge base can become an instruction channel if your assistant treats retrieved text as authoritative steerable content.
Mark where data crosses from untrusted input to privileged action. Common boundaries include:
For each crossing, ask what authenticates the caller, what is authorized, and what is logged. If you cannot answer in one sentence, the boundary is underspecified.
**Confidentiality.** Can prompts, context, or logs expose secrets or personal data to the wrong party—including the model provider and support tooling? Are training or retention settings understood for every vendor in the path?
**Integrity.** Can an attacker cause unsafe actions through prompt injection, poisoned documents, or manipulated tool arguments? Can the model overwrite authoritative fields with fluent nonsense?
**Availability.** Can oversized inputs, recursive tool loops, or expensive retrieval patterns create denial-of-wallet or denial-of-service conditions?
Add safety and misuse questions your domain needs: harmful advice, discriminatory outputs, or policy bypass attempts.
Map features onto existing identity, API gateway, DLP, secrets management, and SIEM capabilities before inventing AI-only controls. Often the highest leverage fix is refusing to give the model a powerful credential, or reusing the same authorization checks your non-AI clients already pass through.
Document control gaps honestly. A red-team finding that restates a known missing ACL filter is still useful—it prioritizes remediation.
Write a short list of attempts you expect:
For each case, define the expected system behavior and a test that would fail if that behavior regresses.
Not every feature needs dual control. Every feature needs a stated oversight rule. Examples: auto-send only for low-risk classifications; require reviewer approval for external messages; block tools entirely in public anonymous sessions. Put the rule in the design doc so engineers and risk partners argue about the same object.
Assume some injections will succeed. Ensure you can answer: Which user triggered the run? Which documents were in context? Which tools fired? Which prompt and model versions were live? Without that telemetry, "we will monitor" is wishful thinking.
Predefine who gets paged for suspected abuse, how to disable a tool quickly, and how to rotate credentials if logs suggest leakage.
Threat models rot when corpora, tools, or vendors change. Tie reviews to release milestones: new tool added, new data source connected, model provider switched, audience expanded from internal to external.
A lightweight session with engineering, security, and the product owner beats a long document nobody updates. Capture decisions as durable checklists attached to the service—not as slides that vanish after the pilot demo.
AI features move faster than traditional releases. The checklist above is a way to keep speed from outrunning accountability.
More approved articles related to this topic.
Tool calling lets models take action through APIs. These patterns keep side effects explicit, authenticated, and reviewable inside existing integration landscapes.
Read article →How to design retrieval-backed assistants that show their work, respect access controls, and stay honest when the corpus does not contain an answer.
Read article →Let’s turn your next technology priority into a clear plan.
Tell us what you are trying to modernize, connect, secure, or automate. We’ll review your goals and recommend a practical next step.
