Generative AI architecture for SaaS: the layered design and the tenant-isolation problem at its core

Generative AI architecture and the core isolation problem

The defining challenge in generative AI architecture for SaaS is multi-tenant data isolation: when one model and one retrieval pipeline serve many customers, the query that grounds the model must return only the calling tenant’s authorized data, every time. A similarity search has no concept of ownership or permission, so isolation has to be enforced in the architecture around it and can never be assumed inside the index. Get this layer right and the rest of the stack is conventional engineering; get it wrong and a single ordinary query can surface another customer’s data.

Start with the isolation model, because it sets the cost and the blast radius for everything above it. Three models recur across independent first-party vendors, which is the strongest signal they are the real design axis. Silo is a store per tenant: the best data and performance isolation and clean per-tenant cost attribution, at higher operational overhead, and Microsoft notes you can hit service limits, so it advises against silo when you have a large number of small tenants.¹ Pool is a shared store with a tenant discriminator in every query: cost-optimized and scales to far more tenants, but Microsoft calls data isolation the most important concern, and the query must include a tenant discriminator.¹ Bridge is the hybrid, with siloed premium or high-compliance tenants alongside a pooled tier for the rest, a naming AWS uses for its multi-tenant agent guidance.²

Per-tenant separation in a vector store

Inside the retrieval layer the decision narrows to how you separate tenants in the vector store, and this is the most technically load-bearing choice on the page. Pinecone’s multitenancy guidance is unambiguous: use one namespace per tenant as the default, because each namespace is stored separately and query cost scales with one tenant’s data rather than the whole index.³ It advises against putting all tenants in a single namespace and filtering by a tenant ID in metadata, because queries scan the entire namespace regardless of the filter, so you pay to scan every tenant’s data and latency grows with total size.³ The underlying reason is a property of approximate-nearest-neighbor indexes: the engine cannot prune by another field before searching, so it finds nearest neighbors first and filters by tenant after, which is both slower and more expensive.⁴ A separate index per tenant is the strongest option, reserved for high-compliance tenants or to split workloads.

One honesty note keeps the takeaway durable: the namespace-versus-metadata cost math is partly Pinecone-specific, because Pinecone meters by namespace. On other engines the exact economics differ, but the isolation hierarchy of separate index, then namespace, then metadata filter holds generally. That hierarchy is the durable rule; the cost figures belong to Pinecone.

Permission-aware retrieval inside a tenant

Tenant isolation keeps customers apart; permission-aware retrieval keeps users within a tenant to the data they are entitled to see. Within one tenant a viewer, an admin, and an auditor have different entitlements, so the authorization rules must be defined first and then used as the basis for retrieval filtering. Microsoft calls this filtering or security trimming, implemented through data-platform features like row-level security or custom access-control metadata on each chunk.¹

The mechanics that prevent cross-tenant leakage are concrete and converge across vendors. Carry the tenant in a signed token: AWS flows tenant context in a JWT and keys its data-layer access policies off the token’s claims, validating the principal against the requested path on every retrieval.² Inject the tenant filter automatically and sanitize results, which AWS states directly as a way to help prevent cross-tenant data leakage.² Route all data access through one gatekeeper API layer: Microsoft is explicit that code which needs tenant data should not be able to query the back-end stores directly, so a single API layer selects only the tenant’s rows, enforces trimming, and logs grounding access for audit.¹ Together these turn isolation from a convention any query could break into an invariant the architecture enforces.

The layered reference architecture

A SaaS AI architecture is a layered AI stack on top of the multi-tenant product: a model gateway and router, an orchestration layer, RAG retrieval, tools over the product API, guardrails on input and output, and evals plus observability, with caching and identity as cross-cutting rails. Present it as a reference architecture and treat it as one valid shape among several. Microsoft, AWS, and the LLM-gateway vendors describe the same primitives under different names, and that convergence is the credibility signal.

The canonical control flow ties the layers together. A client authenticates the user against an identity provider, then calls the orchestrator with the query plus the user’s authorization token; the orchestrator fetches tenant-authorized grounding data with security filtering applied, assembles the prompt, and calls the model’s inferencing API, with results returning to the app.¹ Each layer below maps to a real, citable concern, and the isolation work from the previous section lives inside the retrieval and tools rows. Building this layer for a product is what our AI application development team does, and the tenant-aware data side is where it meets our SaaS engineering work.

Model strategy: gateway, routing, and fallbacks

Most SaaS teams put a single gateway endpoint in front of every model provider and route per request, rather than hard-wiring one model into the product. The gateway fronts providers like OpenAI, Anthropic, Google, and self-hosted models, and provides failover, retries, conditional routing, load-balancing, caching, and budget controls as a standard function set.⁵ That one seam is where cost, reliability, and provider choice get managed instead of being scattered through the codebase.

Routing has three useful dimensions. Route by task complexity, sending easy turns to a small, cheap model and hard reasoning to a frontier model. Route by risk, sending sensitive or regulated prompts to a private or compliant deployment. Route by latency target so interactive paths stay fast. Fallbacks are a related gateway capability: on a provider error, timeout, or rate-limit, the gateway fails over from a primary to a secondary model so a single provider hiccup does not take the feature down.⁵

The hosted-versus-self-hosted choice is an engineering tradeoff with no single right answer. A managed gateway suits teams without spare DevOps capacity and gets a product to market faster; a self-hosted gateway gives infrastructure ownership, deeper routing control, and lower per-request cost at scale, at the price of real operational burden. Many enterprises reach for a private cloud deployment, such as AWS Bedrock or Azure OpenAI, when they need a hard data-residency or no-training guarantee, which is the architectural backstop behind the contractual one covered in the next section.

Security: prompt injection, PII, and training data

The dominant security risk in a SaaS AI feature is prompt injection, ranked LLM01:2025 by OWASP, and the variant that matters most for RAG is indirect injection, where malicious instructions arrive through retrieved or external content. OWASP’s mitigations map directly onto the stack: constrain model behavior in the system prompt, require source citations, filter inputs and outputs, enforce least privilege with scoped tool tokens, segregate untrusted content from instructions, gate high-risk actions behind human approval, and run adversarial testing.⁶ Each of these is a design decision the architecture has to make on day one, never a feature you bolt on at the end.

PII handling belongs at retrieval time as well as at display time: classify and scrub sensitive data before and after generation, which aligns with OWASP’s input and output filtering and with the AWS output guardrail that scans for sensitive-data leakage across tenant boundaries.² The data-not-for-training question has clear first-party answers worth stating plainly. Anthropic’s commercial terms state that Anthropic may not train models on customer content from its services.⁷ OpenAI does not use API inputs or outputs to train models by default, retains data up to thirty days for abuse monitoring, and offers zero-data-retention on eligible endpoints for qualifying organizations.⁸ Private cloud deployments keep data in the customer’s own tenancy and add the architectural backstop to those contractual terms.

Observability and evals in production

Treat the AI layer as a production system you can trace and evaluate, not a prompt you ship and forget. Trace requests, retrievals, prompts, tool calls, cost, latency, and the sources hit end to end, so quality correlates with cost and latency instead of living as anecdote. For a multi-tenant SaaS, scope those traces and eval dashboards per tenant, so a regression or a cost anomaly is attributable to a specific customer, which ties back to Microsoft’s recommendation to log grounding access for audit.¹

Evaluation comes in two modes. Offline evals run curated datasets of inputs and expected outputs as automated experiments before each release or prompt change, catching regressions ahead of users. Online evals sample a small share of live traffic and score it, often with an LLM as a judge plus explicit user feedback, to track qualitative trends in production. The two together give a pre-release gate and a live signal, and per-tenant scoping is what turns both into an audit trail a SaaS customer can trust. This page sits alongside our guides to agentic RAG and AI agents, which go deeper on the retrieval and agent layers named here.

Kanika Mathur

Head of Service Delivery, Resourcifi

Kanika Mathur is Head of Service Delivery at Resourcifi, where her engineering pods design the AI layer for multi-tenant SaaS products and run the tenant-isolation reviews before any retrieval path ships. She has sat in the design sessions where a namespace-versus-metadata decision quietly set a product’s per-query cost and its cross-tenant blast radius for years, which is the vantage point this guide is written from.

Resourcifi on LinkedIn →

Keep reading

Related guides worth your time

← →

Strategy, architecture & ops AI Architecture Patterns Agentic design patterns explained: reflection, tool use, planning, and multi-agent collaboration, with a framework to pic... Read guide → Strategy, architecture & ops AI Cost Optimization A senior-engineer guide to AI cost optimization: where LLM spend comes from, the levers ranked by payoff, the five number... Read guide → Strategy, architecture & ops AI Deployment Checklist: 9 Gates Before You Ship How to deploy AI models to production: a 9-gate pre-launch checklist anchored to the OWASP LLM Top 10 (2025), NIST AI RMF... Read guide → Strategy, architecture & ops AI Evaluation and Evals LLM evaluation and AI evals, explained: the eval taxonomy, how to build an eval suite, LLM-as-a-judge bias, offline vs pr... Read guide → Strategy, architecture & ops AI Features SaaS Customers Actually Want What AI powered SaaS customers actually want: the time-savers and answers they value, the automation they distrust, and h... Read guide → Strategy, architecture & ops AI Security Best Practices Generative AI security best practices: the OWASP Top 10 for LLMs, NIST AI RMF, MITRE ATLAS, lifecycle controls, agentic-A... Read guide → Agents & RAG Agentic RAG: When to Use It and How to Build It Agentic RAG explained: how it differs from naive and advanced RAG, the key patterns like corrective RAG and self-RAG, the... Read guide → Agents & RAG AI Agent for Fintech: Risk, Compliance, Ops, Customer AI agents in finance: fraud, AML, KYC and servicing use cases, how to build with money-movement guardrails and human appr... Read guide → Agents & RAG AI Agent for Healthcare: Use Cases, Governance & Implementation AI agents in healthcare: the use cases that pay off first, how to build one HIPAA-safe on FHIR with clinician review, and... Read guide →