RAG Search for SaaS Platforms: Retrieval, Permissions and Evaluation

A support agent asks, “What does our enterprise cancellation policy say?” The system returns a fluent answer from a two-year-old draft belonging to another tenant. The model did exactly what it was given; the product failed at retrieval, permissions, freshness and evidence.

RAG is useful when the answer must be grounded in changing private knowledge. It is not a truth switch. Quality depends on what enters the index, how documents are divided, which filters run before retrieval, how ranking is evaluated, what context reaches the model and whether the user can inspect the source.

This guide defines a SaaS-grade search system around measurable retrieval rather than a chatbot demonstration.

Define the search job

Support lookup, policy Q&A, document discovery and workflow assistance need different answers.

The failure mode is concrete: one assistant attempts every knowledge task. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, choose a narrow user, corpus, question class and acceptable action. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include user decision, response form, freshness, latency, abstention and escalation. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with representative tasks and out-of-scope prompts. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: product lead
• Release evidence: use-case contract
• Stop condition: success cannot be measured

Inventory authoritative sources

Not every file or message should become searchable truth.

The failure mode is concrete: drafts and duplicated exports outrank approved policy. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, classify systems of record, authority, owner and lifecycle. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include source type, tenant, status, effective date, sensitivity and retention. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with superseded document and deleted source. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: knowledge owner
• Release evidence: source register
• Stop condition: no owner can identify authoritative content

Build repeatable ingestion

Connectors must detect create, update, delete and permission changes.

The failure mode is concrete: a one-time crawl becomes stale. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, use idempotent ingestion with version and deletion propagation. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include source ID, checksum, version, fetched time, status and error. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with duplicate delivery, partial parse and deleted page. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: data engineer
• Release evidence: ingestion reconciliation
• Stop condition: index contains content removed from source

Parse structure before chunking

Headings, tables, lists and page boundaries carry meaning.

The failure mode is concrete: fixed character cuts separate claims from qualifiers. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, preserve document hierarchy and provenance through parsing. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include title path, section, page, table, language and offsets. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with PDF columns, table and malformed file. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: document pipeline owner
• Release evidence: parse quality sample
• Stop condition: citation cannot locate the source passage

Checkpoint 4: reconcile the product promise with the recorded state. Support, finance, security, and delivery teams should be able to reach the same conclusion from the same identifiers without reconstructing events from chat messages or screenshots.

Choose chunks by retrieval behavior

Chunk size trades context against specificity.

The failure mode is concrete: one universal setting is accepted without evaluation. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, compare structure-aware strategies on real questions. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include token length, overlap, heading context, parent link and metadata. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with short fact, multi-section answer and table lookup. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: retrieval engineer
• Release evidence: chunk experiment
• Stop condition: choice has no measured basis

Enforce tenant access before retrieval

Post-filtering generated text is too late.

The failure mode is concrete: another tenant document enters model context. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, resolve identity and permissions before candidate selection. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include tenant, groups, document ACL, expiry and policy version. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with cross-tenant query, revoked user and shared source. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: security owner
• Release evidence: negative isolation tests
• Stop condition: unauthorized text reaches the prompt

Protect embedding and index metadata

Vectors and metadata remain sensitive application data.

The failure mode is concrete: indexes use public namespaces or weak filters. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, apply isolation, encryption, retention and access logging to every store. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include namespace, record tenant, source ACL, key management and deletion. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with guessed ID and backup restore. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: data protection lead
• Release evidence: index control review
• Stop condition: index access bypasses SaaS authorization

Combine lexical and semantic retrieval

Exact identifiers and names often favour lexical search; paraphrases favour semantic similarity.

The failure mode is concrete: vector-only search misses codes or returns conceptually broad text. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, evaluate hybrid retrieval and fusion for the corpus. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include lexical rank, vector score, filters, fusion and candidate count. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with error code, acronym, paraphrase and rare name. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: search lead
• Release evidence: retrieval comparison
• Stop condition: one method wins only on handpicked examples

Checkpoint 8: reconcile the product promise with the recorded state. Support, finance, security, and delivery teams should be able to reach the same conclusion from the same identifiers without reconstructing events from chat messages or screenshots.

Add reranking where evidence supports it

Reranking can improve top results but adds latency and cost.

The failure mode is concrete: a new model is added without measuring value. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, compare candidate recall and final ranking on labelled queries. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include top-k, reranker version, score, timeout and fallback. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with ambiguous query and reranker outage. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: ML engineer
• Release evidence: quality-latency curve
• Stop condition: cost increases without significant retrieval gain

Treat retrieved text as untrusted

Documents can contain instructions aimed at the model.

The failure mode is concrete: retrieved text overrides system policy or triggers tools. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, separate data from instructions and constrain model and tool authority. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include prompt delimiters, tool scopes, output validation, suspicious-content signals and human confirmation. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with embedded prompt injection and poisoned page. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: AI security lead
• Release evidence: adversarial test set
• Stop condition: document text can authorize external action

Require evidence-linked answers

Users need to inspect the passages supporting a claim.

The failure mode is concrete: citations point to a whole document or irrelevant chunk. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, return source title, stable URL or record, section and relevant passage mapping. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include source version, access check, span, effective date and display. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with multiple sources, revoked source and no support. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: UX owner
• Release evidence: citation precision review
• Stop condition: answer remains when all supporting passages are removed

Design abstention and fallback

Some questions have no current or authorized answer.

The failure mode is concrete: the model fills gaps fluently. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, set evidence thresholds and provide search results or escalation when insufficient. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include retrieval score, coverage, conflict, freshness and user messaging. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with unanswerable, conflicting policy and vague query. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: product owner
• Release evidence: abstention benchmark
• Stop condition: system invents an answer without evidence

Checkpoint 12: reconcile the product promise with the recorded state. Support, finance, security, and delivery teams should be able to reach the same conclusion from the same identifiers without reconstructing events from chat messages or screenshots.

Create a retrieval evaluation set

End-to-end thumbs-up data is too noisy for diagnosis.

The failure mode is concrete: quality is judged from demos. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, label representative questions with relevant passages and acceptable outcomes. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include tenant, intent, difficulty, gold sources, freshness and deny cases. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with easy, adversarial, no-answer and cross-tenant items. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: evaluation lead
• Release evidence: versioned eval set
• Stop condition: release has no comparable baseline

Measure retrieval and answer separately

A good passage can still produce a bad answer and vice versa.

The failure mode is concrete: one aggregate score hides the failed layer. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, track candidate recall, ranking, citation support, answer correctness and abstention. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include recall@k, ranking metric, faithfulness rubric, latency, cost and reviewer. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with model change, index change and prompt change. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: quality lead
• Release evidence: layered evaluation report
• Stop condition: team cannot locate a regression

Operate freshness, cost and incidents

Indexes drift, models change and connectors fail.

The failure mode is concrete: the prototype has no monitoring or rollback. This is not solved by adding another screen or background job. The product has to define ownership, permitted transitions, and the evidence retained when the transition occurs.

For the first release, monitor ingestion lag, retrieval shifts, denied access, latency, spend and user corrections. Write that decision as an enforceable server-side rule, not guidance that depends on a client, operator, or AI model remembering the intended boundary.

Implementation should include SLOs, alerts, trace IDs, version inventory, rollback and incident playbook. Keep the public response smaller than the internal record: users need a clear outcome and recovery path, while authorized operators need correlation IDs, policy versions, timestamps, and the before-and-after state.

Validate it with connector outage, bad reindex and model change. Test the ordinary path, then repeat under timeout, duplicate delivery, stale state, partial failure, revoked authority, and concurrent requests. A feature is not ready when only the demonstration succeeds.

• Owner: operations lead
• Release evidence: recovery rehearsal
• Stop condition: a broken index cannot be rolled back

Implementation references

Use Retrieval-Augmented Generation Paper and record the version applied to this release.

Validate decisions against NIST AI RMF Generative AI Profile and record the version applied to this release.

Review OWASP Prompt Injection Guidance and record the version applied to this release.

Compare with OWASP LLM Top 10 and record the version applied to this release.

Continue with AI Development when translating this guide into delivery scope.

Frequently asked questions

Does RAG eliminate hallucinations?

No. It can supply relevant evidence, but retrieval can be wrong or incomplete and generation can still misstate it. Evaluate both layers and support abstention.

What should be indexed first?

A narrow, authoritative corpus with clear ownership, permissions, versioning and deletion behavior—not every available file.

How should chunks be sized?

Test structure-aware chunk strategies against real labelled questions. There is no universal size that is best for every corpus.

Is vector search enough?

Often not. Hybrid lexical and semantic retrieval performs better when the corpus includes exact identifiers, names, codes and paraphrased concepts.

How do we prevent cross-tenant leakage?

Resolve the user and tenant, enforce source ACLs before candidate retrieval, isolate indexes appropriately and test negative cases end to end.

Can retrieved documents contain prompt injections?

Yes. Treat retrieved text as untrusted data, keep tool authority constrained and test adversarial documents.

What should a citation contain?

A stable authorized source reference, version or effective date, section or page, and a mapping to the passage that supports the claim.

What must be monitored in production?

Ingestion freshness, deletion propagation, retrieval quality, authorization denials, abstention, latency, cost, corrections and version changes.

Turn the design into operating evidence

The release is credible when ordinary users can complete the promised journey and operators can explain and recover every important exception from durable state. A polished demonstration is not a substitute for those properties.

Keep the scope narrow, the language accurate and the acceptance evidence visible. That creates a better foundation for expansion than features whose permissions, financial consequences or quality thresholds remain implicit.