All case studies
Case study 02AI Systems

Evidence-Grounded RAG Assistant

Access-aware retrieval with hybrid search, reranking, and citations, measured on a fixed evaluation set.

Role
Senior engineer · retrieval and evaluation
Core stack
LangChainLangGraphHybrid searchRerankingLangSmith
68%→84%
Human-rated answer acceptance
Cited
Answers link back to their source evidence
Access-aware
Results filtered by each user's permissions

Overview

RAG agents that answer only from evidence the user is allowed to see, cite their sources, and are tracked against a fixed evaluation dataset.

Answers had to be accurate, traceable to sources, and respect document permissions. Improvements needed to be measured, not guessed, and inference costs had to stay under control.

Product walkthrough

The flow, screen by screen

Cited answer
01 / 03

Cited answer. Answers are generated only from retrieved evidence, and each claim links to its source passage. Documents the user can't access never appear.

ArchitectureCorePlatformAsync / control
hybrid searchmodelscoredKnowledge sourcesdocuments · recordsIngestionparse · chunk · ACL tagsHybrid indexlexical + vectorUser questionwith identityAccess-aware retrievalpermission filtersRerankertop-k precisionGrounded answerLLM + citationsAnswerwith sourcesModel routingcontext budgetsLangSmithfixed eval set
Ingestion builds a hybrid index; queries pass permission filters and reranking before a cited answer is generated and scored.

Approach

  1. 01Built access-aware retrieval so results are filtered by the user's permissions before they reach the model.
  2. 02Combined lexical and vector search in a hybrid index, then reranked candidates before generation.
  3. 03Generated answers with source citations so every claim can be checked against the evidence.
  4. 04Tracked quality with LangSmith regression suites covering task completion, tool selection, and grounding.
  5. 05Optimized context budgets and model routing to reduce inference cost without lowering answer quality.

Key decisions

Filter by permission before ranking

Applying access rules at retrieval time prevents restricted content from ever entering the prompt.

Hybrid search plus reranking

Lexical search catches exact terms and identifiers; vectors catch meaning; the reranker sharpens the final context.

A fixed evaluation dataset

Holding the dataset constant makes before-and-after comparisons meaningful and catches regressions.

Outcomes

  • Permission-aware results by design, not by prompt instruction.
  • Regression suites run before changes reach users.
  • LangChain
  • LangGraph
  • Hybrid search
  • Reranking
  • LangSmith
  • FastAPI
  • PostgreSQL
  • Redis
  • AWS

Working on something similar?

I work with clients worldwide, across any time zone. Tell me what you're building.

Send a message Email meMore case studies