Retrieval-Augmented Generation (RAG)

What it is

Retrieval-augmented generation connects a language model to your knowledge at the moment a question is asked. Instead of relying on what the model memorized during training, the system searches your documents, databases or tickets, passes the most relevant passages to the model, and asks it to answer from that evidence — ideally with citations.

RAG is the most common enterprise AI pattern because it keeps answers current without retraining, respects existing permissions, and makes answers auditable. It is also where most AI projects succeed or fail: retrieval quality, not the model, is usually the limiting factor.

How it works

1. Ingest

   Parse PDFs, pages, tickets and tables into clean text, preserving structure, metadata and permissions.

2. Chunk

   Split content into passages sized for retrieval, keeping headings and context attached.

3. Embed and index

   Convert chunks into vectors and store them alongside a keyword index and metadata.

4. Retrieve

   Run hybrid (vector plus keyword) search, filter by metadata and user permissions, and rerank the top results.

5. Generate

   Pass the best passages to the model with instructions to answer only from them and cite sources.

6. Evaluate

   Measure retrieval recall, answer faithfulness and citation accuracy on a test set, and keep monitoring.

Key concepts

Chunking strategy

Fixed, structural or semantic splitting; chunk size and overlap strongly affect retrieval quality.

Hybrid search

Combining vector similarity with keyword (BM25) search to catch both meaning and exact terms such as part numbers.

Reranking

A second-stage model that re-scores candidate passages for relevance before they reach the LLM.

Query rewriting

Expanding or decomposing the user's question into better search queries.

Grounding and citations

Constraining answers to retrieved evidence and linking each claim to its source.

Permission-aware retrieval

Filtering results by the user's access rights so the assistant never leaks restricted content.

Agentic and graph RAG

Letting an agent search iteratively, or using a knowledge graph, for multi-hop questions.

Tools & platforms

Skills employers look for

• Building ingestion pipelines for messy real-world documents, including tables and scanned PDFs.
• Choosing chunking, embedding and hybrid retrieval settings based on measured results.
• Adding reranking, metadata filters and permission-aware retrieval.
• Prompting for grounded answers with citations and a clear "I don't know" path.
• Evaluating retrieval recall and answer faithfulness, and debugging which stage failed.
• Keeping indexes fresh with incremental updates and deletions.

Enterprise use cases

Common pitfalls

• Blaming the model for retrieval failures — check whether the right passage was retrieved before changing prompts.
• Ignoring document quality — duplicate, outdated and badly parsed content produces confident wrong answers.
• Skipping access control — an index that ignores permissions is a data leak waiting to happen.
• No evaluation set — without measured recall and faithfulness, every change is a guess.