What is RAG? How AI searches your documents before answering, and why retrieval often goes wrong

What is RAG? RAG (Retrieval-Augmented Generation) is a method of retrieving relevant information from a knowledge base and putting it into a prompt sent to an AI model, so the answer is based on your actual data instead of just relying on the model's memory. The concept was introduced by Patrick Lewis and colleagues in a paper published on 22/5/2020 (accepted at NeurIPS 2020). This post explains what RAG is, how it works, when you don't need RAG, why retrieval often goes wrong and how to fix it according to Anthropic's data, how to choose a vector database and embedding model for Vietnamese, and OPCAIOS's real experience building semantic search with pgvector.

RAG has two halves: preparing the knowledge base (chunking, embedding, storing) and answering (embedding the question, retrieving, generating). Diagram: OPCAIOS.

What is RAG and why was it created?

RAG is a method that combines two types of memory: memory in the model's parameters and external memory that can be looked up. The original paper by Patrick Lewis and colleagues (arXiv:2005.11401, published 22/5/2020) lists three problems with large language models, according to the abstract (read 23/10/2026):

1. The ability to access and manipulate knowledge precisely within parameters is limited.
2. The issue of citing sources for decisions has not been resolved.
3. The issue of updating knowledge has not been resolved.

The authors proposed a pre-trained generative model paired with a dense vector index of Wikipedia. The model accesses the index through a neural retriever. They reported the best results on three open-domain question-answering tasks (NeurIPS 2020). Additionally, on language generation tasks, the RAG model produces more specific, diverse and factually correct language than a model using only parameters. Think of it simply as an open-book exam: for example, instead of forcing the model to remember everything, you let the model flip to the right page of the document before answering.

What steps does RAG go through?

RAG is a process with two parts: preparing the knowledge base and answering when there is a question. According to Anthropic's guide (read 23/10/2026), the steps are as follows:

Preparation (done once, updated when documents change):

1. Cut the document repository into small chunks, usually no more than a few hundred tokens.
2. Use an embedding model to convert each chunk into a vector encoding its meaning.
3. Store the vectors in a database that allows searching by semantic similarity.

When there is a question:
4. Embed the question as a vector.
5. Find the chunks nearest to the question (top k).
6. Add those chunks to the prompt sent to the generative model.
7. The model answers based on those chunks, ideally with sources so users can verify.

For example, OPCAIOS uses pgvector in PostgreSQL as the vector database.

Component	Role	Example
Chunker	Divides documents into retrieval units	By heading, paragraph, few hundred tokens
Embedding model	Converts text into vectors	multilingual-e5-small (384 dimensions)
Vector database	Finds nearest chunks	pgvector in PostgreSQL
Reranker (optional)	Re-filters top chunks by relevance	Reranker model
Generative model	Writes answer from chunks	Claude, GPT, Gemini

When do you not need RAG?

You don't need RAG when your knowledge base is small enough to fit entirely into a prompt. Anthropic says if your base is under 200,000 tokens (around 500 pages), you can put the entire base in the prompt without needing RAG. With prompt caching, according to Anthropic, latency drops more than 2x and cost drops up to 90% (read 23/10/2026). When the base is larger than the context window, RAG is a common solution. Example assumption: a small business with 40 pages of internal procedures should try putting all the documents in the prompt first, then consider RAG. Also, choose simple first, because many small businesses have small enough document bases to start without RAG. On cost depending on usage level, see Which AI model should you choose?.

Why Does RAG Retrieve Wrong Results, and How to Fix It?

RAG retrieval errors happen when the retrieval step fails to provide the right passages to the model. There are two main causes, according to Anthropic (read on 23/10/2026) and the "Contextual Retrieval" article (Anthropic 2024). First, traditional RAG loses context when chunking: a passage like "revenue increased 3% compared to last quarter" doesn't say which company or which quarter, so it gets retrieved incorrectly. Second, embedding models are good at capturing meaning but may miss exact matches: for a question "Error code TS-999" the model might find content about error codes in general and miss the exact "TS-999". Three ways to fix this:

1. Combine BM25 with vector search: BM25 is a ranking function based on word matching, effective with identifiers and technical terms.
2. Contextual Retrieval: add a short context snippet written by a model at the start of each passage before embedding.
3. Reranking: take multiple candidates then filter again, both better and lower cost and latency.

What do Anthropic's test numbers show?

According to Anthropic's data (article "Contextual Retrieval", 19/9/2024), measured by the failed retrieval rate, meaning the rate of relevant passages not in the top 20: Contextual Embeddings reduced it by 35%, from 5.7% to 3.7%. Combining Contextual Embeddings and Contextual BM25 reduced it by 49%, from 5.7% to 2.9%. Adding reranking reduced it by 67%. Anthropic estimates the cost of creating context once at about $1.02 per million document tokens when using prompt cache. This is the result on their datasets (source code, novels, scientific articles), so measure again on your own data; Anthropic themselves also advise always running evaluation on your own case.

How to Choose a Vector Database?

If you already use PostgreSQL, pgvector is the cleanest choice for a vector database. It's an open source extension that adds vector search to Postgres, according to pgvector's README (read on 23/10/2026). The extension stores vectors alongside other data and supports both exact and approximate search. Distances include L2, dot product, cosine, L1, Hamming and Jaccard. You also keep Postgres advantages like ACID, point-in-time recovery and JOINs. There are two types of approximate indexes:

1. HNSW: builds a multi-layer graph, has better speed and accuracy but slower build and higher memory. This type doesn't need existing data to build the index.
2. IVFFlat: builds faster, but needs data beforehand to create the index.

Approximate indexes trade some accuracy for speed, so results may differ when you add an index. For example, if you already use Supabase then that's PostgreSQL ready to go; see What is Supabase?.

Which Embedding Model for Vietnamese?

The embedding model for Vietnamese should be a multilingual model, and you need to test it on your own data. One example is multilingual-e5-small by intfloat on Hugging Face, with a technical report by Liang Wang and others. According to the model card on Hugging Face (read on 23/10/2026), the model has 12 layers, embedding size 384, supports 94 languages and uses MIT license. There is one technical detail easy to get wrong: the model card requires each input text to start with "query: " or "passage: ", even for non-English text. Specifically, use "query: " for questions and "passage: " for document passages. If you forget the prefix, retrieval quality drops but the system won't report an error. OPCAIOS uses this exact model for internal semantic search, as described in the experience section below.

How to Evaluate a RAG System?

Split evaluation into two parts: does retrieval get the right passages, and is the answer correct and sourced. The practical way:

1. Create a set of 30 to 100 real questions each with the correct document passages.
2. Measure whether the correct passage is in top k (recall@k, like Anthropic uses with k = 20).
3. Manually read a sample of answers: are they made up, do they cite sources correctly.
4. Repeat each time you change chunking method, embedding model or prompt.

Risks and Security Considerations When Using RAG

The risk of RAG is that the model can still answer incorrectly or leak data, since RAG reduces rather than eliminates errors. Five points to keep in mind before deploying RAG to real users:

• Access control at retrieval step: only retrieve documents the user is allowed to see; if internal documents are placed in a shared repository, unauthorized users could see them through the response.
• Retrieved content is data, not commands: a document can contain sentences deliberately designed to manipulate the model (prompt injection); see the principles in What is MCP?.
• Personal data: consider whether to include customer data in the repository and in prompts sent to the model provider.
• Outdated documents: the repository must be kept up to date; old passages will produce outdated responses.
• Show sources: always let users see the document passages used so they can verify them.

OPCAIOS experience: semantic search with pgvector

OPCAIOS built the retrieval part of RAG for the internal management area: semantic search across projects, tasks, discussions, support tickets, guidance documents, customers, customer exchanges and sales opportunities. The table storing passages has a 384-dimensional vector column in PostgreSQL with pgvector, using HNSW index with cosine distance; text is embedded using the multilingual-e5-small model running in a separate service that only listens internally, with "query: " and "passage: " prefixes; content is hashed so only re-embedded when a record changes, and each indexing run has a limit on the number of passages. If the embedding service is not configured, the management area falls back to keyword search. Data is indexed only from the management area's own internal data. This is half of "retrieval" (R): we use it to find the right records, we haven't yet built a chatbot that writes responses from those passages, and we haven't published accuracy metrics. At the memory layer of the AIOS system, documents and large knowledge bases are also stored in a vector database following a document, passage and vector structure, for agents to look up.

How to start building RAG?

1. Define questions and documents: pick a document group and 30 real questions.
2. Try without RAG first: if the documents are small, put them in the prompt along with buffer memory.
3. Split passages by meaning: by headings and paragraphs, a few hundred tokens each, with source title attached.
4. Choose a multilingual embedding model and an available vector database (like pgvector).
5. Add keyword search and re-ranking if measurement shows retrieval is missing results.
6. Show sources, set permissions, measure before giving real users access. To automate repository updates, see n8n or Make.

Read more: How to Create a Self-Learning AI Chatbot for Internal Documents (RAG) with Dify and Supabase.

Frequently Asked Questions

What is RAG?

It's a method that retrieves relevant information from a knowledge base and puts it into a prompt so the AI model answers based on real data, often with sources included. The concept came from a paper by Lewis and colleagues in 2020.

How is RAG different from fine-tuning?

RAG brings knowledge in at query time through retrieval and can be updated by changing documents, while fine-tuning changes the model's parameters. RAG works well with knowledge that changes and when you need to cite sources.

Do I need RAG if I have little documentation?

Not necessarily. Anthropic says if your knowledge base is under 200,000 tokens (roughly 500 pages), you can put everything in the prompt and use prompt caching to be faster and cheaper.

Why does RAG give wrong answers even with documents?

RAG gives wrong answers even with documents usually because retrieval is broken, meaning the search step doesn't bring the right passages to the model. There are three common reasons. First, passages lose context when cut, like a passage saying only "revenue grew 3%" without naming the company. Second, embedding models miss exact matches like error codes or proper names. Third, the right passage doesn't land in the top k. Combining BM25, Contextual Retrieval, and reranking helps cut these errors: according to Anthropic, retrieval errors drop 49% with Contextual Embeddings plus Contextual BM25 and drop 67% when adding reranking, on their datasets (read 10/23/2026). Test these numbers on your own data before trusting them.

What should I watch out for with Vietnamese RAG?

Use multilingual embedding models, test with real questions, and follow the model's prefix requirements (like "query: " and "passage: " with E5). Want to learn how to build AI systems using your own data? Check out OPCAIOS's AI Application Training course.

Sources: the paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (arXiv:2005.11401, NeurIPS 2020); Anthropic's "Contextual Retrieval" post (9/19/2024); pgvector README; the intfloat/multilingual-e5-small model card on Hugging Face; direct reading on 10/23/2026. Anthropic's numbers are from their test datasets. Implementation experience is from OPCAIOS, without accuracy metrics.