RAG is a method for generating responses based on your documents rather than relying on vague memories from the training phase. The basic architecture consists of only four steps, but each step has its own potential pitfalls.
Four Steps and Their Corresponding Pitfalls
1. Trim the document. Cutting based on a fixed number of characters is the most common mistake—it breaks sentences, separates tables from their headings, and splits code blocks in half. You should cut along semantic boundaries: headings, paragraphs, and list items. And always include the parent context—at least the document title and the heading hierarchy.
2. Vector embedding. Multilingual embedding models handle Vietnamese differently. You should test them on your own data rather than relying on general rankings.
3. Search. This is the decisive step. Using vectors alone will fall short when it comes to proper nouns, error codes, and product codes. Combining keyword search with vector search and then merging the results almost always yields better rankings than using vectors alone.
4. Generate the answer. Including too many segments makes the answer worse. It’s better to generate 20 segments and then use a ranking model to select the 4 best ones.
Does the long context window eliminate RAG?
No, for three practical reasons. Cramming an entire repository of documents into each question incurs a cost per query. Latency increases with the amount of context. And the quality of attention paid to information buried within a long block of text is still lower than when that information is presented selectively.
Longer context makes RAG easier to work with—you’re allowed to include more passages—but it doesn’t replace it.
What to measure
Most teams only evaluate the final answer. Let’s break it down: does the search phase retrieve the segment containing the answer, and does the generation phase use that exact segment? Almost every time, the score for the first part is lower.
Thảo luận