A three-month record that shattered the illusion that ``RAG is easy''. ~ Debugging of LangChain, LlamaIndex, and Chromadb ~
“If we just hit LangChain, plug it into Vector DB, and build RAG (Retrieval-Augmented Generation), we can create the strongest AI chatbot using our own data.”
...I want to use a time machine to punch my past self who believed in such sweet words right now.
This time, when I started a project to build a RAG system that was trained from in-house manuals and technical documents, I ended up hitting the gates of hell, which involved ``data preprocessing and tuning of search accuracy'' rather than ``implementation''. In this article, I will share the ``real pains of building a RAG'' that I learned while switching back and forth between LangChain and LlamaIndex and struggling with ChromaDB, along with technical knowledge without any presumptions.
1. Why “just a vector search” doesn’t work
First of all, I would like to say that ``Vector Search (Similarity Search) is not a shortcut to the correct answer.''
At first, I thought a simple configuration would be fine.
- Extract text from internal PDF
- Split with LangChain's `RecursiveCharacterTextSplitter`
- Vectorization with OpenAI's `text-embedding-3-small`
- Stored in ChromaDB
- In response to the user's question, extract the Top-K items with high similarity and pass them to LLM
This flow itself is textbook. However, when I actually tried it, the results were disappointing. For example, if you ask someone, ``Please tell me about the company regulations for expense reimbursement,'' the system will pull up an old, irrelevant manual on employee benefits, and the LLM will confidently give the wrong answer.
Why did this happen? The cause is poor chunk design.
This is because simply dividing sentences by the number of characters fragments the context of the sentence, and there are many cases where it is not correctly recognized as a ``group of meaning'' in vector space.
2. Hell of chunking and overlapping
The most important thing in RAG is how to beautifully cut out the "minimum unit of information (chunk)" to be passed to LLM.
At first I was cutting simple fixed lengths. However, with this, a sentence is cut off in the middle and the context is lost.
LangChain's `RecursiveCharacterTextSplitter` is excellent, but to make full use of it, you need to design an ``appropriate chunk_size'' and ``sufficient chunk_overlap.''
from langchain.text_splitter import RecursiveCharacterTextSplitter
# Failure example: Just fixed length segmentation (context is cut off and search accuracy decreases)
# Path to success: Setting with an awareness of cohesion of meaning
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=500, # Appropriate size considering tokens, not number of characters
chunk_overlap=100, # Overlap to preserve context
length_function=len,
is_separator_for_text_split=True
)
# Troubles when actually processing documents
docs = text_splitter.split_documents(raw_documents)
print(f"Generated {len(docs)} chunks.")
"Overlap" here means that the end of the previous chunk overlaps the beginning of the next chunk. This prevents the meaning from becoming fragmented along the dividing line. However, there are cases in which this alone is not sufficient. For example, if the data contains tabular data or bulleted lists, the structure will collapse if the data is simply divided based on the number of characters, and meta-information such as ``what the item is about'' will be lost during vector searches.
To solve this, I introduced the concept of ``Semantic Chunking''. However, this involves computational cost and implementation complexity. In the end, we decided to take the unreliable approach of adding titles and headings and then dividing the pages, reinforcing the context when searching.
3. Pitfalls of ChromaDB and Vector Spaces
The next hurdle we encountered was "data normalization" and "index behavior" in implementing a vector database (ChromaDB).ChromaDB itself is very easy to use, but the results change dramatically depending on what you are using as a search query.
For example, when a user enters "How do I pay my expenses?" and searches for it by converting it into a vector, the result is sometimes noisy responses.
The key here is Hybrid Search.
Combining pure vector similarity with keyword-based searches such as BM25 dramatically improves accuracy when proper nouns or specific product numbers are involved. Even when using LangChain, the stability of the system will depend on how this "fusion of keywords and vectors" is implemented.
Also, when using CheraDB, you should be careful about the distance calculation algorithm (Distance Metric).
In most cases, the default cosine similarity is fine, but depending on the type of embedding model, L2 (Euclidean distance) may be more appropriate. If you ignore this and create something that works for now, you will need to re-index the data later to improve accuracy, which will increase the number of man-hours.
4. LangChain vs LlamaIndex: Which one is better?
During development, I tried both LangChain and LlamaIndex. Both are great frameworks, but because they have different design philosophies, it can be difficult to use them in the wrong way.
The table below summarizes the differences in characteristics between the two that I have observed in my practice.
| Features | LangChain | LlamaIndex |
|---|---|---|
| Primary use | Orchestration of general-purpose LLM applications | Specialized in data structure indexing and advanced searching |
| What we're good at | Building chains, integrating with various tools, controlling complex agents | RAG-specific data connectors, advanced retrieval methods |
| Learning cost | High (high level of abstraction, it takes time to understand the internal structure) | Medium (fast learning as it focuses on RAG) |
| Flexibility | Very high (almost all LLM functions can be incorporated) | Strong optimization in specific domains (especially data search) |
In conclusion, we recommend using LangChain if you want to run a complex agent and LlamaIndex if you want to create a highly accurate RAG system in the shortest possible time. In the end, I adopted a hybrid configuration that incorporates LlamaIndex-like thinking into the data index construction part, while controlling the front-end logic with LangChain.
5. “Last push” by reranking
The most ``effective'' thing was the introduction of Reranking.
After extracting the top 10 results from a vector search, instead of passing them directly to LLM, we re-rank them using a dedicated model such as Cohere Rerank or BGE-Reranker. This has significantly improved the accuracy with which LLM extracts only the information needed to answer by eliminating noise.
By simply adding this step, the "hesitation" of the prompt disappears and the number of hallucinations (lying) is dramatically reduced. However, this is a trade-off at the expense of API cost and latency. How to balance this "cost vs. accuracy" is one of the most important decisions in building a practical RAG system.
Summary: RAG is not a “search technology” but a “data organization technique”
The biggest lesson I learned over the last three months is that 80% of RAG accuracy is determined by the data preprocessing and retrieval design, not by the LLM prompts or model performance.
- Manage chunks as meaningful chunks instead of just cutting them
- Consider hybrid search instead of relying only on vector search
- Incorporate reranking to increase the "purity" of information