This methodology document analyzes a toy project built to practice implementing and using a multilingual RAG pipeline.
Saju-myeongri (Four Pillars of Destiny) was chosen as the topic because it combines statistical analysis with a natural mix of Korean, Chinese, and English source texts. This allows for experiencing all the challenges of multilingual chunking, embedding, and retrieval in a single project.
All experiments and operations were performed in a local environment (M1 Mac), creating a self-contained pipeline without external service dependencies.
This series consists of this overview and four detailed parts.
| Document | Content | Key Question |
|---|---|---|
| This Document (Overview) | Terminology, background, overall architecture, and design principles. | What's RAG, and why was it built this way? |
| Part 1: Data Collection, Cleaning, and Chunking | Data source layers, cleaning rules, chunking strategies, and embedding model bug discovery. | How do you turn raw data into searchable units? |
| Part 2: Embedding, Indexing, and Retrieval | Dense/BM25 dual indexing, hybrid search, adaptive weights, and cross-lingual query expansion. | How is the index created, and how does search work? |
| Part 3: Corpus Management, Auto-Enrichment, and Agents | Corpus degradation types, update strategies, auto-enrichment cycles, and LLM agent utilization. | How do you keep the corpus fresh, and how do LLMs use it? |
| Part 4: Scaling, Production, and IT Application | Scaling from 10K to 1M+, step-by-step implementation options, and IT Knowledge Base application. | How do you apply this to real-world services? |
Here are the core terms used throughout this series. You can skip this if you're already familiar with RAG and embedding concepts.
A technique that lets an LLM (Large Language Model) search for relevant documents before generating an answer. LLMs alone can "hallucinate" inaccurate knowledge not present in their training data. RAG mitigates this by forcing the model to answer based on retrieved source texts.
Standard LLM: User Question → LLM → Answer (Relies on training data, risk of hallucination)
RAG: User Question → [Search] → Relevant Documents → LLM + Docs → Answer (Grounded)
The entire collection of documents that RAG uses as a search target. In this project, 593 PDF, Markdown, and text files related to Saju-myeongri make up the corpus. The quality and scope of the corpus determine the upper limit of RAG performance. No matter how good the search algorithm is, it can't find information that isn't there.
The process of splitting long documents into small units (chunks) suitable for search. Searching an entire book at once makes it hard to pinpoint relevant parts. Instead, we cut it into meaningful units of about 300 to 500 characters, making each an independent search target. If chunks are too large, search precision drops. If they're too small, context is lost.
Converting text into a numerical vector (an array of hundreds of decimal numbers). A sentence like "The character of Gap-mok is straight and strong" becomes an array of 384 numbers like [0.12, -0.34, 0.56, ...]. Sentences with similar meanings get similar vectors. By calculating the distance between vectors, we can numerically compare semantic similarity. This is the principle of "semantic search."
A traditional keyword-based search algorithm. It scores documents based on how often a query word appears (TF) and how rare that word is (IDF). Searching for "甲木" finds documents containing exactly those characters. While embedding finds by "meaning," BM25 finds by "characters." Combining them (hybrid search) covers each other's weaknesses.
A method that combines results from BM25 (keyword matching) and Dense Embedding (semantic search) to determine the final ranking. For example, when searching for "Gap-mok personality":
A second search stage that re-ranks candidates using a more precise model after an initial search. The first stage (BM25+Dense) is fast but broad. The second stage (Cross-Encoder) is slower but highly accurate. It takes the query and document together to judge relevance directly. Since it's slow, it's only applied to the top 20 to 40 candidates from the first stage.
Calculating the distance (similarity) between embedding vectors to find the closest ones. This project calculates the dot product between the query vector and 9,586 chunk vectors. At larger scales, dedicated vector databases like FAISS or Pinecone are used for speed through Approximate Nearest Neighbor (ANN) search.
A setting that determines how many top results to return. If top-k is 5, only the 5 most relevant chunks are retrieved. If K is too small, you might miss relevant info. If it's too large, noise can confuse the LLM. This project uses top-k 5 for single CLI searches and top-k 3 for RAG batch searches.
A tool that splits text into searchable tokens (words or morphemes). Korean needs morphological analysis to split "사주명리학을" into "사주/명리/학." Chinese lacks spaces, so "四柱八字" must be segmented into "四柱/八字." English is mostly handled by splitting at spaces. Using the right tokenizer for each language is vital for BM25 search quality.
LLMs often hallucinate when dealing with systematic but niche professional knowledge like Saju-myeongri. If you ask, "What's the Yong-sin when a Gap-mok person is born in the month of In?", it might confidently give a plausible but wrong answer. In fields requiring precise interpretation based on classical texts, LLMs alone aren't reliable.
This isn't just a Saju problem. The same issue occurs in any domain not sufficiently covered in LLM training data, such as internal service specs, the latest APIs of specific frameworks, legal precedents, or medical guidelines.
Saju-myeongri is inherently a multilingual domain:
| Language | Role | Examples |
|---|---|---|
| Chinese | Original classical texts | 滴天髓, 窮通寶鑑, 子平真詮 |
| Korean | Modern interpretations, textbooks, cases | Theory of structures, Yong-sin theory, 100 practical phrases |
| English | Academic papers, ML research | BaZi career prediction, AI character simulation |
A single query like "Gap-mok In-wol Yong-sin" needs to search Korean textbooks, Chinese classics like 窮通寶鑑, and English papers simultaneously to form a complete answer. This is structurally identical to an IT company needing to search English tech docs, Korean specs, and Japanese partner API docs together.
┌─────────────────────────────────────────────────────────┐
│ Step 0: Data Collection & Cleaning │
│ Raw PDF/MD/TXT → Structured Markup (--- separators) │
│ Dedup → Source Verification → Place in books/ │
│ → Details: Part 1 │
└───────────────┬─────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Step 1: Chunking (chunk.py) │
│ PDF→PyMuPDF / TXT→Direct Read │
│ Lang Detection → Respect Paragraphs → 512-char chunks │
│ Output: chunks.jsonl (9,586 chunks, 4.16MB) │
│ → Details: Part 1 │
└───────────────┬─────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Step 2: Indexing (embed.py) │
│ │
│ ┌──────────────────┐ ┌──────────────────────┐ │
│ │ Dense Embedding │ │ BM25 Sparse Index │ │
│ │ MiniLM-L12-v2 │ │ Korean: Kiwi POS │ │
│ │ 384d, L2 Norm │ │ Chinese: jieba │ │
│ │ → embeddings.npy │ │ English: regex+stop │ │
│ │ (14MB) │ │ → bm25_index.pkl │ │
│ └──────────────────┘ └──────────────────────┘ │
│ → Details: Part 2 │
└───────────────┬─────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Step 3: Hybrid Search (search.py) │
│ │
│ Query → Lang Detect → Cross-lingual Expansion │
│ → BM25 top-K ∪ Dense top-K candidates │
│ → (Optional) bge-reranker-v2-m3 │
│ → Source Diversity Penalty → top-K return │
│ → Details: Part 2 │
└───────────────┬─────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Step 4: LLM Agent Use + Corpus Management │
│ │
│ Claude Code Skill Orchestrator │
│ → 5 Sub-agents in parallel │
│ → Each agent uses search.py for RAG │
│ → Generate reports based on source citations │
│ → Auto Gap Analysis → Collect → Rebuild Cycle │
│ → Details: Part 3 │
└─────────────────────────────────────────────────────────┘
| Item | Value |
|---|---|
| Source Documents | 593 (PDF + MD + TXT) |
| Total Chunks | 9,586 |
| Total Text Volume | ~4.16M characters (chunks.jsonl size 10.4MB) |
| Avg Chunk Size | 434 characters |
| Dense Embeddings | 9,586 × 384 float32 = 14MB |
| BM25 Index | 14.7MB (bm25_index.pkl 7.1MB + tokenized_corpus.pkl 7.6MB) |
| Lang Distribution | Korean 80.5% / Chinese 14.2% / English 5.3% |
These principles, validated in this project, apply regardless of domain or scale.
Always use BM25 (keyword) and Dense (meaning) together in professional domains. Dense alone is risky for terms requiring exact matching, like proper nouns, acronyms, code names, or API endpoints. Dense might return "semantically similar" but wrong results, while BM25 can't find anything without exact keywords. You need both.
Building a table for synonyms, acronyms, and multilingual mappings dramatically improves search recall. This project's mapping of 143 keys (209 pairs) between Korean and Hanja is an example. For legal domains, use "tort, illegal act, 위법행위." For IT, use "K8s, Kubernetes, 쿠버네티스." For medical, use "diabetes mellitus, DM, 당뇨."
Fixed-size chunking is simple, but cutting through semantic units hurts search quality. Using document structures like --- separators or Markdown headings (##) as atomic boundaries is effective. For this to work, data must already be structured during collection. Chunking quality reflects data cleaning quality.
A two-stage pipeline, where Stage 1 uses BM25+Dense for broad candidates and Stage 2 uses a Cross-Encoder for precise ranking, significantly boosts search quality. The cost of re-ranking 40 candidates is negligible, but the precision gain is noticeable.
There's no such thing as a static corpus. You must automate the "detect gap → collect/update → rebuild → verify" cycle to maintain or improve quality over time. Manual management will fail as scale increases.
When passing search results to an LLM, send the original text without summarizing it. Information is lost or distorted during summarization. Letting the LLM see the original text ensures citation accuracy. This is why sub-agents in this project are forced to "cite full original text."
| Component | This Project's Implementation | Production Scaling Options | Details |
|---|---|---|---|
| Data Collection | Manual + saju-research-collector agent | Confluence API, Git webhooks, S3 events, Unstructured.io | Part 1 |
| Data Cleaning | --- separators, mandatory source, no AI content |
HTML to MD, PII masking, version control | Part 1 |
| Chunking | 512 chars, 64 overlap, respect --- |
LangChain TextSplitter, Semantic Chunking | Part 1 |
| Embedding | MiniLM-L12-v2 (384d, local) | bge-m3, OpenAI embedding API, Cohere embed | Part 2 |
| BM25 | rank-bm25 (in-memory pickle) | Elasticsearch, OpenSearch, Weaviate built-in | Part 2 |
| Vector Search | numpy dot product (14MB npy) | FAISS, Pinecone, Qdrant, pgvector | Part 2 |
| Reranker | bge-reranker-v2-m3 (local) | Cohere Rerank, Jina Reranker, LLM reranking | Part 2 |
| Query Expansion | 143 keys (209 pairs) Ko-Hanja dict | WordNet, domain ontology, LLM-based expansion | Part 2 |
| Corpus Management | Auto gap-collect-rebuild cycle | Incremental sync, TTL expiration, user feedback loop | Part 3 |
| Agent | Claude Code Skills (5+5 parallel) | LangChain Agent, LlamaIndex Agent | Part 3 |
| Scaling | numpy + pickle (9.6K chunks) | FAISS → Qdrant → Pinecone | Part 4 |
| IT Application | — | Spec RAG, KB bot, incident response | Part 4 |
The core value of this system is that it's a "small but complete" RAG pipeline. With 593 documents, 9,586 chunks, and 4 Python scripts (chunk.py, embed.py, search.py, dedup.py), it includes multilingual hybrid search, re-ranking, adaptive weights, and auto-enrichment.
To scale up, replace each component with the production options in the table. The design principles and pipeline structure remain the same regardless of scale. It's the infrastructure that changes, not the architecture.