The upper quality limit of RAG is determined by the quality of the corpus. No matter how sophisticated the embedding and search algorithm is, if the original data is not refined, the results will be bad. In this project, the reason the chunking pipeline can respect --- separators and paragraph boundaries is because the data has already been refined into those structures at the collection stage.
books/
├── 0. 자체 정리/ ← 가장 고품질: 1차 수집 → AI 재분류 → 주제별 MD
│ ├── 01_음양론/
│ ├── 02_오행론/
│ ├── ... (총 20개 주제 폴더 + 부록)
│ └── 20_실전사례/
├── 1. 나의 사주명리-심화편/ ← 고품질: 전자책 추출, 장/절 구조 유지
├── 2. 사주명리 실전 100구문/ ← 고품질: 전자책 추출
└── 99. 온라인 자료 (정리 X)/ ← 2차 수집 원본: PDF/TXT, 구조 불균일
Key observation: Tier 0 is the result of precise reclassification by category using AI of data primarily collected online. The original text was divided into topics (Yin-Yang Theory, Five Elements Theory, Gyeok-Guk Theory, etc.) and given a Markdown structure (--- dividing line, ## heading) to maximize chunking quality. Tier 99 is a second online collection that was carried out later and is not yet organized. Chunking quality is high at 0 to 2 and relatively low at 99. In other words, the better the RAG performance is for data that has gone through the following steps: collection → AI-based reclassification → structuring.
Rules enforced when collecting online materials for this project:
1. Source required:
모든 파일에 `출처: URL` 또는 `https://` 포함 필수.
없으면 자동 삭제:
grep -cE '(https?://|출처:)' "$f" → 0이면 rm
Documents without sources are unreliable and are therefore not included in the corpus.
2. --- Separate topics with dividers:
When there are multiple topics in one TXT file, clearly distinguish them with ---. This is a key device that prevents “chunks that cross topic boundaries” during the chunking stage.
# 좋은 예: 주제가 명확히 분리됨
갑목(甲木)은 양의 목기운으로, 큰 나무를 상징한다...
---
을목(乙木)은 음의 목기운으로, 풀이나 덩굴을 상징한다...
3. No AI-generated content:
When the content created by LLM is added to the corpus, the hallucination is reproduced. All documents must be original texts written by humans (academic papers, textbooks, expert articles).
4. 자동 수집 시 파일 크기 제한:
When the automatic collection agent (saju-research-collector) creates a new file, it is limited to 200 lines or less per file. This is due to a stability issue with the agent (failure when generating more than 400 lines), and is not a limitation of the entire corpus. The actual corpus normally includes files of 400 to 1,700 lines. However, if too many topics are mixed in one file, topic contamination may occur during chunking, so it is advisable to separate files by topic if possible.
The same principle applies when an IT company builds a service plan or knowledge base as a RAG corpus:
| principle | Saju Project | IT company application |
|---|---|---|
| structured markup | --- divider, ## heading |
Separation of Confluence headings, Notion blocks, and API docs by endpoint |
| Source tracking | URL, page number | Document ID, version, author, last modified date |
| Indicate whether AI is created | prohibition | Tagging (source: human vs source: ai-draft) |
| Redundancy Management | dedup.py cosine similarity | Document versioning + archive of previous versions |
Example of data collection pipeline (Confluence → RAG):
Confluence API → 페이지 추출
→ HTML → Markdown 변환 (pandoc / html2text)
→ 메타데이터 추출 (제목, 스페이스, 라벨, 최종수정일)
→ 빈 페이지 / 템플릿 필터링
→ 구조화: 헤딩 기준 섹션 분리
→ 이전 버전 대비 diff → 변경분만 재처리
→ chunks.jsonl에 추가
# chunk.py 핵심 파라미터
--chunk-size 512 # 최대 512자 (글자 단위, 토큰이 아니라 char)
--overlap 64 # 64자 오버랩으로 문맥 단절 방지
Overlap means: Including the last 64 characters of the previous chunk at the beginning of the next chunk. For example, if there is a chunk that ends with "...Gapmok has a strong personality.", the next chunk continues with "Gapmok has a strong personality. Therefore, leadership..." This ensures that sentences broken at chunk boundaries can be retrieved with complete context in either chunk.
Reason for choosing 512 characters:
Relationship with Embedding Model — Bugs discovered and lessons learned:
In the process of analyzing this project, a performance loss bug in dense embedding was discovered. The model architecture of paraphrase-multilingual-MiniLM-L12-v2 supports 512 tokens (max_position_embeddings: 512), but the sentence-transformers library defaults to 128 tokens, so only the first part of the chunk is embedded and the rest is truncated.
| item | value |
|---|---|
| Model architecture support | 512 tokens |
| Library defaults (bug, fixed) | 128 tokens |
| Korean 512 characters → actual number of tokens | ~337 tokens |
| 128 token limit reflects actual embedding | First ~200 characters (~39% of total) |
| 128 vs 512 embedding cosine similarity | 0.55 (information is significantly different) |
How to fix: model.max_seq_length = 512 Add one line and rebuild the index. However, since this model may have been trained based on 128 tokens, the degree of quality improvement when expanded to 512 needs to be confirmed with a benchmark.
Why the system worked anyway:
Lesson: Don't blindly trust the library's defaults. The range supported by the model architecture and library settings may differ. When building an embedding pipeline, you must check the number of tokens in the actual chunk with max_seq_length. This type of configuration mismatch is difficult to detect because it silently reduces performance without causing errors.
Reason for choosing 64-character overlap:
It simply doesn't truncate every 512 characters. Hierarchical partitioning that respects semantic units:
Level 1: --- 구분선으로 섹션 분리 (atomic boundary, 절대 넘지 않음)
Level 2: 빈 줄(\n\n)로 단락 분리
Level 3: 문장 단위 분할 (단락이 512자 초과 시)
Core code (chunk.py:116-137):
def _split_into_paragraphs(text: str) -> list[str]:
# --- 구분선이 있으면 섹션별로 분리, 각 섹션을 atomic 단락으로
if _SECTION_SEPARATOR.search(text):
sections = _SECTION_SEPARATOR.split(text)
paragraphs = []
for section in sections:
section = section.strip()
if not section:
continue
paragraphs.append(section)
return paragraphs
# 없으면 빈 줄로 분리
parts = re.split(r"\n{2,}", text)
return [p.strip() for p in parts if p.strip()]
Why --- separator lines are important:
------ as an atomic boundary blocks topic contamination at the source.Same structure in IT services:
The API document (Swagger/OpenAPI) has natural boundaries for each endpoint, and the Confluence page is divided into sections ## 헤딩. For such documents, use the heading (##) as the atomic boundary instead of ---. For example, if GET /api/users and POST /api/users are mixed in the same chunk, it causes confusion during search, but if you chunk by endpoint, you will get accurate results.
Prerequisites for this to work:
As explained in Chapter 4, the data must already contain the --- delimiter during the collection phase. This strategy does not work for unstructured documents without dividers (scanned PDFs, legacy Word files, etc.) and requires additional preprocessing (LLM-based section detection, rule-based heading extraction).
When a paragraph exceeds 512 characters, it is divided into sentences, taking Korean final endings into consideration:
sentences = re.split(
r"(?<=[.!?。！？\n])\s*|(?<=[다요음함임됨것]\.)\s+",
para,
)
.!?。！？ Split from behind (universal)다., 요., 음., 함., etc. Split after Korean final ending + perioddef _detect_lang(text: str) -> str:
"""텍스트 첫 2000자의 문자 유형 비율로 언어 판별"""
# 한글 음절(AC00-D7AF) > 20% → ko
# CJK 통합(4E00-9FFF) > 20% → zh
# ASCII 알파벳이 대다수 → en
Language tags are stored in chunk metadata, which determines tokenizer selection during search and BM25 query expansion direction.