Dense Embedding is a method of converting the meaning of a text into a numeric vector so that "if the meaning is similar, the vector is close," and the BM25 index is a method of scoring based on the frequency of occurrence and rarity of the word itself included in the text. The former can find “synonyms or other expressions,” while the latter can find “documents with exactly that word.”
Reason for selection:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")
embeddings = model.encode(
texts,
batch_size=64,
normalize_embeddings=True, # L2 정규화 → 코사인 유사도 = 내적
)
Why L2 regularization is key:
@)scores = embeddings @ query_embedding.T # (9586,) 벡터, <100ms
@ operator above.Accurate keyword matching is weak through dense embedding alone. I searched for "甲木" and the semantically similar word "yang energy" came up. In domains where accurate term matching is important, BM25 is an essential complement.
# 한국어: Kiwi 형태소 분석기 (명사/동사/형용사만 추출)
def tokenize_ko(text: str) -> list[str]:
kiwi = Kiwi()
tokens = []
for token in kiwi.tokenize(text):
if token.tag.startswith(("NN", "VV", "VA", "SL", "SH", "SN")):
min_len = 1 if token.tag.startswith("NN") else 2
if len(token.form) >= min_len:
tokens.append(token.form)
return tokens
# 중국어: jieba 분절 (불용어 제거)
def tokenize_zh(text: str) -> list[str]:
return [w.strip() for w in jieba.cut(text)
if w.strip() and w not in _ZH_STOP]
# 영어: regex + stopwords (100개)
def tokenize_en(text: str) -> list[str]:
words = re.findall(r"[a-zA-Z]+", text.lower())
return [w for w in words if w not in _ENGLISH_STOP and len(w) > 2]
Reason for allowing Korean one-letter nouns:
One-letter nouns such as the Five Elements (Fri/Wed/Thu/Tues/Sat) and Cheongan (Gap/Eul/Byeong/Jeong) are the key terms of the domain. In general NLP, one-letter nouns are removed as noise, but here only nouns are allowed under the min_len = 1 if NN condition.
Same problem in IT services:
Even in the IT domain, short abbreviations are often key terms: CI, CD, DB, VM, LB, DR, QA, etc. If you apply the len(w) > 2 filter in the English tokenizer, all these two-letter abbreviations will be removed. Depending on your domain, you may need to adjust the tokenizer's minimum length limit or add important abbreviations to the stop word exception list.
JSONL (JSON Lines) is a format that stores one JSON object per line, and can be streamed line by line without loading the entire thing into memory, making it suitable for handling large amounts of chunk data. A regular JSON array ([{...}, {...}]) requires parsing the entire file, but JSONL is free to read only specific lines or append them to the end.
data/
├── chunks.jsonl # 원본 청크 (chunk_id, source, page, lang, text)
├── chunks_meta.jsonl # 검색 결과 반환용 메타+텍스트
├── embeddings.npy # Dense 벡터 (9586 × 384, float32)
├── bm25_index.pkl # BM25Okapi 직렬화
├── tokenized_corpus.pkl # 사전 토큰화 코퍼스 (검색 시 재토큰화 방지)
└── .last_build_count # 마지막 빌드 시 청크 수 (변경 감지용)
Total rebuild time: approximately 3-5 minutes for 9,586 chunks (M1 Mac local)
쿼리: "갑목 인월 용신 조후"
│
┌──────────┼──────────┐
▼ │ ▼
┌──────────┐ │ ┌──────────┐
│ BM25 │ │ │ Dense │
│ top-K │ │ │ top-K │
└────┬─────┘ │ └────┬─────┘
│ │ │
└─────┬─────┘─────────┘
│
합집합 (K=max(20, top_k×4), 기본 top_k=5 시 최대 40개)
│
▼
┌─────────────────────┐
│ Stage 2: Reranker │
│ bge-reranker-v2-m3 │
│ (Cross-Encoder) │
└──────────┬──────────┘
│
top-K 결과 반환
BM25 and Dense search are fundamentally different methods, and which one is more effective will depend on the nature of the query.** When combined at a fixed ratio (e.g. BM25 40% + Dense 60%), you will get suboptimal results for certain types of queries.
Let’s look at the problem specifically:
| query type | What BM25 does well | What Dense does well | The problem of fixed ratios |
|---|---|---|---|
"窮通寶鑑 甲木 寅月" (Chinese character keyword) |
Find documents with exactly these Chinese characters | Import semantically similar Korean interpretations (noise) | If the proportion of Dense is high, incorrect Korean interpretations will be ranked high. |
"갑목 일간이 봄에 태어났을 때 보충해야 할 기운" (descriptive) |
Fragmentary results by matching only the words “Gapmok” and “Spring” | Understand the meaning of the entire sentence and find an answer such as "When throat energy is strong, control it with gold." | If the proportion of BM25 is high, documents containing low-relevance “black” items will be ranked high. |
"도화살" (short jargon) |
Find a document with the exact word "do arrow" | The “Dohwasal” embedding also matches general meanings such as “love” and “attraction.” | If you lean towards one side, you lose accuracy or variety. |
Adaptive weights for this project:
def _adaptive_weights(query, lang):
hanja_ratio = (쿼리 내 한자 비율)
domain_term_count = (도메인 사전에 매칭된 용어 수)
if hanja_ratio > 0.3: # 한자 쿼리
return 0.6, 0.4 # BM25↑: 한자는 정확한 글자 매칭이 핵심
elif len(query) <= 8 and domain_term_count >= 1: # 짧은 전문용어
return 0.5, 0.5 # 균등: 양쪽 모두 필요
elif len(query) > 20 and domain_term_count == 0: # 긴 서술형
return 0.25, 0.75 # Dense↑: 의미 파악이 핵심
else:
return 0.4, 0.6 # 기본: Dense 약간 우세
Production Implications:
Adaptive weights are not only necessary for Saju Myeongrihak. for example:
GET /api/v2/users → BM25 enhancements, “How to get a list of users” → Dense enhancementsBy analyzing the nature of the query at runtime and automatically adjusting the weight, it is possible to respond to a variety of query patterns.
The most core technique of this project. Devices that allow Korean queries to find Chinese sources:
# 143개 키(209쌍)의 한↔漢 매핑 테이블 (일부)
_KOREAN_TO_HANJA = {
"갑목": ["甲木"],
"용신": ["用神"],
"도화살": ["桃花煞", "桃花"],
"합충형파해": ["合沖刑破害", "合冲刑破害"], # 정체자+간체자 모두 포함
"인신충": ["寅申沖", "寅申冲"],
...
}
Query Processing Process:
입력 쿼리: "갑목 인월 용신"
↓ Kiwi 토큰화
토큰: ["갑목", "인", "월", "용신"]
↓ 복합 도메인 용어 추출 (형태소 분석 우회)
추가: ["갑목", "용신"] # 이미 포함됨 확인
↓ 한→漢 확장
확장: ["갑목", "인", "월", "용신", "甲木", "寅", "用神"]
↓ BM25 검색 (확장된 토큰으로)
Same pattern in IT services:
Even in IT environments, terms with the same meaning are used interchangeably in Korean/English/abbreviation. Without query expansion tables, you won't find documents written as "Kubernetes" or "K8s" when you search for "Kubernetes":
_IT_TERM_EXPANSION = {
"쿠버네티스": ["Kubernetes", "K8s", "k8s"],
"도커": ["Docker", "docker"],
"로드밸런서": ["Load Balancer", "LB", "L4", "L7"],
"배포": ["deploy", "deployment", "릴리즈", "release"],
"장애": ["incident", "outage", "장애 대응", "postmortem"],
"인증": ["authentication", "auth", "OAuth", "SSO"],
"캐시": ["cache", "Redis", "Memcached", "CDN"],
"모니터링": ["monitoring", "Grafana", "Prometheus", "옵저버빌리티", "observability"],
...
}
With this table, you can also find English documents where the Korean query “deployment rollback method” is written as “deployment rollback procedure.” Building a domain dictionary is a one-time cost, but the effect of improving search recall is permanent.
Why you need complex domain term extraction:
There are cases where Kiwis separate “Daeun” into “Dae” + “Un” or incorrectly segment “Gungtongbogam”. Compound words registered in the dictionary are matched and preserved first before morphological analysis:
def _extract_compound_terms(query: str) -> list[str]:
# 길이 내림차순 정렬 → 긴 용어 우선 매칭 (합충형파해 > 합충 > 합)
sorted_terms = sorted(_KOREAN_TO_HANJA.keys(), key=len, reverse=True)
found = []
for term in sorted_terms:
if len(term) >= 2 and term in query:
found.append(term)
return found
If multiple chunks are ranked high from the same source, the answers will be biased. Attenuate by 5% starting from the second chunk of the same source:
# 같은 소스 2번째부터 5%씩 감쇠 (최대 15%)
penalty = min(seen * 0.05, 0.15)
adjusted = original_score * (1.0 - penalty)
Example: Even if 5 chunks from 梁湘润_명리전승반필기.pdf are in the top 10, the diversity penalty allows other source chunks to come in and get multiple perspectives.
Same problem in IT services:
When you search for “payment timeout,” if the top 10 chunks from the payments team’s Confluence space dominate the top, you miss out on documents from other perspectives written by the infrastructure or QA teams. By applying the source diversity penalty, you can obtain balanced results such as 3 payment team documents + 2 infrastructure team failure response documents + 1 QA test scenario.