For this project's 9,586 chunks, numpy array dot product + pickle BM25 is sufficient. Search delay is less than 100ms, and memory usage is about 29MB (embedding 14MB + BM25 index 7.1MB + tokenization corpus 7.6MB). But when you scale up:
| Number of chunks | Embedding size (384d) | dot product search time | problem |
|---|---|---|---|
| 10K | 14MB | <100ms | doesn't exist |
| 100K | 140MB | ~500ms | Memory pressure begins |
| 1M | 1.4GB | ~5 seconds | Memory limitations, latency tolerance |
| 10M | 14GB | impossibility | Not possible with numpy |
Minimum change approach. Improves performance while maintaining the current code structure:
| component | today | change |
|---|---|---|
| vector search | numpy dot product | FAISS IndexFlatIP (exact, but optimized) |
| BM25 | rank-bm25 pickle | Same (in-memory up to 50K possible) |
| metadata | JSONL file | SQLite (chunks.db) |
| Create embeddings | PyTorch SentenceTransformer | ONNX conversion (conversion script prepared: convert_to_onnx.py) |
Term Note:
# FAISS 적용 예시 (embed.py 변경 부분만)
import faiss
# 빌드 시
index = faiss.IndexFlatIP(384) # Inner Product = cosine (정규화 후)
index.add(embeddings)
faiss.write_index(index, "data/faiss.index")
# 검색 시
index = faiss.read_index("data/faiss.index")
scores, indices = index.search(query_embedding, top_k) # GPU도 가능
Introducing Approximate Search (ANN). Sacrificing a little bit of accuracy and gaining a lot of speed:
| component | Option A (self-hosted) | Option B (Managed) |
|---|---|---|
| vector search | FAISS IndexIVFFlat (cluster-based) |
Qdrant (Docker) |
| BM25 | Elasticsearch (Docker) | OpenSearch Serverless |
| metadata | PostgreSQL | Supabase (PostgreSQL + REST API) |
| Create Embeddings | Batch processing on GPU servers | OpenAI text-embedding-3-small API |
| Reranker | local bge-reranker | Cohere Rerank API |
# FAISS IVF 적용 (100K+ 청크에 효과적)
nlist = 100 # 클러스터 수 (sqrt(N) 정도)
quantizer = faiss.IndexFlatIP(384)
index = faiss.IndexIVFFlat(quantizer, 384, nlist, faiss.METRIC_INNER_PRODUCT)
index.train(embeddings) # 클러스터 학습
index.add(embeddings)
index.nprobe = 10 # 검색 시 탐색할 클러스터 수 (정확도↔속도 트레이드오프)
Dedicated Vector DB + Search Engine Cluster:
| component | options | characteristic |
|---|---|---|
| vector search | Pinecone | Fully managed, serverless, auto-scaling |
| Weaviate | Built-in hybrid search (BM25+Dense single query) | |
| Qdrant | Rust-based high-performance, self-hosting/cloud | |
| pgvector + PostgreSQL | Add vector search to existing RDB | |
| Keyword search | Elasticsearch cluster | BM25 + built-in morphological analysis (nori/ik) |
| Embedding | API service (OpenAI/Cohere) | No local GPU required, costs are charged per token |
| orchestration | LangChain/LlamaIndex | Search→Reranking→LLM chain automatic configuration |
The current paraphrase-multilingual-MiniLM-L12-v2 (384 dimensions) is lightweight but has lower precision than the latest models:
| model | dimension | size | characteristic | the right size |
|---|---|---|---|---|
| MiniLM-L12-v2 (current) | 384 | 120 MB | Run locally, 50+ languages | ~50K |
bge-m3 (BAAI) |
1024 | 2.2GB | Dense+Sparse+ColBERT 3-in-1, multilingual powerhouse | 50K~500K |
multilingual-e5-large-instruct |
1024 | 2.2GB | Optimized for directed queries | 50K~500K |
text-embedding-3-small (OpenAI) |
1536 | API | Low-cost API, excellent multilingual | 100K+ |
embed-multilingual-v3 (Cohere) |
1024 | API | 100+ languages, special search | 100K+ |
Area of influence when replaced:
embed.py + adjust batch sizeembeddings.npy Regenerate (change dimension → increase file size)search.py does not need to be changed (numpy dot product is dimensionless)| method | merit | disadvantage | suitable situation |
|---|---|---|---|
| Full Rebuild | Simple and consistent | As the scale increases, time increases | <50K chunks, once per day |
| incremental addition | Fast (processes only changes) | Complex delete/edit management | 50K~500K, real-time required |
| Incremental + Cyclic Rebuild | Real-time + quality | Two system management | 500K+, Production |
For this project, a full rebuild is sufficient in 9,586 chunks, as a full rebuild takes 3-5 minutes. However, beyond 100K chunks, incremental indexing becomes essential.
Currently, the entire system operates locally. When expanding this to an external service or sharing it with a team, organize what options are available at each step.
| options | explanation | suitable environment |
|---|---|---|
| File system watching | Directory change detection with watchdog/inotify → automatic reprocessing |
local, small |
| Git webhooks | Automatic rebuild from CI/CD when document repo is pushed | Technical documentation, IaC |
| Confluence/Notion API Synchronization | Detect changed pages with REST API → Incremental processing | corporate environment |
| S3 Events + Lambda | Chunking + embedding with Lambda when uploading documents to S3 | AWS environment, serverless |
| Unstructured.io | PDF/DOCX/PPTX/HTML → Automatic conversion to structured text | Various file formats |
| options | explanation | suitable environment |
|---|---|---|
| Direct implementation (current) | chunk.py: 512 characters, respects paragraph boundaries |
When domain custom control is required |
| LangChain TextSplitter | RecursiveCharacterTextSplitter: Markdown heading recognition, code block preservation |
Universal, quick start |
| LlamaIndex SentenceSplitter | Respect sentence boundaries, automatically extract metadata | universal |
| Unstructured.io | Automatic parsing of document structure followed by element-based chunking (separating title, body, table, and image captions) | unstructured document |
| Semantic Chunking | Divide at the point where meaning switches based on embedding similarity | long text without structure |
| options | expense | delay | suitable environment |
|---|---|---|---|
| Local SentenceTransformer (current) | free | Batch 3-5 minutes | Local development, <100K |
| Local ONNX (conversion script prepared) | free | Deployment ~2 minutes | Local, CPU optimized |
| GPU Server | instance cost | Deployment ~30 seconds | 100K+, frequent rebuilds |
| OpenAI Embedding API | $0.02/1M token | Real time | Production and management burden↓ |
| Cohere Embed API | $0.10/1M token | Real time | Multilingual Specialization |
| Amazon Bedrock Titan | $0.02/1M token | Real time | AWS environment |
| options | BM25 | Dense | hybrid | suitable environment |
|---|---|---|---|---|
| File-based (currently) | rank-bm25 pickle (2 files: bm25_index + tokenized_corpus) | numpy.npy | manual summation | Local, <50K |
| FAISS + Elasticsearch | Elasticsearch | FAISS | Separate summation logic | Self-hosting, 50K~1M |
| Weaviate | Built-in BM25 | Built-in HNSW | Automatically with alpha parameters |
All-in-one, 50K~10M |
| Qdrant | (separate ES required) | Built-in HNSW | FastEmbed integration | High-Performance Vector Search |
| Pinecone | (Separately required) | interior decoration | Sparse+Dense support | Fully managed, scale |
| pgvector + PostgreSQL | pg_trgm/FTS | pgvector | Sum with SQL query | Utilize existing RDB |
| options | expense | performance | suitable environment |
|---|---|---|---|
| bge-reranker-v2-m3 (current) | free | 40 candidates/500ms | Local, multilingual |
| Cohere Rerank | $2/1KSearch | 40 candidates/200ms | Production, API-based |
| Jina Reranker | Free ~ Paid | 40 candidates/300ms | Multilingual, API/Local |
| ColBERT v2 | free | Token unit matching | Long documents, precise matching |
| LLM-based re-ranking | token cost | Slow but extremely precise | Low-volume, high-value searches |
Effective data managed with RAG:
| data type | RAG goodness of fit | reason |
|---|---|---|
| Service Plan | height | Changes frequently, requires versioning, and is shared across teams. |
| API Documentation | very high | Accurate term matching required, differences by version |
| Onboarding Guide | height | Responding to various questions from new employees |
| Fault Response Manual | very high | Quick search is essential in emergency situations |
| Code Review Guidelines | middle | For the code itself, IDE search is better suited |
| proceedings | lowness | Unstructured, requires structuring after extracting only decisions |
| Daily Standup Notes | very low | Short-lived, TTL-based expiration handling |
┌──────────────────────────────────────────────────┐
│ 데이터 소스 │
│ Confluence │ Notion │ Google Docs │ Git (MD) │
└──────┬───────────┬────────────┬──────────────────┘
│ │ │
▼ ▼ ▼
┌──────────────────────────────────────────────────┐
│ 수집 + 정제 레이어 │
│ - API 동기화 (변경분 감지) │
│ - HTML→Markdown 변환 │
│ - 메타데이터 추출 (작성자, 수정일, 라벨, 버전) │
│ - 이전 버전 아카이브 (검색 대상에서 제외) │
│ - 구조 정규화 (헤딩 레벨 통일, 섹션 구분선 삽입) │
└──────────────────┬───────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────┐
│ 청킹 + 인덱싱 │
│ - 헤딩 기반 청킹 (## 단위로 분할) │
│ - 메타데이터 보존 (문서 제목, 스페이스, 태그) │
│ - 증분 인덱싱 (변경분만 재처리) │
│ - 하이브리드 인덱스 (BM25 + Dense) │
└──────────────────┬───────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────┐
│ 검색 + 활용 │
│ - Slack/Teams 봇: "결제 API 타임아웃 몇 초?" │
│ - AI Agent: 기획서 기반 코드 생성 │
│ - 온보딩 봇: 신규 입사자 질문 대응 │
│ - 장애 대응: 에러 메시지로 해결 방법 검색 │
└──────────────────────────────────────────────────┘
Although the project and corporate environment have the same structure, the operational requirements are different:
1. Access permissions (ACL)
Access rights are different for each document. HR documents should not be searchable by engineers.
→ Add access_group field to chunk metadata, filtering when searching
2. Audit Log
You need to keep track of who searched for what.
→ Logging search query + returned results + user ID
3. PII (Personal Information) Masking
Personal information included in the plan must not be exposed as a result of RAG.
→ PII detection before chunking + masking pipeline added
4. Multi-tenancy
Independent corpus for each team or shared corpus + dedicated corpus for each team structure.
→ Separate corpus by namespace, namespace filter when searching