From d1dd9cb24d0537e1abb4fa555356c3f8becb46a9 Mon Sep 17 00:00:00 2001
From: thinhle <thinhlv@vng.com.vn>
Date: Sun, 14 Jun 2026 23:15:23 +0700
Subject: [PATCH] Add README.md

---
 README.md | 135 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 135 insertions(+)
 create mode 100644 README.md

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..842ee25
--- /dev/null
+++ b/README.md
@@ -0,0 +1,135 @@
+# Juniper Technical Documentation RAG Engine
+
+A fast, lightweight, and self-contained Retrieval-Augmented Generation (RAG) and search engine for Juniper technical documentation (KBs) and networking books (PDFs).
+
+---
+
+## 🚀 Features
+
+- **Semantic Section-based Chunking**: Automatically parses Juniper KB metadata and chunks sections (Description, Symptoms, Cause, Solution/Workaround, and Fixed/Upgrade) separately.
+- **Accurate PDF Extraction**: Uses `pdfplumber` with word boundary reconstruction (`x_tolerance=1.5`) to parse text from networking book PDFs without run-together words.
+- **Ultra-Fast Vector Search**: Uses optimized **NumPy** matrix cosine similarity operations, delivering query retrieval times under **5 milliseconds**.
+- **Resumable Indexing**: Remembers indexed files using a SQLite tracking table to ensure indexing is safe to pause and resume.
+- **RAG Generation Endpoint**: Generates detailed, structured network engineering responses using Ollama LLMs by passing retrieved context chunks.
+- **FastAPI REST API Service**: Exposes clean endpoints for `/health`, `/search`, and `/generate` (RAG synthesis) with autogenerated OpenAPI Swagger documentation.
+- **Dockerized Deployment**: Fully containerized environment packaged with a pre-computed 170MB vector database for instant deployment.
+
+---
+
+## 📁 Repository Structure
+
+```text
+├── Dockerfile              # Docker deployment configuration
+├── README.md               # Project documentation
+├── app.py                  # FastAPI REST API web server
+├── embed_docs.py           # Document ingestion, parsing, and embedding script
+├── search_kb.py            # CLI query retrieval engine (numpy powered)
+├── requirements.txt        # Python library dependencies
+├── knowledge_base.db       # SQLite pre-computed vector database (~170MB)
+└── .gitignore              # Configured to exclude raw datasets, cache and log files
+```
+
+---
+
+## ⚙️ Configuration Variables
+
+The RAG Engine can be configured using environment variables:
+
+| Variable | Description | Default |
+| --- | --- | --- |
+| `OLLAMA_URL` | URL to Ollama embedding API | `http://localhost:11434/api/embed` |
+| `OLLAMA_GENERATE_URL` | URL to Ollama text generation API | `http://localhost:11434/api/generate` |
+| `EMBEDDING_MODEL` | Model used to generate embeddings | `qwen3-embedding:0.6b` |
+| `DB_PATH` | Path to the SQLite database | `./knowledge_base.db` |
+| `KB_JSON_PATH` | Path to Juniper KB clean dataset | `./juniper-kb/juniper_kb_data_clean.json` |
+| `BOOKS_DIR` | Path to directory containing PDF books | `./network-books` |
+
+---
+
+## 💻 Local Setup & Usage
+
+### Prerequisites
+1. **Ollama**: Ensure [Ollama](https://ollama.com) is installed and running.
+2. **Embedding Model**: Pull the embedding model:
+   ```bash
+   ollama pull qwen3-embedding:0.6b
+   ```
+3. **LLM Generation Model (Optional for RAG)**:
+   ```bash
+   ollama pull qwen2.5  # or llama3
+   ```
+
+### 1. Ingesting & Indexing Documents
+If you need to re-index the raw KB files and PDFs, configure Ollama, populate `./juniper-kb/` and `./network-books/`, then run:
+```bash
+pip install -r requirements.txt
+python3 embed_docs.py
+```
+
+### 2. Querying via Command Line
+Search the index directly from the CLI:
+```bash
+python3 search_kb.py "BGP down QFX" --top-k 3
+```
+To view the full document text or page content of the matched results, append `--full`:
+```bash
+python3 search_kb.py "EVPN routing instances" --top-k 1 --full
+```
+
+### 3. Running the REST API Server
+Start the FastAPI web server locally:
+```bash
+uvicorn app:app --host 0.0.0.0 --port 8000
+```
+Open your browser and navigate to `http://localhost:8000/docs` to interact with the OpenAPI Swagger interface.
+
+---
+
+## 🐳 Docker Deployment
+
+The project is packaged with the pre-computed database `knowledge_base.db` so you can run it immediately without indexing.
+
+### Build the Image
+```bash
+docker build -t rag-engine:latest .
+```
+
+### Run the Container
+
+**For Linux hosts (accessing host Ollama):**
+```bash
+docker run -d \
+  -p 8000:8000 \
+  --name rag-engine \
+  --add-host=host.docker.internal:host-gateway \
+  rag-engine:latest
+```
+
+**For macOS/Windows hosts:**
+```bash
+docker run -d \
+  -p 8000:8000 \
+  --name rag-engine \
+  rag-engine:latest
+```
+
+### Exposed API Endpoints
+
+#### 1. System Health & Stats (`GET /health`)
+```bash
+curl http://localhost:8000/health
+```
+
+#### 2. Vector Search (`POST /search`)
+```bash
+curl -X POST http://localhost:8000/search \
+  -H "Content-Type: application/json" \
+  -d '{"query": "BGP down SRX", "top_k": 3}'
+```
+
+#### 3. RAG Answer Synthesis (`POST /generate`)
+```bash
+curl -X POST http://localhost:8000/generate \
+  -H "Content-Type: application/json" \
+  -d '{"query": "Tại sao BGP down trên QFX lại ảnh hưởng SRX?", "top_k": 3, "llm_model": "qwen2.5"}'
+```