knowledge-base/README.md

# Juniper Technical Documentation RAG Engine

A fast, lightweight, and self-contained Retrieval-Augmented Generation (RAG) and search engine for Juniper technical documentation (KBs) and networking books (PDFs).

---

## 🚀 Features

- **Semantic Section-based Chunking**: Automatically parses Juniper KB metadata and chunks sections (Description, Symptoms, Cause, Solution/Workaround, and Fixed/Upgrade) separately.
- **Accurate PDF Extraction**: Uses `pdfplumber` with word boundary reconstruction (`x_tolerance=1.5`) to parse text from networking book PDFs without run-together words.
- **Ultra-Fast Vector Search**: Uses optimized **NumPy** matrix cosine similarity operations, delivering query retrieval times under **5 milliseconds**.
- **Resumable Indexing**: Remembers indexed files using a SQLite tracking table to ensure indexing is safe to pause and resume.
- **RAG Generation Endpoint**: Generates detailed, structured network engineering responses using Ollama LLMs by passing retrieved context chunks.
- **FastAPI REST API Service**: Exposes clean endpoints for `/health`, `/search`, and `/generate` (RAG synthesis) with autogenerated OpenAPI Swagger documentation.
- **Dockerized Deployment**: Fully containerized environment packaged with a pre-computed 170MB vector database for instant deployment.

---

## 📁 Repository Structure

```text
├── Dockerfile              # Docker deployment configuration
├── README.md               # Project documentation
├── app.py                  # FastAPI REST API web server
├── embed_docs.py           # Document ingestion, parsing, and embedding script
├── search_kb.py            # CLI query retrieval engine (numpy powered)
├── mcp_server.py           # Model Context Protocol (MCP) stdio server
├── rag_system_prompt.md    # System prompt guide for RAG Copilot integration
├── requirements.txt        # Python library dependencies
├── knowledge_base.db       # SQLite pre-computed vector database (~170MB)
└── .gitignore              # Configured to exclude raw datasets, cache and log files
```

---

## ⚙️ Configuration Variables

The RAG Engine can be configured using environment variables:

| Variable | Description | Default |
| --- | --- | --- |
| `OLLAMA_URL` | URL to Ollama embedding API | `http://localhost:11434/api/embed` |
| `OLLAMA_GENERATE_URL` | URL to Ollama text generation API | `http://localhost:11434/api/generate` |
| `EMBEDDING_MODEL` | Model used to generate embeddings | `qwen3-embedding:0.6b` |
| `DB_PATH` | Path to the SQLite database | `./knowledge_base.db` |
| `KB_JSON_PATH` | Path to Juniper KB clean dataset | `./juniper-kb/juniper_kb_data_clean.json` |
| `BOOKS_DIR` | Path to directory containing PDF books | `./network-books` |

---

## 💻 Local Setup & Usage

### Prerequisites
1. **Ollama**: Ensure [Ollama](https://ollama.com) is installed and running.
2. **Embedding Model**: Pull the embedding model:
   ```bash
   ollama pull qwen3-embedding:0.6b
   ```
3. **LLM Generation Model (Optional for RAG)**:
   ```bash
   ollama pull qwen2.5  # or llama3
   ```

### 1. Ingesting & Indexing Documents
If you need to re-index the raw KB files and PDFs, configure Ollama, populate `./juniper-kb/` and `./network-books/`, then run:
```bash
pip install -r requirements.txt
python3 embed_docs.py
```

### 2. Querying via Command Line
Search the index directly from the CLI:
```bash
python3 search_kb.py "BGP down QFX" --top-k 3
```
To view the full document text or page content of the matched results, append `--full`:
```bash
python3 search_kb.py "EVPN routing instances" --top-k 1 --full
```

### 3. Running the REST API Server
Start the FastAPI web server locally:
```bash
uvicorn app:app --host 0.0.0.0 --port 8000
```
Open your browser and navigate to `http://localhost:8000/docs` to interact with the OpenAPI Swagger interface.

---

## 🐳 Docker Deployment

The project is packaged with the pre-computed database `knowledge_base.db` so you can run it immediately without indexing.

### Build the Image
```bash
docker build -t rag-engine:latest .
```

### Run the Container

**For Linux hosts (accessing host Ollama):**
```bash
docker run -d \
  -p 8010:8000 \
  --name rag-engine \
  --add-host=host.docker.internal:host-gateway \
  rag-engine:latest
```

**For macOS/Windows hosts:**
```bash
docker run -d \
  -p 8010:8000 \
  --name rag-engine \
  rag-engine:latest
```

### Exposed API Endpoints

#### 1. System Health & Stats (`GET /health`)
```bash
curl http://localhost:8010/health
```

#### 2. Vector Search (`POST /search`)
```bash
curl -X POST http://localhost:8010/search \
  -H "Content-Type: application/json" \
  -d '{"query": "BGP down SRX", "top_k": 3}'
```

#### 3. RAG Answer Synthesis (`POST /generate`)
```bash
curl -X POST http://localhost:8010/generate \
  -H "Content-Type: application/json" \
  -d '{"query": "Tại sao BGP down trên QFX lại ảnh hưởng SRX?", "top_k": 3, "llm_model": "qwen2.5:0.5b"}'
```

---

## 🔌 Model Context Protocol (MCP) Server

You can expose the Juniper RAG Engine as tools to your AI Agent (like Claude Desktop) using the Model Context Protocol (MCP). The python script `mcp_server.py` implements the standard JSON-RPC stdio protocol.

### How it Works
The MCP server is a lightweight stdio subprocess. When the AI Agent starts, it spawns `mcp_server.py` which communicates with the running RAG Engine Docker container on `http://localhost:8010`.

### Configure your AI Agent Client

Add the following config block to your MCP client config file (e.g. `claude_desktop_config.json`):

**Linux / macOS / WSL Path**: `~/.config/Claude/claude_desktop_config.json`
**Windows Path**: `%APPDATA%\Claude\claude_desktop_config.json`

```json
{
  "mcpServers": {
    "juniper-rag": {
      "command": "python3",
      "args": ["/root/work/knowledge-base/mcp_server.py"],
      "env": {
        "RAG_API_URL": "http://localhost:8010"
      }
    }
  }
}
```

### Exposed Tools
Once connected, the AI Agent will have access to the following tools:
1. `search_juniper_kb`: Search technical documents and books for relevant snippets.
   - Parameters: `query` (string, required), `top_k` (integer), `source` ("kb" | "book").
2. `ask_juniper_rag`: Ask technical questions and get synthesized answers with citations.
   - Parameters: `query` (string, required), `top_k` (integer), `source` ("kb" | "book"), `llm_model` (string, e.g. "qwen2.5:0.5b").