# Juniper Technical Documentation RAG Engine

A fast, lightweight, and self-contained Retrieval-Augmented Generation (RAG) and search engine for Juniper technical documentation (KBs) and networking books (PDFs).

---

## 🚀 Features

- **Semantic Section-based Chunking**: Automatically parses Juniper KB metadata and chunks sections (Description, Symptoms, Cause, Solution/Workaround, and Fixed/Upgrade) separately.
- **Accurate PDF Extraction**: Uses `pdfplumber` with word boundary reconstruction (`x_tolerance=1.5`) to parse text from networking book PDFs without run-together words.
- **Ultra-Fast Vector Search**: Uses optimized **NumPy** matrix cosine similarity operations, delivering query retrieval times under **5 milliseconds**.
- **Resumable Indexing**: Remembers indexed files using a SQLite tracking table to ensure indexing is safe to pause and resume.
- **RAG Generation Endpoint**: Generates detailed, structured network engineering responses using Ollama LLMs by passing retrieved context chunks.
- **FastAPI REST API Service**: Exposes clean endpoints for `/health`, `/search`, and `/generate` (RAG synthesis) with autogenerated OpenAPI Swagger documentation.
- **Dockerized Deployment**: Fully containerized environment packaged with a pre-computed 170MB vector database for instant deployment.

---

## 📁 Repository Structure

```text
├── Dockerfile              # Docker deployment configuration
├── README.md               # Project documentation
├── app.py                  # FastAPI REST API web server
├── embed_docs.py           # Document ingestion, parsing, and embedding script
├── search_kb.py            # CLI query retrieval engine (numpy powered)
├── requirements.txt        # Python library dependencies
├── knowledge_base.db       # SQLite pre-computed vector database (~170MB)
└── .gitignore              # Configured to exclude raw datasets, cache and log files
```

---

## ⚙️ Configuration Variables

The RAG Engine can be configured using environment variables:

| Variable | Description | Default |
| --- | --- | --- |
| `OLLAMA_URL` | URL to Ollama embedding API | `http://localhost:11434/api/embed` |
| `OLLAMA_GENERATE_URL` | URL to Ollama text generation API | `http://localhost:11434/api/generate` |
| `EMBEDDING_MODEL` | Model used to generate embeddings | `qwen3-embedding:0.6b` |
| `DB_PATH` | Path to the SQLite database | `./knowledge_base.db` |
| `KB_JSON_PATH` | Path to Juniper KB clean dataset | `./juniper-kb/juniper_kb_data_clean.json` |
| `BOOKS_DIR` | Path to directory containing PDF books | `./network-books` |

---

## 💻 Local Setup & Usage

### Prerequisites
1. **Ollama**: Ensure [Ollama](https://ollama.com) is installed and running.
2. **Embedding Model**: Pull the embedding model:
   ```bash
   ollama pull qwen3-embedding:0.6b
   ```
3. **LLM Generation Model (Optional for RAG)**:
   ```bash
   ollama pull qwen2.5  # or llama3
   ```

### 1. Ingesting & Indexing Documents
If you need to re-index the raw KB files and PDFs, configure Ollama, populate `./juniper-kb/` and `./network-books/`, then run:
```bash
pip install -r requirements.txt
python3 embed_docs.py
```

### 2. Querying via Command Line
Search the index directly from the CLI:
```bash
python3 search_kb.py "BGP down QFX" --top-k 3
```
To view the full document text or page content of the matched results, append `--full`:
```bash
python3 search_kb.py "EVPN routing instances" --top-k 1 --full
```

### 3. Running the REST API Server
Start the FastAPI web server locally:
```bash
uvicorn app:app --host 0.0.0.0 --port 8000
```
Open your browser and navigate to `http://localhost:8000/docs` to interact with the OpenAPI Swagger interface.

---

## 🐳 Docker Deployment

The project is packaged with the pre-computed database `knowledge_base.db` so you can run it immediately without indexing.

### Build the Image
```bash
docker build -t rag-engine:latest .
```

### Run the Container

**For Linux hosts (accessing host Ollama):**
```bash
docker run -d \
  -p 8000:8000 \
  --name rag-engine \
  --add-host=host.docker.internal:host-gateway \
  rag-engine:latest
```

**For macOS/Windows hosts:**
```bash
docker run -d \
  -p 8000:8000 \
  --name rag-engine \
  rag-engine:latest
```

### Exposed API Endpoints

#### 1. System Health & Stats (`GET /health`)
```bash
curl http://localhost:8000/health
```

#### 2. Vector Search (`POST /search`)
```bash
curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{"query": "BGP down SRX", "top_k": 3}'
```

#### 3. RAG Answer Synthesis (`POST /generate`)
```bash
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"query": "Tại sao BGP down trên QFX lại ảnh hưởng SRX?", "top_k": 3, "llm_model": "qwen2.5"}'
```