2026-06-14 23:15:23 +07:00

4.7 KiB

Raw Blame History

Juniper Technical Documentation RAG Engine

A fast, lightweight, and self-contained Retrieval-Augmented Generation (RAG) and search engine for Juniper technical documentation (KBs) and networking books (PDFs).

🚀 Features

Semantic Section-based Chunking: Automatically parses Juniper KB metadata and chunks sections (Description, Symptoms, Cause, Solution/Workaround, and Fixed/Upgrade) separately.
Accurate PDF Extraction: Uses pdfplumber with word boundary reconstruction (x_tolerance=1.5) to parse text from networking book PDFs without run-together words.
Ultra-Fast Vector Search: Uses optimized NumPy matrix cosine similarity operations, delivering query retrieval times under 5 milliseconds.
Resumable Indexing: Remembers indexed files using a SQLite tracking table to ensure indexing is safe to pause and resume.
RAG Generation Endpoint: Generates detailed, structured network engineering responses using Ollama LLMs by passing retrieved context chunks.
FastAPI REST API Service: Exposes clean endpoints for /health, /search, and /generate (RAG synthesis) with autogenerated OpenAPI Swagger documentation.
Dockerized Deployment: Fully containerized environment packaged with a pre-computed 170MB vector database for instant deployment.

📁 Repository Structure

├── Dockerfile              # Docker deployment configuration
├── README.md               # Project documentation
├── app.py                  # FastAPI REST API web server
├── embed_docs.py           # Document ingestion, parsing, and embedding script
├── search_kb.py            # CLI query retrieval engine (numpy powered)
├── requirements.txt        # Python library dependencies
├── knowledge_base.db       # SQLite pre-computed vector database (~170MB)
└── .gitignore              # Configured to exclude raw datasets, cache and log files

⚙️ Configuration Variables

The RAG Engine can be configured using environment variables:

Variable	Description	Default
`OLLAMA_URL`	URL to Ollama embedding API	`http://localhost:11434/api/embed`
`OLLAMA_GENERATE_URL`	URL to Ollama text generation API	`http://localhost:11434/api/generate`
`EMBEDDING_MODEL`	Model used to generate embeddings	`qwen3-embedding:0.6b`
`DB_PATH`	Path to the SQLite database	`./knowledge_base.db`
`KB_JSON_PATH`	Path to Juniper KB clean dataset	`./juniper-kb/juniper_kb_data_clean.json`
`BOOKS_DIR`	Path to directory containing PDF books	`./network-books`

💻 Local Setup & Usage

Prerequisites

Ollama: Ensure Ollama is installed and running.
Embedding Model: Pull the embedding model:
```
ollama pull qwen3-embedding:0.6b
```
LLM Generation Model (Optional for RAG):
```
ollama pull qwen2.5  # or llama3
```

1. Ingesting & Indexing Documents

If you need to re-index the raw KB files and PDFs, configure Ollama, populate ./juniper-kb/ and ./network-books/, then run:

pip install -r requirements.txt
python3 embed_docs.py

2. Querying via Command Line

Search the index directly from the CLI:

python3 search_kb.py "BGP down QFX" --top-k 3

To view the full document text or page content of the matched results, append --full:

python3 search_kb.py "EVPN routing instances" --top-k 1 --full

3. Running the REST API Server

Start the FastAPI web server locally:

uvicorn app:app --host 0.0.0.0 --port 8000

Open your browser and navigate to http://localhost:8000/docs to interact with the OpenAPI Swagger interface.

🐳 Docker Deployment

The project is packaged with the pre-computed database knowledge_base.db so you can run it immediately without indexing.

Build the Image

docker build -t rag-engine:latest .

Run the Container

For Linux hosts (accessing host Ollama):

docker run -d \
  -p 8000:8000 \
  --name rag-engine \
  --add-host=host.docker.internal:host-gateway \
  rag-engine:latest

For macOS/Windows hosts:

docker run -d \
  -p 8000:8000 \
  --name rag-engine \
  rag-engine:latest

Exposed API Endpoints

1. System Health & Stats (`GET /health`)

curl http://localhost:8000/health

2. Vector Search (`POST /search`)

curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{"query": "BGP down SRX", "top_k": 3}'

3. RAG Answer Synthesis (`POST /generate`)

curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"query": "Tại sao BGP down trên QFX lại ảnh hưởng SRX?", "top_k": 3, "llm_model": "qwen2.5"}'

4.7 KiB Raw Blame History