> For the complete documentation index, see [llms.txt](https://docs.readyidc.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.readyidc.com/gpu-as-a-service/linux/docker/how-to-using-rag-with-anythingllm.md).

# How to using RAG with AnythingLLM

> Build a RAG (Retrieval-Augmented Generation) application on top of your vLLM endpoint. Chat with your own documents (PDF, DOCX, TXT, web pages). **Prerequisite:** A running vLLM endpoint (see the vLLM Docker Deployment Guide).

## 1. Overview & Architecture

[AnythingLLM](https://anythingllm.com/) handles document ingestion, chunking, embedding, vector storage, and retrieval automatically. It connects to your vLLM endpoint for both chat and (optionally) embeddings.

```
                  ┌─────────────────────────────────────┐
                  │           AnythingLLM                │
   Documents ───► │  (ingest → chunk → embed → store)    │
                  └──────┬───────────────────┬───────────┘
                         │                   │
              embeddings │                   │ chat (LLM)
                         ▼                   ▼
                  ┌──────────────┐    ┌──────────────┐
                  │  Embedding   │    │     vLLM     │
                  │   Server     │    │  (your LLM)  │
                  │  (vLLM)      │    │              │
                  └──────────────┘    └──────────────┘
                         │
                         ▼
                  ┌──────────────┐
                  │  Vector DB   │
                  │  (LanceDB)   │
                  └──────────────┘
```

**Three components:**

1. **LLM** — your existing vLLM (generates answers)
2. **Embedding model** — converts text to vectors (separate small model)
3. **Vector DB** — stores embeddings (LanceDB built-in, no separate container needed)

## 2. Embedding Models

### Built-in vs Dedicated

| Option                       | Pros                                | Cons                  |
| ---------------------------- | ----------------------------------- | --------------------- |
| **AnythingLLM built-in**     | Zero config, runs on CPU            | Slower, basic quality |
| **Dedicated vLLM embedding** | Fast, GPU-accelerated, high quality | Uses extra VRAM       |

For production or Thai documents, a **dedicated embedding server** is strongly recommended.

### Recommended Embedding Models

| Model                            | Use Case                             | VRAM     |
| -------------------------------- | ------------------------------------ | -------- |
| `BAAI/bge-m3`                    | Multilingual (incl. Thai), excellent | \~2-3 GB |
| `intfloat/multilingual-e5-large` | Multilingual                         | \~2-3 GB |
| `BAAI/bge-large-en-v1.5`         | English only, fast                   | \~1-2 GB |

> **`BAAI/bge-m3`** is the top choice for Thai or mixed Thai/English documents. Always match the embedding model to your document language.

## 3. Vector Database Options

| Vector DB              | When to use                                             |
| ---------------------- | ------------------------------------------------------- |
| **LanceDB** (built-in) | Default — single instance, simplest, no extra container |
| **Qdrant**             | Larger scale, shared storage, advanced filtering        |

LanceDB is sufficient for most deployments and requires zero setup.

## 4. Full Stack Deployment (Multi-GPU)

Runs **vLLM (chat) + vLLM (embeddings) + AnythingLLM** together. Chat on GPU 0, embeddings on GPU 1.

### `docker-compose.yml`

```yaml
services:
  # Main chat LLM
  vllm:
    image: vllm/vllm-openai:nightly       # :latest for RTX 3090
    container_name: vllm
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-32B-Instruct-AWQ
      --quantization awq
      --max-model-len 16384
      --gpu-memory-utilization 0.85
      --enforce-eager
      --api-key sk-llm-key
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

  # Embedding server (vLLM in embedding mode)
  vllm-embed:
    image: vllm/vllm-openai:nightly       # :latest for RTX 3090
    container_name: vllm-embed
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=1
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8001:8000"
    ipc: host
    command: >
      --model BAAI/bge-m3
      --task embed
      --gpu-memory-utilization 0.30
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['1']
              capabilities: [gpu]

  # AnythingLLM (RAG app + built-in LanceDB)
  anythingllm:
    image: mintplexlabs/anythingllm:latest
    container_name: anythingllm
    restart: unless-stopped
    depends_on:
      - vllm
      - vllm-embed
    ports:
      - "3001:3001"
    environment:
      - STORAGE_DIR=/app/server/storage

      # LLM (chat)
      - LLM_PROVIDER=generic-openai
      - GENERIC_OPEN_AI_BASE_PATH=http://vllm:8000/v1
      - GENERIC_OPEN_AI_API_KEY=sk-llm-key
      - GENERIC_OPEN_AI_MODEL_PREF=Qwen/Qwen2.5-32B-Instruct-AWQ
      - GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT=16384

      # Embeddings
      - EMBEDDING_ENGINE=generic-openai
      - EMBEDDING_BASE_PATH=http://vllm-embed:8000/v1
      - EMBEDDING_MODEL_PREF=BAAI/bge-m3
      - EMBEDDING_MODEL_MAX_CHUNK_LENGTH=8192
      - GENERIC_OPEN_AI_EMBEDDING_API_KEY=none

      # Vector DB
      - VECTOR_DB=lancedb
    volumes:
      - ./anythingllm-storage:/app/server/storage

volumes:
  anythingllm-storage:
```

### Launch

```bash
echo "HF_TOKEN=hf_your_token_here" > .env
docker compose up -d
docker compose logs -f
```

Wait for all three healthy:

* `vllm` → `Application startup complete`
* `vllm-embed` → `Application startup complete`
* `anythingllm` → `Primary server in HTTP mode listening on port 3001`

## 5. Single-GPU Deployment

Run chat + embedding on one GPU. The embedding model is small (\~2-3GB), so lower the chat model's memory utilization to leave room.

```yaml
services:
  vllm:
    image: vllm/vllm-openai:nightly
    container_name: vllm
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-7B-Instruct-AWQ
      --quantization awq
      --max-model-len 8192
      --gpu-memory-utilization 0.70      # leave room for embedding
      --enforce-eager
      --api-key sk-llm-key
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

  vllm-embed:
    image: vllm/vllm-openai:nightly
    container_name: vllm-embed
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0          # same GPU
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8001:8000"
    ipc: host
    command: >
      --model BAAI/bge-m3
      --task embed
      --gpu-memory-utilization 0.20      # small slice
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

  anythingllm:
    image: mintplexlabs/anythingllm:latest
    container_name: anythingllm
    restart: unless-stopped
    depends_on: [vllm, vllm-embed]
    ports:
      - "3001:3001"
    environment:
      - STORAGE_DIR=/app/server/storage
      - LLM_PROVIDER=generic-openai
      - GENERIC_OPEN_AI_BASE_PATH=http://vllm:8000/v1
      - GENERIC_OPEN_AI_API_KEY=sk-llm-key
      - GENERIC_OPEN_AI_MODEL_PREF=Qwen/Qwen2.5-7B-Instruct-AWQ
      - GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT=8192
      - EMBEDDING_ENGINE=generic-openai
      - EMBEDDING_BASE_PATH=http://vllm-embed:8000/v1
      - EMBEDDING_MODEL_PREF=BAAI/bge-m3
      - GENERIC_OPEN_AI_EMBEDDING_API_KEY=none
      - VECTOR_DB=lancedb
    volumes:
      - ./anythingllm-storage:/app/server/storage
```

> Combined `gpu-memory-utilization` (0.70 + 0.20 = 0.90) must stay under 1.0.

## 6. External Vector DB (Qdrant)

For larger scale or shared vector storage, add Qdrant:

```yaml
  qdrant:
    image: qdrant/qdrant:latest
    container_name: qdrant
    restart: unless-stopped
    ports:
      - "6333:6333"
    volumes:
      - ./qdrant-storage:/qdrant/storage
```

Then change AnythingLLM env:

```yaml
      - VECTOR_DB=qdrant
      - QDRANT_ENDPOINT=http://qdrant:6333
```

## 7. Using AnythingLLM

### First Launch

Open `http://<your-server-ip>:3001`

1. Create your admin account
2. Verify LLM + Embedding settings are pre-filled (from env vars)
3. Create a **Workspace**
4. Upload documents → auto-embedded and ready to chat

### Document Upload

* Supported: PDF, DOCX, TXT, MD, CSV, and web page URLs
* Documents are chunked, embedded, and stored automatically
* Each workspace has its own document set and chat history

### Verify Embedding Server

```bash
curl http://localhost:8001/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "BAAI/bge-m3",
    "input": "Hello world"
  }'
```

You should get an array of numbers (the embedding vector).

## 8. RAG Tuning & Troubleshooting

### Tuning

| Setting           | Where              | Effect                                        |
| ----------------- | ------------------ | --------------------------------------------- |
| Chunk size        | Workspace settings | Smaller = precise, larger = more context      |
| Top-K retrieval   | Workspace settings | How many chunks retrieved per query           |
| `--max-model-len` | vLLM chat          | Must fit retrieved chunks + question + answer |
| Embedding model   | `bge-m3` for Thai  | Match document language                       |

### Common Issues

#### AnythingLLM can't connect to vLLM

* Check containers are on the same Docker network (same compose file = automatic)
* Verify `GENERIC_OPEN_AI_BASE_PATH` uses the **service name** (`http://vllm:8000/v1`), not `localhost`
* Confirm the API key matches between vLLM `--api-key` and AnythingLLM env

#### Embeddings fail / documents won't process

* Verify embedding server: `curl http://localhost:8001/v1/embeddings ...`
* Check `EMBEDDING_BASE_PATH` points to `http://vllm-embed:8000/v1`
* Ensure embedding model loaded: `docker compose logs vllm-embed`

#### Poor answer quality

* Increase Top-K retrieval (more context)
* Use a larger chat model (32B vs 7B)
* For Thai docs, confirm `bge-m3` embedding (not English-only)
* Reduce chunk size for more precise retrieval

#### Out of memory on single GPU

* Lower chat `--gpu-memory-utilization`
* Use a smaller chat model
* Reduce `--max-model-len`


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.readyidc.com/gpu-as-a-service/linux/docker/how-to-using-rag-with-anythingllm.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.