How to using RAG with AnythingLLM

Build a RAG (Retrieval-Augmented Generation) application on top of your vLLM endpoint. Chat with your own documents (PDF, DOCX, TXT, web pages). Prerequisite: A running vLLM endpoint (see the vLLM Docker Deployment Guide).

1. Overview & Architecture

AnythingLLM handles document ingestion, chunking, embedding, vector storage, and retrieval automatically. It connects to your vLLM endpoint for both chat and (optionally) embeddings.

                  ┌─────────────────────────────────────┐
                  │           AnythingLLM                │
   Documents ───► │  (ingest → chunk → embed → store)    │
                  └──────┬───────────────────┬───────────┘
                         │                   │
              embeddings │                   │ chat (LLM)
                         ▼                   ▼
                  ┌──────────────┐    ┌──────────────┐
                  │  Embedding   │    │     vLLM     │
                  │   Server     │    │  (your LLM)  │
                  │  (vLLM)      │    │              │
                  └──────────────┘    └──────────────┘
                         │
                         ▼
                  ┌──────────────┐
                  │  Vector DB   │
                  │  (LanceDB)   │
                  └──────────────┘

Three components:

1. LLM — your existing vLLM (generates answers)

2. Embedding model — converts text to vectors (separate small model)

3. Vector DB — stores embeddings (LanceDB built-in, no separate container needed)

2. Embedding Models

Built-in vs Dedicated

Option	Pros	Cons
AnythingLLM built-in	Zero config, runs on CPU	Slower, basic quality
Dedicated vLLM embedding	Fast, GPU-accelerated, high quality	Uses extra VRAM

For production or Thai documents, a dedicated embedding server is strongly recommended.

Recommended Embedding Models

Model	Use Case	VRAM
BAAI/bge-m3	Multilingual (incl. Thai), excellent	~2-3 GB
intfloat/multilingual-e5-large	Multilingual	~2-3 GB
BAAI/bge-large-en-v1.5	English only, fast	~1-2 GB

BAAI/bge-m3 is the top choice for Thai or mixed Thai/English documents. Always match the embedding model to your document language.

3. Vector Database Options

Vector DB	When to use
LanceDB (built-in)	Default — single instance, simplest, no extra container
Qdrant	Larger scale, shared storage, advanced filtering

LanceDB is sufficient for most deployments and requires zero setup.

4. Full Stack Deployment (Multi-GPU)

Runs vLLM (chat) + vLLM (embeddings) + AnythingLLM together. Chat on GPU 0, embeddings on GPU 1.

docker-compose.yml

services:
  # Main chat LLM
  vllm:
    image: vllm/vllm-openai:nightly       # :latest for RTX 3090
    container_name: vllm
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-32B-Instruct-AWQ
      --quantization awq
      --max-model-len 16384
      --gpu-memory-utilization 0.85
      --enforce-eager
      --api-key sk-llm-key
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

  # Embedding server (vLLM in embedding mode)
  vllm-embed:
    image: vllm/vllm-openai:nightly       # :latest for RTX 3090
    container_name: vllm-embed
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=1
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8001:8000"
    ipc: host
    command: >
      --model BAAI/bge-m3
      --task embed
      --gpu-memory-utilization 0.30
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['1']
              capabilities: [gpu]

  # AnythingLLM (RAG app + built-in LanceDB)
  anythingllm:
    image: mintplexlabs/anythingllm:latest
    container_name: anythingllm
    restart: unless-stopped
    depends_on:
      - vllm
      - vllm-embed
    ports:
      - "3001:3001"
    environment:
      - STORAGE_DIR=/app/server/storage

      # LLM (chat)
      - LLM_PROVIDER=generic-openai
      - GENERIC_OPEN_AI_BASE_PATH=http://vllm:8000/v1
      - GENERIC_OPEN_AI_API_KEY=sk-llm-key
      - GENERIC_OPEN_AI_MODEL_PREF=Qwen/Qwen2.5-32B-Instruct-AWQ
      - GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT=16384

      # Embeddings
      - EMBEDDING_ENGINE=generic-openai
      - EMBEDDING_BASE_PATH=http://vllm-embed:8000/v1
      - EMBEDDING_MODEL_PREF=BAAI/bge-m3
      - EMBEDDING_MODEL_MAX_CHUNK_LENGTH=8192
      - GENERIC_OPEN_AI_EMBEDDING_API_KEY=none

      # Vector DB
      - VECTOR_DB=lancedb
    volumes:
      - ./anythingllm-storage:/app/server/storage

volumes:
  anythingllm-storage:

Launch

echo "HF_TOKEN=hf_your_token_here" > .env
docker compose up -d
docker compose logs -f

Wait for all three healthy:

• vllm → Application startup complete

• vllm-embed → Application startup complete

• anythingllm → Primary server in HTTP mode listening on port 3001

5. Single-GPU Deployment

Run chat + embedding on one GPU. The embedding model is small (~2-3GB), so lower the chat model's memory utilization to leave room.

services:
  vllm:
    image: vllm/vllm-openai:nightly
    container_name: vllm
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-7B-Instruct-AWQ
      --quantization awq
      --max-model-len 8192
      --gpu-memory-utilization 0.70      # leave room for embedding
      --enforce-eager
      --api-key sk-llm-key
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

  vllm-embed:
    image: vllm/vllm-openai:nightly
    container_name: vllm-embed
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0          # same GPU
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8001:8000"
    ipc: host
    command: >
      --model BAAI/bge-m3
      --task embed
      --gpu-memory-utilization 0.20      # small slice
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

  anythingllm:
    image: mintplexlabs/anythingllm:latest
    container_name: anythingllm
    restart: unless-stopped
    depends_on: [vllm, vllm-embed]
    ports:
      - "3001:3001"
    environment:
      - STORAGE_DIR=/app/server/storage
      - LLM_PROVIDER=generic-openai
      - GENERIC_OPEN_AI_BASE_PATH=http://vllm:8000/v1
      - GENERIC_OPEN_AI_API_KEY=sk-llm-key
      - GENERIC_OPEN_AI_MODEL_PREF=Qwen/Qwen2.5-7B-Instruct-AWQ
      - GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT=8192
      - EMBEDDING_ENGINE=generic-openai
      - EMBEDDING_BASE_PATH=http://vllm-embed:8000/v1
      - EMBEDDING_MODEL_PREF=BAAI/bge-m3
      - GENERIC_OPEN_AI_EMBEDDING_API_KEY=none
      - VECTOR_DB=lancedb
    volumes:
      - ./anythingllm-storage:/app/server/storage

Combined gpu-memory-utilization (0.70 + 0.20 = 0.90) must stay under 1.0.

6. External Vector DB (Qdrant)

For larger scale or shared vector storage, add Qdrant:

  qdrant:
    image: qdrant/qdrant:latest
    container_name: qdrant
    restart: unless-stopped
    ports:
      - "6333:6333"
    volumes:
      - ./qdrant-storage:/qdrant/storage

Then change AnythingLLM env:

      - VECTOR_DB=qdrant
      - QDRANT_ENDPOINT=http://qdrant:6333

7. Using AnythingLLM

First Launch

Open http://<your-server-ip>:3001

1. Create your admin account

2. Verify LLM + Embedding settings are pre-filled (from env vars)

3. Create a Workspace

4. Upload documents → auto-embedded and ready to chat

Document Upload

• Supported: PDF, DOCX, TXT, MD, CSV, and web page URLs

• Documents are chunked, embedded, and stored automatically

• Each workspace has its own document set and chat history

Verify Embedding Server

curl http://localhost:8001/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "BAAI/bge-m3",
    "input": "Hello world"
  }'

You should get an array of numbers (the embedding vector).

8. RAG Tuning & Troubleshooting

Tuning

Setting	Where	Effect
Chunk size	Workspace settings	Smaller = precise, larger = more context
Top-K retrieval	Workspace settings	How many chunks retrieved per query
--max-model-len	vLLM chat	Must fit retrieved chunks + question + answer
Embedding model	bge-m3 for Thai	Match document language

Common Issues

AnythingLLM can't connect to vLLM

• Check containers are on the same Docker network (same compose file = automatic)

• Verify GENERIC_OPEN_AI_BASE_PATH uses the service name (http://vllm:8000/v1), not localhost

• Confirm the API key matches between vLLM --api-key and AnythingLLM env

Embeddings fail / documents won't process

• Verify embedding server: curl http://localhost:8001/v1/embeddings ...

• Check EMBEDDING_BASE_PATH points to http://vllm-embed:8000/v1

• Ensure embedding model loaded: docker compose logs vllm-embed

Poor answer quality

• Increase Top-K retrieval (more context)

• Use a larger chat model (32B vs 7B)

• For Thai docs, confirm bge-m3 embedding (not English-only)

• Reduce chunk size for more precise retrieval

Out of memory on single GPU

• Lower chat --gpu-memory-utilization

• Use a smaller chat model

• Reduce --max-model-len