> For the complete documentation index, see [llms.txt](https://docs.readyidc.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.readyidc.com/gpu-as-a-service/linux/docker/how-to-using-local-llm-with-vllm.md).

# How to using Local LLM with vLLM

> Self-hosted LLM serving with vLLM on your GPU VPS\
> OpenAI-compatible API + Open WebUI\
> Covers single-GPU, multi-GPU, Tensor Parallelism (TP) and Pipeline Parallelism (PP)

***

## Supported GPUs

| GPU                    | VRAM  | Architecture        | vLLM Image                                |
| ---------------------- | ----- | ------------------- | ----------------------------------------- |
| RTX 3050               | 8 GB  | Ampere (sm\_86)     | `vllm/vllm-openai:latest` (stable)        |
| RTX 3090               | 24 GB | Ampere (sm\_86)     | `vllm/vllm-openai:latest` (stable)        |
| RTX 5060 Ti            | 16 GB | Blackwell (sm\_120) | `vllm/vllm-openai:nightly` (**required**) |
| RTX 5090               | 32 GB | Blackwell (sm\_120) | `vllm/vllm-openai:nightly` (**required**) |
| RTX PRO 2000 Blackwell | 16 GB | Blackwell (sm\_120) | `vllm/vllm-openai:nightly` (**required**) |
| RTX PRO 4000 Blackwell | 24 GB | Blackwell (sm\_120) | `vllm/vllm-openai:nightly` (**required**) |
| RTX PRO 5000 Blackwell | 48 GB | Blackwell (sm\_120) | `vllm/vllm-openai:nightly` (**required**) |
| RTX PRO 6000 Blackwell | 96 GB | Blackwell (sm\_120) | `vllm/vllm-openai:nightly` (**required**) |

{% hint style="warning" %}
All Blackwell GPUs (RTX 5060 Ti, RTX 5090, RTX PRO 2000 / 4000 / 5000 / 6000 Blackwell) require the **nightly** image. Stable vLLM does not support sm\_120 yet. RTX 3050 and RTX 3090 (Ampere) work with the stable image.
{% endhint %}

> **RTX PRO 6000 (96GB)** is the flagship — its large VRAM runs 70B models on a single GPU (AWQ) or even FP16 70B with TP=2.

> **RTX 5090 (32GB)** is the highest-VRAM consumer card here — the value pick for 32B AWQ on a single GPU, or 70B AWQ with TP=2 across a 2× 5090 VPS.

***

## Model Size ↔ GPU Selector

Pick the **smallest GPU your model fits on** at your target precision and context length. This table summarizes how large a model each GPU in the fleet can run.

### VRAM Rule of Thumb

| Precision   | Model weights  | Per 1B params |
| ----------- | -------------- | ------------- |
| FP16 / BF16 | full precision | \~2.0 GB      |
| FP8         | 8-bit          | \~1.0 GB      |
| AWQ / GPTQ  | 4-bit          | \~0.5–0.6 GB  |

{% hint style="info" %}
Always budget **KV cache** (scales with `--max-model-len` and concurrent requests) plus **2–4 GB** overhead on top of the model weights.
{% endhint %}

### Single-GPU — Max Model per Card

| GPU (VRAM)           | Image      | Comfortable                | Tight / short context | Bigger models          |
| -------------------- | ---------- | -------------------------- | --------------------- | ---------------------- |
| RTX 3050 (8 GB)      | `:latest`  | 3–4B FP16 · 7–8B AWQ       | 7–8B AWQ @ 2–4K ctx   | Use a larger card      |
| RTX 5060 Ti (16 GB)  | `:nightly` | 7–8B FP16 · 14B AWQ        | 14B AWQ @ 4–8K        | 32B → **TP=2**         |
| RTX PRO 2000 (16 GB) | `:nightly` | 7–8B FP16 · 14B AWQ        | 14B AWQ @ 4–8K        | 32B → **TP=2**         |
| RTX 3090 (24 GB)     | `:latest`  | 14B FP16 · 32B AWQ         | 32B AWQ @ 8K          | 70B → **TP=2**         |
| RTX PRO 4000 (24 GB) | `:nightly` | 14B FP16 · 32B AWQ         | 32B AWQ @ 8K          | 70B → **TP=2**         |
| RTX 5090 (32 GB)     | `:nightly` | 32B AWQ · 14B FP16         | 32B AWQ @ 16K         | 70B → **TP=2**         |
| RTX PRO 5000 (48 GB) | `:nightly` | 32B FP16 · 70B AWQ (tight) | 70B AWQ @ 4–8K        | 70B FP16 → **TP=2**    |
| RTX PRO 6000 (96 GB) | `:nightly` | 70B AWQ · 32B FP16         | 70B AWQ @ 32K         | 70B FP16 → 2× PRO 6000 |

### Dual-GPU — When One Card Isn't Enough (TP=2)

| Dual-GPU (total VRAM)    | Runs                     | Notes                         |
| ------------------------ | ------------------------ | ----------------------------- |
| 2× RTX 3090 (48 GB)      | 70B AWQ, TP=2            | Requires `NCCL_P2P_DISABLE=1` |
| 2× RTX 5090 (64 GB)      | 70B AWQ, TP=2 · 32B FP16 | Best consumer TP option       |
| 2× RTX PRO 4000 (48 GB)  | 70B AWQ, TP=2            | `NCCL_P2P_DISABLE=1`          |
| 2× RTX PRO 5000 (96 GB)  | 70B AWQ/FP16, TP=2       | Full-precision 70B headroom   |
| 2× RTX PRO 6000 (192 GB) | 70B FP16, TP=2           | Full-quality 70B FP16         |

### Quick Pick

| Requirement                   | Recommended GPU                                                   |
| ----------------------------- | ----------------------------------------------------------------- |
| Dev/test, small chatbot (≤8B) | **RTX 3050** or **RTX 5060 Ti**                                   |
| General 14–32B, single card   | **RTX 3090 / PRO 4000** (32B AWQ) or **RTX 5090** (most headroom) |
| 70B, single card              | **RTX PRO 6000** (AWQ) or **RTX PRO 5000** (AWQ, tight)           |
| 70B, budget dual-GPU          | **2× RTX 5090 / 2× PRO 4000** (TP=2)                              |
| 70B full precision (FP16)     | **2× RTX PRO 6000** or **2× RTX PRO 5000** (TP=2)                 |

> For deeper flag and VRAM-budget details, see [Section 7](#7-model-sizing-guide) and [Section 8](#8-rtx-pro-6000--running-70b-on-a-single-gpu).

***

## 1. Prerequisites

### Verify GPU is visible

```bash
nvidia-smi
```

You should see your GPU(s) listed. If not, contact support.

### Install Docker + NVIDIA Container Toolkit

```bash
# Docker
curl -fsSL https://get.docker.com | sh

# NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```

### Verify Docker can access GPU

```bash
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
```

### Get a HuggingFace Token (for gated/most models)

Create a token at <https://huggingface.co/settings/tokens> — you'll use it as `HF_TOKEN`.

***

## 2. Understanding TP vs PP

### Tensor Parallelism (TP)

Splits **each layer** across multiple GPUs. All GPUs work on the same request simultaneously.

* **Use when:** model is too large for one GPU, OR you want lower latency
* **Best for:** GPUs in the same machine with fast interconnect
* **Flag:** `--tensor-parallel-size N`
* **Requirement:** number of attention heads must be divisible by N

```
Request → [Layer 1 split across GPU0+GPU1] → [Layer 2 split...] → Output
```

### Pipeline Parallelism (PP)

Splits **different layers** onto different GPUs. Each GPU handles a stage.

* **Use when:** TP isn't efficient (slow interconnect), or scaling across many GPUs
* **Best for:** maximizing throughput with many concurrent requests
* **Flag:** `--pipeline-parallel-size N`

```
Request → [Layers 1-20 on GPU0] → [Layers 21-40 on GPU1] → Output
```

### Quick Decision

| Scenario                                     | Recommendation               |
| -------------------------------------------- | ---------------------------- |
| Model fits in 1 GPU                          | Single GPU (no parallelism)  |
| Model too big for 1 GPU, 2 GPUs same machine | **TP=2**                     |
| Want max throughput, many users              | TP for latency, PP for scale |
| 2 GPUs, low latency priority                 | **TP=2**                     |

{% hint style="info" %}
For 2-GPU setups on a single VPS, TP=2 is almost always the better choice.
{% endhint %}

***

## 3. Single GPU Deployment

Best for 7B–32B models that fit in one GPU's VRAM.

### Directory Structure

```bash
mkdir -p ~/vllm-stack && cd ~/vllm-stack
```

{% tabs %}
{% tab title="RTX 3090 (Ampere, stable image)" %}

```yaml
services:
  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-7B-Instruct-AWQ
      --quantization awq
      --max-model-len 8192
      --gpu-memory-utilization 0.90
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
```

{% endtab %}

{% tab title="Blackwell (RTX 5060 Ti / PRO 4000, nightly image)" %}

```yaml
services:
  vllm:
    image: vllm/vllm-openai:nightly
    container_name: vllm
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-7B-Instruct-AWQ
      --quantization awq
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --enforce-eager
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
```

{% endtab %}
{% endtabs %}

### Create `.env`

```bash
echo "HF_TOKEN=hf_your_token_here" > .env
```

### Launch

```bash
docker compose up -d
docker compose logs -f vllm
```

Wait for: `Application startup complete` and `Uvicorn running on http://0.0.0.0:8000`

***

## 4. Dual GPU — Tensor Parallel (TP)

For models too large for one GPU (e.g. 70B on 2× RTX PRO 4000), or for lower latency.

### `docker-compose.yml` — TP=2

```yaml
services:
  vllm:
    image: vllm/vllm-openai:nightly   # use :latest for RTX 3090
    container_name: vllm-tp2
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0,1
      - NCCL_P2P_DISABLE=1        # required for RTX consumer cards
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-32B-Instruct-AWQ
      --quantization awq
      --tensor-parallel-size 2
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --enforce-eager
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0', '1']
              capabilities: [gpu]
```

### Critical Flags for Multi-GPU on RTX Cards

| Flag                       | Why                                                            |
| -------------------------- | -------------------------------------------------------------- |
| `NCCL_P2P_DISABLE=1`       | RTX consumer cards lack proper P2P; without this, TP **hangs** |
| `--enforce-eager`          | Avoids CUDA graph issues on Blackwell multi-GPU                |
| `--tensor-parallel-size 2` | Splits model across both GPUs                                  |

> **RTX 3090 / 5060 Ti / PRO 4000:** Always set `NCCL_P2P_DISABLE=1` for TP. This is the #1 cause of multi-GPU hangs.

***

## 5. Dual GPU — Pipeline Parallel (PP)

Use PP when you want to run two **separate** model replicas' worth of layers split, typically for throughput scaling. For most 2-GPU VPS cases, TP is preferred — but here's PP for completeness.

### `docker-compose.yml` — PP=2

```yaml
services:
  vllm:
    image: vllm/vllm-openai:nightly   # :latest for RTX 3090
    container_name: vllm-pp2
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0,1
      - NCCL_P2P_DISABLE=1
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-32B-Instruct-AWQ
      --quantization awq
      --pipeline-parallel-size 2
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --enforce-eager
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0', '1']
              capabilities: [gpu]
```

### TP vs PP — Same Hardware, Different Behavior

|                          | TP=2                     | PP=2                           |
| ------------------------ | ------------------------ | ------------------------------ |
| Latency                  | Lower (parallel compute) | Higher (sequential stages)     |
| Throughput               | Good                     | Higher under heavy concurrency |
| Interconnect sensitivity | High (needs fast link)   | Lower                          |
| Best for 2-GPU VPS       | ✅ Usually                | Specific cases                 |

***

## 6. Adding Open WebUI

Open WebUI gives a ChatGPT-style interface in front of your vLLM API.

### Full `docker-compose.yml` (vLLM + Open WebUI)

```yaml
services:
  vllm:
    image: vllm/vllm-openai:nightly   # :latest for RTX 3090
    container_name: vllm
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-7B-Instruct-AWQ
      --quantization awq
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --enforce-eager
      --api-key sk-your-secret-key
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - vllm
    ports:
      - "3000:8080"
    environment:
      - OPENAI_API_BASE_URL=http://vllm:8000/v1
      - OPENAI_API_KEY=sk-your-secret-key
    volumes:
      - ./open-webui-data:/app/backend/data

volumes:
  open-webui-data:
```

### Launch & Access

```bash
docker compose up -d
```

* **API endpoint:** `http://<your-server-ip>:8000/v1`
* **Web UI:** `http://<your-server-ip>:3000`

On first WebUI visit, create an admin account. Models from vLLM appear automatically.

***

## 7. Model Sizing Guide

| GPU (VRAM)              | 7-8B          | 14B           | 32B             | 70B             |
| ----------------------- | ------------- | ------------- | --------------- | --------------- |
| RTX 3050 (8GB)          | ✅ AWQ (tight) | ❌             | ❌               | ❌               |
| RTX 5060 Ti (16GB)      | ✅ 1 GPU       | ✅ 1 GPU (AWQ) | TP=2            | ❌               |
| RTX PRO 2000 (16GB)     | ✅ 1 GPU       | ✅ 1 GPU (AWQ) | TP=2            | ❌               |
| RTX 3090 (24GB)         | ✅ 1 GPU       | ✅ 1 GPU       | ✅ 1 GPU (tight) | TP=2+           |
| RTX PRO 4000 (24GB)     | ✅ 1 GPU       | ✅ 1 GPU       | ✅ 1 GPU (tight) | TP=2+           |
| RTX 5090 (32GB)         | ✅ 1 GPU       | ✅ 1 GPU       | ✅ 1 GPU (AWQ)   | TP=2 (AWQ)      |
| RTX PRO 5000 (48GB)     | ✅ 1 GPU       | ✅ 1 GPU       | ✅ 1 GPU         | TP=2 (AWQ)      |
| RTX PRO 6000 (96GB)     | ✅ 1 GPU       | ✅ 1 GPU       | ✅ 1 GPU         | ✅ 1 GPU (AWQ)   |
| 2× RTX 3090 (48GB)      | ✅             | ✅             | ✅               | TP=2 (AWQ)      |
| 2× RTX 5090 (64GB)      | ✅             | ✅             | ✅               | TP=2 (AWQ)      |
| 2× RTX PRO 4000 (48GB)  | ✅             | ✅             | ✅               | TP=2 (AWQ)      |
| 2× RTX PRO 5000 (96GB)  | ✅             | ✅             | ✅               | TP=2 (AWQ/FP16) |
| 2× RTX PRO 6000 (192GB) | ✅             | ✅             | ✅               | ✅ TP=2 (FP16)   |

> **RTX PRO 6000 (96GB) single-GPU** runs a 70B AWQ model without parallelism — simplest setup for large models. For 70B FP16, use 2× PRO 6000 with TP=2.

### Recommended Models

| Use Case               | Model                                   | Notes            |
| ---------------------- | --------------------------------------- | ---------------- |
| General chat (English) | `Qwen/Qwen2.5-7B-Instruct-AWQ`          | Fast, fits 1 GPU |
| General chat (larger)  | `Qwen/Qwen2.5-32B-Instruct-AWQ`         | 24GB GPU or TP=2 |
| Thai language          | `scb10x/llama-3.1-typhoon2-8b-instruct` | Thai-optimized   |
| Coding                 | `Qwen/Qwen2.5-Coder-7B-Instruct-AWQ`    | Code tasks       |

### Tuning `--max-model-len`

* Longer context = more KV cache memory = fewer concurrent requests
* Start with `8192`; increase only if needed
* If you hit out-of-memory, lower `--max-model-len` or `--gpu-memory-utilization`

***

## 8. RTX PRO 6000 — Running 70B on a Single GPU

The RTX PRO 6000 Blackwell (96GB) is large enough to run a 70B model **without any parallelism** — the simplest possible large-model deployment. No `NCCL_P2P_DISABLE`, no tensor parallel tuning.

### `docker-compose.yml` — 70B AWQ on 1× PRO 6000

```yaml
services:
  vllm:
    image: vllm/vllm-openai:nightly
    container_name: vllm-70b
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-72B-Instruct-AWQ
      --quantization awq
      --max-model-len 16384
      --gpu-memory-utilization 0.92
      --enforce-eager
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
```

### VRAM Budget on 96GB

| Component          | 70B AWQ            | 70B FP16               |
| ------------------ | ------------------ | ---------------------- |
| Model weights      | \~40 GB            | \~140 GB (needs 2 GPU) |
| KV cache (16K ctx) | \~15-20 GB         | —                      |
| Overhead           | \~5 GB             | —                      |
| **Fits in 96GB?**  | ✅ Yes, comfortably | ❌ Needs TP=2           |

> For **70B FP16**, use 2× PRO 6000 with `--tensor-parallel-size 2` (see Section 4). For most use cases, **70B AWQ on a single PRO 6000 is the sweet spot** — full quality with simple setup.

### Larger Context

With 96GB you can push context much further:

```bash
--max-model-len 32768    # 32K context, still fits with 70B AWQ
```

Longer context reduces concurrent capacity but is great for document analysis / RAG.

***

## 9. MIG — Splitting a GPU (PRO 6000)

**MIG (Multi-Instance GPU)** lets you partition one PRO 6000 into multiple isolated GPU instances — useful for running several smaller models or serving multiple isolated workloads on one card.

> Only **RTX PRO 6000 Blackwell** supports MIG among the GPUs here. RTX 3050, 3090, 5060 Ti, 5090, PRO 2000, PRO 4000, PRO 5000 do **not** support MIG.

### When to Use MIG

| Use MIG                                        | Don't use MIG                               |
| ---------------------------------------------- | ------------------------------------------- |
| Run multiple small models (7-8B each) isolated | Running one large model (use full GPU)      |
| Multi-tenant isolation on one card             | Need max performance for single workload    |
| Guaranteed resource partitioning               | Workload varies / needs full VRAM sometimes |

### Enable MIG (on the host/VM with the GPU)

```bash
# Enable MIG mode (requires no running GPU workloads)
sudo nvidia-smi -i 0 -mig 1

# Reboot may be required
sudo reboot
```

### Create MIG Instances

```bash
# List available MIG profiles
nvidia-smi mig -lgip

# Example: create profiles on the 96GB card
# Profile IDs vary; common split into instances:
sudo nvidia-smi mig -cgi 9,9,9,9 -C   # creates multiple instances

# Verify
nvidia-smi -L
```

Output shows MIG device UUIDs:

```
GPU 0: NVIDIA RTX PRO 6000 (UUID: GPU-xxxx)
  MIG 2g.24gb Device 0: (UUID: MIG-xxxx)
  MIG 2g.24gb Device 1: (UUID: MIG-yyyy)
  ...
```

### Assign a MIG Instance to a Container

Use the MIG UUID in `NVIDIA_VISIBLE_DEVICES`:

```yaml
services:
  vllm-model-a:
    image: vllm/vllm-openai:nightly
    environment:
      - NVIDIA_VISIBLE_DEVICES=MIG-xxxx   # specific MIG instance
    # ... rest of config
```

### Disable MIG

```bash
sudo nvidia-smi -i 0 -mig 0
sudo reboot
```

> **Note:** MIG instances have fixed VRAM/compute slices and cannot use NVLink between them. For a single large model, MIG is counterproductive — use the full GPU instead.

***

## 10. Using Other Models

You're not limited to Qwen. vLLM supports most popular open models. Here's how to use any model.

### Finding the Model Name

Use the **HuggingFace repo path** as the `--model` value. Find it on <https://huggingface.co> — the path after `huggingface.co/`.

Example: `meta-llama/Llama-3.1-8B-Instruct` → `--model meta-llama/Llama-3.1-8B-Instruct`

### Popular Models by Category

| Category          | Model (HuggingFace path)                   | Size | Notes                        |
| ----------------- | ------------------------------------------ | ---- | ---------------------------- |
| General (English) | `meta-llama/Llama-3.1-8B-Instruct`         | 8B   | Gated — accept license first |
| General           | `mistralai/Mistral-7B-Instruct-v0.3`       | 7B   | Open                         |
| General (large)   | `meta-llama/Llama-3.1-70B-Instruct`        | 70B  | Needs 2 GPU or PRO 6000      |
| Thai              | `scb10x/llama-3.1-typhoon2-8b-instruct`    | 8B   | Thai-optimized               |
| Thai (large)      | `scb10x/llama-3.1-typhoon2-70b-instruct`   | 70B  | Thai, needs big VRAM         |
| Coding            | `Qwen/Qwen2.5-Coder-32B-Instruct`          | 32B  | Strong coding                |
| Coding (small)    | `Qwen/Qwen2.5-Coder-7B-Instruct`           | 7B   | Fast                         |
| Reasoning         | `deepseek-ai/DeepSeek-R1-Distill-Qwen-32B` | 32B  | Chain-of-thought             |
| Vision (VL)       | `Qwen/Qwen2.5-VL-7B-Instruct`              | 7B   | Image understanding          |

> Model availability and exact names change over time — always verify the repo exists on HuggingFace before deploying.

### Quantized vs Full Precision

**Full precision (FP16)** — use the base repo, no quantization flag:

```yaml
command: >
  --model mistralai/Mistral-7B-Instruct-v0.3
  --max-model-len 8192
  --gpu-memory-utilization 0.90
```

**AWQ quantized** — look for an `-AWQ` variant repo, add `--quantization awq`:

```yaml
command: >
  --model Qwen/Qwen2.5-32B-Instruct-AWQ
  --quantization awq
  --max-model-len 8192
  --gpu-memory-utilization 0.90
```

**GPTQ quantized** — look for `-GPTQ` variant, add `--quantization gptq`:

```yaml
command: >
  --model TheBloke/Llama-2-13B-chat-GPTQ
  --quantization gptq
  --max-model-len 4096
```

### How to Find Quantized Versions

On HuggingFace, search the model name + "AWQ" or "GPTQ". Common quantizers:

* Official repos (e.g. `Qwen/...-AWQ`)
* `TheBloke/...` (many GPTQ/AWQ conversions)
* `casperhansen/...` (AWQ)

### Gated Models (Llama, etc.)

Some models require accepting a license:

1. Visit the model page on HuggingFace (e.g. `meta-llama/Llama-3.1-8B-Instruct`)
2. Click "Agree and access repository"
3. Ensure your `HF_TOKEN` in `.env` belongs to the account that accepted

Without this, you'll get a `401 Gated repo` error in the logs.

### Switching Models — Just Edit and Restart

```bash
# 1. Edit the --model line in docker-compose.yml
nano docker-compose.yml

# 2. Restart
docker compose down && docker compose up -d

# 3. Watch it download + load
docker compose logs -f vllm
```

Models cache in `./models`, so re-using a previously downloaded model is instant.

### Checklist Before Deploying a New Model

| Check                         | How                                                   |
| ----------------------------- | ----------------------------------------------------- |
| Does it fit your VRAM?        | See [Section 7](#7-model-sizing-guide) sizing table   |
| Is there an AWQ/GPTQ version? | Search HF for `-AWQ` / `-GPTQ`                        |
| Is it gated?                  | Check for "Agree to access" on HF page                |
| Blackwell GPU?                | Use `:nightly` image                                  |
| Multi-GPU needed?             | Add `--tensor-parallel-size N` + `NCCL_P2P_DISABLE=1` |

***

## 11. Testing & Troubleshooting

### Test the API

```bash
# List models
curl http://localhost:8000/v1/models

# Chat completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-secret-key" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct-AWQ",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

### Common Issues

#### "No supported device found" / CUDA error on Blackwell

**Cause:** Using stable image on Blackwell GPU.\
**Fix:** Switch to `vllm/vllm-openai:nightly`.

#### Multi-GPU TP hangs at startup

**Cause:** Missing `NCCL_P2P_DISABLE=1`.\
**Fix:** Add `- NCCL_P2P_DISABLE=1` to environment and add `--enforce-eager`.

#### Out of Memory (OOM) / KV cache error

```
ValueError: No available memory for the cache blocks
```

**Fix:**

* Increase `--gpu-memory-utilization` to `0.92`
* OR decrease `--max-model-len`
* OR use a smaller / more quantized model

#### Container exits immediately

```bash
docker compose logs vllm
```

Check for: wrong model name, missing HF\_TOKEN, or gated model access.

#### Model download is slow / fails

* Ensure `HF_TOKEN` is set in `.env`
* For gated models (e.g. Llama), accept the license on HuggingFace first
* Models cache to `./models` — subsequent restarts are fast

### Useful Commands

```bash
# Watch GPU usage live
watch -n1 nvidia-smi

# Container logs
docker compose logs -f vllm

# Restart after config change
docker compose down && docker compose up -d

# Check container status
docker compose ps
```

### GPU Memory Reference (idle vs loaded)

```bash
nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu --format=csv
```

***

## Quick Start Summary

```bash
# 1. Setup
mkdir -p ~/vllm-stack && cd ~/vllm-stack
echo "HF_TOKEN=hf_xxx" > .env

# 2. Create docker-compose.yml (see sections above)
nano docker-compose.yml

# 3. Launch
docker compose up -d

# 4. Test
curl http://localhost:8000/v1/models

# 5. Access Web UI
# http://<server-ip>:3000
```

### Cheat Sheet

| Your Setup                                              | Image      | Key Flags                                                         |
| ------------------------------------------------------- | ---------- | ----------------------------------------------------------------- |
| 1× RTX 3050 / 3090 (Ampere)                             | `:latest`  | `--gpu-memory-utilization 0.90`                                   |
| 1× Blackwell (5060 Ti / 5090 / PRO 2000/4000/5000/6000) | `:nightly` | `+ --enforce-eager`                                               |
| 2× any (TP)                                             | match GPU  | `--tensor-parallel-size 2` `NCCL_P2P_DISABLE=1` `--enforce-eager` |
| 2× any (PP)                                             | match GPU  | `--pipeline-parallel-size 2` `NCCL_P2P_DISABLE=1`                 |

***

*Need help? Contact ReadyIDC Support: <admin@readyidc.com>*


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.readyidc.com/gpu-as-a-service/linux/docker/how-to-using-local-llm-with-vllm.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.