How to using Local LLM with vLLM

Self-hosted LLM serving with vLLM on your GPU VPS OpenAI-compatible API + Open WebUI Covers single-GPU, multi-GPU, Tensor Parallelism (TP) and Pipeline Parallelism (PP)

---

Supported GPUs

GPU	VRAM	Architecture	vLLM Image
RTX 3050	8 GB	Ampere (sm_86)	vllm/vllm-openai:latest (stable)
RTX 3090	24 GB	Ampere (sm_86)	vllm/vllm-openai:latest (stable)
RTX 5060 Ti	16 GB	Blackwell (sm_120)	vllm/vllm-openai:nightly (required)
RTX 5090	32 GB	Blackwell (sm_120)	vllm/vllm-openai:nightly (required)
RTX PRO 2000 Blackwell	16 GB	Blackwell (sm_120)	vllm/vllm-openai:nightly (required)
RTX PRO 4000 Blackwell	24 GB	Blackwell (sm_120)	vllm/vllm-openai:nightly (required)
RTX PRO 5000 Blackwell	48 GB	Blackwell (sm_120)	vllm/vllm-openai:nightly (required)
RTX PRO 6000 Blackwell	96 GB	Blackwell (sm_120)	vllm/vllm-openai:nightly (required)

All Blackwell GPUs (RTX 5060 Ti, RTX 5090, RTX PRO 2000 / 4000 / 5000 / 6000 Blackwell) require the nightly image. Stable vLLM does not support sm_120 yet. RTX 3050 and RTX 3090 (Ampere) work with the stable image.

RTX PRO 6000 (96GB) is the flagship — its large VRAM runs 70B models on a single GPU (AWQ) or even FP16 70B with TP=2.

RTX 5090 (32GB) is the highest-VRAM consumer card here — the value pick for 32B AWQ on a single GPU, or 70B AWQ with TP=2 across a 2× 5090 VPS.

---

Model Size ↔ GPU Selector

Pick the smallest GPU your model fits on at your target precision and context length. This table summarizes how large a model each GPU in the fleet can run.

VRAM Rule of Thumb

Precision	Model weights	Per 1B params
FP16 / BF16	full precision	~2.0 GB
FP8	8-bit	~1.0 GB
AWQ / GPTQ	4-bit	~0.5–0.6 GB

Always budget KV cache (scales with --max-model-len and concurrent requests) plus 2–4 GB overhead on top of the model weights.

Single-GPU — Max Model per Card

GPU (VRAM)	Image	Comfortable	Tight / short context	Bigger models
RTX 3050 (8 GB)	:latest	3–4B FP16 · 7–8B AWQ	7–8B AWQ @ 2–4K ctx	Use a larger card
RTX 5060 Ti (16 GB)	:nightly	7–8B FP16 · 14B AWQ	14B AWQ @ 4–8K	32B → TP=2
RTX PRO 2000 (16 GB)	:nightly	7–8B FP16 · 14B AWQ	14B AWQ @ 4–8K	32B → TP=2
RTX 3090 (24 GB)	:latest	14B FP16 · 32B AWQ	32B AWQ @ 8K	70B → TP=2
RTX PRO 4000 (24 GB)	:nightly	14B FP16 · 32B AWQ	32B AWQ @ 8K	70B → TP=2
RTX 5090 (32 GB)	:nightly	32B AWQ · 14B FP16	32B AWQ @ 16K	70B → TP=2
RTX PRO 5000 (48 GB)	:nightly	32B FP16 · 70B AWQ (tight)	70B AWQ @ 4–8K	70B FP16 → TP=2
RTX PRO 6000 (96 GB)	:nightly	70B AWQ · 32B FP16	70B AWQ @ 32K	70B FP16 → 2× PRO 6000

Dual-GPU — When One Card Isn't Enough (TP=2)

Dual-GPU (total VRAM)	Runs	Notes
2× RTX 3090 (48 GB)	70B AWQ, TP=2	Requires NCCL_P2P_DISABLE=1
2× RTX 5090 (64 GB)	70B AWQ, TP=2 · 32B FP16	Best consumer TP option
2× RTX PRO 4000 (48 GB)	70B AWQ, TP=2	NCCL_P2P_DISABLE=1
2× RTX PRO 5000 (96 GB)	70B AWQ/FP16, TP=2	Full-precision 70B headroom
2× RTX PRO 6000 (192 GB)	70B FP16, TP=2	Full-quality 70B FP16

Quick Pick

Requirement	Recommended GPU
Dev/test, small chatbot (≤8B)	RTX 3050 or RTX 5060 Ti
General 14–32B, single card	RTX 3090 / PRO 4000 (32B AWQ) or RTX 5090 (most headroom)
70B, single card	RTX PRO 6000 (AWQ) or RTX PRO 5000 (AWQ, tight)
70B, budget dual-GPU	2× RTX 5090 / 2× PRO 4000 (TP=2)
70B full precision (FP16)	2× RTX PRO 6000 or 2× RTX PRO 5000 (TP=2)

For deeper flag and VRAM-budget details, see Section 7 and Section 8.

---

1. Prerequisites

Verify GPU is visible

nvidia-smi

You should see your GPU(s) listed. If not, contact support.

Install Docker + NVIDIA Container Toolkit

# Docker
curl -fsSL https://get.docker.com | sh

# NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify Docker can access GPU

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

Get a HuggingFace Token (for gated/most models)

Create a token at https://huggingface.co/settings/tokens — you'll use it as HF_TOKEN.

---

2. Understanding TP vs PP

Tensor Parallelism (TP)

Splits each layer across multiple GPUs. All GPUs work on the same request simultaneously.

• Use when: model is too large for one GPU, OR you want lower latency

• Best for: GPUs in the same machine with fast interconnect

• Flag: --tensor-parallel-size N

• Requirement: number of attention heads must be divisible by N

Request → [Layer 1 split across GPU0+GPU1] → [Layer 2 split...] → Output

Pipeline Parallelism (PP)

Splits different layers onto different GPUs. Each GPU handles a stage.

• Use when: TP isn't efficient (slow interconnect), or scaling across many GPUs

• Best for: maximizing throughput with many concurrent requests

• Flag: --pipeline-parallel-size N

Request → [Layers 1-20 on GPU0] → [Layers 21-40 on GPU1] → Output

Quick Decision

Scenario	Recommendation
Model fits in 1 GPU	Single GPU (no parallelism)
Model too big for 1 GPU, 2 GPUs same machine	TP=2
Want max throughput, many users	TP for latency, PP for scale
2 GPUs, low latency priority	TP=2

For 2-GPU setups on a single VPS, TP=2 is almost always the better choice.

---

3. Single GPU Deployment

Best for 7B–32B models that fit in one GPU's VRAM.

Directory Structure

mkdir -p ~/vllm-stack && cd ~/vllm-stack

RTX 3090 (Ampere, stable image)

services:
  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-7B-Instruct-AWQ
      --quantization awq
      --max-model-len 8192
      --gpu-memory-utilization 0.90
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

Blackwell (RTX 5060 Ti / PRO 4000, nightly image)

services:
  vllm:
    image: vllm/vllm-openai:nightly
    container_name: vllm
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-7B-Instruct-AWQ
      --quantization awq
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --enforce-eager
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

Create .env

echo "HF_TOKEN=hf_your_token_here" > .env

Launch

docker compose up -d
docker compose logs -f vllm

Wait for: Application startup complete and Uvicorn running on http://0.0.0.0:8000

---

4. Dual GPU — Tensor Parallel (TP)

For models too large for one GPU (e.g. 70B on 2× RTX PRO 4000), or for lower latency.

docker-compose.yml — TP=2

services:
  vllm:
    image: vllm/vllm-openai:nightly   # use :latest for RTX 3090
    container_name: vllm-tp2
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0,1
      - NCCL_P2P_DISABLE=1        # required for RTX consumer cards
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-32B-Instruct-AWQ
      --quantization awq
      --tensor-parallel-size 2
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --enforce-eager
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0', '1']
              capabilities: [gpu]

Critical Flags for Multi-GPU on RTX Cards

Flag	Why
NCCL_P2P_DISABLE=1	RTX consumer cards lack proper P2P; without this, TP hangs
--enforce-eager	Avoids CUDA graph issues on Blackwell multi-GPU
--tensor-parallel-size 2	Splits model across both GPUs

RTX 3090 / 5060 Ti / PRO 4000: Always set NCCL_P2P_DISABLE=1 for TP. This is the #1 cause of multi-GPU hangs.

---

5. Dual GPU — Pipeline Parallel (PP)

Use PP when you want to run two separate model replicas' worth of layers split, typically for throughput scaling. For most 2-GPU VPS cases, TP is preferred — but here's PP for completeness.

docker-compose.yml — PP=2

services:
  vllm:
    image: vllm/vllm-openai:nightly   # :latest for RTX 3090
    container_name: vllm-pp2
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0,1
      - NCCL_P2P_DISABLE=1
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-32B-Instruct-AWQ
      --quantization awq
      --pipeline-parallel-size 2
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --enforce-eager
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0', '1']
              capabilities: [gpu]

TP vs PP — Same Hardware, Different Behavior

	TP=2	PP=2
Latency	Lower (parallel compute)	Higher (sequential stages)
Throughput	Good	Higher under heavy concurrency
Interconnect sensitivity	High (needs fast link)	Lower
Best for 2-GPU VPS	✅ Usually	Specific cases

---

6. Adding Open WebUI

Open WebUI gives a ChatGPT-style interface in front of your vLLM API.

Full docker-compose.yml (vLLM + Open WebUI)

services:
  vllm:
    image: vllm/vllm-openai:nightly   # :latest for RTX 3090
    container_name: vllm
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-7B-Instruct-AWQ
      --quantization awq
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --enforce-eager
      --api-key sk-your-secret-key
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - vllm
    ports:
      - "3000:8080"
    environment:
      - OPENAI_API_BASE_URL=http://vllm:8000/v1
      - OPENAI_API_KEY=sk-your-secret-key
    volumes:
      - ./open-webui-data:/app/backend/data

volumes:
  open-webui-data:

Launch & Access

docker compose up -d

• API endpoint: http://<your-server-ip>:8000/v1

• Web UI: http://<your-server-ip>:3000

On first WebUI visit, create an admin account. Models from vLLM appear automatically.

---

7. Model Sizing Guide

GPU (VRAM)	7-8B	14B	32B	70B
RTX 3050 (8GB)	✅ AWQ (tight)	❌	❌	❌
RTX 5060 Ti (16GB)	✅ 1 GPU	✅ 1 GPU (AWQ)	TP=2	❌
RTX PRO 2000 (16GB)	✅ 1 GPU	✅ 1 GPU (AWQ)	TP=2	❌
RTX 3090 (24GB)	✅ 1 GPU	✅ 1 GPU	✅ 1 GPU (tight)	TP=2+
RTX PRO 4000 (24GB)	✅ 1 GPU	✅ 1 GPU	✅ 1 GPU (tight)	TP=2+
RTX 5090 (32GB)	✅ 1 GPU	✅ 1 GPU	✅ 1 GPU (AWQ)	TP=2 (AWQ)
RTX PRO 5000 (48GB)	✅ 1 GPU	✅ 1 GPU	✅ 1 GPU	TP=2 (AWQ)
RTX PRO 6000 (96GB)	✅ 1 GPU	✅ 1 GPU	✅ 1 GPU	✅ 1 GPU (AWQ)
2× RTX 3090 (48GB)	✅	✅	✅	TP=2 (AWQ)
2× RTX 5090 (64GB)	✅	✅	✅	TP=2 (AWQ)
2× RTX PRO 4000 (48GB)	✅	✅	✅	TP=2 (AWQ)
2× RTX PRO 5000 (96GB)	✅	✅	✅	TP=2 (AWQ/FP16)
2× RTX PRO 6000 (192GB)	✅	✅	✅	✅ TP=2 (FP16)

RTX PRO 6000 (96GB) single-GPU runs a 70B AWQ model without parallelism — simplest setup for large models. For 70B FP16, use 2× PRO 6000 with TP=2.

Recommended Models

Use Case	Model	Notes
General chat (English)	Qwen/Qwen2.5-7B-Instruct-AWQ	Fast, fits 1 GPU
General chat (larger)	Qwen/Qwen2.5-32B-Instruct-AWQ	24GB GPU or TP=2
Thai language	scb10x/llama-3.1-typhoon2-8b-instruct	Thai-optimized
Coding	Qwen/Qwen2.5-Coder-7B-Instruct-AWQ	Code tasks

Tuning --max-model-len

• Longer context = more KV cache memory = fewer concurrent requests

• Start with 8192; increase only if needed

• If you hit out-of-memory, lower --max-model-len or --gpu-memory-utilization

---

8. RTX PRO 6000 — Running 70B on a Single GPU

The RTX PRO 6000 Blackwell (96GB) is large enough to run a 70B model without any parallelism — the simplest possible large-model deployment. No NCCL_P2P_DISABLE, no tensor parallel tuning.

docker-compose.yml — 70B AWQ on 1× PRO 6000

services:
  vllm:
    image: vllm/vllm-openai:nightly
    container_name: vllm-70b
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model Qwen/Qwen2.5-72B-Instruct-AWQ
      --quantization awq
      --max-model-len 16384
      --gpu-memory-utilization 0.92
      --enforce-eager
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

VRAM Budget on 96GB

Component	70B AWQ	70B FP16
Model weights	~40 GB	~140 GB (needs 2 GPU)
KV cache (16K ctx)	~15-20 GB	—
Overhead	~5 GB	—
Fits in 96GB?	✅ Yes, comfortably	❌ Needs TP=2

For 70B FP16, use 2× PRO 6000 with --tensor-parallel-size 2 (see Section 4). For most use cases, 70B AWQ on a single PRO 6000 is the sweet spot — full quality with simple setup.

Larger Context

With 96GB you can push context much further:

--max-model-len 32768    # 32K context, still fits with 70B AWQ

Longer context reduces concurrent capacity but is great for document analysis / RAG.

---

9. MIG — Splitting a GPU (PRO 6000)

MIG (Multi-Instance GPU) lets you partition one PRO 6000 into multiple isolated GPU instances — useful for running several smaller models or serving multiple isolated workloads on one card.

Only RTX PRO 6000 Blackwell supports MIG among the GPUs here. RTX 3050, 3090, 5060 Ti, 5090, PRO 2000, PRO 4000, PRO 5000 do not support MIG.

When to Use MIG

Use MIG	Don't use MIG
Run multiple small models (7-8B each) isolated	Running one large model (use full GPU)
Multi-tenant isolation on one card	Need max performance for single workload
Guaranteed resource partitioning	Workload varies / needs full VRAM sometimes

Enable MIG (on the host/VM with the GPU)

# Enable MIG mode (requires no running GPU workloads)
sudo nvidia-smi -i 0 -mig 1

# Reboot may be required
sudo reboot

Create MIG Instances

# List available MIG profiles
nvidia-smi mig -lgip

# Example: create profiles on the 96GB card
# Profile IDs vary; common split into instances:
sudo nvidia-smi mig -cgi 9,9,9,9 -C   # creates multiple instances

# Verify
nvidia-smi -L

Output shows MIG device UUIDs:

GPU 0: NVIDIA RTX PRO 6000 (UUID: GPU-xxxx)
  MIG 2g.24gb Device 0: (UUID: MIG-xxxx)
  MIG 2g.24gb Device 1: (UUID: MIG-yyyy)
  ...

Assign a MIG Instance to a Container

Use the MIG UUID in NVIDIA_VISIBLE_DEVICES:

services:
  vllm-model-a:
    image: vllm/vllm-openai:nightly
    environment:
      - NVIDIA_VISIBLE_DEVICES=MIG-xxxx   # specific MIG instance
    # ... rest of config

Disable MIG

sudo nvidia-smi -i 0 -mig 0
sudo reboot

Note: MIG instances have fixed VRAM/compute slices and cannot use NVLink between them. For a single large model, MIG is counterproductive — use the full GPU instead.

---

10. Using Other Models

You're not limited to Qwen. vLLM supports most popular open models. Here's how to use any model.

Finding the Model Name

Use the HuggingFace repo path as the --model value. Find it on https://huggingface.co — the path after huggingface.co/.

Example: meta-llama/Llama-3.1-8B-Instruct → --model meta-llama/Llama-3.1-8B-Instruct

Popular Models by Category

Category	Model (HuggingFace path)	Size	Notes
General (English)	meta-llama/Llama-3.1-8B-Instruct	8B	Gated — accept license first
General	mistralai/Mistral-7B-Instruct-v0.3	7B	Open
General (large)	meta-llama/Llama-3.1-70B-Instruct	70B	Needs 2 GPU or PRO 6000
Thai	scb10x/llama-3.1-typhoon2-8b-instruct	8B	Thai-optimized
Thai (large)	scb10x/llama-3.1-typhoon2-70b-instruct	70B	Thai, needs big VRAM
Coding	Qwen/Qwen2.5-Coder-32B-Instruct	32B	Strong coding
Coding (small)	Qwen/Qwen2.5-Coder-7B-Instruct	7B	Fast
Reasoning	deepseek-ai/DeepSeek-R1-Distill-Qwen-32B	32B	Chain-of-thought
Vision (VL)	Qwen/Qwen2.5-VL-7B-Instruct	7B	Image understanding

Model availability and exact names change over time — always verify the repo exists on HuggingFace before deploying.

Quantized vs Full Precision

Full precision (FP16) — use the base repo, no quantization flag:

command: >
  --model mistralai/Mistral-7B-Instruct-v0.3
  --max-model-len 8192
  --gpu-memory-utilization 0.90

AWQ quantized — look for an -AWQ variant repo, add --quantization awq:

command: >
  --model Qwen/Qwen2.5-32B-Instruct-AWQ
  --quantization awq
  --max-model-len 8192
  --gpu-memory-utilization 0.90

GPTQ quantized — look for -GPTQ variant, add --quantization gptq:

command: >
  --model TheBloke/Llama-2-13B-chat-GPTQ
  --quantization gptq
  --max-model-len 4096

How to Find Quantized Versions

On HuggingFace, search the model name + "AWQ" or "GPTQ". Common quantizers:

• Official repos (e.g. Qwen/...-AWQ)

• TheBloke/... (many GPTQ/AWQ conversions)

• casperhansen/... (AWQ)

Gated Models (Llama, etc.)

Some models require accepting a license:

1. Visit the model page on HuggingFace (e.g. meta-llama/Llama-3.1-8B-Instruct)

2. Click "Agree and access repository"

3. Ensure your HF_TOKEN in .env belongs to the account that accepted

Without this, you'll get a 401 Gated repo error in the logs.

Switching Models — Just Edit and Restart

# 1. Edit the --model line in docker-compose.yml
nano docker-compose.yml

# 2. Restart
docker compose down && docker compose up -d

# 3. Watch it download + load
docker compose logs -f vllm

Models cache in ./models, so re-using a previously downloaded model is instant.

Checklist Before Deploying a New Model

Check	How
Does it fit your VRAM?	See Section 7 sizing table
Is there an AWQ/GPTQ version?	Search HF for -AWQ / -GPTQ
Is it gated?	Check for "Agree to access" on HF page
Blackwell GPU?	Use :nightly image
Multi-GPU needed?	Add --tensor-parallel-size N + NCCL_P2P_DISABLE=1

---

11. Testing & Troubleshooting

Test the API

# List models
curl http://localhost:8000/v1/models

# Chat completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-secret-key" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct-AWQ",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Common Issues

"No supported device found" / CUDA error on Blackwell

Cause: Using stable image on Blackwell GPU. Fix: Switch to vllm/vllm-openai:nightly.

Multi-GPU TP hangs at startup

Cause: Missing NCCL_P2P_DISABLE=1. Fix: Add - NCCL_P2P_DISABLE=1 to environment and add --enforce-eager.

Out of Memory (OOM) / KV cache error

ValueError: No available memory for the cache blocks

Fix:

• Increase --gpu-memory-utilization to 0.92

• OR decrease --max-model-len

• OR use a smaller / more quantized model

Container exits immediately

docker compose logs vllm

Check for: wrong model name, missing HF_TOKEN, or gated model access.

Model download is slow / fails

• Ensure HF_TOKEN is set in .env

• For gated models (e.g. Llama), accept the license on HuggingFace first

• Models cache to ./models — subsequent restarts are fast

Useful Commands

# Watch GPU usage live
watch -n1 nvidia-smi

# Container logs
docker compose logs -f vllm

# Restart after config change
docker compose down && docker compose up -d

# Check container status
docker compose ps

GPU Memory Reference (idle vs loaded)

nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu --format=csv

---

Quick Start Summary

# 1. Setup
mkdir -p ~/vllm-stack && cd ~/vllm-stack
echo "HF_TOKEN=hf_xxx" > .env

# 2. Create docker-compose.yml (see sections above)
nano docker-compose.yml

# 3. Launch
docker compose up -d

# 4. Test
curl http://localhost:8000/v1/models

# 5. Access Web UI
# http://<server-ip>:3000

Cheat Sheet

Your Setup	Image	Key Flags
1× RTX 3050 / 3090 (Ampere)	:latest	--gpu-memory-utilization 0.90
1× Blackwell (5060 Ti / 5090 / PRO 2000/4000/5000/6000)	:nightly	+ --enforce-eager
2× any (TP)	match GPU	--tensor-parallel-size 2 NCCL_P2P_DISABLE=1 --enforce-eager
2× any (PP)	match GPU	--pipeline-parallel-size 2 NCCL_P2P_DISABLE=1

---

Need help? Contact ReadyIDC Support: [email protected]