For the complete documentation index, see llms.txt. This page is also available as Markdown.

How to using Local LLM with vLLM

Self-hosted LLM serving with vLLM on your GPU VPS OpenAI-compatible API + Open WebUI Covers single-GPU, multi-GPU, Tensor Parallelism (TP) and Pipeline Parallelism (PP)


Supported GPUs

GPU
VRAM
Architecture
vLLM Image

RTX 3050

8 GB

Ampere (sm_86)

vllm/vllm-openai:latest (stable)

RTX 3090

24 GB

Ampere (sm_86)

vllm/vllm-openai:latest (stable)

RTX 5060 Ti

16 GB

Blackwell (sm_120)

vllm/vllm-openai:nightly (required)

RTX 5090

32 GB

Blackwell (sm_120)

vllm/vllm-openai:nightly (required)

RTX PRO 2000 Blackwell

16 GB

Blackwell (sm_120)

vllm/vllm-openai:nightly (required)

RTX PRO 4000 Blackwell

24 GB

Blackwell (sm_120)

vllm/vllm-openai:nightly (required)

RTX PRO 5000 Blackwell

48 GB

Blackwell (sm_120)

vllm/vllm-openai:nightly (required)

RTX PRO 6000 Blackwell

96 GB

Blackwell (sm_120)

vllm/vllm-openai:nightly (required)

RTX PRO 6000 (96GB) is the flagship — its large VRAM runs 70B models on a single GPU (AWQ) or even FP16 70B with TP=2.

RTX 5090 (32GB) is the highest-VRAM consumer card here — the value pick for 32B AWQ on a single GPU, or 70B AWQ with TP=2 across a 2× 5090 VPS.


Model Size ↔ GPU Selector

Pick the smallest GPU your model fits on at your target precision and context length. This table summarizes how large a model each GPU in the fleet can run.

VRAM Rule of Thumb

Precision
Model weights
Per 1B params

FP16 / BF16

full precision

~2.0 GB

FP8

8-bit

~1.0 GB

AWQ / GPTQ

4-bit

~0.5–0.6 GB

Always budget KV cache (scales with --max-model-len and concurrent requests) plus 2–4 GB overhead on top of the model weights.

Single-GPU — Max Model per Card

GPU (VRAM)
Image
Comfortable
Tight / short context
Bigger models

RTX 3050 (8 GB)

:latest

3–4B FP16 · 7–8B AWQ

7–8B AWQ @ 2–4K ctx

Use a larger card

RTX 5060 Ti (16 GB)

:nightly

7–8B FP16 · 14B AWQ

14B AWQ @ 4–8K

32B → TP=2

RTX PRO 2000 (16 GB)

:nightly

7–8B FP16 · 14B AWQ

14B AWQ @ 4–8K

32B → TP=2

RTX 3090 (24 GB)

:latest

14B FP16 · 32B AWQ

32B AWQ @ 8K

70B → TP=2

RTX PRO 4000 (24 GB)

:nightly

14B FP16 · 32B AWQ

32B AWQ @ 8K

70B → TP=2

RTX 5090 (32 GB)

:nightly

32B AWQ · 14B FP16

32B AWQ @ 16K

70B → TP=2

RTX PRO 5000 (48 GB)

:nightly

32B FP16 · 70B AWQ (tight)

70B AWQ @ 4–8K

70B FP16 → TP=2

RTX PRO 6000 (96 GB)

:nightly

70B AWQ · 32B FP16

70B AWQ @ 32K

70B FP16 → 2× PRO 6000

Dual-GPU — When One Card Isn't Enough (TP=2)

Dual-GPU (total VRAM)
Runs
Notes

2× RTX 3090 (48 GB)

70B AWQ, TP=2

Requires NCCL_P2P_DISABLE=1

2× RTX 5090 (64 GB)

70B AWQ, TP=2 · 32B FP16

Best consumer TP option

2× RTX PRO 4000 (48 GB)

70B AWQ, TP=2

NCCL_P2P_DISABLE=1

2× RTX PRO 5000 (96 GB)

70B AWQ/FP16, TP=2

Full-precision 70B headroom

2× RTX PRO 6000 (192 GB)

70B FP16, TP=2

Full-quality 70B FP16

Quick Pick

Requirement
Recommended GPU

Dev/test, small chatbot (≤8B)

RTX 3050 or RTX 5060 Ti

General 14–32B, single card

RTX 3090 / PRO 4000 (32B AWQ) or RTX 5090 (most headroom)

70B, single card

RTX PRO 6000 (AWQ) or RTX PRO 5000 (AWQ, tight)

70B, budget dual-GPU

2× RTX 5090 / 2× PRO 4000 (TP=2)

70B full precision (FP16)

2× RTX PRO 6000 or 2× RTX PRO 5000 (TP=2)

For deeper flag and VRAM-budget details, see Section 7 and Section 8.


1. Prerequisites

Verify GPU is visible

You should see your GPU(s) listed. If not, contact support.

Install Docker + NVIDIA Container Toolkit

Verify Docker can access GPU

Get a HuggingFace Token (for gated/most models)

Create a token at https://huggingface.co/settings/tokens — you'll use it as HF_TOKEN.


2. Understanding TP vs PP

Tensor Parallelism (TP)

Splits each layer across multiple GPUs. All GPUs work on the same request simultaneously.

  • Use when: model is too large for one GPU, OR you want lower latency

  • Best for: GPUs in the same machine with fast interconnect

  • Flag: --tensor-parallel-size N

  • Requirement: number of attention heads must be divisible by N

Pipeline Parallelism (PP)

Splits different layers onto different GPUs. Each GPU handles a stage.

  • Use when: TP isn't efficient (slow interconnect), or scaling across many GPUs

  • Best for: maximizing throughput with many concurrent requests

  • Flag: --pipeline-parallel-size N

Quick Decision

Scenario
Recommendation

Model fits in 1 GPU

Single GPU (no parallelism)

Model too big for 1 GPU, 2 GPUs same machine

TP=2

Want max throughput, many users

TP for latency, PP for scale

2 GPUs, low latency priority

TP=2

For 2-GPU setups on a single VPS, TP=2 is almost always the better choice.


3. Single GPU Deployment

Best for 7B–32B models that fit in one GPU's VRAM.

Directory Structure

Create .env

Launch

Wait for: Application startup complete and Uvicorn running on http://0.0.0.0:8000


4. Dual GPU — Tensor Parallel (TP)

For models too large for one GPU (e.g. 70B on 2× RTX PRO 4000), or for lower latency.

docker-compose.yml — TP=2

Critical Flags for Multi-GPU on RTX Cards

Flag
Why

NCCL_P2P_DISABLE=1

RTX consumer cards lack proper P2P; without this, TP hangs

--enforce-eager

Avoids CUDA graph issues on Blackwell multi-GPU

--tensor-parallel-size 2

Splits model across both GPUs

RTX 3090 / 5060 Ti / PRO 4000: Always set NCCL_P2P_DISABLE=1 for TP. This is the #1 cause of multi-GPU hangs.


5. Dual GPU — Pipeline Parallel (PP)

Use PP when you want to run two separate model replicas' worth of layers split, typically for throughput scaling. For most 2-GPU VPS cases, TP is preferred — but here's PP for completeness.

docker-compose.yml — PP=2

TP vs PP — Same Hardware, Different Behavior

TP=2
PP=2

Latency

Lower (parallel compute)

Higher (sequential stages)

Throughput

Good

Higher under heavy concurrency

Interconnect sensitivity

High (needs fast link)

Lower

Best for 2-GPU VPS

✅ Usually

Specific cases


6. Adding Open WebUI

Open WebUI gives a ChatGPT-style interface in front of your vLLM API.

Full docker-compose.yml (vLLM + Open WebUI)

Launch & Access

  • API endpoint: http://<your-server-ip>:8000/v1

  • Web UI: http://<your-server-ip>:3000

On first WebUI visit, create an admin account. Models from vLLM appear automatically.


7. Model Sizing Guide

GPU (VRAM)
7-8B
14B
32B
70B

RTX 3050 (8GB)

✅ AWQ (tight)

RTX 5060 Ti (16GB)

✅ 1 GPU

✅ 1 GPU (AWQ)

TP=2

RTX PRO 2000 (16GB)

✅ 1 GPU

✅ 1 GPU (AWQ)

TP=2

RTX 3090 (24GB)

✅ 1 GPU

✅ 1 GPU

✅ 1 GPU (tight)

TP=2+

RTX PRO 4000 (24GB)

✅ 1 GPU

✅ 1 GPU

✅ 1 GPU (tight)

TP=2+

RTX 5090 (32GB)

✅ 1 GPU

✅ 1 GPU

✅ 1 GPU (AWQ)

TP=2 (AWQ)

RTX PRO 5000 (48GB)

✅ 1 GPU

✅ 1 GPU

✅ 1 GPU

TP=2 (AWQ)

RTX PRO 6000 (96GB)

✅ 1 GPU

✅ 1 GPU

✅ 1 GPU

✅ 1 GPU (AWQ)

2× RTX 3090 (48GB)

TP=2 (AWQ)

2× RTX 5090 (64GB)

TP=2 (AWQ)

2× RTX PRO 4000 (48GB)

TP=2 (AWQ)

2× RTX PRO 5000 (96GB)

TP=2 (AWQ/FP16)

2× RTX PRO 6000 (192GB)

✅ TP=2 (FP16)

RTX PRO 6000 (96GB) single-GPU runs a 70B AWQ model without parallelism — simplest setup for large models. For 70B FP16, use 2× PRO 6000 with TP=2.

Use Case
Model
Notes

General chat (English)

Qwen/Qwen2.5-7B-Instruct-AWQ

Fast, fits 1 GPU

General chat (larger)

Qwen/Qwen2.5-32B-Instruct-AWQ

24GB GPU or TP=2

Thai language

scb10x/llama-3.1-typhoon2-8b-instruct

Thai-optimized

Coding

Qwen/Qwen2.5-Coder-7B-Instruct-AWQ

Code tasks

Tuning --max-model-len

  • Longer context = more KV cache memory = fewer concurrent requests

  • Start with 8192; increase only if needed

  • If you hit out-of-memory, lower --max-model-len or --gpu-memory-utilization


8. RTX PRO 6000 — Running 70B on a Single GPU

The RTX PRO 6000 Blackwell (96GB) is large enough to run a 70B model without any parallelism — the simplest possible large-model deployment. No NCCL_P2P_DISABLE, no tensor parallel tuning.

docker-compose.yml — 70B AWQ on 1× PRO 6000

VRAM Budget on 96GB

Component
70B AWQ
70B FP16

Model weights

~40 GB

~140 GB (needs 2 GPU)

KV cache (16K ctx)

~15-20 GB

Overhead

~5 GB

Fits in 96GB?

✅ Yes, comfortably

❌ Needs TP=2

For 70B FP16, use 2× PRO 6000 with --tensor-parallel-size 2 (see Section 4). For most use cases, 70B AWQ on a single PRO 6000 is the sweet spot — full quality with simple setup.

Larger Context

With 96GB you can push context much further:

Longer context reduces concurrent capacity but is great for document analysis / RAG.


9. MIG — Splitting a GPU (PRO 6000)

MIG (Multi-Instance GPU) lets you partition one PRO 6000 into multiple isolated GPU instances — useful for running several smaller models or serving multiple isolated workloads on one card.

Only RTX PRO 6000 Blackwell supports MIG among the GPUs here. RTX 3050, 3090, 5060 Ti, 5090, PRO 2000, PRO 4000, PRO 5000 do not support MIG.

When to Use MIG

Use MIG
Don't use MIG

Run multiple small models (7-8B each) isolated

Running one large model (use full GPU)

Multi-tenant isolation on one card

Need max performance for single workload

Guaranteed resource partitioning

Workload varies / needs full VRAM sometimes

Enable MIG (on the host/VM with the GPU)

Create MIG Instances

Output shows MIG device UUIDs:

Assign a MIG Instance to a Container

Use the MIG UUID in NVIDIA_VISIBLE_DEVICES:

Disable MIG

Note: MIG instances have fixed VRAM/compute slices and cannot use NVLink between them. For a single large model, MIG is counterproductive — use the full GPU instead.


10. Using Other Models

You're not limited to Qwen. vLLM supports most popular open models. Here's how to use any model.

Finding the Model Name

Use the HuggingFace repo path as the --model value. Find it on https://huggingface.co — the path after huggingface.co/.

Example: meta-llama/Llama-3.1-8B-Instruct--model meta-llama/Llama-3.1-8B-Instruct

Category
Model (HuggingFace path)
Size
Notes

General (English)

meta-llama/Llama-3.1-8B-Instruct

8B

Gated — accept license first

General

mistralai/Mistral-7B-Instruct-v0.3

7B

Open

General (large)

meta-llama/Llama-3.1-70B-Instruct

70B

Needs 2 GPU or PRO 6000

Thai

scb10x/llama-3.1-typhoon2-8b-instruct

8B

Thai-optimized

Thai (large)

scb10x/llama-3.1-typhoon2-70b-instruct

70B

Thai, needs big VRAM

Coding

Qwen/Qwen2.5-Coder-32B-Instruct

32B

Strong coding

Coding (small)

Qwen/Qwen2.5-Coder-7B-Instruct

7B

Fast

Reasoning

deepseek-ai/DeepSeek-R1-Distill-Qwen-32B

32B

Chain-of-thought

Vision (VL)

Qwen/Qwen2.5-VL-7B-Instruct

7B

Image understanding

Model availability and exact names change over time — always verify the repo exists on HuggingFace before deploying.

Quantized vs Full Precision

Full precision (FP16) — use the base repo, no quantization flag:

AWQ quantized — look for an -AWQ variant repo, add --quantization awq:

GPTQ quantized — look for -GPTQ variant, add --quantization gptq:

How to Find Quantized Versions

On HuggingFace, search the model name + "AWQ" or "GPTQ". Common quantizers:

  • Official repos (e.g. Qwen/...-AWQ)

  • TheBloke/... (many GPTQ/AWQ conversions)

  • casperhansen/... (AWQ)

Gated Models (Llama, etc.)

Some models require accepting a license:

  1. Visit the model page on HuggingFace (e.g. meta-llama/Llama-3.1-8B-Instruct)

  2. Click "Agree and access repository"

  3. Ensure your HF_TOKEN in .env belongs to the account that accepted

Without this, you'll get a 401 Gated repo error in the logs.

Switching Models — Just Edit and Restart

Models cache in ./models, so re-using a previously downloaded model is instant.

Checklist Before Deploying a New Model

Check
How

Does it fit your VRAM?

See Section 7 sizing table

Is there an AWQ/GPTQ version?

Search HF for -AWQ / -GPTQ

Is it gated?

Check for "Agree to access" on HF page

Blackwell GPU?

Use :nightly image

Multi-GPU needed?

Add --tensor-parallel-size N + NCCL_P2P_DISABLE=1


11. Testing & Troubleshooting

Test the API

Common Issues

"No supported device found" / CUDA error on Blackwell

Cause: Using stable image on Blackwell GPU. Fix: Switch to vllm/vllm-openai:nightly.

Multi-GPU TP hangs at startup

Cause: Missing NCCL_P2P_DISABLE=1. Fix: Add - NCCL_P2P_DISABLE=1 to environment and add --enforce-eager.

Out of Memory (OOM) / KV cache error

Fix:

  • Increase --gpu-memory-utilization to 0.92

  • OR decrease --max-model-len

  • OR use a smaller / more quantized model

Container exits immediately

Check for: wrong model name, missing HF_TOKEN, or gated model access.

Model download is slow / fails

  • Ensure HF_TOKEN is set in .env

  • For gated models (e.g. Llama), accept the license on HuggingFace first

  • Models cache to ./models — subsequent restarts are fast

Useful Commands

GPU Memory Reference (idle vs loaded)


Quick Start Summary

Cheat Sheet

Your Setup
Image
Key Flags

1× RTX 3050 / 3090 (Ampere)

:latest

--gpu-memory-utilization 0.90

1× Blackwell (5060 Ti / 5090 / PRO 2000/4000/5000/6000)

:nightly

+ --enforce-eager

2× any (TP)

match GPU

--tensor-parallel-size 2 NCCL_P2P_DISABLE=1 --enforce-eager

2× any (PP)

match GPU

--pipeline-parallel-size 2 NCCL_P2P_DISABLE=1


Need help? Contact ReadyIDC Support: [email protected]

Was this helpful?