How to using api integration guide

Connect to your vLLM endpoint from code. The vLLM API is OpenAI-compatible, so any OpenAI SDK works. Examples in curl, Python, and JavaScript.

Prerequisite: A running vLLM endpoint (see the vLLM Docker Deployment Guide).

1. Endpoint Basics

Item	Value
Base URL	http://<server-ip>:8000/v1 or https://api.yourdomain.com/v1
Auth	Authorization: Bearer <your-api-key>
Format	OpenAI-compatible
Main endpoints	/v1/chat/completions, /v1/completions, /v1/models, /v1/embeddings

List Available Models

curl http://<server-ip>:8000/v1/models \
  -H "Authorization: Bearer sk-your-key"

The id field in the response is the model name to use in requests.

2. curl

Basic Chat Completion

curl http://<server-ip>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-key" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct-AWQ",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain Docker in one sentence."}
    ],
    "temperature": 0.7,
    "max_tokens": 256
  }'

Extract Just the Answer (with jq)

curl -s http://<server-ip>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-key" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct-AWQ",
    "messages": [{"role": "user", "content": "Hello"}]
  }' | jq -r '.choices[0].message.content'

Install the OpenAI SDK

Python

pip install openai

Node.js

npm install openai

3. Python

Basic Chat

from openai import OpenAI

client = OpenAI(
    base_url="http://<server-ip>:8000/v1",
    api_key="sk-your-key",
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct-AWQ",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain Docker in one sentence."},
    ],
    temperature=0.7,
    max_tokens=256,
)

print(response.choices[0].message.content)

Multi-Turn Conversation

from openai import OpenAI

client = OpenAI(base_url="http://<server-ip>:8000/v1", api_key="sk-your-key")

messages = [{"role": "system", "content": "You are a helpful assistant."}]

while True:
    user_input = input("You: ")
    if user_input.lower() in ("quit", "exit"):
        break

    messages.append({"role": "user", "content": user_input})

    response = client.chat.completions.create(
        model="Qwen/Qwen2.5-7B-Instruct-AWQ",
        messages=messages,
    )

    answer = response.choices[0].message.content
    print(f"Assistant: {answer}")
    messages.append({"role": "assistant", "content": answer})

Using requests (no SDK)

import requests

resp = requests.post(
    "http://<server-ip>:8000/v1/chat/completions",
    headers={"Authorization": "Bearer sk-your-key"},
    json={
        "model": "Qwen/Qwen2.5-7B-Instruct-AWQ",
        "messages": [{"role": "user", "content": "Hello"}],
    },
)
print(resp.json()["choices"][0]["message"]["content"])

4. JavaScript / Node.js

Basic Chat

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://<server-ip>:8000/v1",
  apiKey: "sk-your-key",
});

const response = await client.chat.completions.create({
  model: "Qwen/Qwen2.5-7B-Instruct-AWQ",
  messages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: "Explain Docker in one sentence." },
  ],
  temperature: 0.7,
  max_tokens: 256,
});

console.log(response.choices[0].message.content);

Using fetch (no SDK)

const response = await fetch("http://<server-ip>:8000/v1/chat/completions", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Authorization": "Bearer sk-your-key",
  },
  body: JSON.stringify({
    model: "Qwen/Qwen2.5-7B-Instruct-AWQ",
    messages: [{ role: "user", content: "Hello" }],
  }),
});

const data = await response.json();
console.log(data.choices[0].message.content);

5. Streaming Responses

Streaming returns tokens as they're generated (like ChatGPT typing). Set stream: true.

Python (streaming)

from openai import OpenAI

client = OpenAI(base_url="http://<server-ip>:8000/v1", api_key="sk-your-key")

stream = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct-AWQ",
    messages=[{"role": "user", "content": "Write a short poem about servers."}],
    stream=True,
)

for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)
print()

JavaScript (streaming)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://<server-ip>:8000/v1",
  apiKey: "sk-your-key",
});

const stream = await client.chat.completions.create({
  model: "Qwen/Qwen2.5-7B-Instruct-AWQ",
  messages: [{ role: "user", content: "Write a short poem about servers." }],
  stream: true,
});

for await (const chunk of stream) {
  const content = chunk.choices[0]?.delta?.content || "";
  process.stdout.write(content);
}
console.log();

curl (streaming)

curl http://<server-ip>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-key" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct-AWQ",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }' --no-buffer

If using a reverse proxy, ensure proxy_buffering off; is set (see nginx guide) or streaming won't work.

6. Embeddings API

If you run an embedding server (see AnythingLLM/RAG guide), call it the same way.

Python

from openai import OpenAI

client = OpenAI(base_url="http://<server-ip>:8001/v1", api_key="none")

response = client.embeddings.create(
    model="BAAI/bge-m3",
    input=["First document text", "Second document text"],
)

for item in response.data:
    print(len(item.embedding), "dimensions")

curl

curl http://<server-ip>:8001/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "BAAI/bge-m3",
    "input": "Text to embed"
  }'

7. Common Parameters

Parameter	Type	Description
model	string	Model name (from /v1/models)
messages	array	Conversation history (chat endpoint)
temperature	float	Randomness 0.0-2.0 (lower = focused, higher = creative)
max_tokens	int	Max tokens to generate
top_p	float	Nucleus sampling (0.0-1.0)
stream	bool	Stream tokens as generated
stop	array	Stop sequences
frequency_penalty	float	Reduce repetition (-2.0 to 2.0)
presence_penalty	float	Encourage new topics (-2.0 to 2.0)

Recommended Settings by Use Case

Use Case	temperature	Notes
Factual Q&A	0.1-0.3	Consistent, focused
Coding	0.0-0.2	Deterministic
Creative writing	0.7-1.0	More varied
Structured/JSON output	0.0-0.1	Predictable format

JSON Output

Instruct the model and (optionally) use guided decoding:

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct-AWQ",
    messages=[
        {"role": "user", "content": "Return a JSON object with keys 'name' and 'age' for: John, 30"}
    ],
    temperature=0.1,
    extra_body={"guided_json": {
        "type": "object",
        "properties": {
            "name": {"type": "string"},
            "age": {"type": "integer"}
        },
        "required": ["name", "age"]
    }}
)

guided_json is a vLLM extension via extra_body — enforces valid JSON matching your schema.

8. Error Handling

Python

from openai import OpenAI, APIError, APIConnectionError, RateLimitError

client = OpenAI(base_url="http://<server-ip>:8000/v1", api_key="sk-your-key")

try:
    response = client.chat.completions.create(
        model="Qwen/Qwen2.5-7B-Instruct-AWQ",
        messages=[{"role": "user", "content": "Hello"}],
    )
    print(response.choices[0].message.content)
except APIConnectionError:
    print("Cannot reach the server — check URL and that vLLM is running")
except RateLimitError:
    print("Rate limited — slow down requests")
except APIError as e:
    print(f"API error: {e}")

Common Errors

Error	Cause	Fix
Connection refused	vLLM not running / wrong URL	Check docker compose ps, verify URL
401 Unauthorized	Wrong/missing API key	Match --api-key value
404 model not found	Wrong model name	Use exact name from /v1/models
400 context length exceeded	Input too long	Reduce input or raise --max-model-len
Timeout	Long generation	Increase client timeout, reduce max_tokens

Set Client Timeout

client = OpenAI(
    base_url="http://<server-ip>:8000/v1",
    api_key="sk-your-key",
    timeout=120.0,   # seconds
)

const client = new OpenAI({
  baseURL: "http://<server-ip>:8000/v1",
  apiKey: "sk-your-key",
  timeout: 120000,  // milliseconds
});

Quick Reference

# Health check
curl http://<server-ip>:8000/v1/models -H "Authorization: Bearer sk-your-key"

# Quick chat test
curl http://<server-ip>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-key" \
  -d '{"model":"MODEL_NAME","messages":[{"role":"user","content":"hi"}]}' \
  | jq -r '.choices[0].message.content'

Language	SDK Install	Import
Python	pip install openai	from openai import OpenAI
Node.js	npm install openai	import OpenAI from "openai"
Any	—	Direct HTTP to /v1/chat/completions