> For the complete documentation index, see [llms.txt](https://docs.readyidc.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.readyidc.com/gpu-as-a-service/linux/docker/how-to-using-api-integration-guide.md).

# How to using api integration guide

Connect to your vLLM endpoint from code. The vLLM API is **OpenAI-compatible**, so any OpenAI SDK works.\
Examples in curl, Python, and JavaScript.

{% hint style="info" %}
**Prerequisite:** A running vLLM endpoint (see the vLLM Docker Deployment Guide).
{% endhint %}

## 1. Endpoint Basics

| Item           | Value                                                                     |
| -------------- | ------------------------------------------------------------------------- |
| Base URL       | `http://<server-ip>:8000/v1` or `https://api.yourdomain.com/v1`           |
| Auth           | `Authorization: Bearer <your-api-key>`                                    |
| Format         | OpenAI-compatible                                                         |
| Main endpoints | `/v1/chat/completions`, `/v1/completions`, `/v1/models`, `/v1/embeddings` |

### List Available Models

```bash
curl http://<server-ip>:8000/v1/models \
  -H "Authorization: Bearer sk-your-key"
```

The `id` field in the response is the model name to use in requests.

## 2. curl

### Basic Chat Completion

```bash
curl http://<server-ip>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-key" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct-AWQ",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain Docker in one sentence."}
    ],
    "temperature": 0.7,
    "max_tokens": 256
  }'
```

### Extract Just the Answer (with jq)

```bash
curl -s http://<server-ip>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-key" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct-AWQ",
    "messages": [{"role": "user", "content": "Hello"}]
  }' | jq -r '.choices[0].message.content'
```

## Install the OpenAI SDK

{% tabs %}
{% tab title="Python" %}

```bash
pip install openai
```

{% endtab %}

{% tab title="Node.js" %}

```bash
npm install openai
```

{% endtab %}
{% endtabs %}

## 3. Python

### Basic Chat

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://<server-ip>:8000/v1",
    api_key="sk-your-key",
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct-AWQ",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain Docker in one sentence."},
    ],
    temperature=0.7,
    max_tokens=256,
)

print(response.choices[0].message.content)
```

### Multi-Turn Conversation

```python
from openai import OpenAI

client = OpenAI(base_url="http://<server-ip>:8000/v1", api_key="sk-your-key")

messages = [{"role": "system", "content": "You are a helpful assistant."}]

while True:
    user_input = input("You: ")
    if user_input.lower() in ("quit", "exit"):
        break

    messages.append({"role": "user", "content": user_input})

    response = client.chat.completions.create(
        model="Qwen/Qwen2.5-7B-Instruct-AWQ",
        messages=messages,
    )

    answer = response.choices[0].message.content
    print(f"Assistant: {answer}")
    messages.append({"role": "assistant", "content": answer})
```

### Using requests (no SDK)

```python
import requests

resp = requests.post(
    "http://<server-ip>:8000/v1/chat/completions",
    headers={"Authorization": "Bearer sk-your-key"},
    json={
        "model": "Qwen/Qwen2.5-7B-Instruct-AWQ",
        "messages": [{"role": "user", "content": "Hello"}],
    },
)
print(resp.json()["choices"][0]["message"]["content"])
```

## 4. JavaScript / Node.js

### Basic Chat

```javascript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://<server-ip>:8000/v1",
  apiKey: "sk-your-key",
});

const response = await client.chat.completions.create({
  model: "Qwen/Qwen2.5-7B-Instruct-AWQ",
  messages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: "Explain Docker in one sentence." },
  ],
  temperature: 0.7,
  max_tokens: 256,
});

console.log(response.choices[0].message.content);
```

### Using fetch (no SDK)

```javascript
const response = await fetch("http://<server-ip>:8000/v1/chat/completions", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Authorization": "Bearer sk-your-key",
  },
  body: JSON.stringify({
    model: "Qwen/Qwen2.5-7B-Instruct-AWQ",
    messages: [{ role: "user", content: "Hello" }],
  }),
});

const data = await response.json();
console.log(data.choices[0].message.content);
```

## 5. Streaming Responses

Streaming returns tokens as they're generated (like ChatGPT typing). Set `stream: true`.

### Python (streaming)

```python
from openai import OpenAI

client = OpenAI(base_url="http://<server-ip>:8000/v1", api_key="sk-your-key")

stream = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct-AWQ",
    messages=[{"role": "user", "content": "Write a short poem about servers."}],
    stream=True,
)

for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)
print()
```

### JavaScript (streaming)

```javascript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://<server-ip>:8000/v1",
  apiKey: "sk-your-key",
});

const stream = await client.chat.completions.create({
  model: "Qwen/Qwen2.5-7B-Instruct-AWQ",
  messages: [{ role: "user", content: "Write a short poem about servers." }],
  stream: true,
});

for await (const chunk of stream) {
  const content = chunk.choices[0]?.delta?.content || "";
  process.stdout.write(content);
}
console.log();
```

### curl (streaming)

```bash
curl http://<server-ip>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-key" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct-AWQ",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }' --no-buffer
```

{% hint style="warning" %}
If using a reverse proxy, ensure `proxy_buffering off;` is set (see nginx guide) or streaming won't work.
{% endhint %}

## 6. Embeddings API

If you run an embedding server (see AnythingLLM/RAG guide), call it the same way.

### Python

```python
from openai import OpenAI

client = OpenAI(base_url="http://<server-ip>:8001/v1", api_key="none")

response = client.embeddings.create(
    model="BAAI/bge-m3",
    input=["First document text", "Second document text"],
)

for item in response.data:
    print(len(item.embedding), "dimensions")
```

### curl

```bash
curl http://<server-ip>:8001/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "BAAI/bge-m3",
    "input": "Text to embed"
  }'
```

## 7. Common Parameters

| Parameter           | Type   | Description                                             |
| ------------------- | ------ | ------------------------------------------------------- |
| `model`             | string | Model name (from `/v1/models`)                          |
| `messages`          | array  | Conversation history (chat endpoint)                    |
| `temperature`       | float  | Randomness 0.0-2.0 (lower = focused, higher = creative) |
| `max_tokens`        | int    | Max tokens to generate                                  |
| `top_p`             | float  | Nucleus sampling (0.0-1.0)                              |
| `stream`            | bool   | Stream tokens as generated                              |
| `stop`              | array  | Stop sequences                                          |
| `frequency_penalty` | float  | Reduce repetition (-2.0 to 2.0)                         |
| `presence_penalty`  | float  | Encourage new topics (-2.0 to 2.0)                      |

### Recommended Settings by Use Case

| Use Case               | temperature | Notes               |
| ---------------------- | ----------- | ------------------- |
| Factual Q\&A           | 0.1-0.3     | Consistent, focused |
| Coding                 | 0.0-0.2     | Deterministic       |
| Creative writing       | 0.7-1.0     | More varied         |
| Structured/JSON output | 0.0-0.1     | Predictable format  |

### JSON Output

Instruct the model and (optionally) use guided decoding:

```python
response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct-AWQ",
    messages=[
        {"role": "user", "content": "Return a JSON object with keys 'name' and 'age' for: John, 30"}
    ],
    temperature=0.1,
    extra_body={"guided_json": {
        "type": "object",
        "properties": {
            "name": {"type": "string"},
            "age": {"type": "integer"}
        },
        "required": ["name", "age"]
    }}
)
```

{% hint style="info" %}
`guided_json` is a vLLM extension via `extra_body` — enforces valid JSON matching your schema.
{% endhint %}

## 8. Error Handling

### Python

```python
from openai import OpenAI, APIError, APIConnectionError, RateLimitError

client = OpenAI(base_url="http://<server-ip>:8000/v1", api_key="sk-your-key")

try:
    response = client.chat.completions.create(
        model="Qwen/Qwen2.5-7B-Instruct-AWQ",
        messages=[{"role": "user", "content": "Hello"}],
    )
    print(response.choices[0].message.content)
except APIConnectionError:
    print("Cannot reach the server — check URL and that vLLM is running")
except RateLimitError:
    print("Rate limited — slow down requests")
except APIError as e:
    print(f"API error: {e}")
```

### Common Errors

| Error                         | Cause                        | Fix                                          |
| ----------------------------- | ---------------------------- | -------------------------------------------- |
| `Connection refused`          | vLLM not running / wrong URL | Check `docker compose ps`, verify URL        |
| `401 Unauthorized`            | Wrong/missing API key        | Match `--api-key` value                      |
| `404 model not found`         | Wrong model name             | Use exact name from `/v1/models`             |
| `400 context length exceeded` | Input too long               | Reduce input or raise `--max-model-len`      |
| `Timeout`                     | Long generation              | Increase client timeout, reduce `max_tokens` |

### Set Client Timeout

```python
client = OpenAI(
    base_url="http://<server-ip>:8000/v1",
    api_key="sk-your-key",
    timeout=120.0,   # seconds
)
```

```javascript
const client = new OpenAI({
  baseURL: "http://<server-ip>:8000/v1",
  apiKey: "sk-your-key",
  timeout: 120000,  // milliseconds
});
```

## Quick Reference

```bash
# Health check
curl http://<server-ip>:8000/v1/models -H "Authorization: Bearer sk-your-key"

# Quick chat test
curl http://<server-ip>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-key" \
  -d '{"model":"MODEL_NAME","messages":[{"role":"user","content":"hi"}]}' \
  | jq -r '.choices[0].message.content'
```

| Language | SDK Install          | Import                                |
| -------- | -------------------- | ------------------------------------- |
| Python   | `pip install openai` | `from openai import OpenAI`           |
| Node.js  | `npm install openai` | `import OpenAI from "openai"`         |
| Any      | —                    | Direct HTTP to `/v1/chat/completions` |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.readyidc.com/gpu-as-a-service/linux/docker/how-to-using-api-integration-guide.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.