Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 75 additions & 1 deletion agents/build/custom-llm.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,86 @@

Your endpoint speaks the standard OpenAI [chat completions](https://platform.openai.com/docs/api-reference/chat) protocol. The platform sends `POST {base_url}/chat/completions` with `stream: true` and reads the reply as server-sent events, so if your prototype already runs against another voice-agent platform through a custom LLM, the same server works here unchanged.

Each conversation turn, your endpoint receives the same fully assembled context a platform model would see. The `messages` array carries the system prompt with [dynamic variables](/agents/build/dynamic-variables) and [session overrides](/agents/deploy/authenticated-sessions#overrides) applied, the complete conversation history, and any retrieved [knowledge](/agents/build/knowledge-base), and the agent's tools are included in OpenAI function format. When your model returns `tool_calls`, the platform executes the tool and calls you again with the result. Two extra fields ride each request body so your server can look up its own state:
Each conversation turn, your endpoint receives the same fully assembled context a platform model would see. The `messages` array carries the system prompt with [dynamic variables](/agents/build/dynamic-variables) and [session overrides](/agents/deploy/authenticated-sessions#overrides) applied, the complete conversation history, and any retrieved [knowledge](/agents/build/knowledge-base), and the agent's tools are included in OpenAI function format. When your model returns `tool_calls`, the platform executes the tool and calls you again with the result. A few extra top-level fields ride each request body so your server can look up its own state:

| Field | Content |
|---|---|
| `session_id` | The Fish Audio session id. |
| `user_id` | The `end_user_id` you passed when [creating the session](/agents/deploy/authenticated-sessions). Omitted when the session has none. |
| `fishaudio_extra_body` | The `llm_extra_body` object you passed when creating the session, forwarded verbatim on every request. Use it to carry your own identifiers, such as which chat or thread the user is in. Omitted when the session has none. |

<Tip>
If your server proxies requests to an upstream provider such as OpenAI, consider removing `session_id`, `user_id`, and `fishaudio_extra_body` from the body first. Some upstream APIs reject parameters they don't recognize.
</Tip>

### Example request

Suppose the session was created from your backend with `end_user_id: "user-42"` and `llm_extra_body: {"chat_id": "chat-9"}`, the agent has one webhook tool `lookup_order`, and the user has just asked about an order. Your endpoint receives:

```http
POST /v1/chat/completions HTTP/1.1
Host: llm.example.com
Authorization: Bearer sk-your-endpoint-key
Content-Type: application/json
```

```json
{
"model": "persona-70b",
"stream": true,
"stream_options": { "include_usage": true },
"messages": [
{
"role": "system",
"content": "Today is Wednesday, August 26, 2026. Session timezone: America/New_York (UTC-4).\n\n…\n\nYou are Aria, the support assistant for Acme Shoes. Help customers with orders and returns. …"
},
{ "role": "assistant", "content": "Hi, this is Aria from Acme Shoes. How can I help you today?" },
{ "role": "user", "content": "I want to check on my order." },
{ "role": "assistant", "content": "Sure, what's the order number?" },
{ "role": "system", "content": "Current date and time: Wednesday, August 26, 2026, 14:07 (America/New_York)." },
{ "role": "user", "content": "It's A one two three four five." }
],
"tools": [
{
"type": "function",
"function": {
"name": "lookup_order",
"description": "Look up an order by its number.",
"parameters": {
"type": "object",
"properties": {
"order_number": { "type": "string", "description": "The order number, e.g. A12345." }

Check warning on line 61 in agents/build/custom-llm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/build/custom-llm.mdx#L61

Did you really mean 'order_number'?
},
"required": ["order_number"]
}
}
}
],
"session_id": "sess_01j9x4k2m8v3q7n5p6r8t9w0y1",

Check warning on line 68 in agents/build/custom-llm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/build/custom-llm.mdx#L68

Did you really mean 'session_id'?
"user_id": "user-42",

Check warning on line 69 in agents/build/custom-llm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/build/custom-llm.mdx#L69

Did you really mean 'user_id'?
"fishaudio_extra_body": { "chat_id": "chat-9" }

Check warning on line 70 in agents/build/custom-llm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/build/custom-llm.mdx#L70

Did you really mean 'fishaudio_extra_body'?
}
```

A few things to note:

- The first `system` message is the assembled prompt: your agent's [system prompt](/agents/build/configuration#system-prompt) with [dynamic variables](/agents/build/dynamic-variables) and [overrides](/agents/deploy/authenticated-sessions#overrides) applied, plus the platform's own context lines. A short `system` line carrying the current time is inserted before the latest user turn on every request.
- `messages` carries the full transcript so far. Speech recognition output arrives as plain `user` text; your earlier replies come back as `assistant` messages.
- `tools` is present only when the agent has tools configured. The `model`, `stream`, and `stream_options` fields are fixed; no `temperature` or `max_tokens` is sent, so apply your own defaults.

When your model returns a `lookup_order` tool call, the platform executes it and immediately sends the next request with the call and its result appended to `messages`, everything else unchanged:

```json
{
"role": "assistant",
"tool_calls": [
{ "id": "call_1", "type": "function", "function": { "name": "lookup_order", "arguments": "{\"order_number\":\"A12345\"}" } }
]
},
{ "role": "tool", "tool_call_id": "call_1", "content": "{\"status\":\"shipped\",\"eta\":\"2026-08-28\"}" }
```

Reply to that request with the spoken answer as ordinary `delta.content` chunks ending in `finish_reason: "stop"`.

Requests authenticate with `Authorization: Bearer <your API key>`. The endpoint must use `https` on a publicly reachable host, and must support function calling if the agent has tools configured.

Expand Down
1 change: 1 addition & 0 deletions agents/deploy/authenticated-sessions.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
<Note>
No backend, and anyone may talk to the agent? A [public
agent](/agents/deploy/public-agents) lets the SDK create sessions with just an
`agentId`: no token involved, gated by an origin allowlist and rate limits.

Check warning on line 21 in agents/deploy/authenticated-sessions.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/authenticated-sessions.mdx#L21

Did you really mean 'allowlist'?
</Note>

## Create a token on your backend
Expand Down Expand Up @@ -109,7 +109,7 @@

<Note>
`overrides`, `dynamic_variables`, `tool_events`, `timezone`, and
`world_context` belong in your backend's creation request. The SDK forwards

Check warning on line 112 in agents/deploy/authenticated-sessions.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/authenticated-sessions.mdx#L112

Did you really mean 'backend's'?
these options (and its `language` shorthand for `overrides.language`) only in
[public agent](/agents/deploy/public-agents) mode.
</Note>
Expand All @@ -119,12 +119,13 @@
| Field | Type | Description |
| ------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent_id` | string, required | The agent to talk to. It must have a [published version](/agents/deploy/versions-publishing). |
| `name` | string, optional | Display name for this session in the console's Conversations list, up to 128 characters. Omit it to show the session's start time instead. API-key requests only: keyless (public) creation rejects it with `400`. |

Check warning on line 122 in agents/deploy/authenticated-sessions.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/authenticated-sessions.mdx#L122

Did you really mean 'keyless'?
| `overrides` | object, optional | Replace parts of the published configuration for this session. See [Overrides](#overrides). |
| `dynamic_variables` | object, optional | Up to 50 entries of string, number, or boolean values, substituted into `{{placeholders}}`. See [Dynamic variables](/agents/build/dynamic-variables). |
| `tool_events` | boolean, optional | Stream tool lifecycle events (`toolCallStarted` / `toolCallCompleted` / `toolCallFailed`) to the client. Default `true`; set `false` to keep tool inputs and outputs off the client. |
| `end_user_id` | string, optional | Your identifier for the end user, up to 256 characters. Stored on the session and echoed in [webhook](/agents/monitor/webhooks) payloads and [custom LLM](/agents/build/custom-llm) requests. |
| `metadata` | object, optional | Your own key-value namespace. Stored and returned verbatim on session queries and webhooks, never read or interpreted by the platform. |

Check warning on line 127 in agents/deploy/authenticated-sessions.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/authenticated-sessions.mdx#L127

Did you really mean 'namespace'?
| `llm_extra_body` | object, optional | JSON object (at most 16 KB) forwarded verbatim to your [custom LLM](/agents/build/custom-llm) endpoint on every request as `fishaudio_extra_body`, for example the chat or thread the user is in. Not stored or returned on session reads; ignored when the agent uses a platform model. |
| `record_audio` | boolean, optional | Whether to record this session's audio. Overrides the agent's [recording setting](/agents/monitor/conversation-history#what-gets-stored) for this session only; omit it to use the agent's configuration. |
| `timezone` | string, optional | IANA timezone (like `Asia/Shanghai`) for the agent's sense of local time. Invalid names are rejected with `422`. See [Time & timezone](/agents/build/time-timezone). |
| `client_timezone` | string, optional | The end user's browser timezone, filled automatically by the SDK in public-agent mode. A hint, not a demand: it applies only when neither `timezone` nor the agent's configured timezone is set, and invalid values are ignored. See the [resolution order](/agents/build/time-timezone). |
Expand All @@ -149,7 +150,7 @@
"agent_id": "YOUR_AGENT_ID",
"overrides": {
"first_message": "Welcome back, {{name}}, picking up where we left off.",
"voice_id": "802e3bc2b27e49c2995d23ef70e6ac89",

Check warning on line 153 in agents/deploy/authenticated-sessions.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/authenticated-sessions.mdx#L153

Did you really mean 'voice_id'?
"language": "ja"
}
}
Expand Down Expand Up @@ -214,7 +215,7 @@

| Status | Meaning |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `400` | A keyless (public-agent) request sent an override [public sessions don't accept](#overrides), or a `name`. |

Check warning on line 218 in agents/deploy/authenticated-sessions.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/authenticated-sessions.mdx#L218

Did you really mean 'keyless'?
| `401` | Invalid API key. A request with no `Authorization` header at all is treated as a public-agent request instead. |
| `402` | Quota exceeded. |
| `403` | Public-agent request rejected: the agent is not public, or the page's `Origin` is not on the allow-list. |
Expand Down
1 change: 1 addition & 0 deletions agents/telephony/outbound-calls.mdx
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: "Outbound Calls"
description: "Place calls from your phone numbers over the API; the agent speaks when the callee answers"

Check warning on line 3 in agents/telephony/outbound-calls.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/telephony/outbound-calls.mdx#L3

Did you really mean 'callee'?
icon: "phone-arrow-up-right"
---

Dial any allowed number from one of your workspace numbers and the agent takes the call the moment the callee picks up. Outbound calls are ordinary agent sessions with `direction: "outbound"`: they appear in session history, they are [stored](/agents/monitor/conversation-history#what-gets-stored) and analyzed under the same per-agent settings as any other conversation, and they trigger the same webhooks plus one extra, [`phone_call.dial_finished`](/agents/monitor/webhooks), that reports how the dial attempt ended.

Check warning on line 7 in agents/telephony/outbound-calls.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/telephony/outbound-calls.mdx#L7

Did you really mean 'callee'?

<CardGroup cols={3}>
<Card
Expand All @@ -30,7 +30,7 @@

<Steps>
<Step title="Get a number that supports outbound">
Any [purchased number](/agents/telephony/phone-numbers) can place calls. An [imported BYO number](/agents/telephony/byo-sip) can too, once its termination is configured; the number object reports this as `supports_outbound`. The number you dial from is the caller ID the callee sees.

Check warning on line 33 in agents/telephony/outbound-calls.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/telephony/outbound-calls.mdx#L33

Did you really mean 'callee'?
</Step>
<Step title="Publish your agent">
Outbound calls run the agent's published configuration, not the draft. [Publish](/agents/deploy/versions-publishing) before dialing.
Expand Down Expand Up @@ -69,8 +69,9 @@
| `dynamic_variables` | Optional: per-call values for `{{placeholders}}` in the agent's configured text, same rules as [session creation](/agents/build/dynamic-variables). Up to 50 entries. |
| `overrides` | Optional: replace whole configuration fields for this call, subject to the agent's [override allowlist](/agents/deploy/authenticated-sessions#overrides). |
| `metadata` | Optional: your own JSON object, returned verbatim on session reads and in webhook payloads. Never interpreted. |
| `llm_extra_body` | Optional: JSON object (at most 16 KB) forwarded to a [custom LLM](/agents/build/custom-llm) endpoint on every request as `fishaudio_extra_body`. Ignored on platform-model agents. |

The session's [time and timezone context](/agents/build/time-timezone) resolves from the destination number when the agent has no fixed timezone configured, so "tomorrow morning" means the callee's morning.

Check warning on line 74 in agents/telephony/outbound-calls.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/telephony/outbound-calls.mdx#L74

Did you really mean 'callee's'?

### Retry safely with an Idempotency-Key

Expand All @@ -78,7 +79,7 @@

## The dial outcome

Ringing is never billed; metering starts when the callee answers. A call that is never answered is not billed and not analyzed.

Check warning on line 82 in agents/telephony/outbound-calls.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/telephony/outbound-calls.mdx#L82

Did you really mean 'callee'?

You learn how the dial ended in either of two ways:

Expand All @@ -101,7 +102,7 @@

## Allowed destinations

Calls from purchased numbers can reach US, Canada, and Japan numbers. Japanese destinations may keep the domestic trunk zero (`+81080...` is accepted and normalized to `+8180...`), and premium-rate segments (such as `0570` Navi Dial and `0990`) are always refused. Calls from [imported BYO numbers](/agents/telephony/byo-sip) dial out through your own trunk, so the country allowlist does not apply; the destination only has to be valid E.164.

Check warning on line 105 in agents/telephony/outbound-calls.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/telephony/outbound-calls.mdx#L105

Did you really mean 'Navi'?

Check warning on line 105 in agents/telephony/outbound-calls.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/telephony/outbound-calls.mdx#L105

Did you really mean 'allowlist'?

Numbers that live on the platform can never be dialed, so an agent cannot call another agent's number.

Expand Down
Loading