Stream Manual Turns
/v1/audio/speech-to-speech/streamOverview
Requires an enabled REALTIME_ENABLED deployment and audio.write scope.
Routing and provider details
Reviewed models include OpenAI gpt-realtime, 1.5, 2, 2.1, 2.1 Mini,
the pinned 2025-08-28 snapshot, and gemini/gemini-3.1-flash-live-preview.
REALTIME_ENABLED defaults to false. Up to 100 separately funded manual
turns share one native conversation over at most 15 minutes. For one
turn without a socket, use POST /v1/audio/speech-to-speech.
Authenticate the HTTP upgrade with Authorization: Bearer <api-key>.
Send a unique Idempotency-Key header and reuse it only for the same input.
Native browser WebSocket cannot set that header; use a trusted server or
a WebSocket client that supports custom upgrade headers. Never put an API
key in the URL.
The first client text frame must be a session.start object matching
S2sStart, with mono PCM16 at 24000 Hz for OpenAI input or 16000 Hz
for Gemini input, and 24000 Hz for output. Use turn_detection.type=none
and input_transcription=false. Gemini requires voice=Puck (or auto) and
output_transcription=false. It accepts provider.data_region, as a
request does. Set
max_output_tokens between 1 and 1024. Wait for session.ready, send
binary audio frames of at most 256 KiB, then input_audio.commit and
response.create. Input duration must be between 0.1 and 60 seconds.
No automatic VAD, tools, provider events, session updates, input
transcription or response overrides are admitted. After response.done
and input_audio.ready, append and commit the next turn's audio.
Receive output_audio.delta and optional output_transcript.delta events. Gemini uses a persistent native connection with minimal thinking and no transcription, proactivity, tools, context compression or resumption. Its quote reserves 131072 input tokens plus the inclusive output cap: 1024 output tokens reserve USD 0.405504 of provider expense. Native usage must include complete text/audio and thought accounting on the terminal turnComplete frame; earlier periodic usage is insufficient. Gemini response.done contains a request-bound local turn ID and id_origin=local; it does not invent a provider response ID.
Final response.done follows durable native usage recording and wallet settlement for each turn. Send session.finish to close the session. The temporary hold covers the full reviewed context and output cap; settlement uses native text, audio and cache token counters at the published provider tariff minus 15%. Cancellation charges known delivered output. A missing final provider receipt retains the hold; reconnecting with the same idempotency key never repeats generation. Replay returns session.replayed status and usage, not the original audio.
Responses
101WebSocket protocol upgrade accepted.
400Request validation failed.
errorobjectRequiredShow nested fields
codestringRequiredmessagestringRequiredparamstring | nullrequest_idstring | nullmetadataobjectTyped detail for some refusals. budget_exceeded (402): limit (key_budget or workspace_budget) and budget with scope, period, limit, usage and resets_at. key_limit_exceeded (402): limit key_spend_limit. access_denied (403): policy (workspace or key), rule (models, providers or products), product, provider and model.
401Authentication is missing or invalid.
errorobjectRequiredShow nested fields
codestringRequiredmessagestringRequiredparamstring | nullrequest_idstring | nullmetadataobjectTyped detail for some refusals. budget_exceeded (402): limit (key_budget or workspace_budget) and budget with scope, period, limit, usage and resets_at. key_limit_exceeded (402): limit key_spend_limit. access_denied (403): policy (workspace or key), rule (models, providers or products), product, provider and model.
402Credits or billing setup is required.
errorobjectRequiredShow nested fields
codestringRequiredmessagestringRequiredparamstring | nullrequest_idstring | nullmetadataobjectTyped detail for some refusals. budget_exceeded (402): limit (key_budget or workspace_budget) and budget with scope, period, limit, usage and resets_at. key_limit_exceeded (402): limit key_spend_limit. access_denied (403): policy (workspace or key), rule (models, providers or products), product, provider and model.
403Authenticated principal lacks permission.
errorobjectRequiredShow nested fields
codestringRequiredmessagestringRequiredparamstring | nullrequest_idstring | nullmetadataobjectTyped detail for some refusals. budget_exceeded (402): limit (key_budget or workspace_budget) and budget with scope, period, limit, usage and resets_at. key_limit_exceeded (402): limit key_spend_limit. access_denied (403): policy (workspace or key), rule (models, providers or products), product, provider and model.
422Audio could not be processed.
errorobjectRequiredShow nested fields
codestringRequiredmessagestringRequiredparamstring | nullrequest_idstring | nullmetadataobjectTyped detail for some refusals. budget_exceeded (402): limit (key_budget or workspace_budget) and budget with scope, period, limit, usage and resets_at. key_limit_exceeded (402): limit key_spend_limit. access_denied (403): policy (workspace or key), rule (models, providers or products), product, provider and model.
429Rate limit exceeded.
errorobjectRequiredShow nested fields
codestringRequiredmessagestringRequiredparamstring | nullrequest_idstring | nullmetadataobjectTyped detail for some refusals. budget_exceeded (402): limit (key_budget or workspace_budget) and budget with scope, period, limit, usage and resets_at. key_limit_exceeded (402): limit key_spend_limit. access_denied (403): policy (workspace or key), rule (models, providers or products), product, provider and model.
500Internal server error.
errorobjectRequiredShow nested fields
codestringRequiredmessagestringRequiredparamstring | nullrequest_idstring | nullmetadataobjectTyped detail for some refusals. budget_exceeded (402): limit (key_budget or workspace_budget) and budget with scope, period, limit, usage and resets_at. key_limit_exceeded (402): limit key_spend_limit. access_denied (403): policy (workspace or key), rule (models, providers or products), product, provider and model.
502Upstream provider failed after allowed attempts.
errorobjectRequiredShow nested fields
codestringRequiredmessagestringRequiredparamstring | nullrequest_idstring | nullmetadataobjectTyped detail for some refusals. budget_exceeded (402): limit (key_budget or workspace_budget) and budget with scope, period, limit, usage and resets_at. key_limit_exceeded (402): limit key_spend_limit. access_denied (403): policy (workspace or key), rule (models, providers or products), product, provider and model.
503No eligible provider endpoint is currently available, or async processing has no healthy worker polling the durable queue.
errorobjectRequiredShow nested fields
codestringRequiredmessagestringRequiredparamstring | nullrequest_idstring | nullmetadataobjectTyped detail for some refusals. budget_exceeded (402): limit (key_budget or workspace_budget) and budget with scope, period, limit, usage and resets_at. key_limit_exceeded (402): limit key_spend_limit. access_denied (403): policy (workspace or key), rule (models, providers or products), product, provider and model.