Realtime Audio

Choose the Conversation Model

Conversational audio accepts speech and produces a spoken response. Choose the interaction style first: one recorded turn per HTTP request, a continuous microphone session with native turn detection, or an application-controlled sequence of individual turns.

The exact model determines the protocol path. Do not choose a path based only on the provider name. Start with the model catalog and confirm that realtime access is enabled on your deployment.

You need a funded workspace, an audio.write key, and audio capture and playback that match the model’s formats. The WebSocket paths also need a backend WebSocket client; see Streaming for authentication from browser applications.

Send One Turn Over HTTP

POST /v1/audio/speech-to-speech takes one recording and returns the model’s spoken reply with its transcript. Each request is a new conversation with no earlier context.

Send mono 16-bit PCM at 24 kHz, 0.1 to 60 seconds long, as a base64 WAV file or as raw samples with format: "pcm16" and sample_rate: 24000. Convert other recordings first, for example with ffmpeg -i question.m4a -ac 1 -ar 24000 -c:a pcm_s16le question.wav.

The following server-side example uses Node.js 22 or later, with no additional packages. Export the variables from the quickstart, save it as turn.mjs, and put question.wav beside it:

import { randomUUID } from "node:crypto";
import { readFile, writeFile } from "node:fs/promises";

const response = await fetch(
  `${process.env.AUDIOLAB_API_URL}/v1/audio/speech-to-speech`,
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.AUDIOLAB_API_KEY}`,
      "Content-Type": "application/json",
      "Idempotency-Key": randomUUID(),
    },
    body: JSON.stringify({
      model: "openai/gpt-realtime-2.1-mini",
      voice: "marin",
      instructions: "Answer in one short sentence.",
      input_audio: {
        data: (await readFile("question.wav")).toString("base64"),
        format: "wav",
      },
    }),
  },
);

const requestId = response.headers.get("x-request-id");
if (!response.ok) {
  throw new Error(`Turn failed (${response.status}, ${requestId}): ${await response.text()}`);
}

const turn = await response.json();
await writeFile("reply.wav", Buffer.from(turn.audio.data, "base64"));
console.log("Saved reply.wav", { transcript: turn.transcript, cost: turn.usage.cost, requestId });

Run the file and play reply.wav:

node turn.mjs

The reply is a 24 kHz mono WAV, or raw PCM16 with output_audio: { "format": "pcm16" }. status is incomplete when the reply reached max_output_tokens (1 to 1024, default 1024). Voices are alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin and cedar.

Each turn first holds the model’s full context at its highest input rate plus max_output_tokens at its highest output rate: about $1.11 for GPT Realtime 2.1 Mini, $3.54 for 2 and 2.1, and $0.93 for 1.5 at the default cap. The response arrives after OpenAI reports the final usage. You pay that usage at OpenAI’s published token rates minus 15%, shown as usage.cost, and the rest of the hold is released.

Set stream: true to receive the reply as server-sent events while it is generated. This is a protocol sequence:

← {"type":"response.created","request_id":"req_...","model":"openai/gpt-realtime-2.1-mini","provider":"openai"}
← output_audio.delta events with base64 PCM16 at 24 kHz, and output_transcript.delta events
← {"type":"response.done","request_id":"req_...","response":{"id":"resp_...","status":"completed"},"usage":{...}}

A streamed turn ends with response.done or an error event. Audio already streamed counts as delivered, so a turn that fails after a delta is still charged for its usage.

Errors name the field at fault: 400 for a wrong sample rate, length or voice, 422 for controls without a reviewed price such as temperature, and 503 for a model this endpoint does not serve. Reusing an Idempotency-Key returns 409 with the original request ID instead of generating again. If a 502 reports that the outcome could not be completed, check GET /v1/requests/{request_id} before retrying. See the turn reference for every field.

Start a Continuous Session

Connect to the Router’s /v1/audio/conversations/stream WebSocket path. Send your authorization header and use a unique Idempotency-Key for a new logical session. Send session.start within 15 seconds of connecting.

This example selects Nova 2 Sonic for a one-minute session:

{
  "type": "session.start",
  "model": "amazon/nova-2-sonic",
  "voice": "auto",
  "instructions": "Answer briefly and clearly.",
  "max_seconds": 60,
  "input_audio": {
    "format": "pcm16",
    "sample_rate": 16000,
    "channels": 1
  }
}

Wait for session.ready, then send mono PCM16 audio at 16 kHz at its capture rate. The response declares the output format; Hume uses WAV, while the supported Nova and Gemini paths use PCM16 at 24 kHz. Decode or configure playback accordingly.

Continuous sessions support max_seconds from 10 to 300. Send session.finish when done and wait for session.completed, which contains settled usage. If a duplicate session is reported as session.replayed, inspect the original request instead of treating it as a fresh audio stream.

Gemini Live Translate additionally requires target_language and uses its native voice. It does not accept tools or custom VAD settings. Follow the continuous conversation reference for the selected model’s event contract.

Control Individual Turns

The manual-turn protocol lets your application decide when input is complete. Supply audio, commit the input, and request a response using that endpoint’s events. Wait for input_audio.ready before beginning the next turn.

Supported OpenAI profiles use 24 kHz input; supported Gemini profiles use 16 kHz input. The connection supports up to 100 separately funded manual turns. Do not send continuous microphone audio without respecting the ready and commit lifecycle.

Handle Playback, Tools, and Completion

Your application owns playback queues, user interruptions, and any tool execution. Stop queued output when an interruption event arrives so the user does not hear a cancelled answer.

Tool support is model-specific. Validate each supported call, execute it in your application, and return the result with its original identifier. A text model’s tool schema is not automatically valid for an audio session.

Treat the session as complete only after its completion event. On an error or disconnect, preserve the request ID and inspect status and usage before starting another session. See Errors and Debugging for uncertain outcomes.