Methodology

AudioLab measures which model output a listener prefers for the same real-world request. It is a preference benchmark, not a claim that one system is universally more factual, safe, or capable.

Most requests receive one result so AudioLab remains useful as a product. A configurable sample of eligible requests enters an evaluation flow and contributes a pairwise vote.

Text to Speech compares two voice synthesis systems reading the same submitted text. Speech to Text transcribes one spoken request and gives the identical transcript to two text-response models.

Speech to Speech currently generates one shared written answer and compares two speech synthesizers reading that answer. This isolates synthesis preference; it does not yet rank native end-to-end speech models. Rankings are computed independently for each mode.

Spoken prompts are captured in the browser, validated by file signature, MIME type, size, and declared duration, then transcribed server-side. Both candidates receive the exact same normalized prompt. Prompts are moderated before any model generation.

The default evaluation sample is 5%. Recordings shorter than five seconds and low-information fillers such as greetings or acknowledgements are excluded from ranking battles. The sample rate, duration threshold, and active modes are server controlled.

The server considers only enabled, configured, mode-compatible models that are not marked unavailable. Pair weights favor matchups with less coverage and greater outcome uncertainty, while model sampling weights and recent health can adjust eligibility. Active reservations reduce repeated concurrent assignments.

A cryptographic seed randomizes which candidate appears as Response A or Response B. Model IDs, versions, voices, providers, and private storage paths remain server-side until the vote is accepted. Both responses are released together when practical so provider latency does not become a visual identity cue.

A reviewer may prefer Response A, prefer Response B, rate both good, or rate both bad. Speech to Speech votes require at least three seconds of contiguous playback from each response. The server enforces this from bounded playback heartbeats; seeking or leaving a player open does not count as listening.

The primary preference is immutable and stored atomically with the battle state. Optional diagnostic reasons and a short explanation are collected afterward and analyzed separately. Failed, partial, self-matched, duplicate, or administratively invalidated battles are excluded from ranking inputs.

Each mode is fit with a regularized Bradley–Terry maximum-likelihood model. A decisive vote contributes one win and one loss. Both-good and both-bad outcomes remain distinct in the diagnostic record but each contributes a half-win to both models in the ranking fit.

Coefficients are centered so the average fitted model is rated 1000, unless an explicitly configured anchor model is available. Ratings are recomputed from all valid stored outcomes rather than being dependent on vote arrival order.

AudioLab applies Poisson bootstrap weights to aggregated valid outcomes and refits the full model to estimate a nonparametric 95% confidence interval. This is equivalent to vote-level Poisson resampling while remaining practical at scale. A fixed seed makes each algorithm version reproducible, and models with no evidence receive a deliberately wide interval.

Overlapping intervals produce tied or ranged rank presentations instead of false precision. A model is marked preliminary until it reaches 200 valid battles by default. Slices with inadequate support are withheld rather than extrapolated.

AudioLab records side assignment, latency, listening progress, prompt duration, locale, model version, voice, allocation seed, and conversation turn where available. These fields support audits of position, latency, playback-order, language, voice, and conversation-depth effects.

The published score is the unadjusted community preference estimate. A controlled ranking is not displayed until the dataset is large and connected enough to estimate confounds defensibly. Distinctive fixed voices may allow repeat listeners to infer a provider; this remains a known limitation and is disclosed rather than hidden.

Prompt recordings and generated audio are stored in private Supabase buckets under opaque identifiers. Prompt audio is cache-like input data with a default 30-day expiry; deployment operators can shorten the retention window. Signed or authenticated streams do not reveal model IDs.

Database records use hashes and lengths where full content is not operationally required. Provider keys and the Supabase service role never enter the browser. Do not submit sensitive personal information: prompts still pass through moderation, transcription, and the selected model providers.

Production deployments schedule ranking recomputation daily and may trigger an additional run after model or validity changes. The leaderboard reads the latest completed immutable run, so an interrupted recomputation cannot replace a healthy snapshot.

Only models with working server configuration can receive new assignments. Historical outcomes remain attached to the exact stored model version and voice when that metadata is available. Provider outages reduce new sampling without rewriting prior votes.