1,200 Concurrent Voice Agents on One Process, at Flat Latency

With all the talk recently of concurrency numbers around LiveKit and Jambonz, it seemed a good moment to measure our own rather than have an opinion about anyone else’s.

The methodology is deliberately very close to theirs, down to the same public load-testing harness, so this is not a bench built to flatter us. It is also not a comparison, and it quotes nobody else’s results.

We wanted two numbers we could defend: how many concurrent telephony voice-agent sessions one VoiceBlender process carries, and how fast it answers while carrying them. So we assembled the pipeline out of VoiceBlender’s own primitives and walked the concurrency up until the machine generating the load gave out.

It gave out first. VoiceBlender did not.

The headline numbers

The most interesting entry in that table is the one that did not happen. There is no knee. p95 latency at 1,200 concurrent sessions is 1,630 ms, which is the same p95 we measured at 25.

What counts as a session

A session is one inbound SIP call carrying bidirectional RTP through a complete eight-turn conversation - an airline flight-change dialogue of about three minutes, spoken as G.711 from pre-rendered fixtures so every call on every run transmits byte-identical audio. It counts only if setup succeeds and every expected agent reply arrives within its timeout.

The pass/fail bar was fixed before the runs:

The highest tested concurrency at which fewer than 0.5% of calls fail and p95 turn latency stays within 25% of the unloaded baseline.

Two caller utterances contain mid-utterance hesitation pauses of 600, 700, 800 and 900 ms. Those are the point of the workload. They separate a turn model from a silence timer: a timer has to be set beyond 900 ms to ride them out, and then pays that delay at the end of every normal turn.

The pipeline

VoiceBlender does not orchestrate STT → LLM → TTS internally. The agent verb bridges leg audio to an external agent product, and the STT, TTS and event/command primitives are separate building blocks. So the benchmark assembles the pipeline the way an application would, from a controller holding a single /v1/vsi WebSocket:

1. leg.ringing → answer_leg
2. leg.connected → leg_stt_start (Deepgram Flux), then the greeting
3. stt.turn end_of_turn → call the LLM → play the reply
4. leg.disconnected → drop state

STT, TTS and LLM are mock hosts with fixed, published latencies (Flux EndOfTurn 350 ms after speech ends, TTS first audio at 150 ms, LLM first token at 400 ms then 60 tokens/second). No traffic reaches a real vendor, which is what makes the runs repeatable - and what means absolute latencies here are not production figures. Differences between configurations are the meaningful part.

Before believing any capacity number, the measurement chain itself was validated: calibrated LLM time-to-first-token came back 400.8 ms against an injected 400, TTS 150.4 against 150, and a SIP echo responder established the driver and network floor at 4 ms.

Latency: where the 1,478 ms actually goes

The per-turn budget decomposes cleanly against the injected vendor latencies:

Of roughly 1.5 seconds, the media path contributes about 27 milliseconds. Everything else is vendor delay and the model’s turn decision. Nothing unaccounted for is hiding in VoiceBlender.

That framing matters for anyone tuning a voice agent: the remaining reducible term is the LLM token stream, not the platform.

Preflight TTS buys 220 ms, and keeps it under load

v0.12.0 shipped preflight TTS, which exploits the fact that Flux emits eager_end_of_turn about 200 ms before it commits to end_of_turn. The controller generates the reply and stages the audio on the eager signal with leg_tts_preflight, then leg_tts_commit on the real end of turn - so committing starts playback with no synthesis on the critical path. A guess the caller withdraws is discarded on turn_resumed.

p50 drops from 1,478 ms to 1,258 ms, and the advantage holds at every step of the ladder rather than evaporating under load. p50 drifts by 19 ms across a 48x range of concurrency.

It is not free, and the costs are worth stating plainly:

• Token spend. Over the full ladder it generated 117,515 replies for 68,720 turns - 1.71x. Against mocked vendors that is only CPU. Against real ones it is 1.71x the bill, and it belongs in any cost model built on these numbers.
• Memory, not CPU. Staged audio is buffered per leg until committed or discarded, so PSS reaches 595 MB at 1,200 sessions against plain Flux’s 270 MB - 0.44 MB per session against 0.17. Size hosts on that figure.

CPU came out slightly lower than plain Flux (0.0038 against 0.0045 cores per session at N=1,200), most likely because a discarded speculation cancels its synthesis context and a committed one plays from a buffer instead of streaming. That is an explanation consistent with the data rather than a measured mechanism.

The capacity ladder

Three waves of calls per step, 10-second ramp. Unloaded baseline p95 is 1,630 ms, so the +25% clause puts the ceiling at 2,038 ms.

p95 does not rise anywhere in the ladder. It dips slightly mid-range and returns to baseline at the top. Neither clause of the threshold is approached at any step.

The ladder ended because the host ran out of capacity to generate load, not because VoiceBlender degraded. ≥ 1,200 is a floor, not a ceiling.

Two calls failed out of 7,875, both at N=400 - an order of magnitude under the 0.5% clause, and none at the two larger steps, so not load-correlated. The signature is a lost loopback UDP packet on a rig sharing the box with the system under test.

With preflight enabled the same ladder ran zero failures across all 7,875 calls.

The event plane held. This was the main open risk: on Flux, update fires roughly four times a second per active turn, so 1,200 sessions push on the order of 5,000 events per second through a single controller WebSocket. voiceblender_vsi_events_dropped_total was 0 at every step.

Cost per session does not compound

One session costs 0.0044-0.0047 CPU cores, with no trend across a 48x range of concurrency. Cost scales with load; it does not compound. Memory is a ~65 MB base plus roughly 0.17 MB per concurrent session, with no swapping at any step.

This is the signature of a shared media plane rather than a process per call, and it is the property that actually determines what a deployment costs.

Two notes on reading those cores honestly. They were measured on a Zen 4 desktop running near 4.9 GHz, sampled as a fraction of 24 logical CPUs, so they are hyperthread-seconds on fast silicon. Normalised to a typical cloud ARM core at ~2.6 GHz, expect roughly 0.008-0.010 cores per session, or about 100-125 sessions per core consumed, against the 227 the raw figure implies. Use the normalised number when sizing.

Flux also turned out to be about 12% cheaper per session than a silence timer at matched concurrency, because it sends 80 ms frames rather than 20 ms - four times fewer WebSocket writes for the same audio, which more than pays for the extra turn events even at 1,200 sessions.

Flat latency is not the pipeline skipping work

Worth checking, so we checked. Across the Flux ladder: 7,875 calls carrying 69,345 turns; across the preflight ladder, 7,875 calls carrying 68,720 turns from 117,515 stagings (55,125 committed, 54,589 discarded on withdrawn guesses).

The mock TTS increments tts_unknown_text_total for any text it cannot resolve to a scripted fixture. It was 0 at every step of both ladders - every synthesized reply was the right words in the right order, at every concurrency.

The turn state machine is visible in the counters too. For two calls with preflight: 16 StartOfTurn, 30 EagerEndOfTurn, 14 TurnResumed, 16 EndOfTurn - two eager guesses per turn, one withdrawn at each scripted hesitation, exactly as designed.

What this does not say yet

These are strong numbers and we would rather they were read correctly than quoted loosely.

• This is a local shakedown, not a capacity claim. Load generator, mock vendors and VoiceBlender all ran on one 24-core desktop. No knee was reached, so the results bound VoiceBlender from below.
• No knee means no sessions-per-allocated-vCPU figure. Capacity per provisioned vCPU is capacity at the knee divided by the vCPUs you paid for. Without a knee there is no numerator, and we are not going to invent one.
• Vendors are mocked. Absolute latencies are not production figures.
• Turn-detection accuracy is not measured. The configurations were compared on latency and cost only. A scripted hesitation is a gentler test than real speech, so nothing here argues one turn mechanism is better at turn-taking, only that they cost different amounts of time.
• One workload. Single-caller inbound telephony, one codec, one script. Nothing here speaks to rooms, bridging, recording, video, outbound campaigns, or WebSocket and WebRTC legs.
• The LLM call is made by the application tier, not inside VoiceBlender. Its CPU was sampled separately and stayed negligible: 0.096 cores at 1,200 sessions.

One real defect surfaced, in the benchmark controller rather than in VoiceBlender: when a preflight command fails because the caller has already hung up, VoiceBlender replies with an error frame, and the controller only settled its pending callbacks on a result frame. Speculations then waited out a 10-second timeout. It happens only after the call is over and it distorted nothing, but it is being fixed.

Next

The definitive ladders run on cloud instances with the load rig on its own host, which is the only layout where per-provisioned-vCPU capacity is meaningful and where rig validity holds by construction. Then we go looking for the knee, which at ~0.005 cores per session sits far above what a single desktop can generate load for.

After that, the last large reducible latency term: incremental LLM → TTS handoff, so synthesis can start before the model has finished its sentence.

Want the detail behind any of this - the full methodology, the instrumentation and validation runs, the raw result tables, or how your own workload would map onto these figures? Get in touch and we will walk you through it.

Building voice agents on SIP and wondering whether the platform will be your bottleneck? Come talk numbers with us in Discord.