Realtime-Bench Exposes the Latency Gap: Why Top AI Agents Fail at Low-Latency Tasks

The headline result: knowing versus doing in time

The benchmark-wide best wall-clock score is 36%, set by OpenAI’s GPT-6-Astra in its high setting. That is the top of the table, not one model’s isolated result: no system in the lineup did better under real latency. The paused-clock condition, where the environment freezes and the agent can think as long as it likes, lifts high-reasoning models into the 70 to 80% band, a range across that class rather than a single model’s score. The delta is the story: most of the failure is not reasoning failure. It is timing failure.

Read that gap carefully. A model that solves a task with the clock paused has demonstrated it knows what to do. A model that fails the same task at wall clock has demonstrated it cannot do it before the window closes. For teams shipping agents into ticketing queues, checkout flows, or live service consoles, the second number is the one users experience.

Inside Realtime-Bench

The benchmark comprises 50 interactive tasks split into two families. Thirty are web tasks spanning travel, e-commerce, service, and security scenarios. Twenty are game tasks across arcade, puzzle, strategy, and simulation categories. The mix matters: web tasks stress state tracking across asynchronous events, while games compress the feedback loop and make timing errors immediately visible.

The work comes from researchers at the University of Texas at Austin, Ohio State University, and NVIDIA, among others, with additional affiliations at the University of Illinois at Urbana-Champaign, Northwestern University, and Steadyworks, an industry lab. The academic and industry pairing shows in the design. These are not synthetic puzzles; they are the kinds of environments where an agent’s action arrives either just in time or just too late.

Three capabilities under test

Realtime-Bench decomposes realtime competence into three named capabilities, and the decomposition is the most reusable part of the paper for practitioners.

RT-Perceive covers temporal state awareness: using observations to track how the world changes between steps. RT-Deliberate covers adaptive deliberation: scaling reasoning depth to the time actually available. RT-Act covers time-aware action selection: choosing when to act, not just what to do.

The researchers illustrate all three with a ticket-buying scenario. An agent must track sale status as it shifts (RT-Perceive), shorten its deliberation as the purchase window narrows (RT-Deliberate), and decide between waiting for sales to open or buying immediately while inventory remains (RT-Act). Each capability can fail independently, which is precisely why a single pass rate hides so much. An agent that perceives perfectly but deliberates too long fails identically to one that never saw the state change.

Three clocks, three questions

The evaluation design is where the benchmark earns its keep. Each task runs under three conditions, and each condition answers a different question.

The wall clock condition measures performance under real-world latency, including the model’s own inference time. The paused clock condition freezes the environment, establishing a capability baseline free of timing pressure. The token clock condition advances the environment at a fixed rate per generation token, so consuming more tokens delays execution regardless of how fast the underlying hardware runs.

That third clock is the clever one. It isolates the cost of reasoning from the speed of the serving stack. A model that thinks in long chains pays for those tokens in lost execution time regardless of serving speed, because the environment moves per token. Conversely, a model with short reasoning can still lose on the wall clock if inference is slow. Together the three conditions let you attribute a failure to the model’s deliberation policy, the infrastructure, or the underlying capability.

The model lineup

The benchmark casts a wide net across vendors. OpenAI is represented by GPT-6-Astra, GPT-5.6 Sol, GPT-Realtime-2, and OpenAI Decisions. Anthropic contributes Claude Fable 5.1 and Claude Opus 5. Google fields Gemini 3.8 Flash, Gemini 3.1 Flash Live, and Gemini 3.8 Live. The lineup is rounded out by Jev 1.13, DeepSeek V4.1, and Moonshot AI’s Kimi-K3.

The spread is instructive in itself. Realtime-oriented variants sit alongside general-purpose flagships and high-reasoning configurations, which means the benchmark can compare design philosophies, not just raw scores. The headline finding that a high-reasoning configuration tops the wall-clock table at 36% suggests that reasoning depth helps even under time pressure, but nowhere near enough to close the gap to the paused-clock baseline. Reliability questions of the kind explored in our look at Anthropic’s push for trustworthy agents sit underneath these numbers: a confident wrong action taken at the wrong moment is worse than a slow correct one.

What the low scores actually mean

It would be easy to read 36% as a verdict on current models. The more useful reading is cartographic. The low scores identify the specific tasks where agents cannot yet be relied upon, which makes the benchmark a proxy for where agents break in production. That is a different and more actionable artifact than a ranking.

A 36% pass rate is a reliability signal, not a leaderboard curiosity. Systems that succeed roughly seven to eight times in ten when given unlimited time and roughly one time in three under real latency are not ready for unattended operation in time-sensitive workflows. They are ready for supervised assistance, with human checkpoints at exactly the moments the benchmark flags. Teams evaluating agent frameworks now have a concrete way to test whether self-improvement actually helps under the clock, rather than only on static evaluations.

Engineering implications: latency as a first-class constraint

The practical takeaway for agent builders is that inference latency belongs in the design document next to accuracy, not in the infrastructure appendix. RT-Deliberate makes the point explicit: an agent needs a policy for how much thinking it can afford, and that policy has to be conditioned on deadlines the environment imposes, not on a fixed token budget chosen at development time.

The token-clock condition gives teams a clean way to run that analysis. By separating reasoning cost from serving speed, it tells you whether to invest in a faster stack or a shorter chain of thought. Those are different engineering projects with different owners, and conflating them is how latency regressions ship unnoticed. The same discipline applies to any system where timing carries consequence, which is why the benchmark’s vocabulary is worth borrowing even outside its own task set.

The latency gap is the next capability frontier

Realtime-Bench lands at a useful moment. The field has spent years measuring what models know and comparatively little measuring whether they can deploy that knowledge inside a moving environment. The 36% versus 70 to 80% split says the second question has a very different answer, and that inference speed is now a capability axis the benchmark can score directly.

For engineers, the work offers three things worth adopting regardless of vendor: a vocabulary for realtime failure (perceive, deliberate, act), an experimental design that separates model cost from infrastructure speed, and a reminder that a paused clock flatters every agent. The next round of gains may come less from bigger models than from agents that know when to stop thinking and move.