Humanoid Robots Are Learning Tennis Serves from Broadcast Video

Why athletic style and reliable ball play pull in opposite directions

Ball sports are an unusually harsh test for humanoids. The task is reactive: the ball arrives when it arrives, and the robot has milliseconds to read it, move and strike. The style, meanwhile, is a whole-body signature. A professional serve is not a sequence of joint angles chosen for efficiency; it is a coordinated chain of hip rotation, shoulder loading, toss timing and follow-through refined over a career.

Most learning pipelines force a choice. Optimize for task success and the policy converges on whatever motion keeps the ball in play, which tends to look nothing like the reference athlete. Optimize for kinematic similarity and the robot reproduces the shape of the motion while failing to actually hit the ball, because the reference trajectory assumes a body with different mass, different actuator limits and different timing.

The paper frames this as an unresolved problem: preserving a professional athlete’s motion style while maintaining reliable task performance. The two objectives are not merely different, they are in tension, because the corrections that make a robot succeed are exactly the deviations that erase the style.

A two-layer split: planner for style, tracker for execution

The proposed answer is a hierarchical framework with two components trained to stay out of each other’s way.

The planner generates stylistic kinematic motion. It is the layer that carries the athlete’s signature, learned from broadcast video and motion-capture data. The tracker takes that motion and executes it on the physical machine.

Keeping these separate is the core design decision. If a single policy were trained end to end on both objectives, gradient updates aimed at task success would bleed into the stylistic representation and flatten it. By dividing the labor, the planner can be optimized for fidelity to the reference motion while the tracker absorbs the messy business of making a real robot do it. The summary describes the goal as execution “with minimal interference between the two.”

This is a pattern that shows up across embodied AI: a high-level generator proposes, a low-level controller disposes. What is new here is applying it to athletic style specifically, where the high-level signal is a human athlete’s movement rather than a task reward.

Closing the sim-to-real gap with randomized execution speed

The tracker is where these systems usually break. A policy trained in simulation performs well in simulation and degrades on hardware, because the real world contains actuator lag, friction, sensor noise and timing jitter that the simulator did not model faithfully.

The framework’s response is to train the tracker on randomized execution speeds. Rather than learning to reproduce one nominal velocity profile, the tracker sees a distribution of speeds during training and learns to track the reference motion across that range. When the real robot’s actuators respond slower or faster than expected, the policy has already seen the variation.

The second half of the fix targets the planner. Autoregressive planning compounds errors: each predicted step conditions on the previous one, so small inaccuracies accumulate over a long horizon. Noisy perception adds to the drift. To counter this, the planner is conditioned on a learned motion-speed representation, giving it an explicit handle on how fast the motion should unfold. That conditioning reduces the compounding errors that autoregressive planning and imperfect perception otherwise introduce.

Together, the two measures address the same underlying failure mode from opposite ends: the planner is made more robust to its own drift, and the tracker is made more tolerant of the hardware’s.

Real hardware, and a serve without motion capture

The summary reports validation on two fronts. First, real-world testing on a research humanoid platform. Second, transfer to a full-size humanoid robot, including serving in an uncontrolled environment without motion capture. The summary does not name either platform, so there is no way to check how the two bodies differ in size, mass or actuation.

The mocap detail matters. Motion-capture studios are the standard crutch of humanoid locomotion and manipulation research: they give the system an external, high-precision answer to “where am I and what is my body doing.” Remove the mocap rig and the robot has to rely on its onboard perception, which is noisier, occasionally wrong and always delayed. Serving without it is a meaningfully harder test than serving inside a calibrated volume.

The transfer to a full-size platform matters for a similar reason. A policy tuned to one robot’s link lengths, mass distribution and actuator response does not automatically carry over to a larger machine. Demonstrating the transfer suggests the framework is not overfit to a single body.

The pipeline also sits in a broader trend of turning human movement into robot training data. Large-scale motion corpora such as the HiPHI Dataset: 617.5 Hours of Whole-Body Motion for Humanoid Robot Training exist precisely because the behaviors that make a machine useful are messy, varied and physical, and neither simulation nor teleoperation alone supplies enough of them. Broadcast video is an appealing complement: it is abundant, it captures world-class movement, and it requires no rig.

Motion style transfer and sim-to-real: an engineer’s checklist

The summary closes with guidance for teams building future humanoid ball-sports systems. Stripped of the restatement, the actionable points come down to a short checklist.

Randomize the axis you cannot model. Execution speed is the parameter the simulator gets wrong most often, and training across a distribution of speeds is a cheap, general defense against actuator mismatch. It is also the one item here that transfers to almost any sim-to-real pipeline, not just ball sports.

Test without the safety net. Serving without motion capture is the kind of evaluation that separates a demo from a system. If the pipeline only works inside a calibrated volume, it does not work. Treat onboard-only evaluation as the bar, not the stretch goal.

Plan for transfer from day one. Validating on a research platform and then on a full-size robot forces the design to be body-agnostic rather than tuned to one machine’s quirks. If your architecture assumes a fixed body, you will discover it at the worst possible moment.

Engineers working on the prediction side of these systems will recognize a related idea: a model that learns how an environment behaves, rather than just mapping input to output, can anticipate what happens next and plan before committing. The curated path into world models for engineers covers that territory, and it connects naturally to what a tennis robot needs: an internal sense of how the ball and its own body will evolve over the next fraction of a second.

What the public summary leaves open

The accessible version of this work is a promotional summary of a gated white paper, and it should be read as such. The full document requires registration: readers must create a user profile for the hub and log in to access all content.

That gating has real consequences for how much can be verified. Beyond the missing numbers, the summary names no researchers and gives no publication date, so “new” is doing unanchored work. Claims such as “real-world validation” and “transfer to a full-size humanoid robot” are stated without the quantitative detail that would let an outside engineer judge how robust the results are. How many serves landed? How closely did the reproduced motion match the reference athlete’s? How much did performance degrade between the research platform and the full-size machine?

None of that is unusual for a sponsored white paper, and none of it means the work is unsound. It does mean the summary functions as a pointer rather than a result. The architectural ideas are legible and useful on their own; the evidence behind them lives behind a login.

The sponsor, Noitom Robotics, builds ModalityNet, described as “the human-centric data substrate that makes the physical world learnable for embodied AI.” That framing explains the paper’s emphasis on broadcast video and motion capture as training signals: the company’s thesis is that human movement data is the raw material for embodied skill.

A data-driven path from broadcast footage to embodied athletic skill

The most interesting claim here is not that a humanoid can serve a tennis ball. It is that the training signal came from broadcast video, a source that already exists in enormous quantity and captures movement no laboratory could stage. If a two-layer architecture can extract an athlete’s style from that footage, preserve it through a planner, and then execute it on hardware that was never in the recording, the same recipe extends well beyond tennis. Throwing, swinging, jumping and the whole vocabulary of dynamic human movement become candidates.

That is the real shift. Humanoid robotics has spent years solving balance and locomotion; the next phase is about acquiring skill, and skill is expensive to author by hand. Learning it from the movement humans already produce, and then engineering the execution layer so the style survives the transfer to metal, is a more scalable path than writing behaviors one at a time.

The open questions are the ones the gated paper presumably answers: how faithful the style transfer really is, how well the framework generalizes across sports and bodies, and whether the tracker’s tolerance for randomized execution speed holds up outside a controlled test. For now, the architecture is the takeaway, and it is a sound one. Keep the style in the planner, put the noise handling in the tracker, and let broadcast video do the teaching.