A benchmark for multimodal agents vibe coding videos
Evaluating Multimodal Agents on Code-Driven Video Generation
BUPTZGCAPKUJLUKling
18 s · made with code
Like every video on this page, this opening was made entirely with code. No text-to-video model is involved.
Making a video with code is a multimodal task: the agent has to understand text, images, audio and data, plan a sequence in time, and judge its own renders by eye and ear.
A text prompt, on its own or with local files shipped alongside it: notes and scripts, data tables, sprites, level layouts, audio. Everything the agent needs is already in the sandbox.
The agent has to search for and fetch material itself (reference images, photos, facts), as in the Apollo / Saturn V explorer task, where the rocket's real stages and engines have to be right.
Scored tasks by content category · 239
A task prompt, verbatim
Nine agent systems, each in its own native harness at its highest effort, on all 239 tasks.
GPT-6 Astra and Claude Opus 5.5 lead clearly. Opus delivers all 239 videos and passes every program check on 99.6% of tasks; Astra misses a single video. Counting missed videos as 0, they sit about a fifth of a point clear of the next system. Among delivered videos alone the field tightens: Gemini 4 Argon comes within 0.09 of Opus, but on 184 deliveries to Opus’s 239. And even the best system fails at least one core requirement on about a quarter of tasks.
Click a column to sort. No video = 0: a run without a video fails every check and scores 0 on every dimension, and stays in the average. Delivered only: only runs with a video are averaged.
L1 · Program: Delivered Runs that ended with a playable MP4 in the delivery folder within the one-hour budget.
L2 · Rubric: Pass rate pass / (pass + fail) over task-specific binary checks; unknown and n/a are excluded.
Overall · Score Mean of the five quality dimensions (1–5), under the selected scoring rule.
L3 · VQ · VA · CL · TM · AQ Image quality, visual aesthetics, content logic, motion & pacing, audio quality & A/V sync — each 1–5, anchored at 1/3/5.
Caveat Scores are not yet calibrated against human ratings and there is no weighted total; compare systems within one judge panel. Gemini models appear both as judges and as contestants, so the GPT-5.6-sol + Qwen panel is the cross-check.
Look beyond the average
Select a content category to compare all nine systems under the current judge panel.
Category means include runs without a video (scored 0).
Check what the video says and does — not only how it looks.
Each system runs in its own native harness (Claude Code, Codex, Antigravity, Kimi Code…) at maximum effort, inside an identical Docker sandbox with 8 CPUs and no GPU. It has 60 minutes to write code, render frames, compose sound and deliver an MP4 plus an editable project. Rubrics are never shown.
Judges receive the full delivered video with its soundtrack (sampled at 4 fps), the task prompt and its rubric. Every verdict must cite a time range and a channel — visual or audio — and say what was actually seen or heard. Evidence that cannot be observed is marked unknown, never guessed.
Task-specific binary checks measure whether the task's requirements were met: the right objects, the right causal order, the right state changes. Five anchored dimensions (1–5) measure craft. Checks that need the project files rather than the video are proposed as program checks and kept separate.
Three judge panels
Every run is scored by all three panels and the leaderboard never mixes them. Gemini models also compete, so the GPT + Qwen panel is the cross-check against self-preference.
Three levels of metrics
Two scoring rules
Watch the video. Read the prompt. See what the judges caught.
Curated, not random: the team picked these tasks and the video each one opens with. Videos are re-encoded to 480p for the web; judges saw the original 1080p files.
Do judges agree with each other and humans?
Human preferences and all three judge panels produce similar rankings of the nine systems.
Judge panels. All comparisons use the same 239 tasks. Mean scores count no-video runs as 0. Same pass/fail counts rubric checks both panels decided; unknown and n/a are excluded. The summary averages the three panel-pair agreement rates. Absolute scores differ, so the leaderboard keeps panels separate.
| Comparison | Same pass/fail ↑ | System-rank ρ ↑ |
|---|
ρ compares rankings of nine systems. Human rows use A/B preference votes.
If you use VibeVideoBench in your research, please cite:
@misc{diao2026vibevideobenchevaluatingmultimodalagents,
title={VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation},
author={Muxi Diao and Wenbin Xu and Lele Yang and Xuyang Ma and Yuzhe Chen and Jiankang Chen and Kaiyu Jiang and Changyi Liu and Haoyu Zhang and Hongyi Fu and Jianxiong Wang and Tianke Zhang and Haonan Fan and Yingxin Li and Bingyi Li and Xu Wang and Xuanyu Zheng and Bin Wen and Tingting Gao and Han Li and Wentao Zhang and Weiran Xu and Kongming Liang and Zhanyu Ma},
year={2026},
eprint={2610.35864},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2610.35864},
}