Text-to-video systems can paint beautiful clips, but making them playable—reacting smoothly to user inputs over long sequences—has been a brick wall. Tencent Hunyuan’s Hunyuan-GameCraft attacks the problem head-on with a recipe built for game dynamics: unify keyboard/mouse signals into camera-space controls, train with a history-aware objective, and distill the model for real-time latency. The result: long, action-controllable sequences that keep scenes coherent and respond like a game engine—minus the engine.
The trick: turn WASD into camera math
Instead of treating keystrokes as ad-hoc tokens, GameCraft maps keyboard and mouse inputs to a shared, continuous camera representation (translation/rotation directions plus speeds). A lightweight action encoder injects these signals into an MM-DiT video backbone (HunyuanVideo), enabling fine-grained motion like smooth pans or faster strafes without hacking the generator.
Stay coherent over minutes, not seconds
To fight the usual “long-video drift,” the team proposes hybrid history-conditioned training: during autoregressive extension, new chunks are denoised while explicitly conditioning on denoised history with a mask indicator. Compared with training-free or streaming add-ons, this keeps geometry and layout stable across extended play.
Fast enough to feel interactive
A distillation pass (Phased Consistency Model) accelerates inference by 10–20×, cutting latency to <5 s per action in their setup—crucial for anything that calls itself “interactive.”
Trained on real gameplay, then sharpened in 3-D
The dataset is built from 1M+ gameplay clips across 100+ AAA titles (e.g., Assassin’s Creed, Red Dead Redemption, Cyberpunk 2077), segmented with PySceneDetect and annotated with 6-DoF camera trajectories (Monst3R). A synthetic set of rendered motion sequences adds precise camera priors and balances trajectory distributions.
Why this matters
-
Input-to-motion fidelity. Unifying controls in camera space yields smoother, more physical responses to typical WASD/arrow inputs.
-
Long-horizon stability. History conditioning curbs error accumulation that wrecks long, user-driven videos.
-
Path to production. Distillation pushes latency toward “feels responsive,” a precondition for creator tools and AI-assisted level previews.
Availability and what’s next
A project page is live, and the team has released inference code and weights under the Hunyuan-GameCraft-1.0 repository. The arXiv record also notes acceptance to RSS 2025, signaling interest from the robotics community.
Paper link: arXiv 2506.17201 (PDF)
No comments:
Post a Comment