RTFM: Step Inside an AI-Generated World in Real-Time
Table of Contents
A New Kind of World Model
Generative world models are the next frontier in AI, promising to create persistent, interactive, and physically accurate worlds. However, they are incredibly computationally demanding.
RTFM is designed to tackle this challenge head-on, with three key principles in mind:
- Efficiency: RTFM can run at interactive framerates on a single H100 GPU, making it accessible to a wider range of developers and researchers.
- Scalability: The model is designed to scale up with more data and compute, ensuring that it can continue to improve as hardware becomes more powerful.
- Persistence: The worlds generated by RTFM are persistent, meaning they don’t disappear when you’re not looking at them. You can explore for as long as you want and always return to previously visited locations.
World Models as Learned Renderers
Traditional 3D graphics pipelines rely on explicit 3D representations like triangle meshes or Gaussian splats. RTFM takes a different approach, acting as a “learned renderer.”
It’s an autoregressive diffusion transformer that learns to generate new 2D views of a scene based on one or more input images, without ever creating an explicit 3D model. It learns to model complex effects like reflections, shadows, and lens flare simply by observing them in its training data.
Context Juggling for Persistent Worlds
One of the biggest challenges for autoregressive frame models is persistence. As a user explores a world, the model needs to reason over an ever-growing set of frames, which can quickly become computationally expensive.
RTFM solves this problem with a clever technique called “context juggling.” Each frame in the world has a pose (position and orientation) in 3D space. When generating a new frame, the model retrieves nearby frames from its spatial memory to form a custom context. This allows RTFM to maintain a persistent world over long interactions without having to reason over an ever-growing number of frames.
The Future of Interactive AI
RTFM is a glimpse into the future of interactive AI. By creating a world model that is efficient, scalable, and persistent, World Labs is paving the way for a new generation of immersive and interactive experiences. From gaming and entertainment to robotics and simulation, the possibilities are endless.
Frequently Asked Questions
Q: What is the RTFM world model and how does it enable real-time 3D exploration? A: RTFM (Real-Time Frame Model) is a generative system from World Labs that allows users to explore AI-generated 3D worlds in real-time. It is designed for efficiency, running at interactive framerates on a single GPU. This model ensures persistent environments, where previously visited locations remain consistent during long explorations.
Q: How does the “learned renderer” approach in RTFM differ from traditional 3D graphics pipelines? A: Unlike traditional pipelines using meshes, RTFM acts as a learned renderer that generates 2D views from input images without explicit 3D models. It uses an autoregressive diffusion transformer to model complex effects like reflections and shadows. This approach allows the AI to create physically accurate worlds through direct observation.
Q: What is “context juggling” and how does it solve the persistence problem in AI worlds? A: Context juggling maintains a persistent world by retrieving nearby frames from spatial memory to inform new generation. Each frame has a 3D pose, allowing the model to reason over a relevant subset of data without excessive computation. This technique enables long-term interaction within infinite, stable, and explorable environments today.