AgentGarten couples executable scene programs with a shared neural renderer, so agents act in worlds whose rules and state remain explicit while their observations look like video. The engine updates object poses and task variables, then exports depth or surface normals; the renderer uses those conditions, visual history, an appearance reference, and text to generate each observation.
This separation addresses a trade-off between conventional simulators and video world models. Simulators preserve programmable dynamics but require costly assets and rendering pipelines for visual variety, while video models produce rich frames with state and transition rules implicit in generated histories. AgentGarten keeps state updates independent of rendered frames, so visual errors do not accumulate in the simulation state. A related code-based alternative builds physics worlds with PyChrono simulation code, whereas AgentGarten couples an executable environment to a learned visual renderer.
