When a General-Purpose Generative AI Drove Their Car Straight to In-N-Out: How Language Models Are Unlocking Physical World Reasoning

When a General-Purpose Generative AI Drove Their Car Straight to In-N-Out: How Language Models Are Unlocking Physical World Reasoning

Three AI engineers at Bay Area startup Axiom — Aditya Ramabadran, Simon Mahns, and Tobias Gessler — had a very specific lunch craving for In-N-Out Burger not long ago. Instead of hopping in the driver’s seat themselves, they parked their 2024 Toyota Corolla right outside the fast food spot’s drive-thru line, pulled out a laptop, and handed full driving control over to OpenAI’s GPT-6 Astra.

The team linked their chat interface to a server connected to multiple dash-mounted windshield cameras and the car’s electronic power steering system, with a human safety driver positioned in the front seat ready to hit the brakes at a moment’s notice. The model, built natively to generate text, code, and occasional creative imagery, navigated the vehicle slowly and flawlessly up to the order window so the team could pick up their food. “Maybe AGI is here after all,” one of the engineers joked, referencing the much-hyped concept of artificial general intelligence: a machine capable of matching the full scope of human cognitive ability.

Purpose-built self-driving car systems are far from a new innovation, of course — every existing commercial autonomous vehicle runs on algorithms explicitly trained and fine-tuned for road navigation. But this test was unprecedented: the car was controlled by a generalist text-first model, with zero prior custom training or coaching from the three engineers. This fast-food stunt, plus a growing body of parallel experiments, proves that large language models are starting to develop a rudimentary but surprisingly functional understanding of the physical world around them. The team notes that while no dangerous incidents occurred during their lunch run, handing a generalist AI full control over a two-ton hunk of metal moving at road speeds is an inherently high-stakes gamble. As physical reasoning capabilities improve, generalist AI could reach into the real world in transformative — and potentially unforeseen, dangerous — new ways.

Getting Physical

Today’s most advanced state-of-the-art AI models can answer staggeringly complex questions and run entire virtual workflows fully autonomously, but their capabilities were long locked entirely to the digital domain of servers, computers, and the internet. Plop even the most powerful LLM down in an unstructured real-world setting, and it would quickly get completely disoriented.

Despite widespread viral claims that AGI has already arrived, every major AI developer now views physical scene reasoning as one of the last major unconquered frontiers for the field. A growing number of senior researchers have left big tech labs to launch new startups explicitly focused on cracking this problem.

Andrew Dai, CEO of physical reasoning startup Elorian AI and a former Google DeepMind researcher, explains that advanced visual and spatial reasoning will unlock dozens of game-changing new use cases for AI, ranging from systems that can intuit if restaurant diners are enjoying their meals to fully functional home assistant robots. “Physical reasoning is non-negotiable for robotics,” Dai notes. “You simply cannot imagine mass adoption of practical home robotics without this core capability.”

Elorian recently partnered with Scale AI — the industry leader supplying training data to major AI labs — to launch a new standard benchmark called Humanity’s Sixth Sense, designed specifically to measure how well AI models can intuitively parse physical real-world scenes. “Most existing visual AI research only tests for basic object perception,” says Xingang Guo, research scientist at Scale AI and one of the benchmark’s lead developers. “We wanted to measure the thing no one has tested yet: can a model understand a scene intuitively, the exact way a human does without stopping to break every detail down step by step?”

For the three Axiom engineers, the In-N-Out test was a side hobby project, not part of their official work duties, built to measure how well generalist models can perform in the messy, unstructured real world. They landed on the driving idea during a casual weekend hangout, after noticing that multimodal models like Astra were already capable of building hyper-detailed complex 3D simulations, and wondered if that spatial skill would translate to real-world navigation. “We all live in the Bay Area, where you see Teslas and Waymos everywhere you go, so that definitely planted the seed of the idea,” Ramabadran shared in an interview.

The team first tested SpaceXAI’s Grok, before moving on to the latest flagship models from OpenAI and Anthropic. At first, every model refused to take control, firing back responses like “I can help interpret images of roads, but I cannot send motion commands to operate a physical real-world car.” But with carefully worded, safety-aligned prompting, they were able to get past the guardrails and run the test. The three researchers said they were completely stunned when the models began driving the car competently.

They note that major AI firms like OpenAI are clearly investing in improving the spatial reasoning of their latest flagship models, but they are almost certainly not explicitly training those models to drive passenger cars. The unexpected driving ability appears to be a pure emergent side effect of scaling up multimodal training datasets that include large volumes of images, video, and 3D scene data. “This is just an emergent capability that popped up as soon as we scaled the model’s multimodal inputs far enough,” Mahns explains.

Their new open vehicular benchmark, DrivingBench, makes it clear these models still have a very long way to go before they could pass a standard human driving test. The benchmark tests model performance navigating a simple obstacle course set up in an empty parking lot: only GPT-6 Astra was able to complete the full course, and did so at a very slow, cautious pace. Claude Fable 5.1 made it 45% of the way around the loop, and Grok only completed 11% of the course.

Ramabadran notes that the newest flagship models show a surprising ability to correct for errors on the fly, adapting to the quirks of the car’s controls and improving their steering in real time as they drive. “These models clearly demonstrated in-context learning: they watched for their own mistakes, adjusted their response, and figured out how to navigate the car’s controls more smoothly as they went,” he says.

This piece is adapted from Will Knight’s AI Lab newsletter. You can read past editions of the newsletter here.