Robotics is without question the hottest track in AI. In the first quarter of this year, physical-AI startups worldwide attracted roughly $16 billion in investment—an all-time high. Yet the rapid influx of capital hasn’t actually brought robots into the real world, because the hardest bottleneck to cross has never changed: how to get a robot to accomplish something it has never done before, in a world full of variables.
The reason is that what a robot needs is completely different from what generative AI needs. Large language models can understand the world through text, images, and video from the internet, but what a robot has to learn is manipulation. Take the same clip of a person setting a cup down on a table: generative AI can recognize that “someone placed a cup on a tray,” whereas a robot needs to know from which direction the hand should approach, how much force to apply so the cup doesn’t slip, and how to tell when it has a secure grip. This kind of information doesn’t appear in the data on the internet, and it can’t be harvested in bulk by web crawlers.
Many people instinctively assume that if data is what’s lacking, the answer is simply to collect more. But a recent study in robot learning found that what truly limits a robot’s capability may not be the volume of data, but whether that data comes from a sufficiently wide range of different environments. The research found that within a single scene, model performance quickly hits a ceiling even as the number of training repetitions keeps rising. Conversely, simply increasing the diversity of objects, environments, and manipulation scenarios improves generalization far faster—even without a large increase in the amount of data.
This result isn’t hard to understand. Even for the same act of picking up a cup, today it might be a glass, tomorrow a stainless-steel one; today the tabletop is level, tomorrow someone might have just bumped the table. Humans live amid these subtle variations every day, so we never realize just how difficult they are for a robot.
For this reason, many startups are working out how to expose robots to a greater variety of situations. The most direct method is to record human operation in full. Quite a few companies make heavy use of teleoperation, having humans control the robot while every joint angle, force profile, and step of the process is saved—the hope being that the more different ways of operating a robot has seen, the better it can adapt to the real world.
Another path is to let the robot accumulate experience on its own. NVIDIA recently unveiled a learning system called ASPIRE, which analyzes the cause after each failed task, revises its control program, and then saves the successful method into a “skill library.” The next time it encounters a similar situation, the robot can correct itself.
A third approach is more imaginative still: training to the extreme directly inside a high-fidelity virtual world. Sudo AI, a U.S. startup backed by Cherubic Ventures, takes a pure-simulation route, having robots repeatedly face countless combinations of materials, lighting, backgrounds, and placements in a virtual environment before being deployed straight into reality.
At this June’s CVPR conference in the United States, this robot took on random challenges from the audience on the spot. Faced with transparent objects, metals, soft materials, and even tiny pills, it could pick up nearly all of them with composure. The seemingly improvised challenges from the audience had, in fact, already been rehearsed countless times in the simulated world. This year SUDO AI closed a new funding round at a valuation of over $2 billion—a bet on exactly this path.
Over the past few years, the success of large language models led everyone to believe that as long as there’s enough data, AI will keep improving. But robotics has developed in the opposite direction: the thorniest feature of the real world is its unpredictable variability. For a robot, repeating a perfect motion ten thousand times in a perfect laboratory brings no real-world application; only by experiencing ten thousand different challenges in an environment full of randomness—finding order within chaos—will the real solution be found.
