Sim-to-real
426 words · 3 min read · 2 sources
Sim-to-real is the challenge — and the set of techniques — involved in training a robot in a computer simulation and then deploying that trained behaviour on a physical robot in the real world without starting over.
Training a robot to walk by letting it fall over and recover, over and over again, is the most natural way to learn locomotion. But a real robot is expensive, its joints wear out, and a single training run might require millions of attempts. The obvious solution: teach it in a computer simulation, where gravity is free, crashes cost nothing, and you can run
Training a robot to walk by letting it fall over and recover, over and over again, is the most natural way to learn locomotion. But a real robot is expensive, its joints wear out, and a single training run might require millions of attempts. The obvious solution: teach it in a computer simulation, where gravity is free, crashes cost nothing, and you can run a thousand copies of the robot in parallel overnight. The catch comes the moment you switch on the real robot and it does not work.
That gap between simulated competence and real-world performance is called the sim-to-real transfer problem — and the collection of techniques used to bridge it goes by the shorthand sim-to-real.
Why simulations lie
A physics simulation is a mathematical approximation of the real world. It cannot perfectly model every friction coefficient, every motor imperfection, every millimetre of manufacturing tolerance. A robot policy trained in simulation learns to exploit those tiny unrealistic details, and when those details vanish in the real world, performance collapses. The formal name for this is the reality gap.
The main bridging techniques
Domain randomisation deliberately injects noise into the simulation: random friction values, random masses, random lighting, random visual textures. If the policy learns to work across all these variations, it encounters nothing surprising in reality because reality is just one more sample from that range of environments.
Domain adaptation takes the opposite approach — make the simulation look and behave as much like reality as possible, using more accurate physics engines (MuJoCo, Isaac Sim) and photorealistic rendering.
Real-world fine-tuning uses sim training to produce a good starting policy, then refines it with a small amount of real-world experience. The simulation does the heavy lifting; real data provides the correction.
A landmark example
In 2019, OpenAI trained a robotic hand to solve a Rubik's cube in simulation, running the equivalent of 10,000 years of experience in parallel across a large computing cluster, then deployed the trained policy directly on a physical hand. The key to transfer was aggressive domain randomisation — the simulation randomised over hundreds of physical parameters simultaneously.
Why it matters
Without sim-to-real techniques, robot training would be slow, expensive, and limited to simple tasks. With them, researchers can solve problems in days of simulation time that would take years in reality.
The same domain-randomisation technique used to train robot hands is now being applied to train entire humanoid robots to walk on terrain the simulation never showed them.
Next: one minute
Watch how Sim-to-real actually works
A one-minute video, an interactive demo you can poke at, and one question to check it landed.
Watch the one-minute lesson →Plain English. Free. Unsubscribe in one click.
Keep going
Sim-to-Real Transfer in Robotics
Sim-to-Real bridges robot policies trained in simulation to the real world. Solves the data-cost problem of re…
ConceptBehaviour cloning
Behaviour cloning is a machine-learning technique where a robot is trained to copy an expert's actions as clos…
ConceptImitation learning
Imitation learning is a training approach where a robot learns to behave by watching demonstrations of a task …
Last updated · 2026-05-19
Spotted something off? Report an error →