Abstract

In recent years, significant progress has been made in the field of roboticreinforcement learning (RL), enabling methods that handle complex imageobservations, train in the real world, and incorporate auxiliary data, such asdemonstrations and prior experience. However, despite these advances, roboticRL remains hard to use. It is acknowledged among practitioners that theparticular implementation details of these algorithms are often just asimportant (if not more so) for performance as the choice of algorithm. We positthat a significant challenge to widespread adoption of robotic RL, as well asfurther development of robotic RL methods, is the comparative inaccessibilityof such methods. To address this challenge, we developed a carefullyimplemented library containing a sample efficient off-policy deep RL method,together with methods for computing rewards and resetting the environment, ahigh-quality controller for a widely-adopted robot, and a number of challengingexample tasks. We provide this library as a resource for the community,describe its design choices, and present experimental results. Perhapssurprisingly, we find that our implementation can achieve very efficientlearning, acquiring policies for PCB board assembly, cable routing, and objectrelocation between 25 to 50 minutes of training per policy on average,improving over state-of-the-art results reported for similar tasks in theliterature. These policies achieve perfect or near-perfect success rates,extreme robustness even under perturbations, and exhibit emergent recovery andcorrection behaviors. We hope that these promising results and our high-qualityopen-source implementation will provide a tool for the robotics community tofacilitate further developments in robotic RL. Our code, documentation, andvideos can be found at https://serl-robot.github.io/