One bimanual task recorded untrimmed on two -DoF arms, failures included: cameras with depth, joints, poses, wrist force-torque and Hz joint commands on one clock. A sample of every episode, and a full release of a few with everything as recorded, lossless. .
What the overview leaves out: every episode and subtask, the rig and its calibration, every column with its unit and frame, the per-episode record, the file layout, the conventions, known limits and code to load it.
The left arm picks a cable, passes it to the right arm, and the right arm sorts it by color into the two compartments of the target bin. The operator marks each step live with a foot pedal while driving; each clip shows one step of one episode in all four views.
Steps: ✓ success, ✗ failure, in order. Lag: how long after the operator's TCP target the measured
arm follows it; most of it is the arm's own controller, and the hand controller's smoothing is not in it (see
Command lag under “Know before you train”). Live command: share of rows where the arm follows a live teleoperation command
(paddy.action_valid); it is lower while an arm waits for the other arm's turn and while the operator
pauses. This sample was not recorded by a trained expert operator. Click a row to open the episode.
The green shapes are not tracked or fitted to the video. They are the arms, grippers and force-torque sensor of
the URDF that ships with the dataset, posed at each row's measured joint angles and projected through each
camera's calibration in meta/cameras.json, lens distortion included. If the kinematics, a calibration or
the time alignment were off, the outlines would slide off the arms.
Two arms mounted on one frame reach into shared bins. A fixed overhead camera sees the whole scene, a fixed scene camera looks at the bins and the handover from the side, and a camera on each wrist sees the grasp up close. The operator drives each arm with one VR hand controller.
Intrinsics describe the video as delivered (it is not undistorted). Extrinsics place
each camera's color optical frame in an arm's frame and were measured against a ChArUco board; the record of each
calibration ships in meta/cameras.json, and a wrist camera's pose at every row is in
paddy.camera_pose.<camera>.
Each fps row carries the latest reading of every stream at or before its time. Nothing is resampled into the past and nothing is dropped: where a camera missed a slot, the row says how old its frame is.
Every camera records depth aligned to its color image, so a pixel in the video has a distance in the depth file. The full release ships it lossless at each color image's size and rate ( for the overhead camera, fps) as -bit millimetres, with each camera's own depth image beside it; 0 means no measurement.
All features in meta/info.json of the full release, grouped. Units, frames and element
names are read from the dataset itself; open a row's names to see what each element of a vector is. Rows marked
full only are not in the three-camera release.
Besides LeRobot's own episode index (meta/episodes/, which also carries the outcome, scene, labels
and episode id), each episode has a sidecar, meta/paddy/episodes/<episode>.json, with its labels,
subtasks, measured rates, lags and stream quality. The fields most worth knowing:
Standard at the top level, so LeRobotDataset loads either release as is; everything extra
sits beside it.
The rules every column follows, from meta/paddy/info.json.
Every number below is computed from the dataset and recorded in its metadata, per episode.
Plain LeRobot for policy training; pandas and PyAV when you want the raw columns and depth.
Cable sorting is the start. New bimanual tasks will join the sample as we record them.
Larger releases in exactly the same format, so code written against this sample keeps working.
The full release has episodes with everything recorded, lossless; the others are available in the same form.
Tell us the task, the objects and the scene; we collect it on our rig and deliver it in this format.
We collect robot training data and build the collection rigs that record it. More tasks and hours-long releases in this same format are on the way, and we record your task on request. Tell us what your policy needs to learn.