GitHub – rerun-io/pi0-lerobot at hand-kinematic-fitting

A repo to explore training robots with Pi0 and Lerobot and human pose motion retargeting

Currently only linux is supported

Make sure you have the Pixi package manager installed

git clone https://github.com/rerun-io/pi0-lerobot.git
cd pi0-lerobot

Get camera intrinsics/extrinsics, depth maps, and pointcloud for a set of multiview images from hocap

To run camera calibration on a sequence run

pixi run calibrate-cameras-vggt

For HOCap dataset, sample is automatically downloaded from dataset on command run. Checkout example rrd file here

For the full assembly101 dataset go to this link to get/download dataset

To run dataset visualization

pixi run visualize-hocap-dataset # recommended
pixi run visualize-assembly101-dataset

Current Pipeline

To run 2D pose estimation, tracking, and triangulation

pixi run pose-estimation-assembly101

The goal is to create a pipeline that converts time-synced multi-camera footage into axis-angle joint angles (θ) and metric 3D positions for every finger joint. This provides “motion fuel” for robotic retargeting and learning fine motor skills.

Why use a kinematic skeleton with joint angles?

• Rig-agnostic Pose: Captures relative bone rotations, allowing transfer to any avatar or robot hand.
• Easy Retargeting: A single Forward Kinematics (FK) pass transfers motion.
• Compact Storage: Requires only ~60 floats per frame.
• Built-in Constraints: Easily enforce bone lengths, joint limits, etc.
• Gradient-Friendly: Differentiable FK enables optimization from 2D errors.
• Sensor Fusion-Ready: Integrates well with IMUs, mocap markers, etc.

Capture-to-Angles Pipeline:

1. Input: Time-synced, calibrated RGB frames.
2. 2D Keypoints: Detect pixel joint locations using a 2D keypoint detector.
3. 3D Joints: Triangulate or use PnP with camera parameters to get metric 3D joint positions.
4. Inverse Kinematics (IK): Convert 3D joints into axis-angle joint parameters (θ).
5. Forward Kinematics (FK) Check: Rebuild 3D joints from θ for validation.
6. Output: Serialize axis-angle θ and metric 3D joint positions (xyz).

pixi run notebook_tutorial