Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new documentation guide explaining how to use --rename_map to reconcile dataset/environment observation keys with a policy’s expected input keys, and how empty_cameras enables running multi-camera policies on fewer-camera datasets.
Changes:
- Introduces
docs/source/rename_map.mdxcovering rename map usage for training and evaluation. - Documents the “empty cameras” mechanism and the
empty_camerasconfig setting. - Adds a quick reference table and illustrative diagrams.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Policies have a fixed set of **input feature names** (often coming from a pretrained config). For example: | ||
|
|
||
| - **XVLA-base** expects three image keys: `observation.images.image`, `observation.images.image2`, `observation.images.image3`. | ||
| - **pi0-fast-libero** might expect `observation.images.base_0_rgb` and `observation.images.left_wrist_0_rgb`. |
There was a problem hiding this comment.
The policy name here is inconsistent with the rest of the docs/repo naming. Elsewhere we refer to the model as pi0fast-libero (no hyphen after pi0). Using pi0-fast-libero may confuse users and doesn’t match the Hugging Face repo ID shown later in this doc.
| - **pi0-fast-libero** might expect `observation.images.base_0_rgb` and `observation.images.left_wrist_0_rgb`. | |
| - **pi0fast-libero** might expect `observation.images.base_0_rgb` and `observation.images.left_wrist_0_rgb`. |
| - **Keys** = observation keys as produced by your **dataset** (training) or **environment** (evaluation). | ||
| - **Values** = the observation keys your **policy** expects. | ||
|
|
||
| Only keys listed in the map are renamed; everything else is left as-is. Under the hood, the [RenameObservationsProcessorStep](https://github.com/huggingface/lerobot/blob/main/src/lerobot/processor/rename_processor.py) runs in the preprocessor and rewrites observation keys (and keeps normalization stats aligned) so the batch matches the policy’s `input_features`. |
There was a problem hiding this comment.
This sentence implies the rename processor automatically keeps normalization stats aligned, but RenameObservationsProcessorStep only renames observation keys and transforms feature metadata. Stats are not renamed automatically (there’s a separate rename_stats(...) helper used elsewhere, e.g. in lerobot_record.py). Consider rephrasing to avoid implying stats are handled, or explicitly mention that stats must be renamed too if normalization depends on renamed keys.
| Only keys listed in the map are renamed; everything else is left as-is. Under the hood, the [RenameObservationsProcessorStep](https://github.com/huggingface/lerobot/blob/main/src/lerobot/processor/rename_processor.py) runs in the preprocessor and rewrites observation keys (and keeps normalization stats aligned) so the batch matches the policy’s `input_features`. | |
| Only keys listed in the map are renamed; everything else is left as-is. Under the hood, the [RenameObservationsProcessorStep](https://github.com/huggingface/lerobot/blob/main/src/lerobot/processor/rename_processor.py) runs in the preprocessor and rewrites observation keys and updates feature metadata so the batch matches the policy’s `input_features`. If you rely on per-key normalization statistics, make sure to rename those stats consistently with your rename map as well. |
| | Goal | What to do | | ||
| | ------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | ||
| | Dataset keys ≠ policy keys (training) | `--rename_map='{"dataset_key": "policy_key", ...}'` | | ||
| | Env keys ≠ policy keys (eval) | `--rename_map='{"env_key": "policy_key", ...}'` | | ||
| | Fewer cameras than policy expects | Set `empty_cameras` in the policy config (e.g. `1` when you have 2 real cameras and the policy expects 3). | | ||
| | Avoid passing a rename map | Edit the policy’s `config.json` so its observation keys match your dataset or env. | |
There was a problem hiding this comment.
The quick-reference table uses || at the start of each row, which is not valid Markdown table syntax and likely won’t render correctly. It should be a standard pipe table (single leading | per row).
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Title
docs: add rename map and empty cam guide