Add keyboard teleop device to control the end effector robot #1289

michel-aractingi · 2025-06-13T22:05:40Z

What this PR does

Adds KeyboardEndEffectorTeleop as a child of the main keyboard teleop class that is designed to control the end effector of a manipulator in the style of the robot SO100FollowerEndEffector

NOTE: These additions have been tested on an so100 follower arm (teleoperated with my keyboard).

Controls

Arrow keys: Move in X-Y plane
Shift/Shift_R: Move in Z axis
Right/Left Ctrl: Open/close gripper
'i': Toggle intervention mode
'f': End episode with failure
's': End episode with success
'r': End episode with rerecord

Implementation Details

KeyboardEndEffectorTeleop extends base teleop for end-effector specific control keys
KeyboardControlWrapper integrates keyboard control into the gym environment in gym_manipulator.py

Testing

Test with gym manipulator by setting the teleop device in this example config to be:

    "teleop": {
        "type": "keyboard_ee",
        "use_gripper": true
    }

and setting control_mode to "keyboard_ee" in the env config.

Run gym manipulator and teleop with the keyboard (toggle i button to enable/disable interventions)

python lerobot/scripts/rl/gym_manipulator.py --config_path lerobot/configs/env_config_so100.json

…r in gym_manipulator through the keyboard

AdilZouitine

LGTM

…d_ee_teleop_device

michel-aractingi · 2025-06-14T07:09:01Z

Merging after testing the so101_leader and made sure nothing broke in gym_manipulator.py

GGmorello · 2025-06-17T12:55:43Z

Hey does this work in teleoperate.py? I was working at PR #1338 to do so but I was wondering if now it's solved with this.
If it's not I think it would be helpful if we could make it work in teleoperate there and I'm happy to do so.
@michel-aractingi

* fix: sharing predicted chunk with user * [pre-commit.ci] pre-commit autoupdate (#1011) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Revert "[pre-commit.ci] pre-commit autoupdate" (#1025) * fix(ci): Pin draccus (<0.10.0) and torch (<2.7) to fix pipeline (#1022) Co-authored-by: imstevenpmwork <[email protected]> Co-authored-by: Simon Alibert <[email protected]> * fix(ci): Pin `torchcodec` (==0.2.1) to fix pipeline temporarly (#1030) * Update tutorial (#1021) Co-authored-by: Simon Alibert <[email protected]> * Add description motor order SO-101 leader (#1051) * feat(encoding): switching to PyAV for ffmpeg related tasks (#983) * feat(docs): Add new docs build process (#1046) Co-authored-by: Mishig Davaadorj <[email protected]> Co-authored-by: Steven Palma <[email protected]> * Docs: adapt text + fix video code (#1064) * Fix typos (#1070) * docs: minor corrections and clean-up (#1089) * Update 10_use_so100.md; use diff syntax (#944) Co-authored-by: Pepijn <[email protected]> * Update 12_use_so101.md (#1081) Co-authored-by: Pepijn <[email protected]> * bug fix for #1071 When --display_data=true, Failed running control_robot. (#1073) * Add editable -e for feetech install command (#1133) * Fix: emptying action queue between resets (#1117) * fix: typos and grammar (#1148) * Update README.md (#1160) * Update README.md (#1163) * [Fix] Unpin torch beyond 2.6.0 & torchcodec beyond 0.2.1 (#1127) * (hotfix): nightly CI by clipping pymunk version below 7.0.0 (#1182) * [pre-commit.ci] pre-commit autoupdate (#1048) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Simon Alibert <[email protected]> * Add SmolVLA (#1175) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: fracapuano <[email protected]> Co-authored-by: Steven Palma <[email protected]> Co-authored-by: Dana Aubakirova <[email protected]> Co-authored-by: Remi <[email protected]> * Fix SmolVLA loss not sent to wandb (#1198) * Hardware API redesign (#777) Co-authored-by: Pepijn <[email protected]> Co-authored-by: Steven Palma <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Steven Palma <[email protected]> Co-authored-by: Adil Zouitine <[email protected]> Co-authored-by: Pepijn <[email protected]> * fix(smolvla): update record.py, fix populate_queues and remove unused dependencies (#1208) * replaced OBS_ROBOT with OBS_STATE constant (#1211) * Fix test_teleoperate (#1216) * Fix LeKiwi example (#1217) * Fix smolVLA dependencies (#1218) * fix(pyserial): adding pyserial dependency to global ones (#1219) * Update SmolVLA README.md (#1228) * Fix unable to set camera width/height to non-default (#1225) * Update tutorial link (#1250) * update KochFollower.get_observation() so it returns same observation structure as SO101 (#1248) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#1185) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Simon Alibert <[email protected]> * Proposal for fix for enter_pressed on Windows (#1230) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Simon Alibert <[email protected]> * fix: update pi0 dependency version constraint (#1247) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Match motor names with ids lekiwi (#1261) * fix issues: checkpoints keys mismatch and 'task' tokenisation in smolvla (#1256) Co-authored-by: danaaubakirova <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Simon Alibert <[email protected]> Co-authored-by: Simon Alibert <[email protected]> * fix(docs): update realsense documentation (#1268) * Use HF Papers (#1120) * Skip normalization parameters in load_smolvla (#1274) * fix(record): no teleop needed when running with policy (#1284) * Port HIL SERL (#644) Co-authored-by: Michel Aractingi <[email protected]> Co-authored-by: Eugene Mironov <[email protected]> Co-authored-by: s1lent4gnt <[email protected]> Co-authored-by: Ke Wang <[email protected]> Co-authored-by: Yoel Chornton <[email protected]> Co-authored-by: imstevenpmwork <[email protected]> Co-authored-by: Simon Alibert <[email protected]> * fix(docs): SmolVLA fine-tuning getting started (#1201) Co-authored-by: Pepijn <[email protected]> Co-authored-by: danaaubakirova <[email protected]> Co-authored-by: Simon Alibert <[email protected]> Co-authored-by: Francesco Capuano <[email protected]> Co-authored-by: Steven Palma <[email protected]> * chore(teleop): print calibration path saved (#1286) * chore(dependencies): add gamepad support with pygame and hidapi (#1287) * Robot integration tutorial (#1285) * fix(docs): update send_feedback docstrings * Add sim tutorial, fix lekiwi motor config, add notebook links (#1275) Co-authored-by: AdilZouitine <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Michel Aractingi <[email protected]> Co-authored-by: s1lent4gnt <[email protected]> Co-authored-by: Michel Aractingi <[email protected]> Co-authored-by: Eugene Mironov <[email protected]> Co-authored-by: imstevenpmwork <[email protected]> Co-authored-by: Simon Alibert <[email protected]> Co-authored-by: Steven Palma <[email protected]> * Fixes on robot integration tutorial (#1290) * Add keyboard teleop device to control the end effector robot (#1289) * Improve type hints (#1293) * fix(record): no teleop arg in reset environment (#1294) * `learner.py` import so101_leader instead of so100 (#1295) Co-authored-by: Adil Zouitine <[email protected]> * Fixing `PI0` Policy (#1297) * `gym_manipulator.py` Remove None value action_intervention of BaseLeaderTeleoperator (#1299) * (chore): incorrect resume parameter in recording documentation (#1301) * Update lekiwi.mdx (#1229) * bump `pi0` and `hil` transformers version (#1298) * docs: fix imitation learning robots docs command (#1308) * fix(benchmarks): remove .numpy() from frame in benchmark script (#1354) * add smolvla to the supported policies to run tests (: * add: chunk-level access for the policy * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add: smolvla in availables * remove: smolvla from library supported policies * fix: change env for training, xarm is broken as of now * add: predict_action_chunk to all supported policies * fix: add robot type constants * add: predict action chunk in base policy class * restore original Makefile * fix: minor * fix: dict keys come from lerobot/constants * fix: improve act encapsulation, properly supporting temporal ensembling * fix: smolvla action chunking * fix: very minor, but very annoying * fix: minor * fix minor naming Co-authored-by: Steven Palma <[email protected]> Signed-off-by: Francesco Capuano <[email protected]> * fix: refactoring inference for single actions and chunks into different components * fix: minor * fix: temporal ensembling * fix: moving populate queues out of modular component for batch preparation * fix: minor for CI * fix: smovla debug * fix: reward classifier, maybe the last policy lacking? --------- Signed-off-by: Francesco Capuano <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Simon Alibert <[email protected]> Co-authored-by: Adil Zouitine <[email protected]> Co-authored-by: imstevenpmwork <[email protected]> Co-authored-by: Pepijn <[email protected]> Co-authored-by: Caroline Pascal <[email protected]> Co-authored-by: Mishig Davaadorj <[email protected]> Co-authored-by: omahs <[email protected]> Co-authored-by: CharlesCNorton <[email protected]> Co-authored-by: masato-ka <[email protected]> Co-authored-by: Ragnar <[email protected]> Co-authored-by: mshukor <[email protected]> Co-authored-by: Simon Alibert <[email protected]> Co-authored-by: Steven Palma <[email protected]> Co-authored-by: Dana Aubakirova <[email protected]> Co-authored-by: Remi <[email protected]> Co-authored-by: Ben Zhang <[email protected]> Co-authored-by: Pepijn <[email protected]> Co-authored-by: Dhruva <[email protected]> Co-authored-by: Daisuke Sato <[email protected]> Co-authored-by: Sarunas Kalade <[email protected]> Co-authored-by: koenvanwijk <[email protected]> Co-authored-by: Yushun Xiang <[email protected]> Co-authored-by: danaaubakirova <[email protected]> Co-authored-by: Quentin Gallouédec <[email protected]> Co-authored-by: Michel Aractingi <[email protected]> Co-authored-by: Eugene Mironov <[email protected]> Co-authored-by: s1lent4gnt <[email protected]> Co-authored-by: Ke Wang <[email protected]> Co-authored-by: Yoel Chornton <[email protected]> Co-authored-by: Michel Aractingi <[email protected]> Co-authored-by: tidely <[email protected]> Co-authored-by: David <[email protected]>

…face#1289)

michel-aractingi added 3 commits June 13, 2025 23:48

(feature) added new teleop_keyboard device to control the end effecto…

2dbbe05

…r in gym_manipulator through the keyboard

(fix) change the intervention key to i instead of space

842c016

(fix) sign of y axis

f894097

michel-aractingi marked this pull request as ready for review June 13, 2025 22:12

AdilZouitine approved these changes Jun 14, 2025

View reviewed changes

Merge branch 'main' into user/michel-aractingi/2025-06-13-add_keyboar…

6a45242

…d_ee_teleop_device

michel-aractingi merged commit 6007a22 into main Jun 14, 2025
12 checks passed

michel-aractingi deleted the user/michel-aractingi/2025-06-13-add_keyboard_ee_teleop_device branch June 14, 2025 07:10

fracapuano pushed a commit that referenced this pull request Jun 21, 2025

$@fracapuano$

Add keyboard teleop device to control the end effector robot (#1289)

c54e9d4

fracapuano pushed a commit that referenced this pull request Jun 22, 2025

$@fracapuano$

Add keyboard teleop device to control the end effector robot (#1289)

23c28b1

This was referenced Jun 30, 2025

Feature implementation from commits 5350a02..2b71789 codeOwlAI/lerobot#1

Open

Feature implementation from commits edfebd5..ce6a26d codeOwlAI/lerobot#2

Open

cxchhh pushed a commit to cxchhh/lerobot-pi0 that referenced this pull request Jul 3, 2025

Add keyboard teleop device to control the end effector robot (hugging…

e90c94e

…face#1289)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add keyboard teleop device to control the end effector robot #1289

Add keyboard teleop device to control the end effector robot #1289

Uh oh!

michel-aractingi commented Jun 13, 2025 •

edited

Loading

Uh oh!

AdilZouitine left a comment

Uh oh!

michel-aractingi commented Jun 14, 2025

Uh oh!

Uh oh!

GGmorello commented Jun 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

Add keyboard teleop device to control the end effector robot #1289

Add keyboard teleop device to control the end effector robot #1289

Uh oh!

Conversation

michel-aractingi commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Controls

Implementation Details

Testing

Uh oh!

AdilZouitine left a comment

Choose a reason for hiding this comment

Uh oh!

michel-aractingi commented Jun 14, 2025

Uh oh!

Uh oh!

GGmorello commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

michel-aractingi commented Jun 13, 2025 •

edited

Loading

GGmorello commented Jun 17, 2025 •

edited

Loading