Isaac Teleop’s documented hand retargeting maps VR controller trigger and squeeze inputs to robot finger-joint targets. It does not retarget a tracked skeletal hand pose. To get started, load a scenario and YAML profile designed for the same robot, select the matching arm controller, and verify that the profile’s hand prim, joint aliases, and grasp ranges resolve to real joints. You can tune the setup with on-screen debug controls without a headset; live VR input uses a CloudXR-compatible headset.
What Isaac Teleop retargets—and what it does not
The documented TriHand retargeter reads controller trigger and squeeze values and converts them into semantic finger activations. A profile maps those activations to the robot hand’s USD joint names, while a grasp configuration supplies the target range for each joint. NVIDIA distinguishes this input-and-profile mapping from skeletal hand tracking: the documented feature does not capture a tracked hand’s full pose and reproduce it on the robot.
As an Amazon Associate I earn from qualifying purchases.
Hand control is separate from arm control. A floating controller follows a free rigid-body end effector, such as a standalone gripper. An IK controller turns the VR controller’s target pose into position targets for an articulated arm.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Choose how to supply input
| Input option | What it needs | Best for |
|---|---|---|
| Live CloudXR input | A CloudXR-compatible headset, CloudXR running separately, and Isaac Teleop installed with its CloudXR and retargeter extras. | Operating the simulation from tracked VR controllers. |
| Debug mode | No headset, CloudXR, or Isaac Teleop Python package, according to NVIDIA’s tutorial. | Learning the workflow or tuning mappings with draggable markers and sliders. |
| MCAP input replay | The Isaac Teleop package and an MCAP recording of controller and head inputs. | Replaying recorded inputs through the currently selected mapping. |
Start live CloudXR input
NVIDIA’s current tutorial specifies this installation command for Isaac Teleop 1.3.x:
python -m pip install "isaacteleop[cloudxr,retargeters]~=1.3.0"
Start CloudXR separately with python -m isaacteleop.cloudxr --accept-eula, connect a compatible headset on the same network, and launch Isaac Sim. The tutorial’s button mappings target Meta Quest 3. Other headsets can expose different button semantics through OpenXR, so confirm how their controls appear in your setup. Check the Isaac Sim and Teleop versions in use before following version-sensitive instructions. NVIDIA’s installation documentation lists Windows and Linux routes, but the Teleop API documentation says live OpenXR input and MCAP replay require a Linux-only Isaac Teleop prebundle; consult the compatibility matrix for the specific release.
Load a scenario and its matching profile
-
Open a built-in scenario stage that matches the robot and control arrangement you want to use.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy. -
In Isaac Sim, open Tools > Replicator > Teleop and load the YAML profile designed for that stage. For example,
teleop_scenario_floating_xarm_dex3.usdpairs withfloating_xarm_dex3_retargeted.yaml; that profile configures the right Dex3 for TriHand trigger-and-squeeze retargeting.Rank #3
GFTVRCE A pair Original Right and Left handle Hand Controller for Meta Oculus Quest 3- A pair GFTVRCE Original Right and Left handle Hand Controller for Meta Oculus Quest 3
-
Validate the profile before enabling controllers. The stage and profile need to agree on prim paths, and the hand’s aliases must resolve to controllable joints under the configured hand prim. Fix missing or mismatched paths rather than assuming a profile for another robot will work.
NVIDIA also lists floating xArm and single- and dual-UR3e IK examples. Use the supplied matching stage/profile pair as a starting point, then adapt its paths and mappings for a custom robot.
Rank #4
- Genuine Meta Quest Touch Plus Controller – Original right-hand controller designed for Meta Quest 3 and Quest 3S headsets.
- Immersive Haptic Feedback – Advanced vibration and tactile response create more realistic interaction during VR and mixed reality gameplay.
- Precision Input Controls – Features A/B/X/Y buttons, thumbstick, capacitive thumbrest, precision pinch input, and two-stage trigger controls.
- Ergonomic & Comfortable Design – Lightweight controller layout designed for natural hand positioning and extended gaming sessions.
- Accurate VR Tracking Performance – Engineered for responsive movement tracking and smooth interaction across supported Meta Quest applications.
Select the arm controller
| Controller | How it moves the robot | Use it when |
|---|---|---|
| Floating Controller | Tracks the VR controller pose using velocity-based PD control. | The commanded gripper or end effector is a free rigid body, not an articulated arm link. |
| IK Controller | Converts a six-degree-of-freedom target pose into joint-position targets for an articulated arm. | You want an arm to reach the controller’s target pose. Choose the articulation root and an end-effector link; use the wrist when the gripper is controlled separately. |
Inverse-kinematics solver back ends have different prerequisites. Check the selected solver’s requirements for your Isaac Sim release and robot rather than assuming every IK option is interchangeable.
Free tools Windows power users keep installed
One-click scans. No signup required.
Configure trigger, squeeze, and finger-joint mappings
Choose the grasp drive mode
- Trigger: Use the default mode for a conventional gripper whose configured joints are all driven by one squeeze control.
- Retargeted: For the built-in TriHand retargeter, set the grasp drive mode to
retargeted, selecttrihand, and configure the hand prim and grasp configuration. Map the retargeter’s semantic aliases to the actual USD joint names in the robot profile.
The documented TriHand aliases are thumb_rotation, thumb_proximal, thumb_distal, index_proximal, index_distal, middle_proximal, and middle_distal. Do not copy names from a built-in robot blindly: confirm that every alias exists in the grasp configuration and resolves to a controllable joint on your hand. Profile validation checks these relationships.
Understand how TriHand uses the controls
- Trigger drives the index proximal and distal activations.
- Squeeze drives the middle proximal and distal activations.
- Thumb proximal and distal activations use the stronger of trigger and squeeze, with different scaling for each.
- Thumb rotation uses the absolute difference between half the trigger value and half the squeeze value.
These activations are normalized and mapped through the joint-specific target ranges in the grasp configuration. Revolute-joint target ranges are defined in degrees; they are converted internally to radians when required by the articulation tensor backend. Set ranges for the actual robot rather than reusing another hand’s values.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Align tracking coordinates and choose a locomotion target
Correct controller orientation
The documented Isaac Sim setup uses a Z-up coordinate frame. If controller directions do not line up with the robot, first verify the selected coordinate frame. For a persistent yaw correction, author a scene Xform and set it through Session > XR Anchor > Custom Anchor. Do not put the correction under the runtime /Teleop/Markers/TrackingOrigin hierarchy: Teleop recreates that hierarchy.
Choose what thumbstick movement affects
- Target the robot base when thumbsticks should move the robot and its attached arms. You can optionally carry the tracking space to keep the operator anchored relative to the moving robot.
- Target
/Teleop/Markers/TrackingOriginfor a floating gripper without a physical base when you want to move the VR workspace itself. - In Auto mode, Teleop uses velocity for a dynamic rigid-body target and teleport otherwise.
Test the setup and record the right data
Tune without a headset
Enable Debug Mode, then drag the left, right, and head markers. On-screen sliders simulate trigger, squeeze, and thumbstick input. Debug Mode and a live VR connection cannot be active at the same time.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsReplay teleoperation inputs
MCAP records raw controllers and head channels. During playback, Isaac Teleop runs those inputs through the configured retargeting path again; MCAP does not store the resulting robot joint targets or simulation poses. The documented API describes caller-paced playback: while the timeline is playing, provide one input frame per Kit app update. It does not document reliable end-of-file detection, seeking, looping, or timestamp-paced replay.
Keep episode capture separate
The Episode Recorder’s HDF5 workflow records simulation episodes for replay and Replicator dataset generation. That captures simulation episodes, not the raw teleoperation-input stream used by MCAP replay.
Quick Recap
Official documentation
- NVIDIA: Teleoperation Synthetic Data Generation — Isaac Sim Documentation
- NVIDIA: Replicator Teleop — Isaac Sim API
- NVIDIA: Replicator Teleop UI — Isaac Sim API
- NVIDIA: Installation — Isaac Sim Documentation
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
