How should hand pose control a robot when a bad command has consequences?

The control problem

A camera can make robot control feel direct while hiding the exact failures that matter: occlusion, stale frames, depth ambiguity, mode confusion, link delay, and unsafe restart.

The former prototype converted image movement into values labelled as metres, treated handedness probability as tracking confidence, and held the last target after dropout. It had no clutch, independent stop path, command acknowledgement, robot feedback, force sensing, collision planner, or replayable safety decision.

The replacement narrows the claim. The camera estimates a hand skeleton. A calibration frame converts relative geometry into dimensionless X, Y, depth-cue, roll, and grip intent. Those values remain untrusted until a supervisor admits them. Metric motion belongs to a robot-specific gateway with a measured calibration and hazard analysis.

Put the safety boundary near the robot

The testbed deliberately separates a convenient input device from the controls that would make a deployed manipulator safe. Only the first two stages exist in this website. The transport, robot gateway, and independent stop path are interface requirements for a separate edge-control system.

1. 01

   Vision input

   A local MediaPipe model estimates 21 landmarks. Calibration maps image geometry into normalized rate intent; it does not infer metric distance.

   Untrusted

2. 02

   Command supervisor

   A monotonic sample passes calibration, hand-presence, geometry, age, hold-to-run, reacquisition, slew, and workspace checks.

   Tested in browser

3. 03

   Transport contract

   A deployment adapter must authenticate the session and reject expired, duplicated, reordered, or out-of-envelope intent frames.

   Specified / not connected

4. 04

   Robot-side gateway

   A controller beside the robot owns kinematics, collision, force, speed, tool, and task limits. That controller is not implemented by this website.

   Production target

5. 05

   Independent stop path

   Protective stop and emergency stop must not share the camera, browser, network, or application failure path.

   Required / outside browser

Boundary rule: the vision model, browser, and wide-area network are command sources. None is credited as a safety function in the current design.

The rebuilt control bench

The route accepts only local camera inference. Camera access is explicit: the pinned model and WASM runtime load from the same origin, inference runs in a worker, and frames are not uploaded or retained by the application. Admitted camera rates advance a bounded Cartesian setpoint before inverse kinematics and simulated plant feedback are evaluated.

Landmark runtime

MediaPipe Tasks Vision 0.10.35 / local assets

Model

Hand Landmarker full float16 v1 / pinned hash

Inference

Dedicated browser worker / camera stays on device

Command units

Normalized rates and grip request / no metric claim

Supervisor tests

8 deterministic tests / passing 10 July 2026

Physical robot link

Not implemented

Three profile presets change simulator watchdog, reacquisition, command slew, workspace, injected latency, and packet loss. They are bench inputs, not standard limits. A deployment must replace them with task-specific budgets and verified stop behavior.

The command is an expiring request

Position targets can preserve dangerous intent after the operator or link disappears. The rebuilt bench emits a normalized rate request with an explicit validity decision. A failed gate makes every motion rate zero and marks the command invalid. The downstream controller still chooses the task-specific safe action.

{
  "schema": "li.teleop.intent/1",
  "sessionId": "local-sim-…",
  "sequence": 1842,
  "capturedAtMonotonicMs": 91342.5,
  "expiresAfterMs": 140,
  "source": "camera",
  "intent": {
    "linearRate": [0.18, -0.04, 0.00],
    "rollRate": 0.07,
    "grip": 0.61
  },
  "gates": {
    "calibrated": true,
    "handVisible": true,
    "geometryValid": true,
    "clutchHeld": true
  }
}

Supervisor states and recovery

Motion authority, input quality, and stop state are separate. “Tracking” is not “armed,” and “armed” is not “moving.” The operator must deliberately close hold-to-run while every automatic gate remains valid.

“Hold” is only the simulator’s zero-rate behavior. It is not a universal physical safe state. A collision in orbit, a retained subsea load, and a tool inside a radiation field can require different responses: controlled stop, maintain-grasp, retreat, park, release, or torque-off.

One input method, three qualification paths

There is no certificate called “space, subsea, and nuclear compliant.” The complete robot, task, operating environment, communications path, operator, and safety case determine which standards and evidence apply.

What the current tests prove

Tests are reported with their scope. A passing pure-function fixture proves a branch of supervisor behavior. It does not prove browser timing, camera accuracy, robot stopping, or field suitability.

Test baseline: Node 23.9.0 / eight deterministic tests / 10 July 2026. Browser and production-build results are added only after the route is exercised.

The route from tested bench to deployed system

“Production” starts when the robot and task are named. Until then, the useful output is a versioned command contract, safety architecture, hazard log, and test harness that can be tailored without pretending the browser has been certified.