Presa-Mano/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Commands

Build & run (Docker):

docker compose up -d presa-mano        # serves GUI + API on http://localhost:8890

Local dev (uses uv, same as the Dockerfile):

uv sync                                # install deps from pyproject.toml
uv run uvicorn presa_mano.main:app --host 0.0.0.0 --port 8890 --reload

Useful endpoints (all under http://localhost:8890):

• GET /apidocs — Swagger UI; GET /redoc — ReDoc.
• GET/PUT /api/config — read/patch runtime config (deep-merged into config.yaml and persisted).
• Calibration: POST /api/calibration/sample, GET/DELETE /api/calibration/samples, POST /api/calibration/solve, GET /api/calibration/result, POST /api/calibration/preview.
• Perception: POST /api/perception/detect, POST /api/perception/object_points, GET /api/perception/measure.
• Grasp: POST /api/grasp/plan, POST /api/grasp/execute (set dry_run to gate G1 mutations).
• Wrist/EE: GET /api/wrist, POST /api/wrist/{absolute,relative,hand,emergency_stop,unlock}.
• Right arm: GET /api/right_arm/state, POST /api/right_arm/detect_{front,wrist}, POST /api/right_arm/move_toward, POST /api/right_arm/{home,custom_pose}.
• Pose 3D: GET /api/pose_3d/state (wrist + joints + config, incl. cross_camera calibration), POST /api/pose_3d/apply, GET /api/pose_3d/clouds (Nuwa + D435i point clouds), GET /api/pose_3d/tags (ArUco su Nuwa polso + D435i frontale), POST /api/pose_3d/align?align_id=N (fit Umeyama Nuwa→frontale, persiste in pose_3d.cross_camera), CRUD /api/pose_3d/poses/{name}.

Tests live under tests/ and run with uv run pytest. No linter is configured.

Architecture

This is a FastAPI orchestrator that glues together external microservices to implement an eye-to-hand calibration + object-grasping + 3D visualization pipeline for a Unitree G1 humanoid. It owns no hardware itself — every action is an HTTP call.

Upstream services (defaults in config.yaml > services):

Service	Default URL	Role
Nuwa-HP60C	:8877	RGB, depth PNG, point cloud (full + ROI), intrinsics, pixel→3D measure. Eye-in-hand sul polso DX
Front camera (D435i)	services.front_camera	Point cloud frontale fissa sul torso; stessa interfaccia NuwaClient
marker-finder	:8082	ArUco detection (returns corners, position_m, rotation_matrix)
YOLO server	:8879	Object bbox detection from an image URL
G1 API	:8765	Wrist pose read/write, hand open/close, e-stop, /diagnostics (29 joint positions)
SAM21 (NuwaSeg)	:8002	Promptable segmentation: rgb+depth+intrinsics+bbox → 3D point cloud of the object (replaces the bbox-only ROI cloud in the grasp pipeline)

Layered package layout (src/presa_mano/)

• main.py — FastAPI app; mounts web/ at /static, serves web/index.html at /, registers routers under /api/*.
• config.py — Thread-safe singleton config loader. load_config() reads YAML once; update_config(patch) deep-merges and persists. Path comes from PRESA_MANO_CONFIG env (Dockerfile sets /app/config.yaml); data files live under PRESA_MANO_DATA (default /app/data).
• api/ — Routers, one per concern (calibration, config, grasp, home, models, perception, pose_3d, posture, right_arm, robot, work_plane, wrist). Build clients per-request via api/_deps.py (nuwa(), front_camera(), marker(), yolo(), sam21(), g1(dry_run=...)) so runtime PUT /api/config changes take effect without restart. Don't cache clients at module level.
• clients/ — Thin httpx wrappers, one per upstream. Errors are normalized to service-specific exceptions (e.g. G1Unavailable, NuwaError).
• calibration/ — sampler.py builds one (T_base_wrist, T_cam_marker) sample; handeye.py runs cv2.calibrateHandEye and computes residuals; store.py persists samples (JSON) and the result (YAML).
• perception/ — detector.py calls YOLO and normalizes responses across multiple envelope shapes; pointcloud.py handles frame conversion + outlier filtering.
• grasp/ — axis.py PCA → ObjectAxes; planner.py builds a wrist pose from those axes.
• geometry.py — pure-numpy SE(3) helpers (quat↔R, rvec↔R, invert_T, transform_points).
• timing.py — Stopwatch with step(name) context manager; every endpoint annotates its response with timings_ms.
• web/ — vanilla HTML/JS/CSS GUI served as static files; talks only to /api/*. Tab principali: Calibrazione, Calibrazione Piano, Cerca Modello, Movimento EE, Braccio DX, Pose 3D. Il viewport 3D (pose_3d.js) usa Three.js + URDFLoader per il modello robot.

Core data flow: hand-eye calibration (eye-to-hand)

The camera is fixed; an ArUco marker is mounted on the wrist. For each sample:

1. G1.wrist() → T_base_wrist (via T_from_pos_quat).
2. marker-finder.detect_url(image_url=nuwa.rgb_url()) → T_cam_marker from position_m + rotation_matrix.
3. samples.json accumulates samples; HandEyeSolver.solve() calls cv2.calibrateHandEye.

Eye-to-hand trick (calibration/handeye.py): OpenCV documents the eye-in-hand case. We feed invert_T(T_base_wrist) so the solver returns T_camera→base directly.

Residual quality check: with a correct calibration, T_wrist_marker = invert_T(T_base_wrist) @ T_base_camera @ T_cam_marker should be constant across samples (marker is rigidly attached). Mean translation/rotation deviations from sample 0 are reported as residual_translation_mm / residual_rotation_deg.

Core data flow: grasp pipeline

POST /api/grasp/plan (or /execute):

1. YOLO detect on the current Nuwa RGB → top bbox (optionally filtered by class_name).
2. Bbox → pixel ROI → nuwa.pointcloud_roi(...) (Nuwa crops server-side).
3. filter_outliers (n_std rule) → if <10 points, 422.
4. Frame conversion: nuwa_to_cv_frame() flips Y and Z (Nuwa = X-right/Y-up/Z-back; OpenCV = X-right/Y-down/Z-forward). Then transform_points(T_base_camera, ...) lifts to base frame.
5. compute_axes (PCA) → ObjectAxes with sorted extents.
6. GraspPlanner.plan builds R_wrist = [x_hand | y_hand | z_hand] where x_hand = approach direction (currently hard-coded top-down, -Z_base), y_hand = horizontalized closure axis from PCA mid-axis. Returns grasp + pre_grasp poses (the latter offset by approach_offset_m along +Z).
7. /execute sequences: hand_open → pre_grasp → grasp_approach → hand_close, each timed.

Frame conventions (gotcha-prone)

There are three coexisting camera-frame conventions — keep them straight:

• Nuwa native (/api/pointcloud): X-right, Y-up, Z-back (Z<0 in front).
• OpenCV / solvePnP (T_cam_marker from marker-finder, T_base_camera from hand-eye): X-right, Y-down, Z-forward (Z>0 in front).
• Nuwa ROI (/api/pointcloud/roi): already returned in OpenCV convention [x, -y, -z, r, g, b] server-side. The wrapper NuwaClient.pointcloud_roi does not re-flip — but the grasp pipeline still calls nuwa_to_cv_frame on the result. Be aware when adding new code paths that consume Nuwa point clouds: full-frame needs flipping, ROI nominally does not (this is currently inconsistent — verify behavior against actual data when modifying).

Marker-finder caveat: it uses D435i intrinsics baked into its config that don't match the Nuwa. The corners it returns are reusable; the absolute pose position_m/rotation_matrix carries that intrinsics error into the calibration. Marker-finder accuracy is the dominant error source in T_base_camera.

Conventions to preserve

• Dry-run semantics: any G1-mutating endpoint takes a dry_run flag; G1Client._post short-circuits and logs [DRY-RUN] instead of hitting the network. Never silently strip dry-run from a mutating path.
• Per-request client construction (via api/_deps.py) is intentional — runtime PUT /api/config updates would otherwise be invisible.
• SAM21 vs bbox frame convention: the grasp pipeline has two parallel paths in _object_points_in_base. The SAM21 path returns points already in the OpenCV RGB-camera frame and must NOT call nuwa_to_cv_frame; the legacy bbox path (via _points_via_bbox) still calls it because Nuwa's ROI cloud uses a different convention. Don't accidentally add the flip to the SAM21 branch — coordinates would double-flip.
• Pose 3D graceful offline: GET /api/pose_3d/state returns 200 even when G1 is unreachable — wrist and joints are null, robot_error carries the message. The tab must remain usable for editing config / named poses / viewing the URDF model without a live robot.
• Pose 3D clouds: GET /api/pose_3d/clouds acquires from Nuwa (eye-in-hand, polso DX) and D435i (front camera, torso) in parallel. Both are returned in OpenCV optical frame (X right, Y down, Z forward). The client attaches them to URDF links with configurable offset/rpy from config.yaml > pose_3d.clouds.
• Pose 3D joints mirroring: G1Client.diagnostics() → motors[].{name, q} gives real joint positions in radians. The frontend uses these to pose the URDF model directly, bypassing client-side IK.
• Pose 3D cross-camera alignment: POST /api/pose_3d/align fits an Umeyama similarity (rotation + translation + uniform scale) mapping Nuwa-optical → D435-optical from a tag (align_id) seen by both cameras, and persists it to config.yaml > pose_3d.cross_camera. The frontend renders the Nuwa cloud through the D435i transform pipeline (d435LocalMatrix() · alignMatrix) when cross_camera.enabled. If the tag isn't seen by both cameras, the saved matrix is NOT overwritten and the Nuwa cloud falls back to URDF placement. Tag detection (GET /tags) runs on both cameras; the GUI draws Nuwa tags red, D435i tags cyan.
• Endpoints return { ..., "timings_ms": {step: ms, ..., total: ms} }. New endpoints should follow this with Stopwatch().step(name).
• Calibration paths in config (calibration.file, calibration.samples_file) are resolved against PRESA_MANO_DATA when relative — see _deps.calibration_file_path() / samples_file_path().
• Italian is fine in user-facing copy (GUI, README, comments) — it's the project default.