diff --git a/CLAUDE.md b/CLAUDE.md index 33290e7..95f3915 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -22,34 +22,37 @@ Useful endpoints (all under `http://localhost:8890`): - Perception: `POST /api/perception/detect`, `POST /api/perception/object_points`, `GET /api/perception/measure`. - Grasp: `POST /api/grasp/plan`, `POST /api/grasp/execute` (set `dry_run` to gate G1 mutations). - Wrist/EE: `GET /api/wrist`, `POST /api/wrist/{absolute,relative,hand,emergency_stop,unlock}`. +- Right arm: `GET /api/right_arm/state`, `POST /api/right_arm/detect_{front,wrist}`, `POST /api/right_arm/move_toward`, `POST /api/right_arm/{home,custom_pose}`. +- Pose 3D: `GET /api/pose_3d/state` (wrist + joints + config), `POST /api/pose_3d/apply`, `GET /api/pose_3d/clouds` (Nuwa + D435i point clouds), `CRUD /api/pose_3d/poses/{name}`. Tests live under `tests/` and run with `uv run pytest`. No linter is configured. ## Architecture -This is a **FastAPI orchestrator** that glues together four external microservices to implement an eye-to-hand calibration + object-grasping pipeline for a Unitree G1 humanoid using a fixed Nuwa-HP60C depth camera. It owns no hardware itself — every action is an HTTP call. +This is a **FastAPI orchestrator** that glues together external microservices to implement an eye-to-hand calibration + object-grasping + 3D visualization pipeline for a Unitree G1 humanoid. It owns no hardware itself — every action is an HTTP call. Upstream services (defaults in `config.yaml > services`): | Service | Default URL | Role | |---------|------------|------| -| Nuwa-HP60C | `:8877` | RGB, depth PNG, point cloud (full + ROI), intrinsics, pixel→3D measure | +| Nuwa-HP60C | `:8877` | RGB, depth PNG, point cloud (full + ROI), intrinsics, pixel→3D measure. Eye-in-hand sul polso DX | +| Front camera (D435i) | `services.front_camera` | Point cloud frontale fissa sul torso; stessa interfaccia NuwaClient | | marker-finder | `:8082` | ArUco detection (returns `corners`, `position_m`, `rotation_matrix`) | | YOLO server | `:8879` | Object bbox detection from an image URL | -| G1 API | `:8765` | Wrist pose read/write, hand open/close, e-stop | +| G1 API | `:8765` | Wrist pose read/write, hand open/close, e-stop, `/diagnostics` (29 joint positions) | | SAM21 (NuwaSeg) | `:8002` | Promptable segmentation: rgb+depth+intrinsics+bbox → 3D point cloud of the object (replaces the bbox-only ROI cloud in the grasp pipeline) | ### Layered package layout (`src/presa_mano/`) - `main.py` — FastAPI app; mounts `web/` at `/static`, serves `web/index.html` at `/`, registers routers under `/api/*`. - `config.py` — Thread-safe singleton config loader. `load_config()` reads YAML once; `update_config(patch)` deep-merges and persists. Path comes from `PRESA_MANO_CONFIG` env (Dockerfile sets `/app/config.yaml`); data files live under `PRESA_MANO_DATA` (default `/app/data`). -- `api/` — Routers, one per concern. **Build clients per-request via `api/_deps.py`** (`nuwa()`, `marker()`, `yolo()`, `sam21()`, `g1(dry_run=...)`) so runtime `PUT /api/config` changes take effect without restart. Don't cache clients at module level. +- `api/` — Routers, one per concern (calibration, config, grasp, home, models, perception, pose_3d, posture, right_arm, robot, work_plane, wrist). **Build clients per-request via `api/_deps.py`** (`nuwa()`, `front_camera()`, `marker()`, `yolo()`, `sam21()`, `g1(dry_run=...)`) so runtime `PUT /api/config` changes take effect without restart. Don't cache clients at module level. - `clients/` — Thin `httpx` wrappers, one per upstream. Errors are normalized to service-specific exceptions (e.g. `G1Unavailable`, `NuwaError`). - `calibration/` — `sampler.py` builds one (T_base_wrist, T_cam_marker) sample; `handeye.py` runs `cv2.calibrateHandEye` and computes residuals; `store.py` persists samples (JSON) and the result (YAML). - `perception/` — `detector.py` calls YOLO and normalizes responses across multiple envelope shapes; `pointcloud.py` handles frame conversion + outlier filtering. - `grasp/` — `axis.py` PCA → `ObjectAxes`; `planner.py` builds a wrist pose from those axes. - `geometry.py` — pure-numpy SE(3) helpers (quat↔R, rvec↔R, invert_T, transform_points). - `timing.py` — `Stopwatch` with `step(name)` context manager; every endpoint annotates its response with `timings_ms`. -- `web/` — vanilla HTML/JS/CSS GUI (3 tabs: Calibration, Grasp, Movimento EE), served as static files; talks only to `/api/*`. +- `web/` — vanilla HTML/JS/CSS GUI served as static files; talks only to `/api/*`. Tab principali: Calibrazione, Calibrazione Piano, Cerca Modello, Movimento EE, Braccio DX, Pose 3D. Il viewport 3D (pose_3d.js) usa Three.js + URDFLoader per il modello robot. ### Core data flow: hand-eye calibration (eye-to-hand) @@ -88,6 +91,9 @@ Marker-finder caveat: it uses **D435i intrinsics baked into its config** that do - **Dry-run semantics:** any G1-mutating endpoint takes a `dry_run` flag; `G1Client._post` short-circuits and logs `[DRY-RUN]` instead of hitting the network. Never silently strip dry-run from a mutating path. - **Per-request client construction** (via `api/_deps.py`) is intentional — runtime `PUT /api/config` updates would otherwise be invisible. - **SAM21 vs bbox frame convention:** the grasp pipeline has two parallel paths in `_object_points_in_base`. The SAM21 path returns points already in the OpenCV RGB-camera frame and **must NOT call `nuwa_to_cv_frame`**; the legacy bbox path (via `_points_via_bbox`) still calls it because Nuwa's ROI cloud uses a different convention. Don't accidentally add the flip to the SAM21 branch — coordinates would double-flip. +- **Pose 3D graceful offline:** `GET /api/pose_3d/state` returns 200 even when G1 is unreachable — `wrist` and `joints` are `null`, `robot_error` carries the message. The tab must remain usable for editing config / named poses / viewing the URDF model without a live robot. +- **Pose 3D clouds:** `GET /api/pose_3d/clouds` acquires from Nuwa (eye-in-hand, polso DX) and D435i (front camera, torso) in parallel. Both are returned in **OpenCV optical frame** (X right, Y down, Z forward). The client attaches them to URDF links with configurable offset/rpy from `config.yaml > pose_3d.clouds`. +- **Pose 3D joints mirroring:** `G1Client.diagnostics()` → `motors[].{name, q}` gives real joint positions in radians. The frontend uses these to pose the URDF model directly, bypassing client-side IK. - Endpoints return `{ ..., "timings_ms": {step: ms, ..., total: ms} }`. New endpoints should follow this with `Stopwatch().step(name)`. - Calibration paths in config (`calibration.file`, `calibration.samples_file`) are resolved against `PRESA_MANO_DATA` when relative — see `_deps.calibration_file_path()` / `samples_file_path()`. - Italian is fine in user-facing copy (GUI, README, comments) — it's the project default. diff --git a/README.md b/README.md index b7df381..d0838a3 100644 --- a/README.md +++ b/README.md @@ -8,17 +8,21 @@ Microservizio Python (FastAPI) che orchestra: nuvola di punti dell'oggetto. 3. **Calibrazione piano di lavoro** + filtro "sotto piano" applicato a ogni cloud. 4. **Pianificazione presa**: asse dell'oggetto (PCA) + posa polso per il robot attivo. +5. **Visualizzazione 3D** (tab Pose 3D): viewport Three.js con URDF del robot + specchiato sui giunti reali, point cloud live (Nuwa eye-in-hand + D435i front + camera), gizmo di editing pose, named poses salvabili. ## Microservizi usati | Servizio | URL (default) | Scopo | |-------------------|-----------------------|----------------------------------------------------| -| Nuwa-HP60C | http://localhost:8877 | RGB, depth, point cloud, intrinsics | -| Marker-finder | http://localhost:8082 | Detection 2D ArUco (corners + pose) | -| YOLO Server | http://localhost:8879 | Bbox oggetto da prendere | -| SAM21 (NuwaSeg) | http://localhost:8002 | Segmentazione promptable: bbox → cloud 3D filtrata | -| G1 API Server | http://localhost:8765 | Controllo polso/mano G1 (dry-run se down) | -| H1 API Server | http://localhost:8767 | Controllo polso/mano H1 (network host, 4 DoF) | +| Nuwa-HP60C | `:8877` | RGB, depth, point cloud, intrinsics (eye-in-hand polso DX) | +| Front camera (D435i) | config `services.front_camera` | Point cloud frontale fissa sul torso | +| Marker-finder | `:8082` | Detection 2D ArUco (corners + pose) | +| YOLO Server | `:8879` | Bbox oggetto da prendere | +| SAM21 (NuwaSeg) | `:8002` | Segmentazione promptable: bbox → cloud 3D filtrata | +| G1 API Server | `:8765` | Controllo polso/mano G1, diagnostics giunti (dry-run se down) | +| H1 API Server | `:8767` | Controllo polso/mano H1 (network host, 4 DoF) | Il servizio **presa-mano** espone GUI + API su **http://localhost:8890**. Il robot attivo è scelto da `config.yaml > robot.active` (`g1` | `h1`); l'orchestrator fa routing @@ -47,6 +51,13 @@ Tab principali: se SAM21 è down), visualizza bbox + asse, esegui (toggle dry-run). - **Movimento EE**: lettura pose correnti, invio pose assolute/relative, apri/chiudi mano, emergency stop. +- **Braccio DX**: detection via YOLO su front camera + camera polso, move-toward + iterativo, pose salvabili (home, custom), verifica tag ArUco. +- **Pose 3D**: viewport Three.js con URDF G1 specchiato sui giunti reali + (`/diagnostics`), point cloud Nuwa + D435i, modello 3D camera sul polso, + gizmo traslazione/rotazione, editor pose (Apply/Jog), named poses salvabili. + La tab resta utilizzabile anche senza robot (mostra il modello URDF, config, + pose salvate). ## Pipeline cloud-from-bbox