docs: aggiorna README.md e CLAUDE.md con tab Braccio DX, Pose 3D, cloud, diagnostics
Allineati alla rimozione del modulo agent e alle nuove funzionalità: - Tab Braccio DX e Pose 3D documentate in GUI - Servizi front_camera (D435i) e G1 /diagnostics nella tabella upstream - Endpoint right_arm/* e pose_3d/* nella lista API - Convenzioni Pose 3D (graceful offline, clouds, joints mirroring) in CLAUDE.md - Router list e _deps aggiornati (front_camera(), 12 router modules) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
6c2452fe20
commit
ae8e609b7e
2 files changed
+28
-11
No files matched your search
@@ -22,34 +22,37 @@ Useful endpoints (all under `http://localhost:8890`):
|
||||
- Perception: `POST /api/perception/detect`, `POST /api/perception/object_points`, `GET /api/perception/measure`.
|
||||
- Grasp: `POST /api/grasp/plan`, `POST /api/grasp/execute` (set `dry_run` to gate G1 mutations).
|
||||
- Wrist/EE: `GET /api/wrist`, `POST /api/wrist/{absolute,relative,hand,emergency_stop,unlock}`.
|
||||
- Right arm: `GET /api/right_arm/state`, `POST /api/right_arm/detect_{front,wrist}`, `POST /api/right_arm/move_toward`, `POST /api/right_arm/{home,custom_pose}`.
|
||||
- Pose 3D: `GET /api/pose_3d/state` (wrist + joints + config), `POST /api/pose_3d/apply`, `GET /api/pose_3d/clouds` (Nuwa + D435i point clouds), `CRUD /api/pose_3d/poses/{name}`.
|
||||
|
||||
Tests live under `tests/` and run with `uv run pytest`. No linter is configured.
|
||||
|
||||
## Architecture
|
||||
|
||||
This is a **FastAPI orchestrator** that glues together four external microservices to implement an eye-to-hand calibration + object-grasping pipeline for a Unitree G1 humanoid using a fixed Nuwa-HP60C depth camera. It owns no hardware itself — every action is an HTTP call.
|
||||
This is a **FastAPI orchestrator** that glues together external microservices to implement an eye-to-hand calibration + object-grasping + 3D visualization pipeline for a Unitree G1 humanoid. It owns no hardware itself — every action is an HTTP call.
|
||||
|
||||
Upstream services (defaults in `config.yaml > services`):
|
||||
| Service | Default URL | Role |
|
||||
|---------|------------|------|
|
||||
| Nuwa-HP60C | `:8877` | RGB, depth PNG, point cloud (full + ROI), intrinsics, pixel→3D measure |
|
||||
| Nuwa-HP60C | `:8877` | RGB, depth PNG, point cloud (full + ROI), intrinsics, pixel→3D measure. Eye-in-hand sul polso DX |
|
||||
| Front camera (D435i) | `services.front_camera` | Point cloud frontale fissa sul torso; stessa interfaccia NuwaClient |
|
||||
| marker-finder | `:8082` | ArUco detection (returns `corners`, `position_m`, `rotation_matrix`) |
|
||||
| YOLO server | `:8879` | Object bbox detection from an image URL |
|
||||
| G1 API | `:8765` | Wrist pose read/write, hand open/close, e-stop |
|
||||
| G1 API | `:8765` | Wrist pose read/write, hand open/close, e-stop, `/diagnostics` (29 joint positions) |
|
||||
| SAM21 (NuwaSeg) | `:8002` | Promptable segmentation: rgb+depth+intrinsics+bbox → 3D point cloud of the object (replaces the bbox-only ROI cloud in the grasp pipeline) |
|
||||
|
||||
### Layered package layout (`src/presa_mano/`)
|
||||
|
||||
- `main.py` — FastAPI app; mounts `web/` at `/static`, serves `web/index.html` at `/`, registers routers under `/api/*`.
|
||||
- `config.py` — Thread-safe singleton config loader. `load_config()` reads YAML once; `update_config(patch)` deep-merges and persists. Path comes from `PRESA_MANO_CONFIG` env (Dockerfile sets `/app/config.yaml`); data files live under `PRESA_MANO_DATA` (default `/app/data`).
|
||||
- `api/` — Routers, one per concern. **Build clients per-request via `api/_deps.py`** (`nuwa()`, `marker()`, `yolo()`, `sam21()`, `g1(dry_run=...)`) so runtime `PUT /api/config` changes take effect without restart. Don't cache clients at module level.
|
||||
- `api/` — Routers, one per concern (calibration, config, grasp, home, models, perception, pose_3d, posture, right_arm, robot, work_plane, wrist). **Build clients per-request via `api/_deps.py`** (`nuwa()`, `front_camera()`, `marker()`, `yolo()`, `sam21()`, `g1(dry_run=...)`) so runtime `PUT /api/config` changes take effect without restart. Don't cache clients at module level.
|
||||
- `clients/` — Thin `httpx` wrappers, one per upstream. Errors are normalized to service-specific exceptions (e.g. `G1Unavailable`, `NuwaError`).
|
||||
- `calibration/` — `sampler.py` builds one (T_base_wrist, T_cam_marker) sample; `handeye.py` runs `cv2.calibrateHandEye` and computes residuals; `store.py` persists samples (JSON) and the result (YAML).
|
||||
- `perception/` — `detector.py` calls YOLO and normalizes responses across multiple envelope shapes; `pointcloud.py` handles frame conversion + outlier filtering.
|
||||
- `grasp/` — `axis.py` PCA → `ObjectAxes`; `planner.py` builds a wrist pose from those axes.
|
||||
- `geometry.py` — pure-numpy SE(3) helpers (quat↔R, rvec↔R, invert_T, transform_points).
|
||||
- `timing.py` — `Stopwatch` with `step(name)` context manager; every endpoint annotates its response with `timings_ms`.
|
||||
- `web/` — vanilla HTML/JS/CSS GUI (3 tabs: Calibration, Grasp, Movimento EE), served as static files; talks only to `/api/*`.
|
||||
- `web/` — vanilla HTML/JS/CSS GUI served as static files; talks only to `/api/*`. Tab principali: Calibrazione, Calibrazione Piano, Cerca Modello, Movimento EE, Braccio DX, Pose 3D. Il viewport 3D (pose_3d.js) usa Three.js + URDFLoader per il modello robot.
|
||||
|
||||
### Core data flow: hand-eye calibration (eye-to-hand)
|
||||
|
||||
@@ -88,6 +91,9 @@ Marker-finder caveat: it uses **D435i intrinsics baked into its config** that do
|
||||
- **Dry-run semantics:** any G1-mutating endpoint takes a `dry_run` flag; `G1Client._post` short-circuits and logs `[DRY-RUN]` instead of hitting the network. Never silently strip dry-run from a mutating path.
|
||||
- **Per-request client construction** (via `api/_deps.py`) is intentional — runtime `PUT /api/config` updates would otherwise be invisible.
|
||||
- **SAM21 vs bbox frame convention:** the grasp pipeline has two parallel paths in `_object_points_in_base`. The SAM21 path returns points already in the OpenCV RGB-camera frame and **must NOT call `nuwa_to_cv_frame`**; the legacy bbox path (via `_points_via_bbox`) still calls it because Nuwa's ROI cloud uses a different convention. Don't accidentally add the flip to the SAM21 branch — coordinates would double-flip.
|
||||
- **Pose 3D graceful offline:** `GET /api/pose_3d/state` returns 200 even when G1 is unreachable — `wrist` and `joints` are `null`, `robot_error` carries the message. The tab must remain usable for editing config / named poses / viewing the URDF model without a live robot.
|
||||
- **Pose 3D clouds:** `GET /api/pose_3d/clouds` acquires from Nuwa (eye-in-hand, polso DX) and D435i (front camera, torso) in parallel. Both are returned in **OpenCV optical frame** (X right, Y down, Z forward). The client attaches them to URDF links with configurable offset/rpy from `config.yaml > pose_3d.clouds`.
|
||||
- **Pose 3D joints mirroring:** `G1Client.diagnostics()` → `motors[].{name, q}` gives real joint positions in radians. The frontend uses these to pose the URDF model directly, bypassing client-side IK.
|
||||
- Endpoints return `{ ..., "timings_ms": {step: ms, ..., total: ms} }`. New endpoints should follow this with `Stopwatch().step(name)`.
|
||||
- Calibration paths in config (`calibration.file`, `calibration.samples_file`) are resolved against `PRESA_MANO_DATA` when relative — see `_deps.calibration_file_path()` / `samples_file_path()`.
|
||||
- Italian is fine in user-facing copy (GUI, README, comments) — it's the project default.
|
||||
@@ -8,17 +8,21 @@ Microservizio Python (FastAPI) che orchestra:
|
||||
nuvola di punti dell'oggetto.
|
||||
3. **Calibrazione piano di lavoro** + filtro "sotto piano" applicato a ogni cloud.
|
||||
4. **Pianificazione presa**: asse dell'oggetto (PCA) + posa polso per il robot attivo.
|
||||
5. **Visualizzazione 3D** (tab Pose 3D): viewport Three.js con URDF del robot
|
||||
specchiato sui giunti reali, point cloud live (Nuwa eye-in-hand + D435i front
|
||||
camera), gizmo di editing pose, named poses salvabili.
|
||||
|
||||
## Microservizi usati
|
||||
|
||||
| Servizio | URL (default) | Scopo |
|
||||
|-------------------|-----------------------|----------------------------------------------------|
|
||||
| Nuwa-HP60C | http://localhost:8877 | RGB, depth, point cloud, intrinsics |
|
||||
| Marker-finder | http://localhost:8082 | Detection 2D ArUco (corners + pose) |
|
||||
| YOLO Server | http://localhost:8879 | Bbox oggetto da prendere |
|
||||
| SAM21 (NuwaSeg) | http://localhost:8002 | Segmentazione promptable: bbox → cloud 3D filtrata |
|
||||
| G1 API Server | http://localhost:8765 | Controllo polso/mano G1 (dry-run se down) |
|
||||
| H1 API Server | http://localhost:8767 | Controllo polso/mano H1 (network host, 4 DoF) |
|
||||
| Nuwa-HP60C | `:8877` | RGB, depth, point cloud, intrinsics (eye-in-hand polso DX) |
|
||||
| Front camera (D435i) | config `services.front_camera` | Point cloud frontale fissa sul torso |
|
||||
| Marker-finder | `:8082` | Detection 2D ArUco (corners + pose) |
|
||||
| YOLO Server | `:8879` | Bbox oggetto da prendere |
|
||||
| SAM21 (NuwaSeg) | `:8002` | Segmentazione promptable: bbox → cloud 3D filtrata |
|
||||
| G1 API Server | `:8765` | Controllo polso/mano G1, diagnostics giunti (dry-run se down) |
|
||||
| H1 API Server | `:8767` | Controllo polso/mano H1 (network host, 4 DoF) |
|
||||
|
||||
Il servizio **presa-mano** espone GUI + API su **http://localhost:8890**. Il robot attivo
|
||||
è scelto da `config.yaml > robot.active` (`g1` | `h1`); l'orchestrator fa routing
|
||||
@@ -47,6 +51,13 @@ Tab principali:
|
||||
se SAM21 è down), visualizza bbox + asse, esegui (toggle dry-run).
|
||||
- **Movimento EE**: lettura pose correnti, invio pose assolute/relative, apri/chiudi mano,
|
||||
emergency stop.
|
||||
- **Braccio DX**: detection via YOLO su front camera + camera polso, move-toward
|
||||
iterativo, pose salvabili (home, custom), verifica tag ArUco.
|
||||
- **Pose 3D**: viewport Three.js con URDF G1 specchiato sui giunti reali
|
||||
(`/diagnostics`), point cloud Nuwa + D435i, modello 3D camera sul polso,
|
||||
gizmo traslazione/rotazione, editor pose (Apply/Jog), named poses salvabili.
|
||||
La tab resta utilizzabile anche senza robot (mostra il modello URDF, config,
|
||||
pose salvate).
|
||||
|
||||
## Pipeline cloud-from-bbox
|
||||
|
||||
|
||||
Reference in new issue
Block a user