Adriano Dal PastroandClaude Opus 4.7
ac573505b7
Pipeline completa per calibrazione hand-eye e grasp planning su robot Unitree G1/H1 con camera depth Nuwa-HP60C. Esposta come FastAPI + GUI vanilla JS, con assistente AI (OpenRouter + tool calling curato) per pilotare il sistema in linguaggio naturale. Architettura: - FastAPI orchestrator (porta da .env, default 8890) che coordina 4 microservizi upstream: Nuwa-HP60C, marker-finder, YOLO, robot bridge. - Dual-robot: services.g1 e services.h1 in config.yaml; switchabili a runtime via PUT /api/home/robot. - Calibrazione hand-eye eye-to-hand split per-side (right/left). - Marker pose: marker-finder riceve gli intrinseci RGB della Nuwa al boot, PnP server-side con quei dati. Nessun re-PnP locale richiesto. - Depth-to-RGB: registrazione iterativa IR↔RGB per back-projection 3D corretta (Nuwa /api/measure usa coords IR; il nostro /api/perception/ measure_rgb prende pixel RGB e ritorna 3D nel frame RGB OpenCV). - Correzione affine sul depth bias (calibrabile via marker come ground-truth) — riduce errore Z da ~6% a ~1.5% nei test. - AI assistant: 16 tool curati (move_wrist_*, hand_action, get_marker[_3d], detect_object, calibration_*, set_active_robot…), system prompt dinamico che riflette robot attivo + marker config + addendum customizzabile da GUI. Temperature di default 0.1 per evitare invenzioni. GUI (7 tab): - Home: stato comunicazione device + selettore robot attivo G1/H1 - Assistente AI: chat con OpenRouter, history persistita lato browser, task riutilizzabili (prompt salvati), trace tool calls - Impostazioni AI: modello da combo curata, temperature, max iterations, prompt addendum, anteprima system prompt - Calibrazione, Auto-Calibrazione, TrovaUtensile, Servo Visuale, Vista 3D, Movimento EE — invariate dalle versioni precedenti Setup: - Variabili in presa-mano/.env (template in .env.example) - OPENROUTER_API_KEY richiesta solo per il tab Assistente AI - PRESA_MANO_PORT configurabile Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
92 lines
7.3 KiB
Markdown
92 lines
7.3 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Commands
|
|
|
|
Build & run (Docker):
|
|
```bash
|
|
docker compose up -d presa-mano # serves GUI + API on http://localhost:8890
|
|
```
|
|
|
|
Local dev (uses `uv`, same as the Dockerfile):
|
|
```bash
|
|
uv sync # install deps from pyproject.toml
|
|
uv run uvicorn presa_mano.main:app --host 0.0.0.0 --port 8890 --reload
|
|
```
|
|
|
|
Useful endpoints (all under `http://localhost:8890`):
|
|
- `GET /apidocs` — Swagger UI; `GET /redoc` — ReDoc.
|
|
- `GET/PUT /api/config` — read/patch runtime config (deep-merged into `config.yaml` and persisted).
|
|
- Calibration: `POST /api/calibration/sample`, `GET/DELETE /api/calibration/samples`, `POST /api/calibration/solve`, `GET /api/calibration/result`, `POST /api/calibration/preview`, `POST /api/calibration/auto_grid`.
|
|
- Perception: `POST /api/perception/detect`, `POST /api/perception/object_points`, `GET /api/perception/measure`.
|
|
- Grasp: `POST /api/grasp/plan`, `POST /api/grasp/execute` (set `dry_run` to gate G1 mutations).
|
|
- Wrist/EE: `GET /api/wrist`, `POST /api/wrist/{absolute,relative,hand,emergency_stop,unlock}`.
|
|
|
|
There are no tests or linters configured in this repo.
|
|
|
|
## Architecture
|
|
|
|
This is a **FastAPI orchestrator** that glues together four external microservices to implement an eye-to-hand calibration + object-grasping pipeline for a Unitree G1 humanoid using a fixed Nuwa-HP60C depth camera. It owns no hardware itself — every action is an HTTP call.
|
|
|
|
Upstream services (defaults in `config.yaml > services`):
|
|
| Service | Default URL | Role |
|
|
|---------|------------|------|
|
|
| Nuwa-HP60C | `:8877` | RGB, depth PNG, point cloud (full + ROI), intrinsics, pixel→3D measure |
|
|
| marker-finder | `:8082` | ArUco detection (returns `corners`, `position_m`, `rotation_matrix`) |
|
|
| YOLO server | `:8879` | Object bbox detection from an image URL |
|
|
| G1 API | `:8765` | Wrist pose read/write, hand open/close, e-stop |
|
|
|
|
### Layered package layout (`src/presa_mano/`)
|
|
|
|
- `main.py` — FastAPI app; mounts `web/` at `/static`, serves `web/index.html` at `/`, registers routers under `/api/*`.
|
|
- `config.py` — Thread-safe singleton config loader. `load_config()` reads YAML once; `update_config(patch)` deep-merges and persists. Path comes from `PRESA_MANO_CONFIG` env (Dockerfile sets `/app/config.yaml`); data files live under `PRESA_MANO_DATA` (default `/app/data`).
|
|
- `api/` — Routers, one per concern. **Build clients per-request via `api/_deps.py`** (`nuwa()`, `marker()`, `yolo()`, `g1(dry_run=...)`) so runtime `PUT /api/config` changes take effect without restart. Don't cache clients at module level.
|
|
- `clients/` — Thin `httpx` wrappers, one per upstream. Errors are normalized to service-specific exceptions (e.g. `G1Unavailable`, `NuwaError`).
|
|
- `calibration/` — `sampler.py` builds one (T_base_wrist, T_cam_marker) sample; `handeye.py` runs `cv2.calibrateHandEye` and computes residuals; `store.py` persists samples (JSON) and the result (YAML).
|
|
- `perception/` — `detector.py` calls YOLO and normalizes responses across multiple envelope shapes; `pointcloud.py` handles frame conversion + outlier filtering.
|
|
- `grasp/` — `axis.py` PCA → `ObjectAxes`; `planner.py` builds a wrist pose from those axes.
|
|
- `geometry.py` — pure-numpy SE(3) helpers (quat↔R, rvec↔R, invert_T, transform_points).
|
|
- `timing.py` — `Stopwatch` with `step(name)` context manager; every endpoint annotates its response with `timings_ms`.
|
|
- `web/` — vanilla HTML/JS/CSS GUI (3 tabs: Calibration, Grasp, Movimento EE), served as static files; talks only to `/api/*`.
|
|
|
|
### Core data flow: hand-eye calibration (eye-to-hand)
|
|
|
|
The camera is fixed; an ArUco marker is mounted on the wrist. For each sample:
|
|
|
|
1. `G1.wrist()` → `T_base_wrist` (via `T_from_pos_quat`).
|
|
2. `marker-finder.detect_url(image_url=nuwa.rgb_url())` → `T_cam_marker` from `position_m` + `rotation_matrix`.
|
|
3. `samples.json` accumulates samples; `HandEyeSolver.solve()` calls `cv2.calibrateHandEye`.
|
|
|
|
**Eye-to-hand trick** (`calibration/handeye.py`): OpenCV documents the eye-in-hand case. We feed `invert_T(T_base_wrist)` so the solver returns `T_camera→base` directly.
|
|
|
|
Residual quality check: with a correct calibration, `T_wrist_marker = invert_T(T_base_wrist) @ T_base_camera @ T_cam_marker` should be constant across samples (marker is rigidly attached). Mean translation/rotation deviations from sample 0 are reported as `residual_translation_mm` / `residual_rotation_deg`.
|
|
|
|
### Core data flow: grasp pipeline
|
|
|
|
`POST /api/grasp/plan` (or `/execute`):
|
|
1. YOLO detect on the current Nuwa RGB → top bbox (optionally filtered by `class_name`).
|
|
2. Bbox → pixel ROI → `nuwa.pointcloud_roi(...)` (Nuwa crops server-side).
|
|
3. `filter_outliers` (n_std rule) → if <10 points, 422.
|
|
4. **Frame conversion:** `nuwa_to_cv_frame()` flips Y and Z (Nuwa = X-right/Y-up/Z-back; OpenCV = X-right/Y-down/Z-forward). Then `transform_points(T_base_camera, ...)` lifts to base frame.
|
|
5. `compute_axes` (PCA) → `ObjectAxes` with sorted extents.
|
|
6. `GraspPlanner.plan` builds `R_wrist = [x_hand | y_hand | z_hand]` where x_hand = approach direction (currently hard-coded **top-down**, `-Z_base`), y_hand = horizontalized closure axis from PCA mid-axis. Returns `grasp` + `pre_grasp` poses (the latter offset by `approach_offset_m` along +Z).
|
|
7. `/execute` sequences: hand_open → pre_grasp → grasp_approach → hand_close, each timed.
|
|
|
|
### Frame conventions (gotcha-prone)
|
|
|
|
There are **three coexisting camera-frame conventions** — keep them straight:
|
|
- **Nuwa native** (`/api/pointcloud`): X-right, Y-up, Z-back (Z<0 in front).
|
|
- **OpenCV / solvePnP** (`T_cam_marker` from marker-finder, `T_base_camera` from hand-eye): X-right, Y-down, Z-forward (Z>0 in front).
|
|
- **Nuwa ROI** (`/api/pointcloud/roi`): already returned in OpenCV convention `[x, -y, -z, r, g, b]` server-side. The wrapper `NuwaClient.pointcloud_roi` does **not** re-flip — but the grasp pipeline still calls `nuwa_to_cv_frame` on the result. Be aware when adding new code paths that consume Nuwa point clouds: full-frame needs flipping, ROI nominally does not (this is currently inconsistent — verify behavior against actual data when modifying).
|
|
|
|
Marker-finder caveat: it uses **D435i intrinsics baked into its config** that don't match the Nuwa. The `corners` it returns are reusable; the absolute pose `position_m`/`rotation_matrix` carries that intrinsics error into the calibration. Marker-finder accuracy is the dominant error source in `T_base_camera`.
|
|
|
|
### Conventions to preserve
|
|
|
|
- **Dry-run semantics:** any G1-mutating endpoint takes a `dry_run` flag; `G1Client._post` short-circuits and logs `[DRY-RUN]` instead of hitting the network. `auto_grid` keeps `dry_run=True` by default (the G1 DDS has been unstable). Never silently strip dry-run from a mutating path.
|
|
- **Per-request client construction** (via `api/_deps.py`) is intentional — runtime `PUT /api/config` updates would otherwise be invisible.
|
|
- Endpoints return `{ ..., "timings_ms": {step: ms, ..., total: ms} }`. New endpoints should follow this with `Stopwatch().step(name)`.
|
|
- Calibration paths in config (`calibration.file`, `calibration.samples_file`) are resolved against `PRESA_MANO_DATA` when relative — see `_deps.calibration_file_path()` / `samples_file_path()`.
|
|
- Italian is fine in user-facing copy (GUI, README, comments) — it's the project default.
|