diff --git a/docs/superpowers/plans/2026-05-11-work-plane-filter.md b/docs/superpowers/plans/2026-05-11-work-plane-filter.md new file mode 100644 index 0000000..8a24a45 --- /dev/null +++ b/docs/superpowers/plans/2026-05-11-work-plane-filter.md @@ -0,0 +1,1539 @@ +# Work-plane calibration + below-plane filter — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Calibrate the work-plane from a user-drawn rectangle on the RGB image, persist it with a configurable `offset_mm`, and use it to drop points below the plane in every object-cloud-producing endpoint (`/scan`, `/object_points`, `/grasp/plan`, `/grasp/execute`). Repurpose the existing "Servo Visuale" tab as "Calibrazione Piano". + +**Architecture:** A single new module `perception/work_plane.py` owns the `WorkPlane` dataclass (extended with `offset_mm`), `fit_plane_svd`, the disk-backed `get/set_work_plane` singleton, and the new `apply_below_plane_filter` helper. Four new endpoints under `/api/work_plane/*` handle calibration (preview/save/get/delete). The filter is called at three call sites (sam21_bridge, `_points_via_bbox`, `_object_cloud_cv` bbox branch). The visual-servo feature is removed entirely; the GUI tab is rewritten. + +**Tech Stack:** Python 3.10 · FastAPI · `httpx` · `numpy` · `opencv-contrib-python-headless` · `pytest` (existing) · vanilla JS + Three.js (existing in `servo.js`'s viewer code, will be reused in `work_plane.js`). + +**Design doc:** `docs/superpowers/specs/2026-05-11-work-plane-filter-design.md`. + +**Critical engineering facts:** +- `WorkPlane.normal` is in CV/OpenCV camera frame and oriented toward the camera (negative Z component in OpenCV convention). Points "above the plane" (where objects sit) have positive `signed_distance = normal · X + offset`. +- `nuwa.pointcloud_roi` returns points in Nuwa-native frame `[x, -y, -z]` despite its current docstring; the existing `/plane/fit` endpoint applies `nuwa_to_cv_frame` (which does the `(y,z) → (-y,-z)` flip again) before fitting. The new `/api/work_plane/fit` keeps this exact behaviour for continuity with calibrations recorded under the current pipeline. +- The bridge helper `points_via_sam21` already returns points in CV frame (SAM21's native output), so it does NOT need `nuwa_to_cv_frame`. +- The bbox branch of `_object_cloud_cv` and `_points_via_bbox` both call `nuwa_to_cv_frame` on the ROI cloud — the filter goes AFTER that flip. +- Filter math: keep `signed_dist >= -offset_m`, where `offset_m = offset_mm / 1000`. Positive `offset_mm` is a permissive buffer (preserves points up to `offset_mm` below the plane). + +--- + +## File Structure + +| File | Action | Responsibility | +|---|---|---| +| `src/presa_mano/perception/work_plane.py` | create | `WorkPlane` dataclass (with `offset_mm`), `fit_plane_svd`, `set/get_work_plane`, disk persistence, `apply_below_plane_filter` helper, `WorkPlaneError` exception. | +| `tests/test_work_plane.py` | create | Unit tests for fit, filter, persistence (including legacy-file load). | +| `src/presa_mano/perception/sam21_bridge.py` | modify | Call `apply_below_plane_filter` on `pts_cv` after SAM21 returns it; raise `Sam21Fallback` if filtered cloud < 10 points. | +| `src/presa_mano/api/grasp_routes.py` | modify | In `_points_via_bbox`: apply filter after `nuwa_to_cv_frame`. Surface `plane_applied` + `n_points_before_plane` in meta. **Later (Task 5):** remove all visual-servo endpoints, models, helpers, imports. | +| `src/presa_mano/api/perception_routes.py` | modify | In `_object_cloud_cv` bbox branch: apply filter after `nuwa_to_cv_frame`. Surface `plane_applied` + `n_points_before_plane` in `/scan` and `/object_points` responses. | +| `src/presa_mano/api/work_plane_routes.py` | create | Router with `/fit`, `/save`, `GET`, `DELETE` endpoints. | +| `src/presa_mano/main.py` | modify | Register `work_plane_routes` router under `/api/work_plane`. Update cache-bust list (`servo.js` → `work_plane.js`). | +| `src/presa_mano/grasp/visual_servo.py` | delete | Removed in Task 5; the plane bits already migrated to `perception/work_plane.py` in Task 1. | +| `web/index.html` | modify | Rename tab "Servo Visuale" → "Calibrazione Piano". Replace `#tab-servo` panel content with the new layout. | +| `web/servo.js` | delete | Replaced by `web/work_plane.js`. | +| `web/work_plane.js` | create | ROI drawing on RGB, fit-preview call, 3D rendering of plane + colored scatter, offset input, save/clear actions. | + +--- + +## Task 1: `perception/work_plane.py` module (TDD) + +Move and extend the plane code into a focused module, then write the new `apply_below_plane_filter` helper. + +**Files:** +- Create: `src/presa_mano/perception/work_plane.py` +- Create: `tests/test_work_plane.py` + +- [ ] **Step 1: Write the failing tests** + +Create `tests/test_work_plane.py`: + +```python +"""Unit tests for perception.work_plane.""" +from __future__ import annotations + +import json +import tempfile +from pathlib import Path +from unittest.mock import patch + +import numpy as np +import pytest + +from presa_mano.perception.work_plane import ( + WorkPlane, + WorkPlaneError, + apply_below_plane_filter, + fit_plane_svd, + _load_work_plane_from_disk, + _save_work_plane_to_disk, + _work_plane_path, +) + + +def _flat_plane_cloud(z: float = 1.0, n: int = 200, noise_m: float = 0.0): + """Cloud sampled on the plane z=z (camera frame, OpenCV: camera looks +Z).""" + rng = np.random.default_rng(0) + xy = rng.uniform(-0.2, 0.2, size=(n, 2)) + zs = np.full((n, 1), z) + rng.normal(0, noise_m, size=(n, 1)) + return np.hstack([xy, zs]) + + +def test_fit_plane_svd_horizontal_plane(): + """Plane at z=1.0, normal must point toward camera (negative Z).""" + pts = _flat_plane_cloud(z=1.0, n=200, noise_m=0.0) + normal, offset, centroid, rms_mm = fit_plane_svd(pts) + assert abs(normal[2]) > 0.99 + assert normal[2] < 0 + assert abs(offset - 1.0) < 1e-3 + assert rms_mm < 1.0 + + +def test_fit_plane_svd_raises_on_too_few_points(): + with pytest.raises(WorkPlaneError): + fit_plane_svd(np.zeros((5, 3))) + + +def test_apply_below_plane_filter_noop_without_plane(): + pts = _flat_plane_cloud(z=1.0) + out, info = apply_below_plane_filter(pts, None) + assert info["plane_applied"] is False + assert info["n_points_before_plane"] == len(pts) + assert out is pts + + +def test_apply_below_plane_filter_keeps_above_drops_below(): + """Plane at z=1.0, normal toward camera (-Z): points with z<1.0 are above.""" + above = _flat_plane_cloud(z=0.95, n=100) + below = _flat_plane_cloud(z=1.10, n=100) + pts = np.vstack([above, below]) + wp = WorkPlane( + normal=[0.0, 0.0, -1.0], offset=1.0, + centroid=[0.0, 0.0, 1.0], + n_inliers=100, fit_rms_mm=0.0, + roi=[0, 0, 10, 10], + offset_mm=0.0, + created_at=0.0, + ) + out, info = apply_below_plane_filter(pts, wp) + assert info["plane_applied"] is True + assert info["n_points_before_plane"] == 200 + assert len(out) == 100 + assert np.all(out[:, 2] <= 1.0 + 1e-9) + + +def test_apply_below_plane_filter_offset_buffer_keeps_below(): + """offset_mm=50 keeps points up to 50mm below the plane.""" + above = _flat_plane_cloud(z=0.95, n=50) + just_below = _flat_plane_cloud(z=1.03, n=50) + far_below = _flat_plane_cloud(z=1.20, n=50) + pts = np.vstack([above, just_below, far_below]) + wp = WorkPlane( + normal=[0.0, 0.0, -1.0], offset=1.0, + centroid=[0.0, 0.0, 1.0], + n_inliers=50, fit_rms_mm=0.0, + roi=[0, 0, 10, 10], + offset_mm=50.0, + created_at=0.0, + ) + out, info = apply_below_plane_filter(pts, wp) + assert info["plane_applied"] is True + assert len(out) == 100 # above (50) + just_below (50); far_below dropped + + +def test_apply_below_plane_filter_empty_input(): + wp = WorkPlane( + normal=[0.0, 0.0, -1.0], offset=1.0, + centroid=[0.0, 0.0, 1.0], + n_inliers=10, fit_rms_mm=0.0, + roi=[0, 0, 10, 10], + offset_mm=5.0, created_at=0.0, + ) + out, info = apply_below_plane_filter(np.empty((0, 3)), wp) + assert info["plane_applied"] is False # short-circuit: empty input never "ran" + assert info["n_points_before_plane"] == 0 + assert len(out) == 0 + + +def test_work_plane_disk_roundtrip(tmp_path, monkeypatch): + """Save then load preserves all fields including offset_mm.""" + monkeypatch.setenv("PRESA_MANO_DATA", str(tmp_path)) + wp = WorkPlane( + normal=[0.0, 0.0, -1.0], offset=1.0, + centroid=[0.1, 0.2, 1.0], + n_inliers=42, fit_rms_mm=1.5, + roi=[10, 20, 100, 200], + offset_mm=7.5, + created_at=1234567890.0, + ) + _save_work_plane_to_disk(wp) + loaded = _load_work_plane_from_disk() + assert loaded is not None + assert loaded.normal == wp.normal + assert loaded.offset == wp.offset + assert loaded.centroid == wp.centroid + assert loaded.n_inliers == wp.n_inliers + assert loaded.fit_rms_mm == wp.fit_rms_mm + assert loaded.roi == wp.roi + assert loaded.offset_mm == 7.5 + assert loaded.created_at == wp.created_at + + +def test_work_plane_load_legacy_file_defaults_offset(tmp_path, monkeypatch): + """A pre-existing work_plane.json without offset_mm gets the 5.0 default.""" + monkeypatch.setenv("PRESA_MANO_DATA", str(tmp_path)) + path = _work_plane_path() + legacy = { + "normal": [0.0, 0.0, -1.0], "offset": 1.0, + "centroid": [0.0, 0.0, 1.0], + "n_inliers": 10, "fit_rms_mm": 1.0, + "roi": [0, 0, 10, 10], + "created_at": 0.0, + } + path.write_text(json.dumps(legacy)) + loaded = _load_work_plane_from_disk() + assert loaded is not None + assert loaded.offset_mm == 5.0 +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: + +```bash +cd /opt/docker/robotserver/presa-mano +uv run pytest tests/test_work_plane.py -v +``` + +Expected: all tests FAIL with `ModuleNotFoundError: No module named 'presa_mano.perception.work_plane'`. + +- [ ] **Step 3: Implement `perception/work_plane.py`** + +Create `src/presa_mano/perception/work_plane.py`: + +```python +"""Work-plane fit, persistence, and below-plane filter. + +The work plane is fitted from a user-drawn rectangle on the RGB image. The +fit uses SVD on the bbox depth cloud (after `nuwa_to_cv_frame`), so the +plane normal lives in OpenCV RGB-camera frame and is oriented toward the +camera (negative Z component in OpenCV convention). + +The filter `apply_below_plane_filter` removes points more than `offset_mm` +below the plane: keep `(normal · X + offset) >= -offset_mm/1000`. A +positive `offset_mm` is permissive (preserves a buffer below the plane). +""" +from __future__ import annotations + +import json +import logging +from dataclasses import asdict, dataclass +from pathlib import Path + +import numpy as np + +from ..config import data_dir + +log = logging.getLogger(__name__) + + +class WorkPlaneError(RuntimeError): + """Bad input to `fit_plane_svd` (too few points, degenerate cloud).""" + + +@dataclass +class WorkPlane: + """Plane in CV camera frame. `normal · X + offset = 0`; `normal` is + unit-length and oriented toward the camera (negative Z in OpenCV).""" + normal: list[float] + offset: float + centroid: list[float] + n_inliers: int + fit_rms_mm: float + roi: list[int] + offset_mm: float + created_at: float + + +_WORK_PLANE: WorkPlane | None = None +_WORK_PLANE_LOADED = False + + +def _work_plane_path() -> Path: + return data_dir() / "work_plane.json" + + +def _save_work_plane_to_disk(wp: WorkPlane | None) -> None: + path = _work_plane_path() + try: + if wp is None: + if path.exists(): + path.unlink() + return + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(asdict(wp), indent=2)) + except Exception as e: + log.warning("failed to persist work plane: %s", e) + + +def _load_work_plane_from_disk() -> WorkPlane | None: + path = _work_plane_path() + try: + with open(path, "r") as f: + d = json.load(f) + d.setdefault("offset_mm", 5.0) # legacy files + return WorkPlane(**d) + except FileNotFoundError: + return None + except Exception as e: + log.warning("failed to load persisted work plane (%s)", e) + return None + + +def get_work_plane() -> WorkPlane | None: + global _WORK_PLANE, _WORK_PLANE_LOADED + if not _WORK_PLANE_LOADED: + _WORK_PLANE = _load_work_plane_from_disk() + _WORK_PLANE_LOADED = True + return _WORK_PLANE + + +def set_work_plane(wp: WorkPlane | None) -> None: + global _WORK_PLANE, _WORK_PLANE_LOADED + _WORK_PLANE = wp + _WORK_PLANE_LOADED = True + _save_work_plane_to_disk(wp) + + +def fit_plane_svd(points_xyz: np.ndarray + ) -> tuple[np.ndarray, float, np.ndarray, float]: + """Least-squares plane via SVD. Returns (normal, offset, centroid, rms_mm). + + Plane equation: `normal · X + offset = 0`. Normal is oriented toward + the camera origin (so `signed_distance = normal · X + offset` is + positive for points on the camera side of the plane).""" + pts = np.asarray(points_xyz, dtype=np.float64) + if pts.ndim != 2 or pts.shape[1] < 3: + raise WorkPlaneError(f"unexpected points shape for plane fit: {pts.shape}") + xyz = pts[:, :3] + if len(xyz) < 10: + raise WorkPlaneError(f"too few points to fit plane: {len(xyz)}") + c = xyz.mean(axis=0) + centered = xyz - c + _, _, Vt = np.linalg.svd(centered, full_matrices=False) + normal = Vt[-1] + n = float(np.linalg.norm(normal)) + if n < 1e-9: + raise WorkPlaneError("degenerate plane (zero normal)") + normal = normal / n + # Orient toward camera origin: signed distance of origin from plane must + # be positive, i.e. `-n·c > 0`, i.e. `n·c < 0`. + if float(normal @ c) > 0: + normal = -normal + offset = -float(normal @ c) + signed = centered @ normal + rms_mm = float(np.sqrt(np.mean(signed * signed)) * 1000.0) + return normal, offset, c, rms_mm + + +def apply_below_plane_filter( + pts_cv: np.ndarray, + wp: WorkPlane | None, +) -> tuple[np.ndarray, dict]: + """Drop points more than `offset_mm` below the plane. + + Returns `(filtered_pts, info)`. `info` always carries: + plane_applied (bool): true iff a plane was provided AND there were + points to filter + n_points_before_plane (int): len(pts_cv) on entry + """ + info = {"plane_applied": False, + "n_points_before_plane": int(len(pts_cv))} + if wp is None or len(pts_cv) == 0: + return pts_cv, info + n_vec = np.asarray(wp.normal, dtype=np.float64) + d_off = float(wp.offset) + offset_m = float(wp.offset_mm) / 1000.0 + signed = pts_cv @ n_vec + d_off + keep = signed >= -offset_m + info["plane_applied"] = True + return pts_cv[keep], info +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: + +```bash +cd /opt/docker/robotserver/presa-mano +uv run pytest tests/test_work_plane.py -v +``` + +Expected: 8 PASS. + +If any test fails, debug before moving on. The `monkeypatch.setenv("PRESA_MANO_DATA", str(tmp_path))` relies on `data_dir()` reading the env var lazily — if `data_dir()` caches at import time, you may need to import `data_dir` inside the helpers (not at module top). Check `src/presa_mano/config.py` for the actual behaviour. + +- [ ] **Step 5: Commit** + +```bash +git add tests/test_work_plane.py src/presa_mano/perception/work_plane.py +git commit -m "feat(perception): work_plane module with below-plane filter + +Owns WorkPlane (with new offset_mm field), fit_plane_svd, disk-backed +get/set, and the apply_below_plane_filter helper. WorkPlane.normal lives +in OpenCV RGB-camera frame and is oriented toward the camera. The filter +keeps points where signed_distance >= -offset_mm/1000. + +Co-Authored-By: Claude Opus 4.7 (1M context) " +``` + +--- + +## Task 2: Wire the filter into the three call sites + +Apply `apply_below_plane_filter` in `sam21_bridge.points_via_sam21`, `grasp_routes._points_via_bbox`, and `perception_routes._object_cloud_cv` bbox branch. Surface `plane_applied` + `n_points_before_plane` in API responses. + +**Files:** +- Modify: `src/presa_mano/perception/sam21_bridge.py` +- Modify: `src/presa_mano/api/grasp_routes.py` +- Modify: `src/presa_mano/api/perception_routes.py` + +- [ ] **Step 1: Add filter to `points_via_sam21` in sam21_bridge.py** + +In `src/presa_mano/perception/sam21_bridge.py`, add this import near the top with the other relative imports: + +```python +from .work_plane import apply_below_plane_filter, get_work_plane +``` + +Then, after the `pts_cv = sam21.segment_pointcloud(...)` call returns successfully and before the existing `if pts_cv.shape[0] < 10` check, insert: + +```python + pts_cv, plane_info = apply_below_plane_filter(pts_cv, get_work_plane()) + if pts_cv.shape[0] < 10: + log.warning("SAM21 cloud has %d points after plane filter (<10), falling back to bbox", + pts_cv.shape[0]) + raise Sam21Fallback( + f"too few points after plane filter: {pts_cv.shape[0]}" + f" (before plane: {plane_info['n_points_before_plane']})" + ) +``` + +Replace the OLD `<10` check (which compared `pts_cv.shape[0] < 10` against the SAM21-returned cloud) with this new block — the new block does both: applies the filter AND enforces the `<10` minimum. + +Note: `points_via_sam21` does not currently return `plane_info`. Update the function signature: change the return type to `tuple[np.ndarray, dict]` and return `(pts_cv, plane_info)` at the end. Update the docstring to describe the new return. + +The full final tail of `points_via_sam21` should look like: + +```python + pts_cv, plane_info = apply_below_plane_filter(pts_cv, get_work_plane()) + if pts_cv.shape[0] < 10: + log.warning("SAM21 cloud has %d points after plane filter (<10), falling back to bbox", + pts_cv.shape[0]) + raise Sam21Fallback( + f"too few points after plane filter: {pts_cv.shape[0]}" + f" (before plane: {plane_info['n_points_before_plane']})" + ) + return pts_cv, plane_info +``` + +- [ ] **Step 2: Update callers of `points_via_sam21`** + +There are TWO callers: + +**Caller 1**: `src/presa_mano/api/grasp_routes.py` inside `_object_points_in_base`. Current code: + +```python + pts_cv = points_via_sam21(nuwa, _deps.sam21(), bbox, sw) + with sw.step("to_base_frame"): + pts_base = transform_points(T_base_cam, pts_cv) + meta["source"] = "sam21" + meta["n_points"] = int(len(pts_base)) + return pts_base, meta +``` + +Replace with: + +```python + pts_cv, plane_info = points_via_sam21(nuwa, _deps.sam21(), bbox, sw) + with sw.step("to_base_frame"): + pts_base = transform_points(T_base_cam, pts_cv) + meta["source"] = "sam21" + meta["n_points"] = int(len(pts_base)) + meta["plane_applied"] = plane_info["plane_applied"] + meta["n_points_before_plane"] = plane_info["n_points_before_plane"] + return pts_base, meta +``` + +**Caller 2**: `src/presa_mano/api/perception_routes.py` inside `_object_cloud_cv`. Current code: + +```python + if cfg.get("grasp", {}).get("use_sam21", True): + try: + pts_cv = points_via_sam21(nuwa, _deps.sam21(), bbox, sw) + if len(pts_cv) > max_points: + idx = np.random.choice(len(pts_cv), max_points, replace=False) + pts_cv = pts_cv[idx] + return pts_cv, "sam21", extra + except Sam21Fallback as e: + extra["sam21_fallback_reason"] = str(e) +``` + +Replace with: + +```python + if cfg.get("grasp", {}).get("use_sam21", True): + try: + pts_cv, plane_info = points_via_sam21(nuwa, _deps.sam21(), bbox, sw) + extra["plane_applied"] = plane_info["plane_applied"] + extra["n_points_before_plane"] = plane_info["n_points_before_plane"] + if len(pts_cv) > max_points: + idx = np.random.choice(len(pts_cv), max_points, replace=False) + pts_cv = pts_cv[idx] + return pts_cv, "sam21", extra + except Sam21Fallback as e: + extra["sam21_fallback_reason"] = str(e) +``` + +- [ ] **Step 3: Add filter to `_points_via_bbox` in grasp_routes.py** + +In `src/presa_mano/api/grasp_routes.py`, add this import near the existing imports: + +```python +from ..perception.work_plane import apply_below_plane_filter, get_work_plane +``` + +Then, locate `_points_via_bbox`. Current body (around lines 106-130): + +```python +def _points_via_bbox(nuwa, T_base_cam: np.ndarray, bbox: tuple[int, int, int, int], + sw: Stopwatch) -> np.ndarray: + """Legacy path: bbox → Nuwa ROI cloud → filter outliers → base frame. + + Raises HTTPException(422) on <10 valid depth points (existing behaviour). + """ + cfg = get_config() + u_min, v_min, u_max, v_max = bbox + try: + with sw.step("pointcloud_roi"): + pc, _meta = nuwa.pointcloud_roi( + u_min, v_min, u_max, v_max, + step=int(cfg.get("grasp", {}).get("pointcloud_step", 1)), + max_depth_mm=int(cfg.get("grasp", {}).get("pointcloud_max_depth_mm", 2000)), + ) + except Exception as e: + raise HTTPException(status_code=502, detail=f"Nuwa ROI pointcloud failed: {e}") + with sw.step("filter_outliers"): + cropped = filter_outliers(pc) + if len(cropped) < 10: + raise HTTPException(status_code=422, detail=f"too few valid depth points in bbox: {len(cropped)}") + with sw.step("to_base_frame"): + pts_cv = nuwa_to_cv_frame(cropped[:, :3]) + pts_base = transform_points(T_base_cam, pts_cv) + return pts_base +``` + +Change the return type to `tuple[np.ndarray, dict]` and replace the body with: + +```python +def _points_via_bbox(nuwa, T_base_cam: np.ndarray, bbox: tuple[int, int, int, int], + sw: Stopwatch) -> tuple[np.ndarray, dict]: + """Legacy path: bbox → Nuwa ROI cloud → filter outliers → plane filter → base frame. + + Raises HTTPException(422) on <10 valid depth points (after both + outlier filter and below-plane filter). + """ + cfg = get_config() + u_min, v_min, u_max, v_max = bbox + try: + with sw.step("pointcloud_roi"): + pc, _meta = nuwa.pointcloud_roi( + u_min, v_min, u_max, v_max, + step=int(cfg.get("grasp", {}).get("pointcloud_step", 1)), + max_depth_mm=int(cfg.get("grasp", {}).get("pointcloud_max_depth_mm", 2000)), + ) + except Exception as e: + raise HTTPException(status_code=502, detail=f"Nuwa ROI pointcloud failed: {e}") + with sw.step("filter_outliers"): + cropped = filter_outliers(pc) + pts_cv = nuwa_to_cv_frame(cropped[:, :3]) + pts_cv, plane_info = apply_below_plane_filter(pts_cv, get_work_plane()) + if len(pts_cv) < 10: + raise HTTPException( + status_code=422, + detail=f"too few valid depth points in bbox after plane filter: {len(pts_cv)}" + f" (before plane: {plane_info['n_points_before_plane']})" + ) + with sw.step("to_base_frame"): + pts_base = transform_points(T_base_cam, pts_cv) + return pts_base, plane_info +``` + +- [ ] **Step 4: Update the caller of `_points_via_bbox` in `_object_points_in_base`** + +Inside `_object_points_in_base`, change the fall-through path to receive the plane_info tuple. Current: + +```python + pts_base = _points_via_bbox(nuwa, T_base_cam, bbox, sw) + meta["source"] = "bbox_roi" + meta["n_points"] = int(len(pts_base)) + return pts_base, meta +``` + +Replace with: + +```python + pts_base, plane_info = _points_via_bbox(nuwa, T_base_cam, bbox, sw) + meta["source"] = "bbox_roi" + meta["n_points"] = int(len(pts_base)) + meta["plane_applied"] = plane_info["plane_applied"] + meta["n_points_before_plane"] = plane_info["n_points_before_plane"] + return pts_base, meta +``` + +- [ ] **Step 5: Add filter to `_object_cloud_cv` bbox branch in perception_routes.py** + +In `src/presa_mano/api/perception_routes.py`, add the import near the existing imports: + +```python +from ..perception.work_plane import apply_below_plane_filter, get_work_plane +``` + +Locate the bbox branch of `_object_cloud_cv`. Current code (after the SAM21 branch): + +```python + u_min, v_min, u_max, v_max = bbox + step = int(cfg.get("grasp", {}).get("pointcloud_step", 1)) + max_depth = int(cfg.get("grasp", {}).get("pointcloud_max_depth_mm", 2000)) + try: + with sw.step("pointcloud_roi"): + pc, _meta = nuwa.pointcloud_roi(u_min, v_min, u_max, v_max, + step=step, max_depth_mm=max_depth) + except HTTPException: + raise + except Exception as e: + raise HTTPException(status_code=502, detail=f"Nuwa pointcloud ROI failed: {e}") + with sw.step("filter_outliers"): + cropped = filter_outliers(pc) + if len(cropped) > max_points: + idx = np.random.choice(len(cropped), max_points, replace=False) + cropped = cropped[idx] + pts_cv = nuwa_to_cv_frame(cropped[:, :3]) if len(cropped) else np.empty((0, 3), dtype=np.float64) + return pts_cv, "bbox_roi", extra +``` + +Replace with: + +```python + u_min, v_min, u_max, v_max = bbox + step = int(cfg.get("grasp", {}).get("pointcloud_step", 1)) + max_depth = int(cfg.get("grasp", {}).get("pointcloud_max_depth_mm", 2000)) + try: + with sw.step("pointcloud_roi"): + pc, _meta = nuwa.pointcloud_roi(u_min, v_min, u_max, v_max, + step=step, max_depth_mm=max_depth) + except HTTPException: + raise + except Exception as e: + raise HTTPException(status_code=502, detail=f"Nuwa pointcloud ROI failed: {e}") + with sw.step("filter_outliers"): + cropped = filter_outliers(pc) + pts_cv = nuwa_to_cv_frame(cropped[:, :3]) if len(cropped) else np.empty((0, 3), dtype=np.float64) + pts_cv, plane_info = apply_below_plane_filter(pts_cv, get_work_plane()) + extra["plane_applied"] = plane_info["plane_applied"] + extra["n_points_before_plane"] = plane_info["n_points_before_plane"] + if len(pts_cv) > max_points: + idx = np.random.choice(len(pts_cv), max_points, replace=False) + pts_cv = pts_cv[idx] + return pts_cv, "bbox_roi", extra +``` + +- [ ] **Step 6: Surface plane fields in `/scan` and `/object_points` responses** + +In `/scan` (after the `_object_cloud_cv` call), find the section that sets `out["source"] = source` and add the plane fields: + +```python + out["selected"] = target + out["source"] = source + if extra.get("sam21_fallback_reason"): + out["sam21_fallback_reason"] = extra["sam21_fallback_reason"] + if "plane_applied" in extra: + out["plane_applied"] = extra["plane_applied"] + out["n_points_before_plane"] = extra["n_points_before_plane"] + out["n_points"] = int(len(pts_cv)) +``` + +Do the same in `/object_points`: after `"source": source`, add the same two-key block: + +```python + if "plane_applied" in extra: + out["plane_applied"] = extra["plane_applied"] + out["n_points_before_plane"] = extra["n_points_before_plane"] +``` + +- [ ] **Step 7: Smoke-test imports and run pytest** + +```bash +cd /opt/docker/robotserver/presa-mano +uv run python -c "from presa_mano.api.grasp_routes import _object_points_in_base, _points_via_bbox; from presa_mano.api.perception_routes import _object_cloud_cv; from presa_mano.perception.sam21_bridge import points_via_sam21; print('ok')" +uv run pytest -v +``` + +Expected: smoke prints `ok`; all 19 tests pass (11 from before + 8 new from Task 1). + +- [ ] **Step 8: Commit** + +```bash +git add src/presa_mano/perception/sam21_bridge.py src/presa_mano/api/grasp_routes.py src/presa_mano/api/perception_routes.py +git commit -m "feat: apply work-plane filter at all three cloud sites + +points_via_sam21, _points_via_bbox, and _object_cloud_cv (bbox branch) +now call apply_below_plane_filter on the CV-frame cloud before the +<10-points guard. Responses carry plane_applied + n_points_before_plane. +Without a saved plane the filter is a no-op (plane_applied=false). + +Co-Authored-By: Claude Opus 4.7 (1M context) " +``` + +--- + +## Task 3: New `/api/work_plane/*` endpoints + +**Files:** +- Create: `src/presa_mano/api/work_plane_routes.py` +- Modify: `src/presa_mano/main.py` + +- [ ] **Step 1: Create the router module** + +Create `src/presa_mano/api/work_plane_routes.py`: + +```python +"""Work-plane calibration endpoints: fit (preview), save, get, delete.""" +from __future__ import annotations + +import time +from dataclasses import asdict +from typing import Any + +import numpy as np +from fastapi import APIRouter, HTTPException +from pydantic import BaseModel, Field + +from ..perception import filter_outliers, nuwa_to_cv_frame +from ..perception.work_plane import ( + WorkPlane, + WorkPlaneError, + fit_plane_svd, + get_work_plane, + set_work_plane, +) +from ..timing import Stopwatch +from . import _deps + +router = APIRouter() + + +class FitReq(BaseModel): + u_min: int = Field(..., ge=0) + v_min: int = Field(..., ge=0) + u_max: int = Field(..., gt=0) + v_max: int = Field(..., gt=0) + offset_mm: float = Field(5.0, ge=0.0, le=200.0) + max_preview_points: int = Field(5000, ge=100, le=20000) + + +class SaveReq(BaseModel): + plane: dict[str, Any] # accept a serialised WorkPlane + + +@router.post("/fit") +def fit(req: FitReq) -> dict[str, Any]: + """Compute a candidate work plane from a rectangle on the RGB image. + + Does NOT persist. Returns the plane + a downsampled CV-frame cloud for + the GUI 3D preview. + """ + sw = Stopwatch() + nuwa = _deps.nuwa() + try: + with sw.step("pointcloud_roi"): + pc, _meta = nuwa.pointcloud_roi( + req.u_min, req.v_min, req.u_max, req.v_max, + step=1, max_depth_mm=4000, + ) + except Exception as e: + raise HTTPException(status_code=502, detail=f"Nuwa pointcloud_roi failed: {e}") + if len(pc) < 10: + raise HTTPException( + status_code=422, + detail=f"too few depth points in ROI: {len(pc)}", + ) + pts_cv = nuwa_to_cv_frame(pc[:, :3]) + with sw.step("filter_outliers"): + pts_cv = filter_outliers(pts_cv, n_std=2.0) + if len(pts_cv) < 10: + raise HTTPException( + status_code=422, + detail=f"too few inliers after outlier filter: {len(pts_cv)}", + ) + try: + with sw.step("fit_plane_svd"): + normal, offset, centroid, rms_mm = fit_plane_svd(pts_cv) + except WorkPlaneError as e: + raise HTTPException(status_code=422, detail=str(e)) + + wp = WorkPlane( + normal=[float(x) for x in normal], + offset=float(offset), + centroid=[float(x) for x in centroid], + n_inliers=int(len(pts_cv)), + fit_rms_mm=float(rms_mm), + roi=[int(req.u_min), int(req.v_min), int(req.u_max), int(req.v_max)], + offset_mm=float(req.offset_mm), + created_at=time.time(), + ) + + # Downsample for preview. + if len(pts_cv) > req.max_preview_points: + idx = np.random.choice(len(pts_cv), req.max_preview_points, replace=False) + preview = pts_cv[idx] + else: + preview = pts_cv + + return { + "plane": asdict(wp), + "preview": True, + "points_camera": preview.tolist(), + "timings_ms": sw.as_dict(), + } + + +@router.post("/save") +def save(req: SaveReq) -> dict[str, Any]: + """Persist a previously-fitted work plane to disk.""" + try: + d = dict(req.plane) + d.setdefault("offset_mm", 5.0) + wp = WorkPlane(**d) + except Exception as e: + raise HTTPException(status_code=422, detail=f"invalid plane payload: {e}") + set_work_plane(wp) + return {"ok": True, "plane": asdict(wp), "saved": True} + + +@router.get("") +def get_current() -> dict[str, Any]: + wp = get_work_plane() + return {"plane": asdict(wp) if wp is not None else None} + + +@router.delete("") +def delete_current() -> dict[str, Any]: + set_work_plane(None) + return {"ok": True, "plane": None} +``` + +- [ ] **Step 2: Register the router in main.py** + +Open `src/presa_mano/main.py`. Find the existing router-registration block (it calls `app.include_router(...)` for each routes module). Add: + +```python +from .api import work_plane_routes +app.include_router(work_plane_routes.router, prefix="/api/work_plane", tags=["work_plane"]) +``` + +Place it next to the other `include_router` calls (alphabetical or following the order used in the file — match existing style). + +- [ ] **Step 3: Smoke-test in process** + +```bash +cd /opt/docker/robotserver/presa-mano +uv run python -c "from presa_mano.main import app; print([r.path for r in app.routes if '/work_plane' in r.path])" +``` + +Expected output: four routes listed: + +``` +['/api/work_plane/fit', '/api/work_plane/save', '/api/work_plane', '/api/work_plane'] +``` + +- [ ] **Step 4: Run pytest** + +```bash +uv run pytest -v +``` + +Expected: still 19 PASS (Task 3 adds no tests; we'll verify endpoints live in Task 6). + +- [ ] **Step 5: Commit** + +```bash +git add src/presa_mano/api/work_plane_routes.py src/presa_mano/main.py +git commit -m "feat(api): /api/work_plane/{fit,save,get,delete} endpoints + +POST /fit returns a candidate WorkPlane + a downsampled CV-frame cloud +preview, without writing to disk. POST /save persists the plane. GET +returns the current saved plane (or null). DELETE clears it. + +Co-Authored-By: Claude Opus 4.7 (1M context) " +``` + +--- + +## Task 4: Remove the visual-servo feature + +Delete the old visual-servo code from both backend and frontend. The plane endpoints under `/api/grasp/visual_servo/plane*` are gone; the new tab in Task 5 will use `/api/work_plane/*`. + +**Files:** +- Modify: `src/presa_mano/api/grasp_routes.py` +- Delete: `src/presa_mano/grasp/visual_servo.py` +- Delete: `web/servo.js` +- Modify: `web/index.html` +- Modify: `src/presa_mano/main.py` + +- [ ] **Step 1: Strip visual-servo code from grasp_routes.py** + +Open `src/presa_mano/api/grasp_routes.py`. Remove the following: + +1. The import block from `..grasp.visual_servo` (entire `from ..grasp.visual_servo import (...)` group). +2. The Pydantic models `VisualServoReq`, `VisualServoStepReq`, `PlaneFitReq`. +3. The helper `_servo_plan_from_req`. +4. The endpoints decorated with these paths: + - `@router.post("/visual_servo/plan")` + - `@router.post("/visual_servo/step")` + - `@router.post("/visual_servo/plane/fit")` + - `@router.get("/visual_servo/plane")` + - `@router.delete("/visual_servo/plane")` + +To verify exactly which lines to remove, run: + +```bash +grep -n 'visual_servo\|VisualServo\|PlaneFitReq\|WorkPlane\|fit_plane_svd\|set_work_plane\|get_work_plane' src/presa_mano/api/grasp_routes.py +``` + +Every match should either be (a) one of the lines above to delete, or (b) the *already-correct* line `from ..perception.work_plane import apply_below_plane_filter, get_work_plane` (added in Task 2). The line `import time` and `from dataclasses import asdict` may become unused after the deletion — drop them if so. + +- [ ] **Step 2: Verify grasp_routes still imports** + +```bash +cd /opt/docker/robotserver/presa-mano +uv run python -c "from presa_mano.api.grasp_routes import router, _object_points_in_base, _points_via_bbox; print('ok')" +``` + +Expected: `ok`. If you get `ImportError` for a missing symbol, you removed too much; restore as needed. + +- [ ] **Step 3: Delete `grasp/visual_servo.py`** + +```bash +git rm src/presa_mano/grasp/visual_servo.py +``` + +Re-run the smoke import to confirm nothing else depended on it: + +```bash +uv run python -c "from presa_mano.main import app; print('ok')" +``` + +Expected: `ok`. If you see `ModuleNotFoundError: 'presa_mano.grasp.visual_servo'`, search for the lingering import: + +```bash +grep -rn 'visual_servo' src/presa_mano/ +``` + +…and remove each one. + +- [ ] **Step 4: Delete `web/servo.js`** + +```bash +git rm web/servo.js +``` + +- [ ] **Step 5: Strip the servo tab from index.html** + +In `web/index.html`: + +1. Find the line `` and delete it. +2. Find the section `
` and delete the entire section (from `
` up to and including the matching `
`). The section spans roughly lines 237–316 in the current file. +3. Find the script tag `` (near the bottom) and delete it. + +(Task 5 will add a new tab button and panel for "Calibrazione Piano" plus the new script tag. In this task we just remove.) + +- [ ] **Step 6: Update main.py cache-bust list** + +Open `src/presa_mano/main.py` and find the cache-bust loop. Currently: + +```python + for name in ("app.js", "servo.js", "style.css"): +``` + +Change to (anticipating Task 5): + +```python + for name in ("app.js", "work_plane.js", "style.css"): +``` + +- [ ] **Step 7: Run pytest + smoke-test the running app builds** + +```bash +cd /opt/docker/robotserver/presa-mano +uv run pytest -v +uv run python -c "from presa_mano.main import app; print(sorted([r.path for r in app.routes if r.path.startswith('/api/')]))" +``` + +Expected: 19 PASS. The route list should NOT contain any `/api/grasp/visual_servo/...` entry, but should contain `/api/work_plane/*` and the unchanged `/api/grasp/{plan,execute}` etc. + +- [ ] **Step 8: Commit** + +```bash +git add -u src/presa_mano/api/grasp_routes.py src/presa_mano/main.py web/index.html +git commit -m "refactor: remove visual-servo feature + +The marker-relative servo (endpoints /api/grasp/visual_servo/{plan,step, +plane/*}, helpers in grasp_routes.py, the bulk of grasp/visual_servo.py, +web/servo.js, and the Servo Visuale GUI tab) is gone. Plane fitting now +lives in /api/work_plane/* and perception/work_plane.py; Task 5 wires the +new Calibrazione Piano tab. + +Co-Authored-By: Claude Opus 4.7 (1M context) " +``` + +--- + +## Task 5: New "Calibrazione Piano" tab + +Add the tab button and panel to `index.html`, plus a fresh `work_plane.js` with ROI drawing, fit preview, offset slider, save/clear, and a 3D viewer with the plane mesh and color-coded scatter. + +**Files:** +- Modify: `web/index.html` +- Create: `web/work_plane.js` + +- [ ] **Step 1: Add the tab button** + +In `web/index.html`, find the tab nav (` +``` + +- [ ] **Step 2: Add the panel section** + +In `web/index.html`, immediately after the `
` that closes `#tab-ai-settings` (or wherever the previous `#tab-servo` section used to be), insert: + +```html + +
+
+ — +
+
+
+
+
+ Nuwa RGB + +
+
+
RGB live · shift+drag = ROI · drag = pan · wheel = zoom · doppio click = reset
+

Vista 3D (frame camera)

+
+ +
+
piano viola · verde = inlier · giallo = sopra piano+offset · rosso = sotto piano-offset · grigio = buffer · drag = ruota · wheel = zoom
+
+
+

Calibrazione

+
+
+ Disegna con shift+drag un rettangolo sul tavolo (senza oggetti), poi + clicca Calcola piano. Regola l'offset finché i punti + gialli rappresentano l'oggetto e i rossi spariscono. Poi Salva piano. +
+
+ + +
+
+ + + +
+
+ +

Stato

+
+
nessun piano
+
+ +

Dettagli piano

+
—
+
+
+
+``` + +- [ ] **Step 3: Add the script tag at the bottom of index.html** + +Near the bottom of `web/index.html`, find the existing ``. After that line, add: + +```html + +``` + +- [ ] **Step 4: Create `web/work_plane.js`** + +Create `web/work_plane.js`: + +```javascript +// Calibrazione Piano tab: shift+drag ROI on RGB, fit preview, offset +// slider, save / clear, and a 3D scatter with the plane mesh. + +import * as THREE from 'https://unpkg.com/three@0.160.0/build/three.module.js'; +import { OrbitControls } from 'https://unpkg.com/three@0.160.0/examples/jsm/controls/OrbitControls.js'; + +const $ = (id) => document.getElementById(id); +const RGB_REFRESH_MS = 1000; + +let roi = null; // {u0,v0,u1,v1} in image pixel coords +let previewPlane = null; // last response.plane (preview, not saved) +let previewPoints = null; // Float32Array(N*3) in CV camera frame, meters +let savedPlane = null; // last GET /api/work_plane response.plane + +function setChip(text, level) { + const el = $('wpStatusChip'); + if (!el) return; + el.textContent = text; + el.className = 'post-health ' + (level || ''); +} + +function setRoiInfo() { + const el = $('wpRoiInfo'); + if (!roi) { el.textContent = '— · disegna con shift+drag'; el.classList.add('dim'); return; } + el.classList.remove('dim'); + el.textContent = `u=${roi.u0}…${roi.u1} v=${roi.v0}…${roi.v1} (${roi.u1 - roi.u0}×${roi.v1 - roi.v0} px)`; +} + +function updateButtonStates() { + $('wpFitBtn').disabled = !roi; + $('wpSaveBtn').disabled = !previewPlane; + $('wpClearBtn').disabled = !savedPlane; +} + +function refreshRgb() { + $('wpImg').src = `/api/perception/rgb?t=${Date.now()}`; + // Fallback: many builds expose Nuwa RGB through `/static/rgb` or similar. + // If the above 404s the user sees a broken image; that's a known + // limitation of perception_routes.py not exposing a GUI RGB endpoint. +} + +// Note: presa-mano serves Nuwa RGB indirectly. The existing GUI uses the +// upstream URL via `/api/perception/scan` for detection-based refreshes; +// for the static viewer we proxy through Nuwa's own endpoint. Adjust the +// path below if your deployment routes differently. +function rgbUrl() { + // Nuwa is exposed on the host network (per docker-compose), and the GUI + // is same-origin to presa-mano (8890). We can't reach :8877 directly from + // a browser unless the user opens it; instead use the proxy that already + // exists: GET /api/perception/measure backs Nuwa RGB? No — there is no + // RGB proxy in presa-mano. Use the existing servo.js trick: hit Nuwa + // through its own URL using window.location.hostname. + const host = window.location.hostname; + return `http://${host}:8877/api/rgb?t=${Date.now()}`; +} + +function startRgbLoop() { + const tick = () => { $('wpImg').src = rgbUrl(); }; + tick(); + setInterval(tick, RGB_REFRESH_MS); +} + +// ---------------- ROI drawing on the RGB image ---------------- +(function setupRoiDrawing() { + const viewport = $('wpViewport'); + const img = $('wpImg'); + const overlay = $('wpOverlay'); + let drawing = null; + + function imgPixelFromEvent(e) { + const rect = img.getBoundingClientRect(); + const sx = img.naturalWidth / rect.width; + const sy = img.naturalHeight / rect.height; + return { + u: Math.round((e.clientX - rect.left) * sx), + v: Math.round((e.clientY - rect.top) * sy), + }; + } + + function syncOverlaySize() { + overlay.width = img.naturalWidth || 640; + overlay.height = img.naturalHeight || 480; + overlay.style.width = img.clientWidth + 'px'; + overlay.style.height = img.clientHeight + 'px'; + } + img.addEventListener('load', syncOverlaySize); + + function drawOverlay() { + const ctx = overlay.getContext('2d'); + ctx.clearRect(0, 0, overlay.width, overlay.height); + if (roi) { + ctx.strokeStyle = '#9b6dff'; + ctx.lineWidth = 2; + ctx.strokeRect(roi.u0, roi.v0, roi.u1 - roi.u0, roi.v1 - roi.v0); + } + } + + viewport.addEventListener('mousedown', (e) => { + if (!e.shiftKey) return; // shift+drag = ROI; plain drag = pan (no-op here) + const p = imgPixelFromEvent(e); + drawing = { u0: p.u, v0: p.v, u1: p.u, v1: p.v }; + e.preventDefault(); + }); + viewport.addEventListener('mousemove', (e) => { + if (!drawing) return; + const p = imgPixelFromEvent(e); + drawing.u1 = p.u; + drawing.v1 = p.v; + roi = { + u0: Math.min(drawing.u0, drawing.u1), + v0: Math.min(drawing.v0, drawing.v1), + u1: Math.max(drawing.u0, drawing.u1), + v1: Math.max(drawing.v0, drawing.v1), + }; + drawOverlay(); + setRoiInfo(); + updateButtonStates(); + }); + viewport.addEventListener('mouseup', () => { drawing = null; }); + viewport.addEventListener('mouseleave', () => { drawing = null; }); +})(); + +// ---------------- Three.js 3D viewer ---------------- +const three = (function setupThree() { + const canvas = $('wp3dCanvas'); + const scene = new THREE.Scene(); + const camera = new THREE.PerspectiveCamera(45, canvas.width / canvas.height, 0.01, 20); + camera.position.set(0.5, -0.5, -1.2); + camera.lookAt(0, 0, 1); + const renderer = new THREE.WebGLRenderer({ canvas, antialias: true }); + renderer.setSize(canvas.width, canvas.height); + const controls = new OrbitControls(camera, canvas); + scene.add(new THREE.AxesHelper(0.1)); + const planeMesh = new THREE.Mesh( + new THREE.PlaneGeometry(0.4, 0.4), + new THREE.MeshBasicMaterial({ color: 0x9b6dff, transparent: true, opacity: 0.3, side: THREE.DoubleSide }), + ); + planeMesh.visible = false; + scene.add(planeMesh); + let scatter = null; + + function render() { renderer.render(scene, camera); } + (function loop() { controls.update(); render(); requestAnimationFrame(loop); })(); + + function setPlane(plane) { + if (!plane) { planeMesh.visible = false; return; } + const c = plane.centroid; + const n = plane.normal; + planeMesh.position.set(c[0], c[1], c[2]); + // Align plane to its normal. + planeMesh.lookAt(c[0] + n[0], c[1] + n[1], c[2] + n[2]); + planeMesh.visible = true; + } + + function setScatter(pointsCv, plane, offsetMm) { + if (scatter) { scene.remove(scatter); scatter.geometry.dispose(); scatter.material.dispose(); scatter = null; } + if (!pointsCv || pointsCv.length === 0) return; + const n = pointsCv.length; + const positions = new Float32Array(n * 3); + const colors = new Float32Array(n * 3); + const nx = plane ? plane.normal[0] : 0; + const ny = plane ? plane.normal[1] : 0; + const nz = plane ? plane.normal[2] : 0; + const d = plane ? plane.offset : 0; + const offsetM = (offsetMm || 0) / 1000.0; + const rmsM = plane ? (plane.fit_rms_mm / 1000.0) * 2.0 : 0.01; + for (let i = 0; i < n; i++) { + const x = pointsCv[i][0], y = pointsCv[i][1], z = pointsCv[i][2]; + positions[3 * i] = x; positions[3 * i + 1] = y; positions[3 * i + 2] = z; + const signed = plane ? (nx * x + ny * y + nz * z + d) : 0; + let r = 0.5, g = 0.5, b = 0.5; + if (plane) { + if (signed > offsetM) { r = 1.0; g = 0.83; b = 0.31; } // yellow: above plane+offset + else if (signed < -offsetM) { r = 0.93; g = 0.27; b = 0.27; } // red: below plane-offset + else if (Math.abs(signed) <= rmsM) { r = 0.30; g = 0.78; b = 0.40; } // green: inlier + else { r = 0.55; g = 0.55; b = 0.55; } // grey: buffer + } + colors[3 * i] = r; colors[3 * i + 1] = g; colors[3 * i + 2] = b; + } + const geom = new THREE.BufferGeometry(); + geom.setAttribute('position', new THREE.BufferAttribute(positions, 3)); + geom.setAttribute('color', new THREE.BufferAttribute(colors, 3)); + const mat = new THREE.PointsMaterial({ size: 0.005, vertexColors: true }); + scatter = new THREE.Points(geom, mat); + scene.add(scatter); + } + + return { setPlane, setScatter }; +})(); + +// ---------------- State + button handlers ---------------- +async function fetchSaved() { + try { + const r = await fetch('/api/work_plane').then((r) => r.json()); + savedPlane = r.plane || null; + } catch (e) { + savedPlane = null; + } + if (savedPlane && !previewPlane) { + three.setPlane(savedPlane); + renderDetails(savedPlane); + $('wpStateBlock').innerHTML = `piano salvato · RMS ${savedPlane.fit_rms_mm.toFixed(1)} mm · n_inliers ${savedPlane.n_inliers} · offset ${savedPlane.offset_mm} mm`; + setChip('piano salvato', 'ok'); + } else if (!savedPlane && !previewPlane) { + $('wpStateBlock').innerHTML = '
nessun piano
'; + setChip('nessun piano'); + } + updateButtonStates(); +} + +function renderDetails(plane) { + $('wpDetails').textContent = JSON.stringify(plane, null, 2); +} + +async function doFit() { + if (!roi) return; + setChip('fit…'); + $('wpFitBtn').disabled = true; + try { + const body = { + u_min: roi.u0, v_min: roi.v0, u_max: roi.u1, v_max: roi.v1, + offset_mm: parseFloat($('wpOffset').value || '5'), + }; + const r = await fetch('/api/work_plane/fit', { + method: 'POST', headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(body), + }).then((r) => r.json()); + if (r.detail) throw new Error(r.detail); + previewPlane = r.plane; + previewPoints = r.points_camera; + three.setPlane(previewPlane); + three.setScatter(previewPoints, previewPlane, previewPlane.offset_mm); + renderDetails(previewPlane); + $('wpStateBlock').innerHTML = `preview · RMS ${previewPlane.fit_rms_mm.toFixed(1)} mm · n_inliers ${previewPlane.n_inliers} · offset ${previewPlane.offset_mm} mm · non salvato`; + setChip('preview pronto', 'ok'); + } catch (e) { + setChip('fit failed: ' + e.message, 'err'); + } finally { + updateButtonStates(); + } +} + +async function doSave() { + if (!previewPlane) return; + setChip('save…'); + try { + const r = await fetch('/api/work_plane/save', { + method: 'POST', headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ plane: previewPlane }), + }).then((r) => r.json()); + if (r.detail) throw new Error(r.detail); + savedPlane = r.plane; + previewPlane = null; + $('wpStateBlock').innerHTML = `piano salvato · RMS ${savedPlane.fit_rms_mm.toFixed(1)} mm · n_inliers ${savedPlane.n_inliers} · offset ${savedPlane.offset_mm} mm`; + renderDetails(savedPlane); + setChip('piano salvato', 'ok'); + } catch (e) { + setChip('save failed: ' + e.message, 'err'); + } finally { + updateButtonStates(); + } +} + +async function doClear() { + if (!confirm('Rimuovere il piano salvato?')) return; + setChip('clear…'); + try { + await fetch('/api/work_plane', { method: 'DELETE' }).then((r) => r.json()); + savedPlane = null; + previewPlane = null; + previewPoints = null; + three.setPlane(null); + three.setScatter(null); + $('wpStateBlock').innerHTML = '
nessun piano
'; + $('wpDetails').textContent = '—'; + setChip('nessun piano'); + } catch (e) { + setChip('clear failed: ' + e.message, 'err'); + } finally { + updateButtonStates(); + } +} + +// ---------------- Offset slider: re-color in real time ---------------- +$('wpOffset').addEventListener('input', () => { + if (previewPlane && previewPoints) { + const off = parseFloat($('wpOffset').value || '0'); + previewPlane.offset_mm = off; + three.setScatter(previewPoints, previewPlane, off); + $('wpStateBlock').innerHTML = `preview · RMS ${previewPlane.fit_rms_mm.toFixed(1)} mm · n_inliers ${previewPlane.n_inliers} · offset ${previewPlane.offset_mm.toFixed(1)} mm · non salvato`; + } +}); + +$('wpFitBtn').addEventListener('click', doFit); +$('wpSaveBtn').addEventListener('click', doSave); +$('wpClearBtn').addEventListener('click', doClear); + +// Initial load. +startRgbLoop(); +fetchSaved(); +``` + +- [ ] **Step 5: Smoke build the Docker image** + +```bash +docker compose -f /opt/docker/robotserver/docker-compose.yml up -d --build presa-mano +until curl -fsS http://localhost:8890/health >/dev/null 2>&1; do sleep 2; done +echo healthy +``` + +Expected: container builds and reports healthy. + +- [ ] **Step 6: Manual GUI smoke** + +Open `http://localhost:8890/` in a browser and hard-refresh (`Ctrl+Shift+R`). Verify: + +1. The tab bar contains "Calibrazione Piano" (no more "Servo Visuale"). +2. Clicking it shows the new panel with the RGB image (live), a 3D canvas with axes only, and three buttons. +3. `GET /api/work_plane` is called on load and returns `{plane: null}` (or the existing saved plane if any). + +If anything is broken: open DevTools, check the Console for errors. Common issues: +- Three.js import failed: the unpkg URLs may be blocked offline. Use a local copy if needed. +- RGB image 404: the `rgbUrl()` helper assumes Nuwa is reachable at `host:8877`. If the deployment routes differently, adjust. + +- [ ] **Step 7: Commit** + +```bash +git add web/index.html web/work_plane.js +git commit -m "feat(gui): Calibrazione Piano tab (ROI + fit preview + 3D viewer) + +New tab replaces the removed Servo Visuale. Shift+drag on the RGB image +defines a rectangle, 'Calcola piano' calls POST /api/work_plane/fit and +renders the plane + a colored scatter in 3D. Adjusting the offset slider +re-colors the scatter client-side. 'Salva piano' persists; 'Rimuovi piano +salvato' clears. + +Co-Authored-By: Claude Opus 4.7 (1M context) " +``` + +--- + +## Task 6: Live integration verification + +Verify the full flow end-to-end against a running stack (Nuwa + SAM21 + presa-mano + at least one object on the table). + +**Files:** none modified. + +- [ ] **Step 1: Ensure all upstream services are healthy** + +```bash +curl -fsS http://localhost:8877/api/health && echo " | nuwa ok" +curl -fsS http://localhost:8002/api/health && echo " | sam21 ok" +curl -fsS http://localhost:8879/health && echo " | yolo ok" +curl -fsS http://localhost:8890/health && echo " | presa-mano ok" +``` + +Expected: all four print `ok`. + +- [ ] **Step 2: Verify a clean baseline (no plane)** + +```bash +curl -s -X DELETE http://localhost:8890/api/work_plane | jq . +curl -s -X POST http://localhost:8890/api/grasp/plan -H 'Content-Type: application/json' -d '{}' \ + | jq '{source, n_points, plane_applied, n_points_before_plane}' +``` + +Expected: `plane: null` from delete. `/grasp/plan` returns `plane_applied: false`, `n_points_before_plane == n_points` (no filtering). + +- [ ] **Step 3: Calibrate a plane from the GUI** + +In the browser: + +1. Open the "Calibrazione Piano" tab. +2. Place a flat region of the table in view (move any objects out of a corner of the frame). +3. Shift+drag a rectangle over the bare table area. +4. Click "Calcola piano (preview)". Expected: 3D view shows the plane mesh and a tight green cluster of inliers along it; RMS is < 5 mm. +5. Click "Salva piano". + +Verify via API: + +```bash +curl -s http://localhost:8890/api/work_plane | jq . +ls -la /opt/docker/robotserver/presa-mano/data/work_plane.json +``` + +Expected: GET returns the saved plane; the JSON file exists on disk. + +- [ ] **Step 4: Verify the filter on the grasp pipeline** + +Put an object on the table: + +```bash +curl -s -X POST http://localhost:8890/api/grasp/plan -H 'Content-Type: application/json' -d '{}' \ + | jq '{source, n_points, plane_applied, n_points_before_plane, sam21_fallback_reason}' +``` + +Expected: +- `source: "sam21"` +- `plane_applied: true` +- `n_points` < `n_points_before_plane` (table surface points are dropped) +- Typical: a Mortaio segmented to ~2400 points before plane, ~1500 after (depending on object height and offset). + +- [ ] **Step 5: Verify the filter on /scan and /object_points** + +```bash +curl -s -X POST http://localhost:8890/api/perception/scan -H 'Content-Type: application/json' -d '{"include_points":true}' \ + | jq '{source, n_points, plane_applied, n_points_before_plane}' +curl -s -X POST http://localhost:8890/api/perception/object_points -H 'Content-Type: application/json' -d '{}' \ + | jq '{source, n_points, plane_applied, n_points_before_plane}' +``` + +Expected: both responses carry `plane_applied: true` and a non-trivial reduction in `n_points` vs `n_points_before_plane`. + +- [ ] **Step 6: Adjust offset and re-fit (preview-only)** + +In the GUI, click "Calcola piano" again with the same ROI, change the offset to 20 mm, observe the scatter recolor (more yellow points appear as the threshold shifts up). Note: the recolor happens client-side; `n_points` returned by `/grasp/plan` does NOT change until "Salva piano" is clicked. + +Click "Salva piano". Re-run `/api/grasp/plan` and confirm `n_points` increases compared to step 4 (because the more permissive offset keeps more points). + +- [ ] **Step 7: Verify clear** + +```bash +curl -s -X DELETE http://localhost:8890/api/work_plane | jq . +curl -s -X POST http://localhost:8890/api/grasp/plan -H 'Content-Type: application/json' -d '{}' \ + | jq '{plane_applied, n_points_before_plane, n_points}' +ls -la /opt/docker/robotserver/presa-mano/data/work_plane.json 2>&1 | head -1 +``` + +Expected: `plane_applied: false`, `n_points == n_points_before_plane`, the JSON file is gone. + +- [ ] **Step 8: Tail logs for ERROR-level lines** + +```bash +docker compose -f /opt/docker/robotserver/docker-compose.yml logs --since=5m presa-mano | grep -E 'ERROR|Traceback' || echo "no errors" +``` + +Expected: `no errors`. + +- [ ] **Step 9: Final commit (only if anything new emerged)** + +If verification revealed a real bug that needed fixing during this task, commit that fix here. Otherwise, this task ends without a commit and the feature is done. + +--- + +## Done criteria + +- All pytest tests pass: `uv run pytest -v` → 19 green. +- `data/work_plane.json` is created by "Salva piano" and removed by "Rimuovi piano salvato". +- `/api/grasp/plan` carries `plane_applied: true` and `n_points < n_points_before_plane` when a plane is saved. +- The "Servo Visuale" tab and all its endpoints are gone; the "Calibrazione Piano" tab renders correctly with no console errors. +- No ERROR lines in `presa-mano` logs during normal operation. +- 6 commits on `main` (or on the feature branch if one is created).