The 0.4x0.4 PlaneGeometry square was a fixed-size proxy that did not
reflect the actual ROI extent — visually misleading. Replace it with
an explicit blue point cloud of the inliers fed to fit_plane_svd, so
the user can see *exactly* which depth pixels the plane was fit on.
- Backend: fit response now returns fit_points_camera (CV frame) instead
of the (parallax-shifted) RGB pixel projection.
- Frontend: drop the purple plane mesh; new three.setFitInliers paints
the inliers in blue. The RGB overlay no longer carries blue dots.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
NuwaClient.pointcloud_roi's docstring claims OpenCV frame, but in
practice the raw payload arrives in Nuwa native frame (Z<0 forward).
nuwa_to_cv_frame is what actually flips it. Using `pts_cv` (post-flip
+ post-outlier-mask) makes the projection drop none of the inliers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The fit endpoint now returns fit_points_px (the post-outlier-filter
inliers projected to RGB pixels via Nuwa's RGB intrinsics). The GUI
draws them on the same overlay as the ROI rectangle so the user can
visually verify which depth pixels were fed to fit_plane_svd.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop the ge=0 constraint on FitReq.offset_mm (negative values are a
documented usage: aggressive noise cull above the plane).
- HTML input min lowered to -200 to match.
- Render FastAPI 422 validation arrays as a human-readable string in the
status chip (was showing "[object Object]").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Setting canvas.width/height clears its bitmap; the live RGB tick was
firing the img.load handler every second, wiping the ROI rectangle and
breaking the two-click flow (third click was interpreted as restart).
- Only mutate canvas size when image resolution actually changes.
- Re-paint the overlay after each load so the ROI persists.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Right side has no calibration in data/calibration_right.yaml;
the H1 left arm is the one with samples + solved hand-eye.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Open3DModel returns the same envelope for /api/recipes as for /api/pose;
the bare-list code path stays as a fallback for older mocks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brainstorming → approved spec for the GUI/backend redesign of tab "Cerca
modello": YOLO+SAM21 cluster feeds Open3DModel /api/pose, presa-mano
samples the recipe STL, projects onto a snapshot RGB with togglable
overlays for accepted/excluded points and the fitted model points.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add SAM21 (NuwaSeg :8002) to the upstream services table.
- New "Calibrazione Piano" tab documented: 2-click ROI, fit preview, 3D
camera-POV viewer with red = excluded points, offset_mm semantics
(positive = permissive buffer below, negative = aggressive cut above for
depth noise), save persists to data/work_plane.json.
- "TrovaUtensile" tab renamed to "Cerca modello".
- New section "Pipeline cloud-from-bbox" describing the SAM21-first flow
with bbox fallback and the work-plane filter, plus the response meta
fields (source, plane_applied, n_points_before_plane,
sam21_fallback_reason).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 3D viewer was only seeing points inside the ROI rectangle used for
the plane fit. Now /fit also fetches nuwa.pointcloud (step=4, full frame),
flips to CV frame, and returns it as points_camera (downsampled to
max_preview_points). The user sees the entire scene with red = excluded
by the below-plane filter, cyan = kept.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Camera at origin (CV frame) looking at +Z with screen-up = CV −Y, so
the rendered scene matches the camera's vantage point. OrbitControls
retargets to the plane centroid on fit so rotation feels natural.
- Simplify scatter coloring to two tones: red for filter-excluded points
(signed_distance < −offset_mm/1000), cyan-white for everything kept.
Matches the backend filter contract exactly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User asked for "2 punti" — replace the drag-to-rectangle interaction with
two discrete clicks: first click places corner A (cross marker), second
click defines the rectangle, third click starts a new selection.
Rectangles smaller than 10×10 px are rejected. Status chip guides the
flow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Direct fetch to http://localhost:8877/api/rgb fails from the browser due
to CORS (page is served from :8890). Switch to /api/calibration/rgb.jpg
which presa-mano already proxies same-origin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Rename "TrovaUtensile" tab → "Cerca modello".
- Remove the "Precisione (ultime 5 detection)" stats panel and the
"Dettagli" JSON dump from the grasp tab; keep hidden stubs so the
existing app.js continuous-loop code doesn't null-deref.
- Add work_plane_routes to api/__init__.py imports + __all__ (consistency
with the other routers).
- Trim the misleading "drag = pan · wheel = zoom · doppio click = reset"
hint from the Calibrazione Piano RGB label — pan/zoom isn't wired.
- Disable wpClearBtn initially so it doesn't flash clickable before
fetchSaved() resolves.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New tab replaces the removed Servo Visuale. Shift+drag on the RGB image
defines a rectangle, 'Calcola piano' calls POST /api/work_plane/fit and
renders the plane + a colored scatter in 3D. Adjusting the offset slider
re-colors the scatter client-side. 'Salva piano' persists; 'Rimuovi piano
salvato' clears.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The marker-relative servo (endpoints /api/grasp/visual_servo/{plan,step,
plane/*}, helpers in grasp_routes.py, the bulk of grasp/visual_servo.py,
web/servo.js, and the Servo Visuale GUI tab) is gone. Plane fitting now
lives in /api/work_plane/* and perception/work_plane.py; Task 5 wires the
new Calibrazione Piano tab.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
POST /fit returns a candidate WorkPlane + a downsampled CV-frame cloud
preview, without writing to disk. POST /save persists the plane. GET
returns the current saved plane (or null). DELETE clears it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Until Task 4 removes the duplicate WorkPlane code in grasp/visual_servo.py,
the visual-servo plane endpoints must also update perception.work_plane's
singleton so the below-plane filter (which reads from the perception side)
sees the newly-calibrated plane within the same process lifetime.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
points_via_sam21, _points_via_bbox, and _object_cloud_cv (bbox branch)
now call apply_below_plane_filter on the CV-frame cloud before the
<10-points guard. Responses carry plane_applied + n_points_before_plane.
Without a saved plane the filter is a no-op (plane_applied=false).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Owns WorkPlane (with new offset_mm field), fit_plane_svd, disk-backed
get/set, and the apply_below_plane_filter helper. WorkPlane.normal lives
in OpenCV RGB-camera frame and is oriented toward the camera. The filter
keeps points where signed_distance >= -offset_mm/1000.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
6 task TDD-first: perception/work_plane.py + 8 unit tests, filter wiring
ai 3 call sites con plane_applied/n_points_before_plane nelle response,
nuovi endpoint /api/work_plane/{fit,save,get,delete}, rimozione totale
del visual-servo (backend + GUI), nuova tab Calibrazione Piano con
work_plane.js (ROI + fit preview + 3D viewer con scatter colorato +
offset live-recolor), verifica live end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Repurpose the "Servo Visuale" tab into "Calibrazione Piano". Reuses the
existing WorkPlane + fit_plane_svd infrastructure (moved out of
grasp/visual_servo.py into perception/work_plane.py), extends WorkPlane
with offset_mm, splits fit (preview) from save (commit), and applies a
below-plane + offset filter to every object cloud (sam21_bridge,
_points_via_bbox, _object_cloud_cv bbox branch). Visual-servo feature
removed entirely; the only survivors are the plane bits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extract the SAM21 wiring out of grasp_routes.py into a shared module
perception/sam21_bridge.py (Sam21Fallback + points_via_sam21). Both
/api/perception/scan and /api/perception/object_points now try SAM21
first and fall back to the bbox ROI path on any Sam21Fallback. Response
now carries 'source' (sam21 | bbox_roi) and optional 'sam21_fallback_reason'.
Side-effect: 'points_camera' on both endpoints is now in OpenCV
RGB-camera frame (same as 'points_base' minus the T_base_camera
transform), regardless of which path produced the cloud. Previously the
bbox path stored points in Nuwa-native frame here. The web GUI plots
points_base when available, so the scatter views automatically pick up
the SAM21-segmented cloud.
Verified live: /scan → source=sam21 n=2414 sam21=111ms;
/object_points → source=sam21 n=2347 sam21=118ms;
/grasp/plan regression → source=sam21 n=2390 sam21=119ms;
/scan with SAM21 down → source=bbox_roi fallback='unreachable: ...'
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Status line on "Pianifica grasp" now shows source=sam21 / source=bbox_roi
(with fallback reason if any) plus n_points, so SAM21 vs fallback path is
visible at a glance without opening DevTools.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
12 commits — spec, plan, implementation, fixes, live integration.
- New: Sam21Client (clients/sam21.py), align_depth_to_rgb
(perception/depth_align.py), _points_via_sam21 helper in
grasp_routes.py, services.sam21 + grasp.use_sam21 config keys, pytest
scaffold + 11 unit tests.
- _object_points_in_base: try SAM21 first (default on), fall back to the
legacy bbox path on any non-fatal failure. Nuwa failures during prep
raise 502 immediately. Single-log discipline.
- Verified live: source=sam21 with ~2200 segmented points at 128 ms GPU
inference; SAM21 down → automatic bbox_roi fallback; class filter +
recovery on restart all work.
- Note that tests now exist and run with uv run pytest.
- Add SAM21 to the upstream services table.
- Add sam21() to the _deps factory list.
- Add a convention bullet about the SAM21 vs bbox frame asymmetry so
future contributors don't add nuwa_to_cv_frame to the SAM21 branch.
- Move 'import httpx' into the third-party imports block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Nuwa /api/intrinsics omits width/height from the rgb and ir blocks, so
SAM21 rejected the payload with KeyError('width') and we silently fell
back to bbox on every request. Derive the dimensions from the decoded
BGR + IR depth shapes and augment intr_full before building
CameraRegistration and calling SAM21. Verified end-to-end: source=sam21,
~2200 segmented points on Mortaio at 128 ms GPU inference.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- All SAM21-path logging now lives in _points_via_sam21 at the right
level (WARNING for operational, ERROR for bug codes / unexpected).
_object_points_in_base no longer double-logs on fallback.
- Nuwa failures during SAM21 prep now raise HTTPException(502)
immediately instead of running the futile bbox fallback (which would
also fail because Nuwa is down).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When grasp.use_sam21=true (default), _object_points_in_base feeds the YOLO
bbox to SAM21 and uses the segmented cloud directly. Any failure
(transport, segmentation empty, no valid depth, payload bug) logs and falls
back to the legacy bbox path. Response meta includes source ∈ {sam21,
bbox_roi} and sam21_fallback_reason when applicable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure refactor in preparation for the SAM21 branch. _object_points_in_base
keeps its signature and the bbox→ROI→base-frame logic moves into the
private helper. Meta now includes source='bbox_roi'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Align sam21() with nuwa()/marker()/yolo(): require base_url to be present
in config rather than silently defaulting. Misconfiguration now raises
KeyError at factory call instead of masking as a transport error.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Catch httpx.RequestError so DecodingError etc. trigger the bbox fallback
instead of propagating raw. Lock the intrinsics dict shape in the
happy-path test, drop unused imports.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Posts (rgb_png, depth_png, intrinsics, bbox) and returns (N,3) points in
meters in OpenCV RGB-camera frame. Normalises transport vs semantic errors
into Sam21Unavailable / Sam21Error.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 2× focal-length collision test wasn't actually colliding (adjacent IR
pixels rounded to different RGB columns). 4× makes them collapse, so the
np.minimum.at reduce is now exercised. Also drop unused H_ir, W_ir.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
