Tre miglioramenti al move-toward dopo il primo uso live:
1. checkbox "force" accanto a "Muovi": bypassa max_step_m quando
l'utente sa che il delta richiesto e' grande ma corretto (es. primo
avvicinamento all'oggetto).
2. camera_offset_wrist_mm: il modello assumeva camera e TCP coincidenti
(sole sign flips). La camera Nuwa e' invece montata con traslazione
costante rispetto al polso. La formula diventa:
delta_wrist = sign * (centroid_cam - offset_cam) + t_wrist_cam
Esposto in config (default 0,0,0), GUI (3 input "Camera rispetto a
polso (mm)") e in /state. Il termine compensa il fatto che muovere
il polso porta la camera, non il TCP, sul punto di presa.
3. filter_nearest_cluster: il centroide era media di tutti i punti
SAM21 della mask. In presenza di sfondo lontano visibile attraverso
buchi della depth del primo piano la media veniva trascinata a
metri di distanza. Ora si tiene il cluster piu' vicino:
z_near = p5(Z)
keep = punti con Z <= z_near + cluster_depth_window_m
Default window 0.20 m, configurabile. La risposta di /detect/wrist
espone n_sam21_points, n_centroid_points e z_near/z_cutoff per
diagnosi.
Aggiornata anche la persistenza: /offset accetta camera_offset_wrist_mm.
GUI: i campi Offset/Sign/CamOff rispettano "user touched" (refresh non
li cancella mentre l'utente digita).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tre problemi della tab Braccio DX dopo il primo rodaggio:
1. /detect/front e /detect/wrist ritornavano bbox raw dal YOLO server,
che lo manda come lista [x1,y1,x2,y2]; la GUI legge bbox.x1 e mostra
NaN. Ora il router normalizza il payload via perception/detector
._normalize, allineandosi al resto del codice. Aggiunta regressione
con payload reale del server.
2. La tab gateava su robot.active==g1 && wrist_side==right con banner
"Attiva G1 right" e POST /activate. Confonde: la selezione del
robot e' globale e non si flippa per-feature. Rimossi banner,
bottone, endpoint /activate e _guard_g1_right(); gli endpoint
operano sul polso destro del robot attivo qualunque esso sia.
3. image_size: [0,0] perche' Nuwa /api/intrinsics non include
width/height; vivono in /api/status > data.resolution. Aggiunto
NuwaClient.image_size() e usato in detect_front. Aggiornato
docstring di intrinsics() per coerenza con il payload reale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ping anche services.front_camera in /api/home/status. Etichetta nuwa
rinominata in 'Nuwa-HP60C (camera polso)' per chiarezza ora che ci sono
due camere distinte.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pre-SAM21 depth filter and post-SAM21 partition_by_plane were
fighting each other and drowning the actual object cluster. Per
operator request, send Open3DModel the raw SAM21 cluster and let
RANSAC handle outliers itself. The accepted/excluded/SAM21 overlays
in the GUI continue to work; excluded is now always empty and
accepted == SAM21 cluster.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The find_model response now returns plane_polygon_px (4 corners of a
30 cm square anchored at the work plane centroid, projected to RGB
pixels via the same K used for the SAM21 cloud). The GUI gains a
'Piano (viola)' checkbox that draws the polygon as a translucent
violet quad with an outline, helping the operator see whether the
plane sits where expected relative to the object.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously SAM21 ran on the raw aligned depth and frequently leaked the
mask onto the table around the object: ~88% of segmented points (2136
of 2440 in a sample run) were table pixels filtered away by the post
partition_by_plane step. Open3DModel got a noisy cluster mostly off
the object.
New pipeline: with a calibrated work plane, depth pixels whose 3D
back-projection sits below the plane are zeroed BEFORE SAM21 sees the
depth PNG. SAM21 cannot back-project zero-depth pixels, so only true
above-plane points end up in the cloud regardless of mask outline.
Same sample run after the change: 513 SAM21 -> 513 accepted, 0 excluded.
The post-partition still runs (now a no-op on a calibrated plane), so
the find_model response shape is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a 4th checkbox 'Punti SAM21 (gialli)' that draws the full SAM21
segmentation (input to both the plane partition and Open3DModel) on
the RGB result canvas. The backend now returns sam21_points_px and
n_sam21 alongside the existing accepted/excluded splits.
Useful as a sanity check: visualise what was actually segmented vs
what the plane filter kept/rejected vs where the STL got placed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 0.4x0.4 PlaneGeometry square was a fixed-size proxy that did not
reflect the actual ROI extent — visually misleading. Replace it with
an explicit blue point cloud of the inliers fed to fit_plane_svd, so
the user can see *exactly* which depth pixels the plane was fit on.
- Backend: fit response now returns fit_points_camera (CV frame) instead
of the (parallax-shifted) RGB pixel projection.
- Frontend: drop the purple plane mesh; new three.setFitInliers paints
the inliers in blue. The RGB overlay no longer carries blue dots.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
NuwaClient.pointcloud_roi's docstring claims OpenCV frame, but in
practice the raw payload arrives in Nuwa native frame (Z<0 forward).
nuwa_to_cv_frame is what actually flips it. Using `pts_cv` (post-flip
+ post-outlier-mask) makes the projection drop none of the inliers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The fit endpoint now returns fit_points_px (the post-outlier-filter
inliers projected to RGB pixels via Nuwa's RGB intrinsics). The GUI
draws them on the same overlay as the ROI rectangle so the user can
visually verify which depth pixels were fed to fit_plane_svd.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop the ge=0 constraint on FitReq.offset_mm (negative values are a
documented usage: aggressive noise cull above the plane).
- HTML input min lowered to -200 to match.
- Render FastAPI 422 validation arrays as a human-readable string in the
status chip (was showing "[object Object]").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Open3DModel returns the same envelope for /api/recipes as for /api/pose;
the bare-list code path stays as a fallback for older mocks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 3D viewer was only seeing points inside the ROI rectangle used for
the plane fit. Now /fit also fetches nuwa.pointcloud (step=4, full frame),
flips to CV frame, and returns it as points_camera (downsampled to
max_preview_points). The user sees the entire scene with red = excluded
by the below-plane filter, cyan = kept.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Rename "TrovaUtensile" tab → "Cerca modello".
- Remove the "Precisione (ultime 5 detection)" stats panel and the
"Dettagli" JSON dump from the grasp tab; keep hidden stubs so the
existing app.js continuous-loop code doesn't null-deref.
- Add work_plane_routes to api/__init__.py imports + __all__ (consistency
with the other routers).
- Trim the misleading "drag = pan · wheel = zoom · doppio click = reset"
hint from the Calibrazione Piano RGB label — pan/zoom isn't wired.
- Disable wpClearBtn initially so it doesn't flash clickable before
fetchSaved() resolves.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The marker-relative servo (endpoints /api/grasp/visual_servo/{plan,step,
plane/*}, helpers in grasp_routes.py, the bulk of grasp/visual_servo.py,
web/servo.js, and the Servo Visuale GUI tab) is gone. Plane fitting now
lives in /api/work_plane/* and perception/work_plane.py; Task 5 wires the
new Calibrazione Piano tab.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
POST /fit returns a candidate WorkPlane + a downsampled CV-frame cloud
preview, without writing to disk. POST /save persists the plane. GET
returns the current saved plane (or null). DELETE clears it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Until Task 4 removes the duplicate WorkPlane code in grasp/visual_servo.py,
the visual-servo plane endpoints must also update perception.work_plane's
singleton so the below-plane filter (which reads from the perception side)
sees the newly-calibrated plane within the same process lifetime.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
points_via_sam21, _points_via_bbox, and _object_cloud_cv (bbox branch)
now call apply_below_plane_filter on the CV-frame cloud before the
<10-points guard. Responses carry plane_applied + n_points_before_plane.
Without a saved plane the filter is a no-op (plane_applied=false).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Owns WorkPlane (with new offset_mm field), fit_plane_svd, disk-backed
get/set, and the apply_below_plane_filter helper. WorkPlane.normal lives
in OpenCV RGB-camera frame and is oriented toward the camera. The filter
keeps points where signed_distance >= -offset_mm/1000.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extract the SAM21 wiring out of grasp_routes.py into a shared module
perception/sam21_bridge.py (Sam21Fallback + points_via_sam21). Both
/api/perception/scan and /api/perception/object_points now try SAM21
first and fall back to the bbox ROI path on any Sam21Fallback. Response
now carries 'source' (sam21 | bbox_roi) and optional 'sam21_fallback_reason'.
Side-effect: 'points_camera' on both endpoints is now in OpenCV
RGB-camera frame (same as 'points_base' minus the T_base_camera
transform), regardless of which path produced the cloud. Previously the
bbox path stored points in Nuwa-native frame here. The web GUI plots
points_base when available, so the scatter views automatically pick up
the SAM21-segmented cloud.
Verified live: /scan → source=sam21 n=2414 sam21=111ms;
/object_points → source=sam21 n=2347 sam21=118ms;
/grasp/plan regression → source=sam21 n=2390 sam21=119ms;
/scan with SAM21 down → source=bbox_roi fallback='unreachable: ...'
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Note that tests now exist and run with uv run pytest.
- Add SAM21 to the upstream services table.
- Add sam21() to the _deps factory list.
- Add a convention bullet about the SAM21 vs bbox frame asymmetry so
future contributors don't add nuwa_to_cv_frame to the SAM21 branch.
- Move 'import httpx' into the third-party imports block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Nuwa /api/intrinsics omits width/height from the rgb and ir blocks, so
SAM21 rejected the payload with KeyError('width') and we silently fell
back to bbox on every request. Derive the dimensions from the decoded
BGR + IR depth shapes and augment intr_full before building
CameraRegistration and calling SAM21. Verified end-to-end: source=sam21,
~2200 segmented points on Mortaio at 128 ms GPU inference.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- All SAM21-path logging now lives in _points_via_sam21 at the right
level (WARNING for operational, ERROR for bug codes / unexpected).
_object_points_in_base no longer double-logs on fallback.
- Nuwa failures during SAM21 prep now raise HTTPException(502)
immediately instead of running the futile bbox fallback (which would
also fail because Nuwa is down).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When grasp.use_sam21=true (default), _object_points_in_base feeds the YOLO
bbox to SAM21 and uses the segmented cloud directly. Any failure
(transport, segmentation empty, no valid depth, payload bug) logs and falls
back to the legacy bbox path. Response meta includes source ∈ {sam21,
bbox_roi} and sam21_fallback_reason when applicable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure refactor in preparation for the SAM21 branch. _object_points_in_base
keeps its signature and the bbox→ROI→base-frame logic moves into the
private helper. Meta now includes source='bbox_roi'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Align sam21() with nuwa()/marker()/yolo(): require base_url to be present
in config rather than silently defaulting. Misconfiguration now raises
KeyError at factory call instead of masking as a transport error.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Catch httpx.RequestError so DecodingError etc. trigger the bbox fallback
instead of propagating raw. Lock the intrinsics dict shape in the
happy-path test, drop unused imports.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Posts (rgb_png, depth_png, intrinsics, bbox) and returns (N,3) points in
meters in OpenCV RGB-camera frame. Normalises transport vs semantic errors
into Sam21Unavailable / Sam21Error.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 2× focal-length collision test wasn't actually colliding (adjacent IR
pixels rounded to different RGB columns). 4× makes them collapse, so the
np.minimum.at reduce is now exercised. Also drop unused H_ir, W_ir.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
