Plan dettagliato per la tab "Pose 3D" (writing-plans skill),
basato sullo spec del 2026-05-21. 15 task suddivisi in backend
(routes + tests TDD) e frontend (Three.js viewport + gizmo +
mode toggle Editor/Jog).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lavoro accumulato nella sessione 2026-05-21 prima del brainstorming
della tab Pose 3D:
- compute_delta_wrist generalizzato con cam_to_wrist_perm: la mappa
cam->polso non e' solo sign-flip ma una permutazione (default
identita' per backward compat). DeltaOversize ora espone X/Y/Z
nella stringa d'errore. Test aggiornati.
- G1Client.wrist_absolute / wrist_relative: precise=True di default
(era False). Stessa cosa per wrist_routes.AbsoluteReq Pydantic
e per agent.tools move_wrist_*. Movimenti reali ora chiedono
controllo accurato salvo override esplicito.
- Tab Braccio DX:
- /verify_tag endpoint: snapshot wrist FK + tag id=1 dalla front
camera. Per verifica relativa moveto vs marker (no calibrazione).
- GUI: pannello "Verifica tag id=1" + campi camera_offset_wrist_mm.
- markUserTouchedOnInput esteso a raOff/raSgn/raCamOff.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds CRUD for named wrist poses persisted in config.yaml under
pose_3d.named_poses. PUT validates the name against ^[A-Za-z0-9_-]{1,40}$
and normalizes the quaternion. DELETE returns 404 if the pose is missing.
Bypasses update_config's deep-merge for wholesale named_poses replacement
(deep-merge cannot remove nested keys) via a small helper that mutates
the in-memory singleton then persists with an empty patch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Specifica per nuova tab GUI con viewport Three.js, gizmo
translate/rotate sui polsi DX e SX, modalita' Editor (preview) /
Jog (auto-move al mouseup), safety guard sui movimenti grandi,
checkbox precise toggle, save/load named poses in config.yaml.
Brainstorming nella sessione 2026-05-21.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tuning da prima sessione live: move_offset_camera_mm.x=100 (lateral
verso il pestello), camera_offset_wrist_mm.z=100 (mounting verticale
della Nuwa sopra il TCP destro), cluster_depth_window_m=0.20.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tre miglioramenti al move-toward dopo il primo uso live:
1. checkbox "force" accanto a "Muovi": bypassa max_step_m quando
l'utente sa che il delta richiesto e' grande ma corretto (es. primo
avvicinamento all'oggetto).
2. camera_offset_wrist_mm: il modello assumeva camera e TCP coincidenti
(sole sign flips). La camera Nuwa e' invece montata con traslazione
costante rispetto al polso. La formula diventa:
delta_wrist = sign * (centroid_cam - offset_cam) + t_wrist_cam
Esposto in config (default 0,0,0), GUI (3 input "Camera rispetto a
polso (mm)") e in /state. Il termine compensa il fatto che muovere
il polso porta la camera, non il TCP, sul punto di presa.
3. filter_nearest_cluster: il centroide era media di tutti i punti
SAM21 della mask. In presenza di sfondo lontano visibile attraverso
buchi della depth del primo piano la media veniva trascinata a
metri di distanza. Ora si tiene il cluster piu' vicino:
z_near = p5(Z)
keep = punti con Z <= z_near + cluster_depth_window_m
Default window 0.20 m, configurabile. La risposta di /detect/wrist
espone n_sam21_points, n_centroid_points e z_near/z_cutoff per
diagnosi.
Aggiornata anche la persistenza: /offset accetta camera_offset_wrist_mm.
GUI: i campi Offset/Sign/CamOff rispettano "user touched" (refresh non
li cancella mentre l'utente digita).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
YAML riformattato (block style) dal side-effect di update_config —
contenuto invariato a parte robot.{active,wrist_side}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tre problemi della tab Braccio DX dopo il primo rodaggio:
1. /detect/front e /detect/wrist ritornavano bbox raw dal YOLO server,
che lo manda come lista [x1,y1,x2,y2]; la GUI legge bbox.x1 e mostra
NaN. Ora il router normalizza il payload via perception/detector
._normalize, allineandosi al resto del codice. Aggiunta regressione
con payload reale del server.
2. La tab gateava su robot.active==g1 && wrist_side==right con banner
"Attiva G1 right" e POST /activate. Confonde: la selezione del
robot e' globale e non si flippa per-feature. Rimossi banner,
bottone, endpoint /activate e _guard_g1_right(); gli endpoint
operano sul polso destro del robot attivo qualunque esso sia.
3. image_size: [0,0] perche' Nuwa /api/intrinsics non include
width/height; vivono in /api/status > data.resolution. Aggiunto
NuwaClient.image_size() e usato in detect_front. Aggiornato
docstring di intrinsics() per coerenza con il payload reale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ping anche services.front_camera in /api/home/status. Etichetta nuwa
rinominata in 'Nuwa-HP60C (camera polso)' per chiarezza ora che ci sono
due camere distinte.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Sostituisce colori hardcoded con CSS variables (--bg/--panel/--border/--dim/--warn/--ok/--danger)
- Scope .dim a #tab-right-arm per non alterare altre tab che usano la classe
- Aggiunge white-space pre-wrap a .ra-result per testi multi-riga
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
13 task TDD per il nuovo tab: config, geometry, math puro, 6 endpoint
router /api/right_arm/*, markup HTML + CSS, JS module. Spec:
docs/superpowers/specs/2026-05-20-right-arm-tab-design.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Design del nuovo tab "Braccio DX": fronte (depth cam fissa sul robot)
+ polso (Nuwa eye-in-hand) + controlli move/home/custom-pose. Router
/api/right_arm/*, JS dedicato, persistenza in config tab_right_arm.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pre-SAM21 depth filter and post-SAM21 partition_by_plane were
fighting each other and drowning the actual object cluster. Per
operator request, send Open3DModel the raw SAM21 cluster and let
RANSAC handle outliers itself. The accepted/excluded/SAM21 overlays
in the GUI continue to work; excluded is now always empty and
accepted == SAM21 cluster.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The find_model response now returns plane_polygon_px (4 corners of a
30 cm square anchored at the work plane centroid, projected to RGB
pixels via the same K used for the SAM21 cloud). The GUI gains a
'Piano (viola)' checkbox that draws the polygon as a translucent
violet quad with an outline, helping the operator see whether the
plane sits where expected relative to the object.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously SAM21 ran on the raw aligned depth and frequently leaked the
mask onto the table around the object: ~88% of segmented points (2136
of 2440 in a sample run) were table pixels filtered away by the post
partition_by_plane step. Open3DModel got a noisy cluster mostly off
the object.
New pipeline: with a calibrated work plane, depth pixels whose 3D
back-projection sits below the plane are zeroed BEFORE SAM21 sees the
depth PNG. SAM21 cannot back-project zero-depth pixels, so only true
above-plane points end up in the cloud regardless of mask outline.
Same sample run after the change: 513 SAM21 -> 513 accepted, 0 excluded.
The post-partition still runs (now a no-op on a calibrated plane), so
the find_model response shape is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a 4th checkbox 'Punti SAM21 (gialli)' that draws the full SAM21
segmentation (input to both the plane partition and Open3DModel) on
the RGB result canvas. The backend now returns sam21_points_px and
n_sam21 alongside the existing accepted/excluded splits.
Useful as a sanity check: visualise what was actually segmented vs
what the plane filter kept/rejected vs where the STL got placed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 0.4x0.4 PlaneGeometry square was a fixed-size proxy that did not
reflect the actual ROI extent — visually misleading. Replace it with
an explicit blue point cloud of the inliers fed to fit_plane_svd, so
the user can see *exactly* which depth pixels the plane was fit on.
- Backend: fit response now returns fit_points_camera (CV frame) instead
of the (parallax-shifted) RGB pixel projection.
- Frontend: drop the purple plane mesh; new three.setFitInliers paints
the inliers in blue. The RGB overlay no longer carries blue dots.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
NuwaClient.pointcloud_roi's docstring claims OpenCV frame, but in
practice the raw payload arrives in Nuwa native frame (Z<0 forward).
nuwa_to_cv_frame is what actually flips it. Using `pts_cv` (post-flip
+ post-outlier-mask) makes the projection drop none of the inliers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The fit endpoint now returns fit_points_px (the post-outlier-filter
inliers projected to RGB pixels via Nuwa's RGB intrinsics). The GUI
draws them on the same overlay as the ROI rectangle so the user can
visually verify which depth pixels were fed to fit_plane_svd.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop the ge=0 constraint on FitReq.offset_mm (negative values are a
documented usage: aggressive noise cull above the plane).
- HTML input min lowered to -200 to match.
- Render FastAPI 422 validation arrays as a human-readable string in the
status chip (was showing "[object Object]").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Setting canvas.width/height clears its bitmap; the live RGB tick was
firing the img.load handler every second, wiping the ROI rectangle and
breaking the two-click flow (third click was interpreted as restart).
- Only mutate canvas size when image resolution actually changes.
- Re-paint the overlay after each load so the ROI persists.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Right side has no calibration in data/calibration_right.yaml;
the H1 left arm is the one with samples + solved hand-eye.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Open3DModel returns the same envelope for /api/recipes as for /api/pose;
the bare-list code path stays as a fallback for older mocks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brainstorming → approved spec for the GUI/backend redesign of tab "Cerca
modello": YOLO+SAM21 cluster feeds Open3DModel /api/pose, presa-mano
samples the recipe STL, projects onto a snapshot RGB with togglable
overlays for accepted/excluded points and the fitted model points.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add SAM21 (NuwaSeg :8002) to the upstream services table.
- New "Calibrazione Piano" tab documented: 2-click ROI, fit preview, 3D
camera-POV viewer with red = excluded points, offset_mm semantics
(positive = permissive buffer below, negative = aggressive cut above for
depth noise), save persists to data/work_plane.json.
- "TrovaUtensile" tab renamed to "Cerca modello".
- New section "Pipeline cloud-from-bbox" describing the SAM21-first flow
with bbox fallback and the work-plane filter, plus the response meta
fields (source, plane_applied, n_points_before_plane,
sam21_fallback_reason).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 3D viewer was only seeing points inside the ROI rectangle used for
the plane fit. Now /fit also fetches nuwa.pointcloud (step=4, full frame),
flips to CV frame, and returns it as points_camera (downsampled to
max_preview_points). The user sees the entire scene with red = excluded
by the below-plane filter, cyan = kept.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Camera at origin (CV frame) looking at +Z with screen-up = CV −Y, so
the rendered scene matches the camera's vantage point. OrbitControls
retargets to the plane centroid on fit so rotation feels natural.
- Simplify scatter coloring to two tones: red for filter-excluded points
(signed_distance < −offset_mm/1000), cyan-white for everything kept.
Matches the backend filter contract exactly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User asked for "2 punti" — replace the drag-to-rectangle interaction with
two discrete clicks: first click places corner A (cross marker), second
click defines the rectangle, third click starts a new selection.
Rectangles smaller than 10×10 px are rejected. Status chip guides the
flow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Direct fetch to http://localhost:8877/api/rgb fails from the browser due
to CORS (page is served from :8890). Switch to /api/calibration/rgb.jpg
which presa-mano already proxies same-origin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Rename "TrovaUtensile" tab → "Cerca modello".
- Remove the "Precisione (ultime 5 detection)" stats panel and the
"Dettagli" JSON dump from the grasp tab; keep hidden stubs so the
existing app.js continuous-loop code doesn't null-deref.
- Add work_plane_routes to api/__init__.py imports + __all__ (consistency
with the other routers).
- Trim the misleading "drag = pan · wheel = zoom · doppio click = reset"
hint from the Calibrazione Piano RGB label — pan/zoom isn't wired.
- Disable wpClearBtn initially so it doesn't flash clickable before
fetchSaved() resolves.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New tab replaces the removed Servo Visuale. Shift+drag on the RGB image
defines a rectangle, 'Calcola piano' calls POST /api/work_plane/fit and
renders the plane + a colored scatter in 3D. Adjusting the offset slider
re-colors the scatter client-side. 'Salva piano' persists; 'Rimuovi piano
salvato' clears.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The marker-relative servo (endpoints /api/grasp/visual_servo/{plan,step,
plane/*}, helpers in grasp_routes.py, the bulk of grasp/visual_servo.py,
web/servo.js, and the Servo Visuale GUI tab) is gone. Plane fitting now
lives in /api/work_plane/* and perception/work_plane.py; Task 5 wires the
new Calibrazione Piano tab.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
POST /fit returns a candidate WorkPlane + a downsampled CV-frame cloud
preview, without writing to disk. POST /save persists the plane. GET
returns the current saved plane (or null). DELETE clears it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Until Task 4 removes the duplicate WorkPlane code in grasp/visual_servo.py,
the visual-servo plane endpoints must also update perception.work_plane's
singleton so the below-plane filter (which reads from the perception side)
sees the newly-calibrated plane within the same process lifetime.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
points_via_sam21, _points_via_bbox, and _object_cloud_cv (bbox branch)
now call apply_below_plane_filter on the CV-frame cloud before the
<10-points guard. Responses carry plane_applied + n_points_before_plane.
Without a saved plane the filter is a no-op (plane_applied=false).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Owns WorkPlane (with new offset_mm field), fit_plane_svd, disk-backed
get/set, and the apply_below_plane_filter helper. WorkPlane.normal lives
in OpenCV RGB-camera frame and is oriented toward the camera. The filter
keeps points where signed_distance >= -offset_mm/1000.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
6 task TDD-first: perception/work_plane.py + 8 unit tests, filter wiring
ai 3 call sites con plane_applied/n_points_before_plane nelle response,
nuovi endpoint /api/work_plane/{fit,save,get,delete}, rimozione totale
del visual-servo (backend + GUI), nuova tab Calibrazione Piano con
work_plane.js (ROI + fit preview + 3D viewer con scatter colorato +
offset live-recolor), verifica live end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Repurpose the "Servo Visuale" tab into "Calibrazione Piano". Reuses the
existing WorkPlane + fit_plane_svd infrastructure (moved out of
grasp/visual_servo.py into perception/work_plane.py), extends WorkPlane
with offset_mm, splits fit (preview) from save (commit), and applies a
below-plane + offset filter to every object cloud (sam21_bridge,
_points_via_bbox, _object_cloud_cv bbox branch). Visual-servo feature
removed entirely; the only survivors are the plane bits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extract the SAM21 wiring out of grasp_routes.py into a shared module
perception/sam21_bridge.py (Sam21Fallback + points_via_sam21). Both
/api/perception/scan and /api/perception/object_points now try SAM21
first and fall back to the bbox ROI path on any Sam21Fallback. Response
now carries 'source' (sam21 | bbox_roi) and optional 'sam21_fallback_reason'.
Side-effect: 'points_camera' on both endpoints is now in OpenCV
RGB-camera frame (same as 'points_base' minus the T_base_camera
transform), regardless of which path produced the cloud. Previously the
bbox path stored points in Nuwa-native frame here. The web GUI plots
points_base when available, so the scatter views automatically pick up
the SAM21-segmented cloud.
Verified live: /scan → source=sam21 n=2414 sam21=111ms;
/object_points → source=sam21 n=2347 sam21=118ms;
/grasp/plan regression → source=sam21 n=2390 sam21=119ms;
/scan with SAM21 down → source=bbox_roi fallback='unreachable: ...'
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Status line on "Pianifica grasp" now shows source=sam21 / source=bbox_roi
(with fallback reason if any) plus n_points, so SAM21 vs fallback path is
visible at a glance without opening DevTools.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
12 commits — spec, plan, implementation, fixes, live integration.
- New: Sam21Client (clients/sam21.py), align_depth_to_rgb
(perception/depth_align.py), _points_via_sam21 helper in
grasp_routes.py, services.sam21 + grasp.use_sam21 config keys, pytest
scaffold + 11 unit tests.
- _object_points_in_base: try SAM21 first (default on), fall back to the
legacy bbox path on any non-fatal failure. Nuwa failures during prep
raise 502 immediately. Single-log discipline.
- Verified live: source=sam21 with ~2200 segmented points at 128 ms GPU
inference; SAM21 down → automatic bbox_roi fallback; class filter +
recovery on restart all work.
- Note that tests now exist and run with uv run pytest.
- Add SAM21 to the upstream services table.
- Add sam21() to the _deps factory list.
- Add a convention bullet about the SAM21 vs bbox frame asymmetry so
future contributors don't add nuwa_to_cv_frame to the SAM21 branch.
- Move 'import httpx' into the third-party imports block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Nuwa /api/intrinsics omits width/height from the rgb and ir blocks, so
SAM21 rejected the payload with KeyError('width') and we silently fell
back to bbox on every request. Derive the dimensions from the decoded
BGR + IR depth shapes and augment intr_full before building
CameraRegistration and calling SAM21. Verified end-to-end: source=sam21,
~2200 segmented points on Mortaio at 128 ms GPU inference.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- All SAM21-path logging now lives in _points_via_sam21 at the right
level (WARNING for operational, ERROR for bug codes / unexpected).
_object_points_in_base no longer double-logs on fallback.
- Nuwa failures during SAM21 prep now raise HTTPException(502)
immediately instead of running the futile bbox fallback (which would
also fail because Nuwa is down).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When grasp.use_sam21=true (default), _object_points_in_base feeds the YOLO
bbox to SAM21 and uses the segmented cloud directly. Any failure
(transport, segmentation empty, no valid depth, payload bug) logs and falls
back to the legacy bbox path. Response meta includes source ∈ {sam21,
bbox_roi} and sam21_fallback_reason when applicable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure refactor in preparation for the SAM21 branch. _object_points_in_base
keeps its signature and the bbox→ROI→base-frame logic moves into the
private helper. Meta now includes source='bbox_roi'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Align sam21() with nuwa()/marker()/yolo(): require base_url to be present
in config rather than silently defaulting. Misconfiguration now raises
KeyError at factory call instead of masking as a transport error.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Catch httpx.RequestError so DecodingError etc. trigger the bbox fallback
instead of propagating raw. Lock the intrinsics dict shape in the
happy-path test, drop unused imports.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Posts (rgb_png, depth_png, intrinsics, bbox) and returns (N,3) points in
meters in OpenCV RGB-camera frame. Normalises transport vs semantic errors
into Sam21Unavailable / Sam21Error.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
