feat: initial commit — orchestratore presa-mano con AI assistant + dual-robot
Pipeline completa per calibrazione hand-eye e grasp planning su robot Unitree G1/H1 con camera depth Nuwa-HP60C. Esposta come FastAPI + GUI vanilla JS, con assistente AI (OpenRouter + tool calling curato) per pilotare il sistema in linguaggio naturale. Architettura: - FastAPI orchestrator (porta da .env, default 8890) che coordina 4 microservizi upstream: Nuwa-HP60C, marker-finder, YOLO, robot bridge. - Dual-robot: services.g1 e services.h1 in config.yaml; switchabili a runtime via PUT /api/home/robot. - Calibrazione hand-eye eye-to-hand split per-side (right/left). - Marker pose: marker-finder riceve gli intrinseci RGB della Nuwa al boot, PnP server-side con quei dati. Nessun re-PnP locale richiesto. - Depth-to-RGB: registrazione iterativa IR↔RGB per back-projection 3D corretta (Nuwa /api/measure usa coords IR; il nostro /api/perception/ measure_rgb prende pixel RGB e ritorna 3D nel frame RGB OpenCV). - Correzione affine sul depth bias (calibrabile via marker come ground-truth) — riduce errore Z da ~6% a ~1.5% nei test. - AI assistant: 16 tool curati (move_wrist_*, hand_action, get_marker[_3d], detect_object, calibration_*, set_active_robot…), system prompt dinamico che riflette robot attivo + marker config + addendum customizzabile da GUI. Temperature di default 0.1 per evitare invenzioni. GUI (7 tab): - Home: stato comunicazione device + selettore robot attivo G1/H1 - Assistente AI: chat con OpenRouter, history persistita lato browser, task riutilizzabili (prompt salvati), trace tool calls - Impostazioni AI: modello da combo curata, temperature, max iterations, prompt addendum, anteprima system prompt - Calibrazione, Auto-Calibrazione, TrovaUtensile, Servo Visuale, Vista 3D, Movimento EE — invariate dalle versioni precedenti Setup: - Variabili in presa-mano/.env (template in .env.example) - OPENROUTER_API_KEY richiesta solo per il tab Assistente AI - PRESA_MANO_PORT configurabile Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
ac573505b7
121 files changed
+67199
No files matched your search
@@ -0,0 +1,68 @@
|
||||
"""Client for the marker-finder service (port 8082).
|
||||
|
||||
Marker-finder accepts OpenCV intrinsics (`camera_matrix` 3x3 + `dist_coeffs`)
|
||||
either as a per-request `intrinsics` form field on `/detect`, or as a stored
|
||||
default via `POST /intrinsics`. We push the live Nuwa intrinsics at startup
|
||||
(see main.py) so every subsequent `/detect` produces correct poses with no
|
||||
extra work, and we still expose a per-call override for unusual cases.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
class MarkerClient:
|
||||
def __init__(self, base_url: str, timeout_s: float = 5.0):
|
||||
self.base_url = base_url.rstrip("/")
|
||||
self.timeout_s = timeout_s
|
||||
|
||||
def detect_url(self, markers: list[dict[str, Any]], image_url: str,
|
||||
intrinsics: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""Returns {'found': [...], 'not_found': [...]}.
|
||||
|
||||
Each found item includes 'corners' (Nx2), 'id', 'size_mm', 'position_m',
|
||||
'rotation_matrix', 'quaternion_xyzw'. Pose is computed by marker-finder
|
||||
with the active default intrinsics (or the per-request override).
|
||||
|
||||
Args:
|
||||
intrinsics: optional per-call override, OpenCV format
|
||||
`{"camera_matrix": [[fx,0,cx],[0,fy,cy],[0,0,1]],
|
||||
"dist_coeffs": [k1,k2,p1,p2,k3]}`
|
||||
(mostly unused now that we push the Nuwa defaults at startup;
|
||||
kept for tools that want to detect against a custom camera).
|
||||
"""
|
||||
data = {"markers": json.dumps(markers), "image_url": image_url}
|
||||
if intrinsics is not None:
|
||||
data["intrinsics"] = json.dumps(intrinsics)
|
||||
with httpx.Client(timeout=self.timeout_s) as c:
|
||||
r = c.post(f"{self.base_url}/detect", data=data)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
def get_intrinsics(self) -> dict[str, Any]:
|
||||
"""Currently-active default intrinsics on marker-finder."""
|
||||
with httpx.Client(timeout=self.timeout_s) as c:
|
||||
r = c.get(f"{self.base_url}/intrinsics")
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
def set_intrinsics(self, camera_matrix: list[list[float]],
|
||||
dist_coeffs: list[float],
|
||||
persist: bool = False) -> dict[str, Any]:
|
||||
"""Replace marker-finder's default intrinsics.
|
||||
|
||||
`persist=True` writes them into marker-finder's config.yaml so they
|
||||
survive a restart. We default to in-memory only — presa-mano repushes
|
||||
them at every startup, so persisting would just create config drift.
|
||||
"""
|
||||
body = {"camera_matrix": camera_matrix, "dist_coeffs": dist_coeffs}
|
||||
with httpx.Client(timeout=self.timeout_s) as c:
|
||||
r = c.post(f"{self.base_url}/intrinsics",
|
||||
params={"persist": "true" if persist else "false"},
|
||||
json=body)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
Reference in new issue
Block a user