The 100 ms cache of get_rgb/get_depth/cached_frame served the latest
aligned frame, so an /api/capture from another client between /api/rgb and
/api/depth made the depth come from a newer frame than the color (with two
clients on PC2, the multi-camera service and the pestle detector, this can
happen). Older than R01: capture() already refreshed _last_frame then.
The cache is now its own slot, renewed only by the cached readers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JPEG of the latest color frame at the camera rate, not aligned to depth.
after=N waits up to timeout_ms (default 1000, max 5000) for a frame numbered
above N, so a client reads every frame without polling on empty; 504 when
none arrives. Headers: X-Frame-Seq, X-Camera-Timestamp-Ms (camera clock),
X-Camera-Timestamp-Domain, X-Host-Timestamp (arrival on the host).
Each color frame is encoded once for all clients (EncodedColor, own cache of
4: the color rate would flush the aligned frames' cache). New timing phase
jpeg_color. /api/rgb is unchanged: still the color of the aligned frame, the
same frame as /api/depth (README says so next to the new row).
color_frame() without after waits for the very first frame after open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two threads while the camera is open:
- realsense-raw (continuous, FrameAcquisition(on_demand=False)): the only
caller of wait_for_frames. For every frameset it publishes the color image
(own BGR copy) with the camera timestamp, its domain and the host arrival
time, and keeps the SDK frameset. No align and no depth conversion at the
camera rate: per frame it costs wait + one color copy (YUYV decode on USB 2).
- realsense-align (on demand, as in R01): when someone asks for color+depth
it aligns the newest frameset not aligned yet, converts the depth, copies.
Concurrent callers share one align. capture(), the 100 ms cache of
get_rgb/get_depth/cached_frame and every existing endpoint read this
aligned frame, so /api/rgb stays the color of the same frame as /api/depth.
The aligned color reuses the raw color pixels (align maps depth onto color).
At most 3 SDK framesets are referenced at a time (latest, receiving,
aligning); ColorFrame and CameraFrame hold no SDK memory.
New timing phase color_convert. One commit for items 1 and 2 of the request:
once the continuous thread owns the pipeline, the aligned path must take
its framesets, so they cannot be split without a broken intermediate state.
color_frame(after_seq, timeout_ms) gives the fast channel to the API.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fake frames now hand out a buffer (memoryview) like pyrealsense2's
BufData, so np.asanyarray() of it is a view, as on the robot; a control test
proves it. Then: the captured rgb/depth share no memory with the fake SDK
frame and own their data (without the fix both shared it), and _owned()
copies views only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the acquisition thread, frames outlive the grab (the latest frame,
the 100 ms cache, the 4 shared encodings). rgb/depth were views on the SDK
frame memory (np.asanyarray of get_data), so each kept an SDK or rs.align
pool frame busy; on the Jetson an exhausted pool shows as wait_for_frames
timeouts. _grab_once now copies them (_owned: only arrays that are views;
the YUYV decode, USB 2 resize and scaled depth already own theirs).
New phase "copy" in the timings, to see its cost on PC2.
capture() docstring: the frame is the next one COMPLETED after the call;
a grab already in progress may have started before it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/fake_realsense.py: a pipeline whose wait_for_frames renders a textured
640x480 frameset after 100 ms (no RealSense on the development PC).
tests/test_parallel_capture.py:
- 3 parallel /api/capture take < 2x one alone (medians of 5): the criterion
declared for optimize-streams R01. On the previous code: 391 vs 128 ms.
- /api/capture keeps its fields; parallel requests share one frame and its
bytes, frame_id still differs per response; one thread calls the pipeline.
- /api/rgb, /api/depth, /api/depth/color decode; /ws streams; close() stops
the acquisition thread.
Skipped when pyrealsense2 is not importable, like the existing driver tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Endpoints that wait for the camera or encode/compute (capture, rgb, depth,
depth/color, pointcloud, pointcloud/roi, measure, snapshot, and the SDK
calls discover/open/close) are plain `def`: FastAPI runs them in its
thread pool, so one slow request no longer holds the others. Plain `def`
rather than asyncio.to_thread in each route: it is FastAPI's own mechanism
and keeps the route bodies unchanged.
- /ws and the NATS snapshot stay async and use asyncio.to_thread for the
frame wait, the encoding and the recording write.
- EncodedFrame: the JPEG q85, 16-bit PNG, colormap JPEG q90, their base64
and the depth stats of a frame are computed once, on first use, and
shared: parallel requests, /ws and NATS on the same frame get the same
bytes. The last 4 frames' encodings are kept.
- camera.cached_frame(): the whole cached frame (get_rgb/get_depth unchanged).
Contract unchanged: same endpoints, same JSON fields. frame_id still counts
responses (now under a lock); two parallel /api/capture served by one frame
have different frame_id and the same timestamp and images.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FrameAcquisition: a single thread calls wait_for_frames + align (the old
capture() body, now _grab_once) and keeps the latest frame, numbered.
capture() no longer touches the pipeline: it waits for the next frame of
that thread; requests waiting at the same moment get the same frame, so N
callers cost one grab. get_rgb()/get_depth() keep their 100 ms cache,
now on the thread's latest frame (_last_frame is gone).
The thread grabs on demand, only while someone waits for a newer frame:
grabbing continuously at 30 fps would align frames nobody reads, and align
on the Jetson CPU is the main suspect of the capture cost. A grab error is
logged and retried, the thread never dies. close() stops it before the
pipeline and joins it after (pipeline.stop() ends a wait in progress).
open/close and camera_open are unchanged, so the healthcheck + autoheal
recovery of ede70a2 sees the same state as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One log line per phase every PHASE_LOG_EVERY samples (default 100, 0 = off):
mean and max of wait_frames, align, convert, each JPEG/PNG encoding,
colormap, depth_stats, base64, json and ws_send. Meant for the measurement
on PC2 via `docker logs` (optimize-streams R01): where a capture spends its
~130 ms of CPU is not known yet.
capture() moves its numpy conversion into _frame_from() unchanged, only so
the block can be timed; _depth_colormap() likewise wraps _jet_colormap().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The depth stream is aligned to the color frame via rs.align(rs.stream.color),
so the depth map lives on the color pixel grid. /api/measure, /api/pointcloud
and /api/pointcloud/roi were de-projecting those pixels with the *depth* (ir)
intrinsics, inflating X/Y by fx_rgb/fx_ir (~1.54x on the D435i). Z was correct
since it only depends on the depth value, not the intrinsics.
A 103mm ArUco marker at ~0.73m measured ~157mm; back-projected point clouds did
not overlap other calibrated cameras.
Fix: centralize intrinsics selection in _backprojection_intrinsics(), keyed on
the new RealSenseD435.is_depth_aligned_to_color(). Aligned (USB 3 default) ->
rgb intrinsics; no alignment (USB 2 fallback, where color is resized onto the
depth grid) -> ir intrinsics. An unconditional ir->rgb swap would have broken
the USB 2 path symmetrically.
Verified live: /api/measure?u=377&v=68 now returns x=+0.0686 (rgb) instead of
+0.1092 (ir). Documented root cause + non-regression check in README and
bug_d435i_intrinsics_mismatch.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Read width/height/fps from the live pipeline profile instead of
hardcoding 640x480, so /status reflects the firmware-accepted
resolution (e.g. the USB 2.x 640x480@15 fallback) rather than the
requested config.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mount mscli + master_service.sock into the container and stop
video_hub_pc4 at startup (and every 10s) so the RealSense pipeline
can open the D435i V4L2 node without EBUSY.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sostituisce il healthcheck banale (urlopen /api/health) con uno che
fallisce quando camera_open != true, e aggiunge il container willfarrell/autoheal
che restarta il backend quando Docker lo marca unhealthy. Recovery <1min
da SDK stuck senza intervento SSH.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This repo is now an independent project tracking the G1 deploy. The
upstream remote points to RealSense-D435i-G1 on git.humion.it and the
default branch is `main`. The H1 repo (RealSense-D435-H1) is now a
separate sibling project, linked from the README under "Repo correlati".
Sweeps the README for stale references to `branch camera-g1`, updates
the architecture diagram and repo-tree header to the new project name,
and updates rsync/cd paths to the canonical RealSense-D435i-G1 folder.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Updates the README from the H1 description to the current G1 state:
D435i over USB 3.2, librealsense pinned to v2.50.0 for FW 05.15.x
compatibility, host-side React build, and the video_hub_pc4 stop
prerequisite. Adds a G1-vs-H1 hardware comparison table and three
new troubleshooting entries (wait_for_frames timeout, resolve
request failure, npm DNS).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The G1's head camera is a D435i (USB 3.2, FW 05.15.01.55) — same family
as the H1's D435 but on a different bus and with older firmware. The
existing image opened the device but wait_for_frames timed out
indefinitely; this turned out to be a librealsense v2.55.x regression
against the D4xx firmware on this robot. Pinning the SDK back to v2.50.0
(the version known to work on the host install at /home/unitree/librealsense)
restores frame delivery.
While in there, refactor the runtime stage to copy librealsense2.so.*
straight from the builder instead of apt-installing it, so the binding
ABI always matches the .so version. Also drop the now-unused Intel apt
repo + GPG key dance.
Frontend stage removed: npm-in-docker can't resolve registry.npmjs.org
on this Jetson (docker's default DNS 127.0.0.53 doesn't work from inside
containers). Build the React app on the host one-shot via:
docker run --rm --dns 8.8.8.8 -v $(pwd)/src/frontend/react:/app \\
-w /app node:22-alpine sh -c 'npm install && npm run build'
then `docker compose build` just copies the prebuilt dist/.
Tested on the G1: all REST endpoints (rgb, depth, depth/color, capture,
intrinsics, pointcloud, measure, snapshot) return 200 with valid
payloads at 640x480@30 via USB 3.2.
Note: requires videohub_pc4 stopped (it holds /dev/video4):
/unitree/sbin/mscli stopservice video_hub_pc4
Drop-in replacement for the Nuwa-HP60C camera container, exposing RGB +
aligned depth + intrinsics via REST/WebSocket so the existing
pestello-pose-detector pipeline can consume frames without changes.
Key adaptations for D4xx on USB 2.0 (this robot has no USB 3 port exposed
externally — see STATUS.md):
- pyrealsense2 built from source; the librealsense v2.55.1 install rule
doesn't ship the main .so, so we copy it from the build dir and add an
__init__.py that re-exports the submodule.
- Color stream uses YUYV format on USB 2.x (BGR8/RGB8 time out on this
bus); converted to BGR via cv2.cvtColor in capture().
- rs.align is skipped on USB 2 (incompatible with YUYV); color is
cv2-resized to depth shape for approximate per-pixel alignment.
- Driver auto-detects usb_type_descriptor and falls back to the
full-resolution BGR8 + rs.align pipeline when on USB 3.
Tested: /api/{health,status,discover,rgb,depth,depth/color,capture,
intrinsics} all return 200 with valid payloads at 640x480.
OliviaCandClaude Opus 5.5