For aligning on the host: on the Jetson rs.align.process costs ~104 ms and
holds the GIL (optimize-streams R04/R05). JSON in the usual envelope, one
response: frameset_seq, camera_timestamp_ms, timestamp_domain,
host_timestamp, color_jpeg_b64 (color grid), depth_png_b64 (16-bit PNG, mm,
depth grid), color_shape, depth_shape, depth_units. after=N / timeout_ms as
/api/color/latest. JSON + base64 like /api/capture: every client already
parses that envelope, one request gets both images, base64 costs ~0.1 ms.
Built once per frameset (EncodedRaw, last 4 kept); the color JPEG is the
color channel's own (same bytes). The depth is converted only on request
(camera.raw_frame, phase raw_depth; PNG in phase png_raw_depth).
camera: _published() shared by color_frame() and raw_frame(); _depth_mm()
extracted unchanged from _aligned_frame().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 100 ms cache of get_rgb/get_depth/cached_frame served the latest
aligned frame, so an /api/capture from another client between /api/rgb and
/api/depth made the depth come from a newer frame than the color (with two
clients on PC2, the multi-camera service and the pestle detector, this can
happen). Older than R01: capture() already refreshed _last_frame then.
The cache is now its own slot, renewed only by the cached readers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JPEG of the latest color frame at the camera rate, not aligned to depth.
after=N waits up to timeout_ms (default 1000, max 5000) for a frame numbered
above N, so a client reads every frame without polling on empty; 504 when
none arrives. Headers: X-Frame-Seq, X-Camera-Timestamp-Ms (camera clock),
X-Camera-Timestamp-Domain, X-Host-Timestamp (arrival on the host).
Each color frame is encoded once for all clients (EncodedColor, own cache of
4: the color rate would flush the aligned frames' cache). New timing phase
jpeg_color. /api/rgb is unchanged: still the color of the aligned frame, the
same frame as /api/depth (README says so next to the new row).
color_frame() without after waits for the very first frame after open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two threads while the camera is open:
- realsense-raw (continuous, FrameAcquisition(on_demand=False)): the only
caller of wait_for_frames. For every frameset it publishes the color image
(own BGR copy) with the camera timestamp, its domain and the host arrival
time, and keeps the SDK frameset. No align and no depth conversion at the
camera rate: per frame it costs wait + one color copy (YUYV decode on USB 2).
- realsense-align (on demand, as in R01): when someone asks for color+depth
it aligns the newest frameset not aligned yet, converts the depth, copies.
Concurrent callers share one align. capture(), the 100 ms cache of
get_rgb/get_depth/cached_frame and every existing endpoint read this
aligned frame, so /api/rgb stays the color of the same frame as /api/depth.
The aligned color reuses the raw color pixels (align maps depth onto color).
At most 3 SDK framesets are referenced at a time (latest, receiving,
aligning); ColorFrame and CameraFrame hold no SDK memory.
New timing phase color_convert. One commit for items 1 and 2 of the request:
once the continuous thread owns the pipeline, the aligned path must take
its framesets, so they cannot be split without a broken intermediate state.
color_frame(after_seq, timeout_ms) gives the fast channel to the API.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the acquisition thread, frames outlive the grab (the latest frame,
the 100 ms cache, the 4 shared encodings). rgb/depth were views on the SDK
frame memory (np.asanyarray of get_data), so each kept an SDK or rs.align
pool frame busy; on the Jetson an exhausted pool shows as wait_for_frames
timeouts. _grab_once now copies them (_owned: only arrays that are views;
the YUYV decode, USB 2 resize and scaled depth already own theirs).
New phase "copy" in the timings, to see its cost on PC2.
capture() docstring: the frame is the next one COMPLETED after the call;
a grab already in progress may have started before it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Endpoints that wait for the camera or encode/compute (capture, rgb, depth,
depth/color, pointcloud, pointcloud/roi, measure, snapshot, and the SDK
calls discover/open/close) are plain `def`: FastAPI runs them in its
thread pool, so one slow request no longer holds the others. Plain `def`
rather than asyncio.to_thread in each route: it is FastAPI's own mechanism
and keeps the route bodies unchanged.
- /ws and the NATS snapshot stay async and use asyncio.to_thread for the
frame wait, the encoding and the recording write.
- EncodedFrame: the JPEG q85, 16-bit PNG, colormap JPEG q90, their base64
and the depth stats of a frame are computed once, on first use, and
shared: parallel requests, /ws and NATS on the same frame get the same
bytes. The last 4 frames' encodings are kept.
- camera.cached_frame(): the whole cached frame (get_rgb/get_depth unchanged).
Contract unchanged: same endpoints, same JSON fields. frame_id still counts
responses (now under a lock); two parallel /api/capture served by one frame
have different frame_id and the same timestamp and images.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FrameAcquisition: a single thread calls wait_for_frames + align (the old
capture() body, now _grab_once) and keeps the latest frame, numbered.
capture() no longer touches the pipeline: it waits for the next frame of
that thread; requests waiting at the same moment get the same frame, so N
callers cost one grab. get_rgb()/get_depth() keep their 100 ms cache,
now on the thread's latest frame (_last_frame is gone).
The thread grabs on demand, only while someone waits for a newer frame:
grabbing continuously at 30 fps would align frames nobody reads, and align
on the Jetson CPU is the main suspect of the capture cost. A grab error is
logged and retried, the thread never dies. close() stops it before the
pipeline and joins it after (pipeline.stop() ends a wait in progress).
open/close and camera_open are unchanged, so the healthcheck + autoheal
recovery of ede70a2 sees the same state as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One log line per phase every PHASE_LOG_EVERY samples (default 100, 0 = off):
mean and max of wait_frames, align, convert, each JPEG/PNG encoding,
colormap, depth_stats, base64, json and ws_send. Meant for the measurement
on PC2 via `docker logs` (optimize-streams R01): where a capture spends its
~130 ms of CPU is not known yet.
capture() moves its numpy conversion into _frame_from() unchanged, only so
the block can be timed; _depth_colormap() likewise wraps _jet_colormap().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The depth stream is aligned to the color frame via rs.align(rs.stream.color),
so the depth map lives on the color pixel grid. /api/measure, /api/pointcloud
and /api/pointcloud/roi were de-projecting those pixels with the *depth* (ir)
intrinsics, inflating X/Y by fx_rgb/fx_ir (~1.54x on the D435i). Z was correct
since it only depends on the depth value, not the intrinsics.
A 103mm ArUco marker at ~0.73m measured ~157mm; back-projected point clouds did
not overlap other calibrated cameras.
Fix: centralize intrinsics selection in _backprojection_intrinsics(), keyed on
the new RealSenseD435.is_depth_aligned_to_color(). Aligned (USB 3 default) ->
rgb intrinsics; no alignment (USB 2 fallback, where color is resized onto the
depth grid) -> ir intrinsics. An unconditional ir->rgb swap would have broken
the USB 2 path symmetrically.
Verified live: /api/measure?u=377&v=68 now returns x=+0.0686 (rgb) instead of
+0.1092 (ir). Documented root cause + non-regression check in README and
bug_d435i_intrinsics_mismatch.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Read width/height/fps from the live pipeline profile instead of
hardcoding 640x480, so /status reflects the firmware-accepted
resolution (e.g. the USB 2.x 640x480@15 fallback) rather than the
requested config.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop-in replacement for the Nuwa-HP60C camera container, exposing RGB +
aligned depth + intrinsics via REST/WebSocket so the existing
pestello-pose-detector pipeline can consume frames without changes.
Key adaptations for D4xx on USB 2.0 (this robot has no USB 3 port exposed
externally — see STATUS.md):
- pyrealsense2 built from source; the librealsense v2.55.1 install rule
doesn't ship the main .so, so we copy it from the build dir and add an
__init__.py that re-exports the submodule.
- Color stream uses YUYV format on USB 2.x (BGR8/RGB8 time out on this
bus); converted to BGR via cv2.cvtColor in capture().
- rs.align is skipped on USB 2 (incompatible with YUYV); color is
cv2-resized to depth shape for approximate per-pixel alignment.
- Driver auto-detects usb_type_descriptor and falls back to the
full-resolution BGR8 + rs.align pipeline when on USB 3.
Tested: /api/{health,status,discover,rgb,depth,depth/color,capture,
intrinsics} all return 200 with valid payloads at 640x480.
OliviaCandClaude Opus 5.5