The previous commit's shared-JPEG test failed when run with the others:
the module-level caches kept entries numbered 1 from earlier tests' fake
cameras (each numbers its framesets from 1). The service is not affected:
a cache hit also requires the same frame object. The fixture now clears
the caches.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the fake camera: /api/raw/latest holds color and depth of one frameset
(frame number in the pixels), waits with after=N and times out with 504,
shares its color JPEG with the color channel; /api/capture's frameset_seq;
/api/intrinsics has sizes, models and depth_scale_m; compare_depths counts
and differences; the comparison script end to end through the API with
identical geometry (median |diff| 0, > 90% exact: rs.align itself changes
~3% of the fake's pure-noise depth where pixel footprints overlap).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read-only (GET only). For each frameset: /api/raw/latest and /api/capture
of the same frameset_seq (retried until they match), the raw depth aligned
on the host with pyrealsense2 (software_device built from /api/intrinsics:
both streams' intrinsics, distortion model and coefficients, extrinsics
depth->color; then syncer + rs.align), and a pixel-by-pixel comparison of
the two aligned depths: pixels valid in both, only in one, exact-equal
fraction, |diff| mean/median/p99/max in mm. One JSON line per frameset.
The syncer may emit the first pair split, so a pair is pushed again (up
to 5 times) until a complete frameset of that frame number comes out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Already there: fx/fy/cx/cy and k1,k2,k3,p1,p2 of both streams, extrinsics
depth->color (rotation row-major, translation mm). Added, as new keys only:
width, height and distortion model name (an rs.distortion member) in "ir"
and "rgb", and depth_scale_m at the top. Docstring states the extrinsics
convention: p_rgb = R p_ir + t, R row-major (the SDK's is column-major).
Same keys in /api/status and /api/open, which share the payload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The number of the camera frameset an aligned frame comes from, in the same
numbering as /api/raw/latest and /api/color/latest: it lets a tool compare
Intel's align on PC2 with an align on the host on the SAME frameset.
Existing fields unchanged (frame_id still counts responses); the field-set
tests now say "the R01 fields plus frameset_seq only".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For aligning on the host: on the Jetson rs.align.process costs ~104 ms and
holds the GIL (optimize-streams R04/R05). JSON in the usual envelope, one
response: frameset_seq, camera_timestamp_ms, timestamp_domain,
host_timestamp, color_jpeg_b64 (color grid), depth_png_b64 (16-bit PNG, mm,
depth grid), color_shape, depth_shape, depth_units. after=N / timeout_ms as
/api/color/latest. JSON + base64 like /api/capture: every client already
parses that envelope, one request gets both images, base64 costs ~0.1 ms.
Built once per frameset (EncodedRaw, last 4 kept); the color JPEG is the
color channel's own (same bytes). The depth is converted only on request
(camera.raw_frame, phase raw_depth; PNG in phase png_raw_depth).
camera: _published() shared by color_frame() and raw_frame(); _depth_mm()
extracted unchanged from _aligned_frame().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fake_realsense: clocked mode (a frameset every 1/fps s, like the device),
FakeAlign, camera timestamps, and the frame number written in the pixels
(color block, depth[0, 0]) so a test knows which frame an image came from.
test_two_channels:
- /api/rgb, /api/capture, /api/depth: rgb and depth from one frame (criterion
B of optimize-streams R03; failed before the previous commit: 1 vs 2);
- /api/color/latest serves strictly newer frames, the image is the frame its
headers name (camera clock), 504 on timeout, each frame encoded once;
- color >= 25 new frames/s while /api/capture loops (criteria A1/A2, 3 s).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 100 ms cache of get_rgb/get_depth/cached_frame served the latest
aligned frame, so an /api/capture from another client between /api/rgb and
/api/depth made the depth come from a newer frame than the color (with two
clients on PC2, the multi-camera service and the pestle detector, this can
happen). Older than R01: capture() already refreshed _last_frame then.
The cache is now its own slot, renewed only by the cached readers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JPEG of the latest color frame at the camera rate, not aligned to depth.
after=N waits up to timeout_ms (default 1000, max 5000) for a frame numbered
above N, so a client reads every frame without polling on empty; 504 when
none arrives. Headers: X-Frame-Seq, X-Camera-Timestamp-Ms (camera clock),
X-Camera-Timestamp-Domain, X-Host-Timestamp (arrival on the host).
Each color frame is encoded once for all clients (EncodedColor, own cache of
4: the color rate would flush the aligned frames' cache). New timing phase
jpeg_color. /api/rgb is unchanged: still the color of the aligned frame, the
same frame as /api/depth (README says so next to the new row).
color_frame() without after waits for the very first frame after open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two threads while the camera is open:
- realsense-raw (continuous, FrameAcquisition(on_demand=False)): the only
caller of wait_for_frames. For every frameset it publishes the color image
(own BGR copy) with the camera timestamp, its domain and the host arrival
time, and keeps the SDK frameset. No align and no depth conversion at the
camera rate: per frame it costs wait + one color copy (YUYV decode on USB 2).
- realsense-align (on demand, as in R01): when someone asks for color+depth
it aligns the newest frameset not aligned yet, converts the depth, copies.
Concurrent callers share one align. capture(), the 100 ms cache of
get_rgb/get_depth/cached_frame and every existing endpoint read this
aligned frame, so /api/rgb stays the color of the same frame as /api/depth.
The aligned color reuses the raw color pixels (align maps depth onto color).
At most 3 SDK framesets are referenced at a time (latest, receiving,
aligning); ColorFrame and CameraFrame hold no SDK memory.
New timing phase color_convert. One commit for items 1 and 2 of the request:
once the continuous thread owns the pipeline, the aligned path must take
its framesets, so they cannot be split without a broken intermediate state.
color_frame(after_seq, timeout_ms) gives the fast channel to the API.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fake frames now hand out a buffer (memoryview) like pyrealsense2's
BufData, so np.asanyarray() of it is a view, as on the robot; a control test
proves it. Then: the captured rgb/depth share no memory with the fake SDK
frame and own their data (without the fix both shared it), and _owned()
copies views only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the acquisition thread, frames outlive the grab (the latest frame,
the 100 ms cache, the 4 shared encodings). rgb/depth were views on the SDK
frame memory (np.asanyarray of get_data), so each kept an SDK or rs.align
pool frame busy; on the Jetson an exhausted pool shows as wait_for_frames
timeouts. _grab_once now copies them (_owned: only arrays that are views;
the YUYV decode, USB 2 resize and scaled depth already own theirs).
New phase "copy" in the timings, to see its cost on PC2.
capture() docstring: the frame is the next one COMPLETED after the call;
a grab already in progress may have started before it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/fake_realsense.py: a pipeline whose wait_for_frames renders a textured
640x480 frameset after 100 ms (no RealSense on the development PC).
tests/test_parallel_capture.py:
- 3 parallel /api/capture take < 2x one alone (medians of 5): the criterion
declared for optimize-streams R01. On the previous code: 391 vs 128 ms.
- /api/capture keeps its fields; parallel requests share one frame and its
bytes, frame_id still differs per response; one thread calls the pipeline.
- /api/rgb, /api/depth, /api/depth/color decode; /ws streams; close() stops
the acquisition thread.
Skipped when pyrealsense2 is not importable, like the existing driver tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Endpoints that wait for the camera or encode/compute (capture, rgb, depth,
depth/color, pointcloud, pointcloud/roi, measure, snapshot, and the SDK
calls discover/open/close) are plain `def`: FastAPI runs them in its
thread pool, so one slow request no longer holds the others. Plain `def`
rather than asyncio.to_thread in each route: it is FastAPI's own mechanism
and keeps the route bodies unchanged.
- /ws and the NATS snapshot stay async and use asyncio.to_thread for the
frame wait, the encoding and the recording write.
- EncodedFrame: the JPEG q85, 16-bit PNG, colormap JPEG q90, their base64
and the depth stats of a frame are computed once, on first use, and
shared: parallel requests, /ws and NATS on the same frame get the same
bytes. The last 4 frames' encodings are kept.
- camera.cached_frame(): the whole cached frame (get_rgb/get_depth unchanged).
Contract unchanged: same endpoints, same JSON fields. frame_id still counts
responses (now under a lock); two parallel /api/capture served by one frame
have different frame_id and the same timestamp and images.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FrameAcquisition: a single thread calls wait_for_frames + align (the old
capture() body, now _grab_once) and keeps the latest frame, numbered.
capture() no longer touches the pipeline: it waits for the next frame of
that thread; requests waiting at the same moment get the same frame, so N
callers cost one grab. get_rgb()/get_depth() keep their 100 ms cache,
now on the thread's latest frame (_last_frame is gone).
The thread grabs on demand, only while someone waits for a newer frame:
grabbing continuously at 30 fps would align frames nobody reads, and align
on the Jetson CPU is the main suspect of the capture cost. A grab error is
logged and retried, the thread never dies. close() stops it before the
pipeline and joins it after (pipeline.stop() ends a wait in progress).
open/close and camera_open are unchanged, so the healthcheck + autoheal
recovery of ede70a2 sees the same state as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One log line per phase every PHASE_LOG_EVERY samples (default 100, 0 = off):
mean and max of wait_frames, align, convert, each JPEG/PNG encoding,
colormap, depth_stats, base64, json and ws_send. Meant for the measurement
on PC2 via `docker logs` (optimize-streams R01): where a capture spends its
~130 ms of CPU is not known yet.
capture() moves its numpy conversion into _frame_from() unchanged, only so
the block can be timed; _depth_colormap() likewise wraps _jet_colormap().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sostituisce il healthcheck banale (urlopen /api/health) con uno che
fallisce quando camera_open != true, e aggiunge il container willfarrell/autoheal
che restarta il backend quando Docker lo marca unhealthy. Recovery <1min
da SDK stuck senza intervento SSH.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
OliviaCandClaude Opus 5.5