Skip to content

tools/lib: add auto_camera_source for multi-source camera file resolution#38103

Open
Ininsico wants to merge 1 commit into
commaai:masterfrom
Ininsico:camera-auto-source
Open

tools/lib: add auto_camera_source for multi-source camera file resolution#38103
Ininsico wants to merge 1 commit into
commaai:masterfrom
Ininsico:camera-auto-source

Conversation

@Ininsico

Copy link
Copy Markdown

Summary

Adds auto_camera_source() — the camera equivalent of the existing auto_source() for logs — enabling smart multi-source resolution of camera files (fcamera, dcamera, ecamera, qcamera) across all data sources.

Motivation

Resolves the TODO at tools/lib/logreader.py:149:

# TODO this should apply to camera files as well

Previously, camera file resolution had no multi-source fallback mechanism. Tools like replay, clip, and cabana had to construct camera file URLs manually through Route, without any of the smart fallback behavior that logs already enjoyed (rlog → qlog across internal API, comma API, openpilot CI, commaCarSegments).

Changes

tools/lib/file_sources.py

Extended comma_api_source() to serve all four camera file types through the Route API:

  • FileName.FCAMERA → route.camera_paths()
  • FileName.DCAMERA → route.dcamera_paths()
  • FileName.ECAMERA → route.ecamera_paths()
  • FileName.QCAMERA → route.qcamera_paths()

The previous else branch (which handled both QLOG and unknown types) is now explicit per type, with a ValueError raised for truly unknown file types.

tools/lib/logreader.py

Added auto_camera_source(identifier, sources, camera_type) function that:

  1. Parses route identifiers (supporting all existing formats: /slice, --segment, pipe |, etc.)
  2. Iterates through provided sources in order, collecting camera file URLs
  3. Falls back across sources when a source only partially covers the requested segments
  4. Returns a flat list of resolved file URLs
  5. Raises LogsUnavailable with a descriptive error (including per-source exception details) when files can't be found

tools/lib/tests/test_logreader.py

Added TestCameraSource class with 12 test cases:

Test What it covers
test_auto_camera_source_no_source Empty source list raises LogsUnavailable
test_auto_camera_source_no_source_default_camera Default camera type (FCAMERA)
test_auto_camera_source_found Single segment, single source
test_auto_camera_source_multiple_segments Multiple segments resolved at once
test_auto_camera_source_source_fallback First source empty, second succeeds
test_auto_camera_source_partial_fallback Each source provides some segments
test_auto_camera_source_error_then_success Source raises exception, fallback succeeds
test_auto_camera_source_all_fail All sources fail → LogsUnavailable
test_auto_camera_source_route_with_slash Pipe-separated route name
test_auto_camera_source_route_with_segment --N segment notation
test_comma_api_source_extended All 6 file types work through comma_api_source
test_comma_api_source_unknown_type Unknown type raises ValueError

Compatibility

  • auto_camera_source() works with all existing sources (comma_api_source, openpilotci_source, internal_source) — they already construct URLs from the FileNames tuple parameter
  • comma_car_segments_source is unaffected (it doesn't use the fns parameter)
  • Backwards compatible: existing comma_api_source callers passing RLOG/QLOG see zero behavior change

Usage Example

from openpilot.tools.lib.logreader import auto_camera_source
from openpilot.tools.lib.file_sources import comma_api_source, openpilotci_source

# Resolve fcamera.hevc for segment 5 of a route
urls = auto_camera_source(
    a2a0ccea32023010/2023-07-27--13-01-19/5,
    sources=[comma_api_source, openpilotci_source],
    camera_type=FileName.FCAMERA,
)
# urls == [https://.../5/fcamera.hevc]

Verification

All 12 new tests exercise the function logic through mocks. The test_comma_api_source_extended test validates that comma_api_source correctly dispatches to Route.camera_paths(), Route.dcamera_paths(), Route.ecamera_paths(), and Route.qcamera_paths().

@github-actions github-actions Bot added the tools label May 29, 2026
@Ininsico Ininsico force-pushed the camera-auto-source branch from 94e6744 to c018128 Compare May 29, 2026 06:01
@github-actions

github-actions Bot commented May 29, 2026

Copy link
Copy Markdown
Contributor

Process replay diff report

Replays driving segments through this PR and compares the behavior to master.
Please review any changes carefully to ensure they are expected.

✅ 0 changed, 66 passed, 0 errors

…tion

Implements the TODO at logreader.py:149 ('this should apply to camera
files as well') by introducing auto_camera_source(), the camera
equivalent of the existing auto_source() for log files.

- New auto_camera_source() function resolves fcamera/dcamera/ecamera/
  qcamera files across multiple sources with fallback, matching the
  pattern used for rlog/qlog resolution
- Extended comma_api_source() in file_sources.py to serve all four
  camera file types through the Route API
- Added ValueError guard for unknown file types
- 12 new tests covering single/multi-segment resolution, source
  fallback, error recovery, full-route parsing, and comma_api_source
  integration
@Ininsico Ininsico force-pushed the camera-auto-source branch from c018128 to 13c059d Compare May 29, 2026 06:14
@github-actions

Copy link
Copy Markdown
Contributor

This PR has had no activity for 24 days. It will be automatically closed in 7 days if there is no activity.

@github-actions github-actions Bot added the stale label Jun 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant