Regular Out of Memory occurances (due to camera unavailable?)

[MartinAirN]

Running the Cuda image, in this case in Podman (but similar experiences with worse logging in Docker), I experience quite a few OOM restarts of the container.

viseron.service: A process of this unit has been killed by the OOM killer. is a regular statement.

Setup:
Container: docker.io/roflcoopter/amd64-cuda-viseron:latest bafe75d0e1a9
RAM allocated: --memory 10g --shm-size 1g
Normal RAM usage: 12% - 3.94 GB - 37% of 10.7 GB limit
System:

• CPU: Intel(R) Core(TM) i5-14600K
• GPU: GA107 [GeForce RTX 3050 6GB]
• RAM: 32GB DDR4 (2x 16GB)

Analysis
If I look at the logs there is no clear "this is happening, so that might increase ram usage" thing going on. Though it might be related with unavailable cameras. As in this log camera_6 seems to have issues. It could however also be a symptom of the OOM state. Though if I go back further in the logs there is nothing different in remarks than this. So camera_6 might just be out of reach.
In the logs at the bottom I have included the first OOM remark present. The camera timeout is there many, many times before it.

Server config:

ffmpeg:
  camera:
    camera_6:
      name: KitchenDining
      host: 192.168.1.58
      port: 554
      path: /stream1
      username: <camerauser>
      password: <camerapass>

    camera_4:
      name: Living
      host: 192.168.1.59
      port: 554
      path: /stream1
      username: <camerauser>
      password: <camerapass>

    camera_7:
      name: Driveway
      host: 192.168.1.249
      port: 554
      path: /stream1
      username: <camerauser>
      password: <camerapass>

    camera_8:
      name: Garden
      host: 192.168.1.244
      port: 554
      path: /stream1
      username: <camerauser>
      password: <camerapass>

    camera_bedroom:
      name: Bedroom
      host: 192.168.1.39
      port: 554
      path: /s0
      username: <camerauser>
      password: <camerapass>

codeprojectai:
  host: <ai.hostname>
  license_plate_recognition:
    cameras:
      camera_7:
    known_plates:
      - lic3ns3

yolo:
  object_detector:
    model_path: /detectors/models/yolo/yolo26n.pt
    cameras:
      camera_4:
        fps: 5
        log_all_objects: true
        labels:
        - label: person
          confidence: 0.4
          trigger_event_recording: false
        - label: cat
          confidence: 0.2
          trigger_event_recording: false
      camera_6:
        fps: 5
        log_all_objects: true
        zones:
        - name: HallDining
          coordinates:
          - x: 1339
            y: 4
          - x: 1325
            y: 1296
          - x: 2140
            y: 1296
          - x: 2304
            y: 1117
          - x: 2304
            y: 653
          - x: 1886
            y: 496
          - x: 1912
            y: 0
          labels:
          - label: person
            confidence: 0.7
            height_min: 0.45
            trigger_event_recording: false
          - label: cat
            confidence: 0.35
            trigger_event_recording: false
        - name: Kitchen
          coordinates:
          - x: 0
            y: 752
          - x: 4
            y: 4
          - x: 1333
            y: 0
          - x: 1322
            y: 1296
          - x: 218
            y: 1296
          labels:
          - label: person
            confidence: 0.4
            height_min: 0.3
            trigger_event_recording: false
          - label: cat
            confidence: 0.35
            trigger_event_recording: false
      camera_7:
        fps: 5
        scan_on_motion_only: true
        log_all_objects: true
        zones:
        - name: Driveway
          coordinates:
          - x: 0
            y: 652
          - x: 597
            y: 327
          - x: 2337
            y: 334
          - x: 2880
            y: 941
          - x: 2868
            y: 1606
          - x: 9
            y: 1611
        labels:
        - label: person
          confidence: 0.5
          height_min: 0.3
          trigger_event_recording: true
        - label: cat
          confidence: 0.2
          trigger_event_recording: false
      camera_8:
        fps: 3
        log_all_objects: true
        zones:
        - name: Garden
          coordinates:
          - x: 105
            y: 1144
          - x: 2051
            y: 334
          - x: 2305
            y: 185
          - x: 2880
            y: 602
          - x: 2454
            y: 1620
          - x: 290
            y: 1620
          labels:
          - label: person
            confidence: 0.4
            height_min: 0.3
            trigger_event_recording: true
          - label: cat
            confidence: 0.2
            trigger_event_recording: false
      camera_bedroom:
        fps: 1
        log_all_objects: true
        labels:
        - label: person
          confidence: 0.6
          trigger_event_recording: true
        - label: cat
          confidence: 0.2
          trigger_event_recording: false

mog2:
  motion_detector:
    cameras:
      camera_6:
        fps: 5
        threshold: 10
        area: 0.02
      camera_4:
        fps: 1
        threshold: 25
      camera_7:
        fps: 1
        threshold: 25
        mask:
        - coordinates:
          - x: 14
            y: 20
          - x: 2873
            y: 20
          - x: 2868
            y: 1503
          - x: 2434
            y: 651
          - x: 583
            y: 643
          - x: 16
            y: 1429
      camera_8:
        fps: 1
        threshold: 25
        mask:
        - coordinates:
          - x: 0
            y: 866
          - x: 2622
            y: 31
          - x: 2880
            y: 128
          - x: 2880
            y: 0
          - x: 0
            y: 0
      camera_bedroom:
        fps: 1
        threshold: 10
        area: 0.02
nvr:
  camera_6:
  camera_4:
  camera_7:
  camera_8:
  camera_bedroom:

mqtt:
  broker: <address>
  port: <port>
  username: <mqttusername>
  password: <password>
  home_assistant:

storage:
  recorder:
    tiers:
    - path: /
      events:
        max_size:
          gb: 100
      continuous:
        max_size:
          gb: 125
      check_interval:
        minutes: 5
  snapshots:
    tiers:
    - path: /
      max_age:
        days: 60
      check_interval:
        minutes: 5

Logs (reverse order):

systemd
15:07
viseron.service: Consumed 1h 13min 31.148s CPU time, 10G memory peak, 7G memory swap peak.
systemd
15:07
2026-07-03 15:07:40.226166488 +0200 CEST m=+0.127052340 container remove 3960eb3d71b51b5991ad7034b49f7f182b15392b0860ae964eb696aca0bf639f (image=docker.io/roflcoopter/amd64-cuda-viseron:latest, name=viseron, com.docker.compose.service=amd64-cuda-viseron, com.docker.compose.version=5.1.1, com.nvidia.cudnn.version=9.10.2.21-1, maintainer=NVIDIA CORPORATION <cudatools@nvidia.com>, org.opencontainers.image.ref.name=ubuntu, org.opencontainers.image.version=22.04, PODMAN_SYSTEMD_UNIT=viseron.service, com.docker.compose.project=azure-pipelines)
podman
15:07
2026-07-03 15:07:40.117941638 +0200 CEST m=+0.018827528 container died 3960eb3d71b51b5991ad7034b49f7f182b15392b0860ae964eb696aca0bf639f (image=docker.io/roflcoopter/amd64-cuda-viseron:latest, name=viseron, com.docker.compose.version=5.1.1, com.nvidia.cudnn.version=9.10.2.21-1, maintainer=NVIDIA CORPORATION <cudatools@nvidia.com>, org.opencontainers.image.ref.name=ubuntu, org.opencontainers.image.version=22.04, PODMAN_SYSTEMD_UNIT=viseron.service, com.docker.compose.project=azure-pipelines, com.docker.compose.service=amd64-cuda-viseron)
podman
15:07
[s6-finish] sending all processes the KILL signal and exiting.
viseron
15:07
2026-07-03 13:07:09.926 UTC [580] LOG: database system is shut down
viseron
15:07
2026-07-03 13:07:09.871 UTC [610] LOG: shutting down
viseron
15:07
2026-07-03 13:07:09.855 UTC [580] LOG: background worker "logical replication launcher" (PID 615) exited with exit code 1
viseron
15:07
2026-07-03 13:07:09.846 UTC [1127] FATAL: terminating connection due to administrator command
viseron
15:07
2026-07-03 13:07:09.844 UTC [580] LOG: received smart shutdown request
viseron
15:07
2026-07-03 13:07:09.844 UTC [1133] FATAL: terminating connection due to administrator command
viseron
15:07
2026-07-03 13:07:09.843 UTC [13084] FATAL: terminating connection due to administrator command
viseron
15:07
2026-07-03 13:07:09.843 UTC [680] FATAL: terminating connection due to administrator command
viseron
15:07
2026-07-03 13:07:09.843 UTC [673] FATAL: terminating connection due to administrator command
viseron
15:07
2026-07-03 13:07:09.843 UTC [1129] FATAL: terminating connection due to administrator command
viseron
15:07
2026-07-03 13:07:09.838 UTC [580] LOG: received SIGHUP, reloading configuration files
viseron
15:07
[s6-finish] sending all processes the TERM signal.
viseron
15:07
[s6-finish] waiting for services.
viseron
15:07
[cont-finish.d] done.
viseron
15:07
[cont-finish.d] 10-postgres: exited 99.
viseron
15:07
s6-maximumtime: warning: unable to wait for child process: Operation timed out
viseron
15:06
[0;32mWait for Viseron to stop...[0m
viseron
15:06
[cont-finish.d] 10-postgres: executing...
viseron
15:06
[cont-finish.d] executing container finish scripts...
viseron
15:06
[viseron-finish] Viseron received signal 9
viseron
15:06
viseron
15:06
[viseron-finish] Shutdown completed at Fri Jul 3 13:06:39 UTC 2026
viseron
15:06
[viseron-finish] Viseron exit code 256
viseron
15:06
viseron.service: A process of this unit has been killed by the OOM killer.
systemd
15:03
[80D[1A[K[31m2026-07-03 13:03:50.319 [ERROR ] [viseron.watchdog.subprocess_watchdog] - Process subprocess.viseron.components.storage.storage_subprocess.tier_check_worker.process has exited, restarting, message repeated 3 times[0m
viseron
15:03
viseron.service: A process of this unit has been killed by the OOM killer.
systemd
15:01
[80D[1A[K[31m2026-07-03 13:01:05.319 [ERROR ] [viseron.watchdog.subprocess_watchdog] - Process subprocess.viseron.components.storage.storage_subprocess.tier_check_worker.process has exited, restarting, message repeated 2 times[0m
viseron
15:00
viseron.service: A process of this unit has been killed by the OOM killer.
systemd
14:58
[31m2026-07-03 12:58:05.319 [ERROR ] [viseron.watchdog.subprocess_watchdog] - Process subprocess.viseron.components.storage.storage_subprocess.tier_check_worker.process has exited, restarting[0m
viseron
14:57
viseron.service: A process of this unit has been killed by the OOM killer.
systemd
14:56
[31m2026-07-03 12:56:14.926 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743779.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743779.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.926 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743775.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743775.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.925 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743769.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743769.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.924 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743763.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743763.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.923 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743759.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743759.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.923 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743755.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743755.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.922 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743749.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743749.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.922 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743745.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743745.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.921 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743739.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743739.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.921 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743734.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743734.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.920 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743729.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743729.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.919 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743719.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743719.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.919 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743725.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743725.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.918 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743713.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743713.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.918 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743703.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743703.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.918 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743699.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743699.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.918 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743709.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743709.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.917 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743693.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743693.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.916 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743689.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743689.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:14.915 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743685.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743685.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:11.324 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743679.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743679.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:07.783 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743675.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743675.m4s'[0m
viseron
14:56
[31m2026-07-03 12:56:06.939 [ERROR ] [viseron.components.storage.storage_subprocess.subprocess] - Error processing command: DataItemDeleteFile(cmd='delete_file', src='/segments/camera_8/1782743669.m4s', callback_id=None, error=None), error: [Errno 2] No such file or directory: '/segments/camera_8/1782743669.m4s'[0m
viseron
14:55
[80D[1A[K[31m2026-07-03 12:55:20.319 [ERROR ] [viseron.watchdog.subprocess_watchdog] - Process subprocess.viseron.components.storage.storage_subprocess.tier_check_worker.process has exited, restarting, message repeated 6 times[0m
viseron
14:55
2026-07-03 12:55:12.971 UTC [94178] LOG: unexpected EOF on client connection with an open transaction
viseron
14:55
viseron.service: A process of this unit has been killed by the OOM killer.
systemd
14:51
[80D[1A[K[31m2026-07-03 12:51:50.319 [ERROR ] [viseron.watchdog.subprocess_watchdog] - Process subprocess.viseron.components.storage.storage_subprocess.tier_check_worker.process has exited, restarting, message repeated 5 times[0m
viseron
14:51
viseron.service: A process of this unit has been killed by the OOM killer.
systemd
14:49
[80D[1A[K[31m2026-07-03 12:49:20.319 [ERROR ] [viseron.watchdog.subprocess_watchdog] - Process subprocess.viseron.components.storage.storage_subprocess.tier_check_worker.process has exited, restarting, message repeated 4 times[0m
viseron
14:49
2026-07-03 12:49:17.142 UTC [84824] LOG: unexpected EOF on client connection with an open transaction
viseron
14:49
viseron.service: A process of this unit has been killed by the OOM killer.
systemd
14:45
[80D[1A[K[31m2026-07-03 12:45:50.320 [ERROR ] [viseron.watchdog.subprocess_watchdog] - Process subprocess.viseron.components.storage.storage_subprocess.tier_check_worker.process has exited, restarting, message repeated 3 times[0m
viseron
14:45
viseron.service: A process of this unit has been killed by the OOM killer.
systemd
14:43
[80D[1A[K[31m2026-07-03 12:43:20.319 [ERROR ] [viseron.watchdog.subprocess_watchdog] - Process subprocess.viseron.components.storage.storage_subprocess.tier_check_worker.process has exited, restarting, message repeated 2 times[0m
viseron
14:43
viseron.service: A process of this unit has been killed by the OOM killer.
systemd
14:42
[80D[1A[K[33m2026-07-03 12:42:24.282 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac., message repeated 2 times[0m
viseron
14:42
[33m2026-07-03 12:42:24.282 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac.[0m
viseron
14:42
[31m2026-07-03 12:42:24.281 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Restarting frame pipe[0m
viseron
14:42
[31m2026-07-03 12:42:19.277 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Did not receive a frame[0m
viseron
14:42
[31m2026-07-03 12:42:13.929 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Connection timed out[0m
viseron
14:41
[80D[1A[K[33m2026-07-03 12:41:14.179 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac., message repeated 2 times[0m
viseron
14:41
[33m2026-07-03 12:41:14.174 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac.[0m
viseron
14:41
[31m2026-07-03 12:41:14.172 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Restarting frame pipe[0m
viseron
14:41
[31m2026-07-03 12:41:09.167 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Did not receive a frame[0m
viseron
14:41
[31m2026-07-03 12:41:08.773 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Connection reset by peer[0m
viseron
14:40
[31m2026-07-03 12:40:50.839 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Connection timed out[0m
viseron
14:39
[31m2026-07-03 12:39:35.320 [ERROR ] [viseron.watchdog.subprocess_watchdog] - Process subprocess.viseron.components.storage.storage_subprocess.tier_check_worker.process has exited, restarting[0m
viseron
14:39
2026-07-03 12:39:21.568 UTC [1800] LOG: unexpected EOF on client connection with an open transaction
viseron
14:39
viseron.service: A process of this unit has been killed by the OOM killer.
systemd
14:37
[80D[1A[K[33m2026-07-03 12:37:41.397 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac., message repeated 2 times[0m
viseron
14:37
[33m2026-07-03 12:37:41.396 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac.[0m
viseron
14:37
[31m2026-07-03 12:37:41.334 [ERROR ] [viseron.watchdog.thread_watchdog] - Thread viseron.camera.camera_6.relay_frame is dead, restarting[0m
viseron
14:37
[31m2026-07-03 12:37:35.319 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Timeout waiting for frame[0m
viseron
14:36
[31m2026-07-03 12:36:24.740 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Connection timed out[0m
viseron
14:36
[31m2026-07-03 12:36:19.631 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Input/output error[0m
viseron
14:36
[31m2026-07-03 12:36:11.439 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Connection timed out[0m
viseron
14:33
[80D[1A[K[33m2026-07-03 12:33:58.780 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac., message repeated 2 times[0m
viseron
14:33
[33m2026-07-03 12:33:58.776 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac.[0m
viseron
14:33
[31m2026-07-03 12:33:58.771 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Restarting frame pipe[0m
viseron
14:33
[31m2026-07-03 12:33:53.766 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Did not receive a frame[0m
viseron
14:33
[31m2026-07-03 12:33:53.498 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Input/output error[0m
viseron
14:33
[31m2026-07-03 12:33:21.878 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Connection timed out[0m
viseron
14:30
[80D[1A[K[33m2026-07-03 12:30:54.237 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac., message repeated 2 times[0m
viseron
14:30
[33m2026-07-03 12:30:54.236 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac.[0m
viseron
14:30
[31m2026-07-03 12:30:54.236 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Restarting frame pipe[0m
viseron
14:30
[31m2026-07-03 12:30:49.231 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Frame reader process has exited[0m
viseron
14:30
[31m2026-07-03 12:30:49.228 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Invalid data found when processing input[0m
viseron
14:30
[80D[1A[K[33m2026-07-03 12:30:43.773 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac., message repeated 2 times[0m
viseron
14:30
[33m2026-07-03 12:30:43.765 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac.[0m
viseron
14:30
[31m2026-07-03 12:30:43.763 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Restarting frame pipe[0m
viseron
14:30
[31m2026-07-03 12:30:38.758 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Did not receive a frame[0m
viseron
14:30
[31m2026-07-03 12:30:38.550 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Connection reset by peer[0m
viseron
14:30
[31m2026-07-03 12:30:03.756 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Connection timed out[0m
viseron
14:28
[31m2026-07-03 12:28:54.194 [ERROR ] [viseron.domains.camera.fragmenter.subprocess.camera_6] - Failed to get extinf for 1783081730.m4s[0m
viseron
14:28
[80D[1A[K[33m2026-07-03 12:28:40.850 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac., message repeated 2 times[0m
viseron
14:28
[33m2026-07-03 12:28:40.850 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac.[0m
viseron
14:28
[31m2026-07-03 12:28:40.803 [ERROR ] [viseron.watchdog.thread_watchdog] - Thread viseron.camera.camera_6.relay_frame is dead, restarting[0m
viseron
14:28
[31m2026-07-03 12:28:35.319 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Timeout waiting for frame[0m
viseron
14:27
[31m2026-07-03 12:27:09.951 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Connection timed out[0m
viseron
14:25
[80D[1A[K[33m2026-07-03 12:25:10.947 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac., message repeated 2 times[0m
viseron
14:25
[33m2026-07-03 12:25:10.946 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac.[0m
viseron
14:25
[31m2026-07-03 12:25:10.891 [ERROR ] [viseron.watchdog.thread_watchdog] - Thread viseron.camera.camera_6.relay_frame is dead, restarting[0m
viseron
14:25
[31m2026-07-03 12:25:05.319 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Timeout waiting for frame[0m
viseron
14:23
[31m2026-07-03 12:23:35.720 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Connection timed out[0m
viseron
14:17
[80D[1A[K[33m2026-07-03 12:17:27.442 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac., message repeated 2 times[0m
viseron
14:17
[33m2026-07-03 12:17:27.441 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac.[0m
viseron
14:17
[31m2026-07-03 12:17:27.441 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Restarting frame pipe[0m
viseron
14:17
[31m2026-07-03 12:17:22.436 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Did not receive a frame[0m
viseron
14:16
[31m2026-07-03 12:16:17.488 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Connection timed out[0m
viseron
14:15
[80D[1A[K[33m2026-07-03 12:15:32.118 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac., message repeated 2 times[0m
viseron
14:15
[33m2026-07-03 12:15:32.117 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac.[0m
viseron
14:15
[31m2026-07-03 12:15:32.116 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Restarting frame pipe[0m
viseron
14:15
[31m2026-07-03 12:15:27.115 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Frame reader process has exited[0m
viseron
14:15
[31m2026-07-03 12:15:27.110 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Invalid data found when processing input[0m
viseron
14:15
[80D[1A[K[33m2026-07-03 12:15:18.475 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac., message repeated 2 times[0m
viseron
14:15
[33m2026-07-03 12:15:18.473 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac.[0m
viseron
14:15
[31m2026-07-03 12:15:18.470 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Restarting frame pipe[0m
viseron
14:15
[31m2026-07-03 12:15:13.465 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Did not receive a frame[0m
viseron
14:15
[31m2026-07-03 12:15:13.165 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Connection reset by peer[0m
viseron
14:14
[31m2026-07-03 12:14:34.566 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Connection timed out[0m
viseron
14:14
[31m2026-07-03 12:14:29.455 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - [NULL @ 0x562aa31c0940] missing picture in access unit with size 38148[0m
viseron
14:11
[32m2026-07-03 12:11:56.624 [INFO ] [viseron.components.ffmpeg.recorder.camera_7] - Stopping recorder[0m
viseron
14:11
[32m2026-07-03 12:11:56.233 [INFO ] [viseron.components.nvr.nvr.camera_7] - Stopping recording in: 0s[0m
viseron
14:11
[32m2026-07-03 12:11:55.280 [INFO ] [viseron.components.nvr.nvr.camera_7] - Stopping recording in: 1s[0m
viseron
14:11
[32m2026-07-03 12:11:54.215 [INFO ] [viseron.components.nvr.nvr.camera_7] - Stopping recording in: 2s[0m
viseron
14:11
[32m2026-07-03 12:11:53.296 [INFO ] [viseron.components.nvr.nvr.camera_7] - Stopping recording in: 3s[0m
viseron
14:11
[32m2026-07-03 12:11:52.253 [INFO ] [viseron.components.nvr.nvr.camera_7] - Stopping recording in: 4s[0m
viseron
14:11
[31m2026-07-03 12:11:51.496 [ERROR ] [viseron.components.codeprojectai.license_plate_recognition.camera_7] - Error calling CodeProject.AI: CodeProject.AI Server connection timeout. Current timeout is {timeout} seconds, try increasing this[0m
viseron
14:11
[32m2026-07-03 12:11:51.285 [INFO ] [viseron.components.nvr.nvr.camera_7] - Stopping recording in: 5s[0m
viseron
14:11
[32m2026-07-03 12:11:50.235 [INFO ] [viseron.components.nvr.nvr.camera_7] - Stopping recording in: 6s[0m
viseron
14:11
[32m2026-07-03 12:11:49.321 [INFO ] [viseron.components.nvr.nvr.camera_7] - Stopping recording in: 7s[0m
viseron
14:11
[32m2026-07-03 12:11:48.094 [INFO ] [viseron.components.nvr.nvr.camera_7] - Stopping recording in: 8s[0m
viseron
14:11
[32m2026-07-03 12:11:47.305 [INFO ] [viseron.components.nvr.nvr.camera_7] - Stopping recording in: 9s[0m
viseron
14:11
[32m2026-07-03 12:11:46.592 [INFO ] [viseron.components.nvr.nvr.camera_7] - Stopping recording in: 10s[0m
viseron
14:11
[80D[1A[K[33m2026-07-03 12:11:41.751 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac., message repeated 2 times[0m
viseron
14:11
[33m2026-07-03 12:11:41.750 [WARNING ] [viseron.components.ffmpeg.stream.camera_6] - Container mp4 does not support pcm_alaw audio codec. Audio will be transcoded as aac.[0m
viseron
14:11
[31m2026-07-03 12:11:41.749 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Restarting frame pipe[0m
viseron
14:11
[32m2026-07-03 12:11:41.488 [INFO ] [viseron.components.ffmpeg.recorder.camera_7] - Starting recorder[0m
viseron
14:11
[31m2026-07-03 12:11:36.747 [ERROR ] [viseron.components.ffmpeg.camera.camera_6] - Did not receive a frame[0m
viseron
14:11
[31m2026-07-03 12:11:13.524 [ERROR ] [viseron.components.ffmpeg.stream.camera_6] - rtsp://*****:*****@192.168.1.58:554/stream1: Connection timed out[0m

[roflcoopter]

Have you tried increasing the memory limits? Depending on the resolution of your cameras it might not be a memory leak, it simply needs that much memory to process large frames.

If you still experience OOM issues, we might be able to get a hint on what is using the memory by setting the environment variable VISERON_PROFILE_MEMORY=true when starting the container, and enable debug logging:

logger:
  default_level: debug

This will periodically print out the memory usage and what piece of code has allocated it

[MartinAirN]

@roflcoopter
With more RAM (16GB) it takes a bit longer for the OOM to appear, but they are there it seems.

The log is now about 100MB though, showing startup at about 08:47, to the first OOM at 12:54. And the final OOM errors followed by a reboot at 18:14.

Are there any snippets you need or if you want the whole file, do you have a place to send it to instead.

[MartinAirN]

I went through the log a bit, whilst I dont now if the memory profiler entries are per camera or not. The highest i could find is this:

jul 10 12:00:38 ParaServer viseron[2751386]: Top 5 lines
jul 10 12:00:38 ParaServer viseron[2751386]: #1: camera/shared_frames.py:113: 1273555.4 KiB
jul 10 12:00:38 ParaServer viseron[2751386]:     decoded_frame = cv2.cvtColor(
jul 10 12:00:38 ParaServer viseron[2751386]: #2: multiprocessing/queues.py:122: 1130884.3 KiB
jul 10 12:00:38 ParaServer viseron[2751386]:     return _ForkingPickler.loads(res)
jul 10 12:00:38 ParaServer viseron[2751386]: #3: yolo/object_detector.py:58: 50909.0 KiB
jul 10 12:00:38 ParaServer viseron[2751386]:     return np.array(frame)
jul 10 12:00:38 ParaServer viseron[2751386]: #4: motion_detector/__init__.py:442: 16972.2 KiB
jul 10 12:00:38 ParaServer viseron[2751386]:     decoded_frame = self._get_frame_function(shared_frame).copy()
jul 10 12:00:38 ParaServer viseron[2751386]: #5: nvr/nvr.py:878: 8750.1 KiB
jul 10 12:00:38 ParaServer viseron[2751386]:     ).copy(),
jul 10 12:00:38 ParaServer viseron[2751386]: 5651 other: 10033.2 KiB
jul 10 12:00:38 ParaServer viseron[2751386]: Total allocated size: 2491104.1 KiB

It however does not seem to correlate to the OOM moments.

The top ten usages:

2491104
1843431
1753101
1584193
1573217
1541777
1509075
1504852
1470342
1438344

As there are 5 cameras, if the logs are per camera, that would mean a max of 12.5 GB. But that has not, which realistically doesn't happen according to these logs. That combined with that it seems to happen more when a camera (or more) are not available leads me to be believe it might be a leak of processes trying to reconnect.