Skip to content

metadata-update.md

Latest commit

 

History

History
356 lines (289 loc) · 19.8 KB

metadata-update.md

File metadata and controls

356 lines (289 loc) · 19.8 KB
Raw

📖 Metadata Update & FFprobe (metadata_update)

This chapter covers the metadata_update block inside config.yml, which determines how aggressively or gently Tuliprox manages background tasks for resolving metadata and technical stream properties.

Tuliprox utilizes three distinct mechanisms to ensure perfect library quality (especially for Plex/Jellyfin compatibility):

  1. Resolve (Xtream API): Fetching missing VOD details (Cast, Director, Plot) directly from the provider's API.
  2. TMDB: Supplementing missing Release Years and high-resolution Covers/Backdrops via The Movie Database.
  3. Probe (FFprobe): Physically opening the stream to analyze the exact A/V codecs (HEVC, H264) and resolution.

Top-level entries

metadata_update:
  cache_path: metadata
  retry_delay: 2s
  worker_idle_timeout: 1m
  max_queue_size: 100000
  no_change_cache_ttl_secs: 3600
  probe_fairness_resolve_burst: 200
  log:
  resolve:
  probe:
  tmdb:
  ffprobe:

Global Options (Flat Keys)

Parameter Type Default Technical Impact & Background
cache_path String "metadata" Directory where TMDB cache files and metadata are stored. Relative paths are resolved against storage_dir. Used by all metadata resolution paths (Xtream VOD/Series, local library).
retry_delay Duration "2s" General minimum wait time when the worker encounters temporary runtime errors (e.g. socket timeout). Prevents very fast retry loops in case of transient network issues.
worker_idle_timeout Duration "1m" Time of inactivity (empty queue) after which the background worker kills itself to free RAM/CPU resources.
max_queue_size Int 100000 RAM Safety Limit: Maximum number of metadata tasks kept in memory per input simultaneously. New tasks are rejected once this limit is reached.
no_change_cache_ttl_secs Int 3600 How long (seconds) a "No Change" status is cached to avoid unnecessary DB checks across subsequent playlist updates. Identical reason sets are skipped while valid.
probe_fairness_resolve_burst Int 200 After 200 consecutive Resolve tasks (API), 1 Probe task (FFprobe) is forcibly prioritized so probes don't starve in large libraries.

Note on Durations: Duration fields support s, m, h, d or plain seconds (e.g. 30s, 10m, 1h, 7d). Exception: ffprobe.analyze_duration and ffprobe.live_analyze_duration require explicit unit suffix. Size fields support B, KB, MB, GB, TB or plain bytes.

1. Logging (log)

Controls background worker logging verbosity to monitor the metadata queue status.

metadata_update:
  log:
    queue_interval: 30s
    progress_interval: 15s
Parameter Type Default Description
queue_interval Duration "30s" Interval to log the current queue size and pending task status of the metadata worker.
progress_interval Duration "15s" Interval for progress reports while tasks are being processed (successful/failed resolves).

2. API Resolve Limits (resolve)

Controls API requests for pure metadata (Xtream Info / TMDB).

metadata_update:
  resolve:
    max_retry_backoff: 1h
    min_retry_base: 5s
    max_attempts: 3
    exhaustion_reset_gap: 1h
Parameter Type Default Description
max_retry_backoff Duration "1h" Upper limit for exponential wait time between repeated API failures.
min_retry_base Duration "5s" Initial wait time on the very first failure before exponential backoff kicks in.
max_attempts Int (u8) 3 Max attempts per cycle before a resolve task is marked as "exhausted" for the current run.
exhaustion_reset_gap Duration "1h" Time window after a cycle completes before "exhausted" states are reset for the next run.

3. FFprobe Retries & Limits (probe)

Controls technical stream probing retries via FFprobe.

metadata_update:
  probe:
    cooldown: 7d
    retry_load_retry_delay: 1m
    retry_backoff_step_1: 10m
    retry_backoff_step_2: 30m
    retry_backoff_step_3: 1h
    max_attempts: 3
    backoff_jitter_percent: 20
    user_priority: 127
Parameter Type Default Description
cooldown Duration "7d" Hard lock time (cooldown) after probe attempts are exhausted. The stream is ignored for this period to protect the provider.
retry_load_retry_delay Duration "1m" Wait time before re-attempting to load the internal metadata_retry_state.db after a load failure.
retry_backoff_step_1 Duration "10m" Wait time after the 1st FFprobe failure.
retry_backoff_step_2 Duration "30m" Wait time after the 2nd FFprobe failure.
retry_backoff_step_3 Duration "1h" Wait time from the 3rd FFprobe failure onwards.
max_attempts Int (u8) 3 Max failures to probe a stream before it enters global long-term cooldown.
backoff_jitter_percent Int (u8) 20 Random time deviation in percent (Jitter) so parallel retries don't hit the provider at the same second.
user_priority Int (i8) 127 Priority of the probe task on the Unix Nice-Scale. 127 is the absolute lowest. Probes at 127 are immediately cancelled/preempted if a real user needs the slot.

4. TMDB API Integration (tmdb)

Controls how Tuliprox interacts with The Movie Database API.

metadata_update:
  tmdb:
    enabled: false
    api_key: "YOUR_KEY"
    rate_limit_ms: 250
    cache_duration_days: 30
    language: en-US
    cooldown: 7d
    match_threshold: 86
Parameter Type Default Description
enabled Bool false Global master switch for TMDB resolution.
api_key String (Internal) Your own TMDB API Key. If omitted, Tuliprox uses a built-in default placeholder.
rate_limit_ms Int (u64) 250 Minimum wait between TMDB API calls to prevent IP rate-limiting or bans (milliseconds).
cache_duration_days Int (u32) 30 How long successful TMDB results are kept in the cache. Use 0 for permanent caching.
language String "en-US" Preferred metadata language (e.g., "de-DE") for plot and titles.
cooldown Duration "7d" Lock time for a movie if the TMDB search was successful but returned "no match" for the title.
match_threshold Int (u16) 86 Minimum similarity score (Jaro-Winkler) for a result to be accepted as a valid "Match".

5. FFprobe Process Rules (ffprobe)

Detailed settings for the technical stream analysis process.

metadata_update:
  ffprobe:
    enabled: true
    timeout: 60
    analyze_duration: 10s
    probe_size: 10MB
    live_analyze_duration: 5s
    live_probe_size: 5MB
Parameter Type Default Description
enabled Bool false Global master switch for ALL stream probing. Must be true for input flags like probe_vod to work.
timeout Int (u64) 60 Hard timeout (in seconds) for the OS FFprobe process. Prevents zombie processes.
analyze_duration Duration "10s" Passes -analyzeduration to FFprobe for VODs/Series. Warning: Requires unit suffix (s, m)!
probe_size Size "10MB" Passes -probesize to FFprobe for VODs/Series (Data Limit). Supports units like KB, MB, GB.
live_analyze_duration Duration "5s" Stricter time limit for Live-TV streams to minimize latency and provider traffic. Warning: Requires unit suffix!
live_probe_size Size "5MB" Stricter data limit for Live-TV streams.

Why are there 4 FFprobe fields? ffprobe.analyze_duration + ffprobe.probe_size are the default pair for non-live probes (VOD/Series). ffprobe.live_analyze_duration + ffprobe.live_probe_size are the live-specific pair for all live probes. This split is intentional because live probing usually needs lower values (less provider load / lower latency), while VOD/Series can use higher values for better metadata extraction quality.

Global Activation of Video Analysis & TMDB API Fallback

While metadata_update section in config.yml configures the global behavior, you must explicitly activate analysis per input.

Input Config (source.yml):

inputs:
  - name: my-provider
    type: xtream
    options:
      # Attempts to resolve missing TMDB IDs and Release Date via TMDB API based on title
      resolve_tmdb: true
      # Probes stream if video/audio info is missing in provider data
      probe_stream: true

Target Config (source.yml): If add_quality_to_filename is set for STRM output, the analyzed quality tags are used in filenames.

targets:
  - name: my-library
    output:
      - type: strm
        directory: /media/strm
        style: jellyfin
        # Adds tags like [2160p 4K HEVC HDR] to the filename
        add_quality_to_filename: true
        # Groups different versions of the same movie into one folder (based on TMDB ID)
        flat: true

Note on Probing: Probing respects the max_connections limit of your provider input. If no connection slot is available, the item is skipped and retried during the next update cycle.

Glossary & Data Types

To ensure precise configuration, the following terms and data formats are used throughout the metadata module:

Special Data Types (Strings)

  • Duration: Specifies time intervals. Supported units are s (seconds), m (minutes), h (hours), and d (days).
    • Example: 3600 (plain seconds) or 1h (suffixed).
    • Note: ffprobe.analyze_duration strictly requires a unit suffix.
  • Size: Specifies data volumes for stream analysis. Supported units are B, KB, MB, GB, and TB.
    • Example: 10485760 (plain bytes) or 10MB (suffixed).

Terminology

  • Resolve task: A metadata job that fetches or enriches item metadata (e.g., VOD/Series details, TMDB IDs, or release dates) via provider APIs or TMDB.
  • Probe task: A technical analysis job that physically inspects stream properties (codecs, resolution, audio tracks) using the FFprobe engine.
  • TMDB cooldown: A per-item lock set when a TMDB lookup completes successfully but returns "no match." This prevents redundant API calls for items not found on TMDB.
  • Attempt: A single execution try of a task. If it fails, the attempt counter for that specific item is incremented.
  • Retry: A subsequent re-attempt of a failed task after a waiting period.
  • Backoff delay: The waiting time before the next retry is allowed after a failure.
  • Exponential backoff: A strategy where the backoff delay grows with each failed attempt (e.g., 5s, 10s, 20s) up to a configured max_retry_backoff.
  • Jitter: A small random variation added to the backoff delay to prevent "thundering herd" issues, where hundreds of tasks retry at the exact same millisecond.
  • Transient error: A temporary failure (e.g., socket timeout, 502 Bad Gateway) that is likely to resolve itself in a future attempt.
  • Exhausted: The state reached when max_attempts for a task type are met. The task will not be retried within the current cycle.
  • Cooldown: A mandatory skip period applied after a task is Exhausted (primarily used for Probes). No retries are allowed until this period expires.
  • Update cycle: A full metadata processing run for a specific input, spanning from the first queued item to the point where the queue is idle.
  • Resolve exhaustion reset gap: The time window after a cycle completion after which "Exhausted" states for resolve tasks are cleared for the next run.
  • Pending queue: The in-memory list of metadata tasks currently waiting for an available background worker slot.
  • Worker idle timeout: The duration a metadata worker stays active without work before shutting down to release system resources.

Persistence: All retry, exhaustion, and cooldown states are persisted per input in the metadata_retry_state.db. This ensures that Tuliprox remembers the status of broken streams across server restarts.

 

Additional Information

Tuliprox is not just a proxy; it is a highly intelligent Playlist Processing Engine. A core part of this is the asynchronous update and metadata process that loads information from the provider, updates local databases, and fully automatically supplements missing metadata.

1. The Complete Processing Pipeline

When a playlist update starts (via Scheduler, API, or Boot), Tuliprox runs this pipeline:

  1. Download & Cache Check: For each configured input, it checks if provider data needs re-downloading (controlled by cache_duration).
  2. Input Storage (B+Tree): The raw data (M3U, Xtream categories) is written to a local, extremely fast B+Tree database (input_name.db). This drastically saves RAM.
  3. Target Processing: For each defined target (output playlist), data is loaded from the input and routed through the pipeline (processing_order, e.g., Filter ➔ Rename ➔ Map).
  4. Metadata Resolve & Probe: Tuliprox analyzes the filtered entries. If data is missing (e.g., TMDB IDs, Video Codecs), these are dispatched as "Jobs" to the MetadataUpdateManager.
  5. Target Storage & EPG: The finished playlist is written to the Target databases. Only then is XMLTV EPG data matched and assigned.

2. The MetadataUpdateManager (Architecture)

The MetadataUpdateManager is an asynchronous background engine (if resolve_background: true is set on the input) that prevents blocking the main playlist update.

Architecture & Logic

  • Per-Input Worker: A dedicated, isolated Tokio Task (Worker) is started for each Provider-Input. This prevents a slow provider from blocking another.
  • Task-Merging: If a stream requires both TMDB info and an FFprobe, they are merged into a single Task.
  • Rate-Limiting & Connection-Locks: The manager strictly respects the max_connections of your input. An FFprobe ( Stream Analysis) is only initiated if a provider connection is free. A probe task runs at the absolute lowest priority (user_priority: 127). If a real user starts streaming, the FFprobe process is immediately aborted/preempted to free the slot for the user!
  • Smart Retry & Cooldown: If a fetch or probe fails (e.g., HTTP 502), an exponential backoff with Jitter (random deviation) kicks in. If the max attempts (max_attempts) are reached, the task enters a global cooldown (e.g., 7 days) to stop harassing the provider.
  • Persistence (Retry State): The status of failed tasks is stored locally in metadata_retry_state.db. A server restart does not cause Tuliprox to immediately bombard the provider with requests for broken streams.
  • Cascading Updates: Once a worker collects a batch of metadata, it saves it in the Input DB and immediately cascades (inherits) the updates into all Target DBs, without requiring a full playlist rebuild.

3. Metadata Collection Mechanisms

How is a stream queued for analysis?

Tuliprox checks every stream for completeness (has_details()). A task is queued if the switches in inputs.options are active and one of these conditions is met:

  • Info-Resolve (VOD/Series): Missing provider info (Cast, Plot, Director) retrievable via Xtream API (get_vod_info / get_series_info).
  • TMDB/Date-Resolve: Missing tmdb_id or release_date.
  • Probing (FFprobe): * VOD/Series: Missing technical A/V parameters (video_codec, audio_codec, resolution).
    • Live-TV: The last_probed_timestamp is older than probe_live_interval_hours.

Collection Engines

  • Release Year / Date (PTT): Tuliprox uses a highly optimized internal parser (PTT - Parse Torrent Title). It locally analyzes the stream name and extracts the year (e.g., from "My Movie (2023)"). If this fails, it queries the TMDB API.
  • TMDB Information: Via TMDB API and a Jaro-Winkler distance comparison (similarity scoring), it fetches IDs, release years, covers, backdrops, genres, directors, and actors.
  • Video & Audio (FFprobe): Tuliprox briefly opens the stream via ffprobe. It extracts and normalizes:
    • Resolution: SD, 720p, 1080p, 1440p, 4K, 8K
    • Video: Codec (H264, HEVC, AV1), Bit-Depth (8bit, 10bit), Dynamic Range (HDR10, Dolby Vision, HLG)
    • Audio: Codec (AAC, AC3, EAC3, DTS, TrueHD) and Channels (2.0, 5.1, 7.1)
    • Tuliprox uses these tags later for the add_quality_to_filename target feature (e.g., My Movie [2160p 4K HEVC HDR].strm).
  • Seasons & Episodes: For series, the Xtream API delivers a structure of seasons and episodes. Tuliprox "flattens" these into individually playable streams (PlaylistItemType::Series). Each episode is treated individually during probing, as codecs and resolutions can change from episode to episode.