Skip to content

out_prometheus_exporter ignores HTTP server worker and buffer options #12110

Description

@stondo

Bug Report

Describe the bug

out_prometheus_exporter parses the generic HTTP server configuration, but
prom_http_server_create() does not pass buffer_chunk_size or workers to
flb_http_server_init_with_options().

Consequently, http_server.workers is accepted without creating the requested worker
runtime. Under a large metrics workload, synchronous metrics decoding and Prometheus
text encoding in cb_prom_flush() can occupy the caller event loop and delay HTTP
response handling.

The omission is present in Fluent Bit 5.0.2, current 5.0, and current master.

To Reproduce

  1. Configure the exporter with explicit HTTP options:
service:
  flush: 1

pipeline:
  inputs:
    - name: fluentbit_metrics
      scrape_interval: 1

  outputs:
    - name: prometheus_exporter
      match: "*"
      host: 127.0.0.1
      port: 2021
      http_server.buffer_chunk_size: 64K
      http_server.workers: 2
  1. Start Fluent Bit and inspect its threads.
  2. Observe that no HTTP worker runtime or listener worker threads are created.
  3. Apply the two missing assignments and repeat.
  4. Observe two HTTP worker threads and a non-null worker runtime.

A deterministic runtime test reproduces the defect without timing thresholds:

server.buffer_chunk_size == ins->http_server_config->buffer_chunk_size
server.workers == 2
server.runtime != NULL

All three checks fail as applicable before the fix and pass afterward.

Expected behavior

The parsed http_server.buffer_chunk_size and http_server.workers settings should
reach the embedded HTTP server.

Your Environment

  • Version used: 5.0.2 and master at fd5ea1f0d503
  • Operating systems: embedded Linux arm64 and Debian 12 x86_64 test container
  • Plugins: process_exporter_metrics, fluentbit_metrics,
    prometheus_exporter

Additional context

With approximately 1500 synthetic processes and a 2.47 MB metrics chunk, one
correlated reproduction showed:

request TTFB:             673.957 ms
cb_prom_flush total:      746.633 ms
cb_prom_flush decode:     740.716 ms
HTTP handler total:         8.083 ms
HTTP payload mutex wait:  ~0.002-0.003 ms

After applying the option-propagation fix and configuring two HTTP workers:

requests:             762
worst TTFB:            46 ms
concurrent flush:     ~954 ms

The fix does not reduce encoding CPU time. It makes the configured HTTP worker runtime
effective, allowing request handling to proceed independently.

A separate follow-up enhancement can default the exporter to one output worker so
cb_prom_flush() does not run on the main engine event loop even when HTTP workers are
not explicitly configured.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions