in_tail & out_s3: Support for continuously growing files with strict ordering, sequential delivery, and atomic completeness

[arunkvelu]

Is your feature request related to a problem? Please describe.

It would be helpful if the S3 output plugin could better support continuously growing (append-only) files when uploading to object storage.

One common use case is collecting continuously growing log files. Depending on the application and log rotation policy, a log file may remain open and grow for an extended period, or new log files may be created periodically as existing files reach a configured size or age. In either case, log data is generated continuously, and the S3 output plugin may trigger uploads due to upload_timeout or total_file_size before log generation has completed. Periodic uploads allow the latest logs to be available in object storage without waiting for the application or logging process to finish.

Background

In our deployment, Fluent Bit tails event log files and uploads them to an S3 compatible object store.

The log collection flow is:

• Tail input plugin monitors the Spark event log and forwards newly appended data to the Fluent Bit engine.
• The engine buffers the records.
• The S3 output plugin uploads buffered data when either upload_timeout or total_file_size is reached.

When successive uploads target the same object key (for example, using static_file_path On with a constant object key), each upload replaces the previously stored object with only the data contained in the current upload.

As a result, earlier portions of the log are no longer present in the destination object.

Example

At time T1, the event log contains:

event1
event2
event3

upload_timeout expires, and Fluent Bit uploads:

event1
event2
event3

The application continues running and appends additional events:

event1
event2
event3
event4
event5

At the next upload cycle, Fluent Bit uploads only the newly buffered records:

event4
event5

Since the destination object key is unchanged, the object stored in object storage now contains only:

event4
event5

The previously uploaded events are no longer part of the object. Changing the object key breaks the application that reads the uploaded content.

Use case
This behavior is particularly relevant for append-only log files that remain open for long periods, such as:

• Spark event logs
• Database audit logs
• Long-running application logs
• Other continuously growing log files

For these workloads, users may want both:

• Near real-time availability of logs in object storage.
• A single object that always represents the complete log collected so far.

Current workaround
At the moment, the available options appear to be:

• Configure a large upload_timeout so uploads occur only after the application has mostly finished, reducing the chance of partial objects but delaying log availability.
• Configure a small upload_timeout to make logs available sooner, but the destination object only reflects the most recently uploaded portion of the log.

Both approaches involve trade-offs depending on the workload.

Describe the solution you'd like
Would it be possible to consider a mode that better supports continuously growing files?

Some possible approaches could include:

• Re-uploading the complete file contents when a static object path is used.
• Providing an optional mode that maintains a complete object for append-only source files.
• Supporting backend-specific append semantics where they are available.

One approach I considered:

1. Tail plugin reading whole file when a certain flag is set (Flag is used to avoid changing default behaviour)
2. S3 plugin receives whole file after clearing existing buffer

This way, the whole file is written to target storage every time. Trade-off: A higher CPU / IO usage from fluent bit container / process.

These are only suggestions - the maintainers may have better approaches that fit the existing architecture.

Describe alternatives you've considered

Additional context