Skip to content

fix(http): consume Content-Length bodies exactly + honor Expect: 100-continue#47

Open
jrjones wants to merge 1 commit into
librespeed:masterfrom
rimrock-systems:upstream-pr/upload-body-read-and-expect
Open

fix(http): consume Content-Length bodies exactly + honor Expect: 100-continue#47
jrjones wants to merge 1 commit into
librespeed:masterfrom
rimrock-systems:upstream-pr/upload-body-read-and-expect

Conversation

@jrjones

@jrjones jrjones commented Jun 25, 2026

Copy link
Copy Markdown

Fixes #46

Summary

Two bugs in the HTTP request-body path (src/http/request.rs), both affecting uploads (POST /empty):

  1. Fixed-length bodies are read in rigid 1024-byte read_exact chunks, counting buffer.len() (always 1024) per iteration. A Content-Length that is not a multiple of 1024 therefore (a) over-reads the final chunk, consuming bytes of the next pipelined request, and (b) against a peer that keeps the connection open, blocks forever in read_exact waiting for the missing tail of the final chunk — hanging the upload until the client times out / disconnects. Clients pad upload payloads to a 1024 boundary to avoid this; the fix removes that need.
  2. Expect: 100-continue is never answered. Clients that use it (curl and many HTTP libraries) wait out their continue-timeout (~1s for curl) before sending the body on every upload.

Root cause

// BodyType::Fixed (before)
let mut buffer = [0; 1024];
let mut len: usize = 0;
loop {
    let bytes_read = buf_reader.read_exact(&mut buffer).await; // demands a FULL 1024 bytes
    match bytes_read {
        Ok(_) => { len += buffer.len();                        // always +1024, not actual
                   if len >= body_size as usize { break; } }
        Err(_) => break,
        // ...
    }
}

read_exact completes only once the whole 1024-byte buffer is filled, so the last partial chunk of a non-aligned body never completes against a still-open connection. And len += buffer.len() counts 1024 regardless of what was actually read, so the loop can stop 1–1023 bytes past the body. The Expect header is simply never inspected before the body read.

Fix

  • Read the body with read (not read_exact) into a 64 KiB buffer, decrementing by the actual bytes returned and stopping exactly at Content-Length. Safe for any length; the larger buffer also cuts per-1 KiB read-syscall overhead.
  • Before reading the body, if the request carries Expect: 100-continue, write the interim HTTP/1.1 100 Continue\r\n\r\n (RFC 9110 §10.1.1).

Validation

New regression tests (src/http/request.rs) that fail on master and pass with the fix:

test before after
non-1024-multiple body not over-read into a pipelined request FAIL — 1 response instead of 2 (the pipelined GET is swallowed) ok
Expect: 100-continue receives an interim 100 FAIL — no 100 Continue emitted ok

End-to-end (release build, vs unpatched master on a second port):

scenario unpatched patched
non-1024-multiple upload, client keeps socket open hangs — no response in 8s 200 OK, immediate
upload with Expect: 100-continue (curl default) ~1.07 s ~0.03 s
upload throughput (Expect disabled, 50 MB) ~8 Gbps ~14 Gbps
GET /empty ping · 100 MB garbage download unchanged

cargo build --release and cargo clippy are clean.

…continue

The fixed-length body reader looped `read_exact` into a `[0; 1024]` buffer and
added `buffer.len()` (always 1024) per iteration. A Content-Length that is not a
multiple of 1024 therefore:

  - over-read the final chunk, consuming bytes of the next pipelined request; and
  - against a peer that keeps the connection open, blocked forever inside
    `read_exact` waiting for the missing tail of the final 1024-byte chunk —
    hanging the upload until the client times out / disconnects.

Clients work around this by padding upload payloads up to a 1024 boundary; this
removes that need. Read with `read` and count the bytes actually returned,
stopping exactly at Content-Length. The larger 64 KiB buffer also cuts the
per-1 KiB read-syscall overhead.

Separately, the server never answered `Expect: 100-continue`, so clients that
send it (curl and many HTTP libraries) waited out their continue-timeout (~1s for
curl) before sending the body on every upload. Send the interim `100 Continue`
before reading the body, per RFC 9110 §10.1.1.

Adds regression tests for both: a non-1024-multiple body is not over-read into a
following pipelined request, and `Expect: 100-continue` receives an interim 100.
@jrjones

jrjones commented Jun 25, 2026

Copy link
Copy Markdown
Author

Have been running some private speedtest servers for my own client, ran into a couple relatively edge-casey issues in production that were easy behavior-preserving fixes under normal circumstances. LMK if questions/concerns. Thanks to the maintainers, we've gotten a lot of utility out of this repo. Once we are out of beta I will open at least one of our servers and add it to the list of public servers.)

--- JRJ

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

POST upload hangs on non-1024-multiple Content-Length; Expect: 100-continue not answered

1 participant