Skip to content

feat(proxy): add opt-in Fetch Relay transport - #975

Open
w0fv1 wants to merge 3 commits into
tashfeenahmed:mainfrom
w0fv1:feat/fetch-relay-transport
Open

feat(proxy): add opt-in Fetch Relay transport#975
w0fv1 wants to merge 3 commits into
tashfeenahmed:mainfrom
w0fv1:feat/fetch-relay-transport

Conversation

@w0fv1

@w0fv1 w0fv1 commented Aug 22, 2026

Copy link
Copy Markdown

Closes #974

Problem

Standard proxy transports cannot use URL-based Fetch Relays such as Cloudflare Workers. FreeLLMAPI currently interprets every HTTPS proxy URL as a CONNECT-capable forward proxy.

Summary

This PR adds an opt-in global fetch-relay transport beneath the existing proxyFetch() abstraction:

FreeLLMAPI -> user-controlled Fetch Relay -> LLM provider
  • adds the shared forward | fetch-relay mode, settings API persistence, PROXY_MODE, Dashboard selector, and draft Test support
  • sends the provider URL in X-FreeLLMAPI-Target-URL; supports an encoded {url} template for existing URL-query relays
  • preserves method, provider headers, body, AbortSignal, status, headers, and response stream
  • forces manual redirect handling so a relay redirect cannot silently become a direct provider request
  • includes protocol documentation and a standalone, deployable Cloudflare Worker with Wrangler configuration
  • redacts proxy/relay paths, queries, and credentials from logs

Default behavior is unchanged

  • no proxyMode -> forward
  • legacy PROXY_URL -> forward unless PROXY_MODE=fetch-relay is explicit
  • ALL_PROXY, HTTPS_PROXY, HTTP_PROXY, NO_PROXY, HTTP/HTTPS/SOCKS, and per-key proxies keep their existing behavior
  • disabled, per-platform bypass, and local-destination bypass still connect directly
  • Fetch Relay is global-only in this first version

Relay protocol

Preferred form:

POST https://relay.example.workers.dev/a-long-random-secret
X-FreeLLMAPI-Target-URL: https://api.provider.example/v1/chat/completions
Authorization: Bearer provider-key
Content-Type: application/json

Compatibility form:

https://relay.example.workers.dev/secret?url={url}

The header form keeps provider query parameters out of edge URL logs. Relay responses are returned as the original Response, without calling text(), json(), or arrayBuffer().

Security

  • Fetch Relay URLs are limited to HTTP/HTTPS
  • relay paths, queries, and userinfo are excluded from logs
  • the relay request uses redirect: manual
  • the reference Worker requires a secret path and exact HTTPS upstream-host allowlisting
  • relay-only and hop-by-hop headers are removed before the provider request
  • the reference Worker does not follow redirects or buffer response streams
  • documentation states that the relay operator can see provider credentials and request content

Tests

Added 16 regression tests, including a real local path through a fake relay and mock upstream.

  • JSON POST, Authorization, headers, and body preservation
  • SSE first chunk arrives before the upstream stream ends
  • AbortSignal propagation
  • disabled and per-platform bypass remain direct
  • {url} encoding and Header-first dispatch
  • redirect remains manual
  • settings validation, persistence, startup restore, env precedence, and draft Test
  • relay secret paths and queries never appear in logs
  • existing proxy regression suite
  • server suite: 227 files, 2,576 passed, 8 skipped
  • client suite: 182 passed; i18n validation passed for 60 locales
  • npm run build
  • Wrangler 4.125.0 production bundle dry-run

Live Cloudflare smoke test

The exact example Worker in this PR was temporarily deployed together with a separate mock-upstream Worker and exercised through FreeLLMAPI's built proxyFetch():

FreeLLMAPI proxyFetch() -> live Relay Worker -> live mock upstream Worker
  • JSON POST returned 200 with the exact body, content type, and provider Authorization
  • SSE returned data: first after 115 ms and data: second after 1,614 ms (the upstream deliberately waited 1,500 ms), proving incremental delivery
  • upstream 302 remained 302 instead of bypassing the Relay
  • an incorrect secret path returned 404
  • a non-allowlisted target returned 403

Both temporary Workers were deleted after the smoke test. The test also caught the need for Cloudflare's global_fetch_strictly_public flag when an upstream is itself hosted on Workers; the checked-in Wrangler configuration now includes it.

Local Windows-only test limitations, unrelated to this diff:

  • root npm test stops in the bootstrap fixture because its spawned Windows PowerShell does not expose Get-FileHash
  • CLI Linux-path golden tests render backslashes on a Windows host (78 pass, 17 path/snapshot failures)

This lowers the infrastructure barrier for developers, educators, nonprofits, and researchers who cannot operate an additional proxy server.

@w0fv1

w0fv1 commented Aug 22, 2026

Copy link
Copy Markdown
Author

Added a deployable Cloudflare Worker example in 8dfe1b4 and completed a live edge smoke test using the built FreeLLMAPI proxyFetch(): JSON POST/Authorization/body round-tripped exactly; SSE chunks arrived at 115 ms and 1,614 ms around a deliberate 1,500 ms upstream delay; 302 remained manual; wrong secret path returned 404; disallowed host returned 403. The test caught and fixed the Workers-to-Workers routing edge case via global_fetch_strictly_public. Both temporary Workers were deleted afterward. Full evidence is now in the PR description.

@w0fv1

w0fv1 commented Aug 22, 2026

Copy link
Copy Markdown
Author

Follow-up verification with a real provider and the user-facing Dashboard is complete (commit 5c8284a).

End-to-end path

Browser Playground
  -> FreeLLMAPI /v1/chat/completions
  -> live Cloudflare Fetch Relay
  -> Groq Chat Completions
  -> openai/gpt-oss-20b
  • Dashboard draft Relay test succeeded against api.groq.com in 580 ms.
  • The saved fetch-relay mode was then used by a real streaming Playground conversation.
  • Groq returned the requested response successfully; the Playground identified groq · openai/gpt-oss-20b and reported 1,451 ms end-to-end.
  • The request used a real Groq credential stored in FreeLLMAPI. No credential was placed in the Worker, repository, PR text, or custom logs.
  • The deployed Worker used an exact api.groq.com allowlist and a Cloudflare secret binding for RELAY_PATH.

Production observability

The reference Worker now emits structured events for request acceptance, upstream headers, stream completion, policy rejection, configuration errors, and upstream failures. Fields are limited to request ID, method, target hostname, colo/country, status, content type, duration, byte count, and fixed error categories. Workers Logs is enabled in wrangler.jsonc; the README documents access control and sampling guidance.

The Relay secret is compared using SHA-256 plus crypto.subtle.timingSafeEqual(). Responses still pass through a TransformStream, so completion metrics are collected without buffering SSE or other response bodies.

Additional checks after this change:

  • Wrangler 4.125.0 production dry-run passed (5.19 KiB upload / 1.78 KiB gzip).
  • The live deployment accepted the Dashboard probe and Playground stream using the checked-in Worker source.
  • Existing automated Fetch Relay coverage remains 21/21 passing, and the repository build passed before this follow-up.

@w0fv1

w0fv1 commented Aug 22, 2026

Copy link
Copy Markdown
Author

Reference implementation

These are commit-pinned links to the exact Worker source used for the live provider and Playground verification above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: add Fetch Relay as an outbound transport

1 participant