Skip to content

Add env variable to control aws identity cache timeout#1189

Closed
bbalser wants to merge 1 commit into
mainfrom
bbalser/aws-identity-timeout
Closed

Add env variable to control aws identity cache timeout#1189
bbalser wants to merge 1 commit into
mainfrom
bbalser/aws-identity-timeout

Conversation

@bbalser
Copy link
Copy Markdown
Collaborator

@bbalser bbalser commented May 8, 2026

Summary

  • Set a 30 s load_timeout on the AWS SDK identity cache in file_store::new_client, replacing the SDK default of 5 s. Allow override via the FILE_STORE_IDENTITY_LOAD_TIMEOUT_SECS env var.
  • Fixes sporadic mobile-packet-verifier backfill-session failures of the form aws error: get_object (...) → dispatch failure → identity resolver timed out after 5s.

Why

The S3 client is cached, but credentials inside it are loaded lazily by IdentityCache::lazy() and re-fetched when they expire (~hourly on EC2/ECS via IMDS). The SDK's default 5 s identity-load timeout is too tight: when IMDS / STS briefly stalls, the next in-flight get_object is failed.

The backfill CLI hits this far more than the daemon because it issues get_object back-to-back at high rate over long runs, virtually guaranteeing a refresh will land on an active request.

This restores the behavior of the previous load_timeout(30s) setting (commit 19b20186) that was removed in commit 73d5511d during an AWS SDK upgrade — that commit's message explicitly anticipated re-adding it "if we're still getting timeout failures." We are.

Changes

  • file_store/src/lib.rs
    • Add IdentityCache import and Duration to existing imports.
    • Add FILE_STORE_IDENTITY_LOAD_TIMEOUT_SECS env constant + identity_load_timeout() helper (default 30 s, falls back to default on parse failure).
    • In new_client, chain .identity_cache(IdentityCache::lazy().load_timeout(...).build()) onto aws_config::defaults(...).

Single call site, so every file_store consumer (price tracker, all verifiers, backfill CLIs, tests) inherits the longer timeout — no per-config plumbing.

@bbalser bbalser requested a review from michaeldjeffrey May 8, 2026 14:19
@bbalser bbalser marked this pull request as draft May 8, 2026 14:30
@bbalser bbalser closed this May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant