docs(xet): document shard/chunk cache and other Xet environment variables

[rajatarya]

What

Expands the Environment Variables section of Using Xet Storage with the Xet client's user-tunable variables that weren't yet documented. Three new subsections:

• General — HF_XET_HIGH_PERFORMANCE (alias HF_XET_HP), HF_XET_CACHE
• Shard Cache — HF_XET_SHARD_CACHE_SIZE_LIMIT, HF_XET_SHARD_CHUNK_INDEX_TABLE_MAX_SIZE, HF_XET_SHARD_CACHE_SUBDIR
• Chunk Cache — HF_XET_CHUNK_CACHE_SIZE_BYTES

Why

HF_XET_SHARD_CACHE_SIZE_LIMIT came up in a customer migration: with HF_XET_HIGH_PERFORMANCE=1, the on-disk shard cache can grow enough to fill the disk, and capping it below free space is the fix — but the variable wasn't documented here. This change also folds in the "other Xet environment variables" listed on the huggingface_hub reference page so the Hub page is self-contained.

Accuracy notes (verified against xet-core v1.5.1 and main)

• Shard cache soft limit is 16 GB (xet_runtime/src/config/groups/shard.rs), not 4 GB. The huggingface_hub env-vars page still says 4 GB — stale; worth a follow-up there.
• Chunk cache default is 0 (disabled): hf_xet's Cargo default enables no-default-cache. Per @seanses, the 10 GB build-feature default doesn't apply to Git Xet either — Git Xet v0 goes through the bridge, where the variable has no effect. Documented as a single default plus that caveat.
• Two variables dropped after review (@seanses): HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY (writes are now always buffered sequential writes pinned to one writer thread) and HF_XET_NUM_CONCURRENT_RANGE_GETS (superseded by adaptive concurrency). Neither exists in xet-core anymore; both still appear on the huggingface_hub env-vars page — stale, follow-up needed there too.

Scope

Limited to user-facing knobs. Internal/testing fields (CAS endpoint/scheme/prefix, session xorb-flush internals, dedup fragmentation estimators and global-dedup query controls, shard target sizes, progress-aggregation internals, USE_VECTORED_WRITE, HF_HUB_DISABLE_XET) are intentionally left out. Purely additive — no existing rows changed.

🤖 Generated with Claude Code

https://claude.ai/code/session_012XmgpfqHUeQtJthDXJddq9

---

Note

Low Risk
Documentation-only additions to the Hub Xet guide; no runtime, auth, or data-path code changes.

Overview
Expands the Environment Variables section in using-xet-storage.md so Hub readers can tune Xet without cross-linking only to huggingface_hub.

Adds a General table for HF_XET_HIGH_PERFORMANCE / HF_XET_HP and HF_XET_CACHE, plus new Shard Cache and Chunk Cache sections. Shard docs call out the 16 GB soft on-disk limit for HF_XET_SHARD_CACHE_SIZE_LIMIT, pruning at run start, and when to lower the cap (e.g. under high-performance mode filling disk). Chunk cache docs note hf_xet defaults to disabled (0) and that HF_XET_CHUNK_CACHE_SIZE_BYTES does not apply to Git Xet v0.

Existing adaptive concurrency, network, buffer, and logging tables are unchanged.

^{Reviewed by Cursor Bugbot for commit cebcfd0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[seanses]

LGTM!