diff --git a/docs/hub/storage-buckets-integrations.md b/docs/hub/storage-buckets-integrations.md index b6c322ae2e..fd789c3d92 100644 --- a/docs/hub/storage-buckets-integrations.md +++ b/docs/hub/storage-buckets-integrations.md @@ -51,6 +51,10 @@ from datasets import load_dataset ds = load_dataset("buckets/username/my-bucket", data_files=["data.parquet"]) ``` +## Transformers + +The [`Trainer`](https://huggingface.co/docs/transformers/trainer_recipes) can push and resume training checkpoints directly to a bucket, so a run can resume on a fresh machine without keeping checkpoints in a Git repo. See the [Trainer checkpointing docs](https://huggingface.co/docs/transformers/trainer_recipes) for setup. + ## Filesystem operations For direct file operations, `huggingface_hub` exposes a pre-instantiated [filesystem object](/docs/huggingface_hub/guides/hf_file_system), `hffs`: diff --git a/docs/hub/storage-buckets.md b/docs/hub/storage-buckets.md index d367d5305e..b1798e4f34 100644 --- a/docs/hub/storage-buckets.md +++ b/docs/hub/storage-buckets.md @@ -334,6 +334,9 @@ hf sync ./checkpoints hf://buckets/my-org/training-run-42/checkpoints Because buckets are built on [Xet](./xet/index), successive checkpoints where large parts of the model are frozen benefit from chunk-level deduplication. Only the changed chunks are uploaded. +> [!TIP] +> Transformers' [`Trainer`](https://huggingface.co/docs/transformers/trainer_recipes) can push and resume training checkpoints directly to a bucket — no manual `sync` step needed. See the Trainer docs for setup. + ### Data processing pipelines Buckets serve as staging areas for data processing workflows. Process raw data, write intermediate outputs to a bucket, then promote the final artifact to a versioned [Dataset](./datasets) repository when the pipeline completes. This keeps your versioned repo clean while giving your pipeline fast mutable storage.