Skip to content
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/hub/storage-buckets-integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,10 @@ from datasets import load_dataset
ds = load_dataset("buckets/username/my-bucket", data_files=["data.parquet"])
```

## 🤗 Transformers
Comment thread
davanstrien marked this conversation as resolved.
Outdated

The [`Trainer`](https://huggingface.co/docs/transformers/trainer_recipes) can push and resume training checkpoints directly to a bucket, so a run can resume on a fresh machine without keeping checkpoints in a Git repo. See the [Trainer checkpointing docs](https://huggingface.co/docs/transformers/trainer_recipes) for setup.

## Filesystem operations

For direct file operations, `huggingface_hub` exposes a pre-instantiated [filesystem object](/docs/huggingface_hub/guides/hf_file_system), `hffs`:
Expand Down
3 changes: 3 additions & 0 deletions docs/hub/storage-buckets.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,9 @@ hf sync ./checkpoints hf://buckets/my-org/training-run-42/checkpoints

Because buckets are built on [Xet](./xet/index), successive checkpoints where large parts of the model are frozen benefit from chunk-level deduplication. Only the changed chunks are uploaded.

> [!TIP]
> 🤗 Transformers' [`Trainer`](https://huggingface.co/docs/transformers/trainer_recipes) can push and resume training checkpoints directly to a bucket — no manual `sync` step needed. See the Trainer docs for setup.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice 👍

Comment thread
davanstrien marked this conversation as resolved.
Outdated

### Data processing pipelines

Buckets serve as staging areas for data processing workflows. Process raw data, write intermediate outputs to a bucket, then promote the final artifact to a versioned [Dataset](./datasets) repository when the pipeline completes. This keeps your versioned repo clean while giving your pipeline fast mutable storage.
Expand Down
Loading