feat: BROS-353: Databricks storage integration (#8245) (#8542)

yyassi-heartex · robot-ci-heartex · makseq · CLupoW · commit a9122d019902 · 2025-10-04T23:32:17.000+02:00
Co-authored-by: robot-ci-heartex &lt;87703623+robot-ci-heartex@users.noreply.github.com&gt;
Co-authored-by: makseq &lt;makseq@gmail.com&gt;
Co-authored-by: robot-ci-heartex &lt;robot-ci-heartex@users.noreply.github.com&gt;
Co-authored-by: makseq &lt;makseq@users.noreply.github.com&gt;
diff --git a/docs/source/guide/storage.md b/docs/source/guide/storage.md
@@ -13,37 +13,13 @@ section: "Import & Export"
 
 Integrate popular cloud and external storage systems with Label Studio to collect new items uploaded to the buckets, containers, databases, or directories and return the annotation results so that you can use them in your machine learning pipelines.
 
-<div class="opensource-only">
-
-| Storage | Community | Enterprise |
-|---|---|---|
-| [Amazon S3](#Amazon-S3) | ✅ | ✅ |
-| [Amazon S3 with IAM role](https://docs.humansignal.com/guide/storage#Set-up-an-S3-connection-with-IAM-role-access) | ❌ | ✅ |
-| [Google Cloud Storage](#Google-Cloud-Storage) | ✅ | ✅ |
-| [Google Cloud Storage WIF Auth](https://docs.humansignal.com/guide/storage#Google-Cloud-Storage-with-Workload-Identity-Federation-WIF) | ❌ | ✅ |
-| [Microsoft Azure Blob Storage](#Microsoft-Azure-Blob-storage) | ✅ | ✅ |
-| [Microsoft Azure Blob Storage with Service Principal](https://docs.humansignal.com/guide/storage#Azure-Blob-Storage-with-Service-Principal-authentication) | ❌ | ✅ |
-| [Databricks Files (UC Volumes)](https://docs.humansignal.com/guide/storage#Databricks-Files-UC-Volumes) | ❌ | ✅ |
-| [Redis database](#Redis-database)| ✅ | ✅ |
-| [Local storage](#Local-storage) | ✅ | ✅ |
- 
-</div>
-
-<div class="enterprise-only">
-
-| Storage | Community | Enterprise |
-|---|---|---|
-| [Amazon S3](#Amazon-S3) | ✅ | ✅ |
-| [Amazon S3 with IAM role](#Set-up-an-S3-connection-with-IAM-role-access) | ❌ | ✅ |
-| [Google Cloud Storage](#Google-Cloud-Storage) | ✅ | ✅ |
-| [Google Cloud Storage WIF Auth](#Google-Cloud-Storage-with-Workload-Identity-Federation-WIF) | ❌ | ✅ |
-| [Microsoft Azure Blob Storage](#Microsoft-Azure-Blob-storage) | ✅ | ✅ |
-| [Microsoft Azure Blob Storage with Service Principal](#Azure-Blob-Storage-with-Service-Principal-authentication) | ❌ | ✅ |
-| [Databricks Files (UC Volumes)](#Databricks-Files-UC-Volumes) | ❌ | ✅ |
-| [Redis database](#Redis-database)| ✅ | ✅ |
-| [Local storage](#Local-storage) (on-prem only) | ✅ | ✅ |
-
-</div>
+Set up the following cloud and other storage systems with Label Studio:
+- [Amazon S3](#Amazon-S3)
+- [Google Cloud Storage](#Google-Cloud-Storage)
+- [Microsoft Azure Blob storage](#Microsoft-Azure-Blob-storage)
+- [Redis database](#Redis-database)
+- [Local storage](#Local-storage) <div class="enterprise-only">(for On-prem only)</div>
+- [Databricks Files (UC Volumes)](#Databricks-Files-UC-Volumes)
 
 
 ## Troubleshooting
@@ -1294,7 +1270,7 @@ Complete the following fields and then click **Test connection**:
 | | |
 | --- | --- |
 | Storage Title | Enter a name for the storage connection to appear in Label Studio. | 
-| Storage Name | Enter the name of your Azure storage account. |
+| Storage Name | Enter the name of your Azure storage sccount. |
 | Container Name | Enter the name of a container within the Azure storage account. |
 | Tenant ID | Specify the **Directory (tenant) ID** from your App Registration. |
 | Client ID | Specify the **Application (client) ID** from your App Registration. |
@@ -1516,92 +1492,60 @@ If you're using Label Studio in Docker, you need to mount the local directory th
 
 <div class="enterprise-only">
 
-Connect Label Studio Enterprise to Databricks Unity Catalog (UC) Volumes to import files as tasks and export annotations as JSON back to your volumes. This connector uses the Databricks Files API and operates only in proxy mode (presigned URLs are not supported by Databricks).
+Connect Label Studio Enterprise to Databricks Unity Catalog (UC) Volumes to import files as tasks and export annotations as JSON back to your volumes. This connector uses the Databricks Files API and operates only in proxy mode (no presigned URLs are supported by Databricks).
 
 ### Prerequisites
-- A Databricks workspace URL (Workspace Host), for example `https://adb-12345678901234.1.databricks.com` (or Azure domain).
-
-    See [Create a workspace](https://docs.databricks.com/aws/en/admin/workspace/) and [Get identifiers for workspace objects](https://docs.databricks.com/aws/en/workspace/workspace-details#workspace-url).
-- A Databricks Personal Access Token (PAT) with permission to access the Files API. 
-
-    You can generate tokens from **Settings > Developer**. See [Databricks personal access token authentication](https://docs.databricks.com/en/dev-tools/auth/pat.html). 
-- A UC Volume path under `/Volumes/<catalog>/<schema>/<volume>` with files you want to label. 
-  
-    See [What are Unity Catalog volumes?](https://docs.databricks.com/aws/en/volumes/).
-
-### Create a source storage connection in the Label Studio UI
-
-From Label Studio, open your project and select **Settings > Cloud Storage > Add Source Storage**.
-
-Select **Databricks Files (UC Volumes)** and click **Next**.
-
-#### Configure Connection
-
-Complete the following fields and then click **Test connection**:
-
-<div class="noheader rowheader">
-
-| | |
-| --- | --- |
-| Storage Title | Enter a name for the storage connection to appear in Label Studio. | 
-| Workspace Host | Enter your workspace URL, for example `https://<workspace-identifier>.cloud.databricks.com` |
-| Access Token | Enter your personal access token that you generated in Databricks. |
-| Catalog <br> Schema <br> Volume | Specify your volume path (UC coordinates). You can find this from the **Catalog Explorer** in Databricks (see screenshot below). |
-
-</div>
-
-![Screenshot of Databricks UI and LS UI](/images/storages/databricks-volume.png)
+- A Databricks workspace URL (Workspace Host), for example `https://adb-12345678901234.1.databricks.com` (or Azure domain)
+- A Databricks Personal Access Token (PAT) with permission to access the Files API
+- A UC Volume path under `/Volumes/<catalog>/<schema>/<volume>` with files you want to label
 
-#### Import Settings & Preview
+References:
+- Databricks workspace: https://docs.databricks.com/en/getting-started/index.html
+- Personal access tokens: https://docs.databricks.com/en/dev-tools/auth/pat.html
+- Unity Catalog and Volumes: https://docs.databricks.com/en/files/volumes.html
 
-Complete the following fields and then click **Load preview** to ensure you are syncing the correct data:
-
-<div class="noheader rowheader">
-
-| | |
-| --- | --- |
-| Bucket Prefix | Optionally, enter the directory name within the volume that you would like to use.  For example, `data-set-1` or `data-set-1/subfolder-2`.  | 
-| Import Method | Select whether you want create a task for each file in your container or whether you would like to use a JSON/JSONL/Parquet file to define the data for each task. |
-| File Name Filter | Specify a regular expression to filter bucket objects. Use `.*` to collect all objects. |
-| Scan all sub-folders | Enable this option to perform a recursive scan across subfolders within your container. |
-
-</div>
-
-#### Review & Confirm
-
-If everything looks correct, click **Save & Sync** to sync immediately, or click **Save** to save your settings and sync later.
+### Set up connection in the Label Studio UI
+1. Open Label Studio → project → **Settings > Cloud Storage**.
+2. Click **Add Source Storage**. Select **Databricks Files (UC Volumes)**.
+3. Configure the connection:
+   - Workspace Host: your Databricks workspace base URL (no trailing slash)
+   - Access Token: your PAT
+   - Catalog / Schema / Volume: Unity Catalog coordinates
+   - Click **Next** to open Import Settings & Preview
+4. Import Settings & Preview:
+   - Bucket Prefix (optional): relative subpath under the volume (e.g., `images/train`)
+   - File Name Filter (optional): regex to filter files (e.g., `.*\.json$`)
+   - Scan all sub-folders: enable for recursive listing; disable to list only current folder
+   - Click **Load preview** to verify files
+5. Click **Save** (or **Save & Sync**) to create the connection and sync tasks.
+
+### Target storage (export)
+1. Open **Settings > Cloud Storage** → **Add Target Storage** → **Databricks Files (UC Volumes)**.
+2. Use the same Workspace Host/Token and UC coordinates.
+3. Set an Export Prefix (e.g., `exports/${project_id}`).
+4. Click **Save** and then **Sync** to push annotations as JSON files to your volume.
 
 !!! note "URI schema"
-    To reference Databricks files directly in task JSON (without using source storage), use Label Studio’s Databricks URI scheme:
+    To reference Databricks files directly in task JSON (without using an Import Storage), use Label Studio’s Databricks URI scheme:
     
     `dbx://Volumes/<catalog>/<schema>/<volume>/<path>`
     
     Example:
     
-    `{ "image": "dbx://Volumes/main/default/dataset/images/1.jpg" }`
-    
+    ```
+    { "image": "dbx://Volumes/main/default/dataset/images/1.jpg" }
+    ```
 
 
 !!! note "Troubleshooting"
-    - If your file preview returns zero files, verify the path under `/Volumes/<catalog>/<schema>/<volume>/<prefix?>` and your PAT permissions.
+    - If listing returns zero files, verify the path under `/Volumes/<catalog>/<schema>/<volume>/<prefix?>` and your PAT permissions.
     - Ensure the Workspace Host has no trailing slash and matches your workspace domain.
-    - If previews work but media fails to load, confirm proxy mode is allowed for your organization in Label Studio (**Organization > Usage & License > Features**) and network egress allows Label Studio to reach Databricks.
+    - If previews work but media fails to load, confirm proxy mode is allowed for your organization in Label Studio and network egress allows Label Studio to reach Databricks.
 
 
 !!! warning "Proxy and security"
     This connector streams data **through the Label Studio backend** with HTTP Range support. Databricks does not support presigned URLs, so this option is also not available in Label Studio.
 
-### Create a target storage connection in the Label Studio UI
-
-Repeat the steps from the previous section but using **Add Target Storage**. Use the same workspace host, token, and volume path (UC coordinates). 
-
-For your **Bucket Prefix**, set an export folder to use (e.g., `exports/${project_id}`) and determine whether you want to allow files to be deleted from target storage. 
-
-When file deletion is enabled, if you delete an annotation in Label Studio (via UI or API), Label Studio will also delete the corresponding exported JSON file from your target storage for this storage connection. 
-
-Note that this only affects files that were exported by that target storage, not your source media or tasks. Your PAT permissions must also allow deletion.
-
-After adding, click **Sync** to export annotations as JSON files to your target volume.
 
 </div>
 
@@ -1615,11 +1559,7 @@ Databricks Unity Catalog (UC) Volumes integration is available in Label Studio E
 - Stream media securely via the platform proxy (no presigned URLs)
 - Export annotations back to your Databricks Volume as JSON
 
-Learn more and see the full setup guide in the Enterprise documentation: 
-
-[Databricks Files (UC Volumes)](https://docs.humansignal.com/guide/storage#Databricks-Files-UC-Volumes). 
-
-If your organization needs governed access to Databricks data with Unity Catalog, consider [Label Studio Enterprise](https://humansignal.com/).
+Learn more and see the full setup guide in the Enterprise documentation: [Databricks Files (UC Volumes)](https://docs.humansignal.com/guide/storage#Databricks-Files-UC-Volumes). If your organization needs governed access to Databricks data with Unity Catalog, consider [Label Studio Enterprise](https://humansignal.com/).
 
 </div>
 
