Conversation
Implement a REST API that wraps DSpace items as HuggingFace-compatible model repositories, allowing HuggingFace client tools (transformers.from_pretrained, huggingface_hub snapshot_download, hf CLI) to consume models stored in DSpace by setting HF_ENDPOINT environment variable. Implementation: - New @RestController at /api/hf with routes: * GET /api/models/{handle} - model metadata (sha, files, tags) * GET /api/models/{handle}/revision/{revision} - revision-specific metadata * GET /api/models/{handle}/tree/{revision} - file listing * GET|HEAD /{handle}/resolve/{revision}/{filename} - file download with Range/206 support * GET /api/models?search=&limit= - model listing with filtering * GET /api/whoami-v2 - authentication status (401 phase-1) - Exposure filter: items must match metadata-field filter (default dc.type=machineLearningModel) or owning collection must be in hf.api.exposed.collections; fail-closed by default - SHA computation: deterministic 40-hex SHA-1 from item UUID + last-modified + bitstream MD5s, enables client-side caching and resumable downloads - File download semantics: via BitstreamResource machinery (Range/206/304/412 support), UsageEvent/Matomo tracking, S3 direct-download redirect when enabled - Authorization: existing CLARIN license-gating and READ policies apply; anonymous access only for non-gated bitstreams (phase 1) Configuration (default disabled): - hf.api.enabled (boolean, default false) - hf.api.filter.metadata-field (default dc.type) - hf.api.filter.value (default machineLearningModel) - hf.api.exposed.collections (handle list) Validation: - 14 unit tests (service logic, sha determinism, revision matching) - 18 integration tests (endpoints, auth, error codes, Range requests, encoded filenames) - Full unit suite: 1,538 tests, 0 failures - Full IT suite: 69 tests, 0 failures - Live E2E: real huggingface_hub client (snapshot_download, cache semantics verified)
Member
Author
Live E2E validation scriptThis script was used to validate the facade against the real Run this after deploying the feature to a test instance with the facade enabled ( import os, hashlib, pathlib
from huggingface_hub import model_info, hf_hub_download, snapshot_download
os.environ["HF_ENDPOINT"] = "https://<your-host>/server/api/hf"
os.environ["HF_HUB_DISABLE_TELEMETRY"] = "1"
os.environ["HF_HUB_CACHE"] = "/tmp/hf_e2e_cache"
REPO = "<handle-prefix>/<handle-suffix>"
EXPECTED = {"file1.bin": "<md5-hex>", "file2.json": "<md5-hex>"}
# 1. Probe metadata
info = model_info(REPO)
assert info.sha and len(info.sha) == 40, f"Invalid sha: {info.sha}"
assert {s.rfilename for s in info.siblings} == set(EXPECTED), f"Files mismatch: {[s.rfilename for s in info.siblings]}"
# 2. Download single file
hf_hub_download(repo_id=REPO, filename=list(EXPECTED.keys())[0])
# 3. Download all files (snapshot)
cache_dir = snapshot_download(repo_id=REPO)
# 4. Verify MD5s match
for name, expected_md5 in EXPECTED.items():
actual_md5 = hashlib.md5((pathlib.Path(cache_dir) / name).read_bytes()).hexdigest()
assert actual_md5 == expected_md5, f"{name}: expected {expected_md5}, got {actual_md5}"
# 5. Second download = cache hit (proves sha/ETag semantics work)
second_dir = snapshot_download(repo_id=REPO)
assert second_dir == cache_dir, "Cache miss on second call (ETag/SHA semantics broken)"
print("✅ E2E OK", cache_dir)Test environment notes:
What gets validated:
|
There was a problem hiding this comment.
Pull request overview
This PR introduces a HuggingFace Hub–compatible, read-only REST API facade under /api/hf which maps DSpace Items (and their ORIGINAL bitstreams) to “model repositories” consumable by HuggingFace client tooling via HF_ENDPOINT. The implementation is backed by a dedicated service layer and is controlled via new hf.api.* configuration keys.
Changes:
- Added a new
ClarinHuggingFaceControllerproviding HuggingFace-shaped endpoints for model info, tree listing, file resolve/download (with Range support), and awhoami-v2placeholder. - Added
ClarinHuggingFaceServiceplus DTOs (HuggingFaceModelInfo,HuggingFaceTreeEntry,HuggingFaceSibling) to compute deterministic repository “sha” and transform Items/bitstreams into HuggingFace response shapes. - Added configuration defaults (
hf.api.enabled, exposure filters / allow-list) and new unit + integration tests covering sha/revision logic and endpoint behavior.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| dspace/config/clarin-dspace.cfg | Adds hf.api.* configuration keys to enable/disable and control exposure filtering for the HuggingFace facade. |
| dspace-server-webapp/src/main/java/org/dspace/app/rest/hf/ClarinHuggingFaceController.java | Implements the /api/hf HuggingFace-compatible REST endpoints including resolve/download behavior. |
| dspace-server-webapp/src/main/java/org/dspace/app/rest/hf/ClarinHuggingFaceService.java | Implements exposure checks, item discovery/listing, sha computation, and DTO mapping from DSpace Items/bitstreams. |
| dspace-server-webapp/src/main/java/org/dspace/app/rest/hf/HuggingFaceModelInfo.java | DTO representing HuggingFace model_info JSON shape. |
| dspace-server-webapp/src/main/java/org/dspace/app/rest/hf/HuggingFaceSibling.java | DTO for siblings entries in model_info. |
| dspace-server-webapp/src/main/java/org/dspace/app/rest/hf/HuggingFaceTreeEntry.java | DTO representing HuggingFace tree entries. |
| dspace-server-webapp/src/test/java/org/dspace/app/rest/hf/ClarinHuggingFaceServiceTest.java | Unit tests for sha determinism and revision validation helpers. |
| dspace-server-webapp/src/test/java/org/dspace/app/rest/ClarinHuggingFaceControllerIT.java | Integration tests validating endpoint shapes, exposure filtering, revision handling, resolve semantics, Range requests, and auth behavior. |
Comment on lines
+230
to
+233
| for (Item item : huggingFaceService.findExposedItems(context, search, limit)) { | ||
| String sha = huggingFaceService.computeShaForItem(context, item); | ||
| models.add(huggingFaceService.toModelInfo(context, item, sha)); | ||
| } |
Member
Author
There was a problem hiding this comment.
Addressed in 863a197: ClarinHuggingFaceService#findExposedItems() now skips items the current context user cannot READ (authorizeActionBoolean), applied before the limit cap so unauthorized items never count against pagination. Added ClarinHuggingFaceControllerIT#listModelsExcludesItemsWithoutReadAccess covering anonymous vs. admin visibility of a restricted-but-exposed item.
Copilot review on PR #1381 flagged that listModels() returned metadata for every exposed item without checking READ on the Item, letting anonymous users see the existence/title/tags of embargoed or private items via the facade's model listing. ClarinHuggingFaceService#findExposedItems() now skips any candidate the current context user cannot READ (authorizeActionBoolean), applied before the result is capped at the requested limit so unauthorized items never count against pagination. This mirrors the READ check already done by resolveAuthorizedRepository()/resolveFile() for the single-model and resolve/download endpoints, keeping all HF facade read paths consistent. Adds ClarinHuggingFaceControllerIT#listModelsExcludesItemsWithoutReadAccess covering the anonymous vs. admin visibility of a restricted-but-exposed item.
Comment on lines
+230
to
+237
| Context context = ContextUtil.obtainContext(request); | ||
| List<HuggingFaceModelInfo> models = new ArrayList<>(); | ||
| for (Item item : huggingFaceService.findExposedItems(context, search, limit)) { | ||
| String sha = huggingFaceService.computeShaForItem(context, item); | ||
| models.add(huggingFaceService.toModelInfo(context, item, sha)); | ||
| } | ||
| context.complete(); | ||
| return ResponseEntity.ok(models); |
Comment on lines
+234
to
+238
| Iterator<Item> items = itemService.findByCollection(context, (Collection) dso); | ||
| while (items.hasNext()) { | ||
| Item item = items.next(); | ||
| candidates.putIfAbsent(item.getID(), item); | ||
| } |
| List<Bundle> bundles = item.getBundles(Constants.CONTENT_BUNDLE_NAME); | ||
| for (Bundle bundle : bundles) { | ||
| for (Bitstream bitstream : bundle.getBitstreams()) { | ||
| result.putIfAbsent(bitstream.getName(), bitstream); |
Comment on lines
+418
to
+422
| List<HuggingFaceSibling> siblings = new ArrayList<>(); | ||
| for (String filename : getOriginalFiles(item).keySet()) { | ||
| siblings.add(new HuggingFaceSibling(filename)); | ||
| } | ||
| modelInfo.setSiblings(siblings); |
Comment on lines
+440
to
+451
| public List<HuggingFaceTreeEntry> toTreeEntries(Context context, Item item) { | ||
| List<HuggingFaceTreeEntry> entries = new ArrayList<>(); | ||
| for (Map.Entry<String, Bitstream> entry : getOriginalFiles(item).entrySet()) { | ||
| HuggingFaceTreeEntry treeEntry = new HuggingFaceTreeEntry(); | ||
| treeEntry.setType("file"); | ||
| treeEntry.setPath(entry.getKey()); | ||
| treeEntry.setSize(entry.getValue().getSizeBytes()); | ||
| treeEntry.setOid(entry.getValue().getChecksum()); | ||
| entries.add(treeEntry); | ||
| } | ||
| return entries; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement a REST API that wraps DSpace items as HuggingFace-compatible model repositories, allowing HuggingFace client tools (transformers.from_pretrained, huggingface_hub snapshot_download, hf CLI) to consume models stored in DSpace by setting HF_ENDPOINT environment variable.
Implementation:
New
@RestControllerat /api/hf with routes:Exposure filter: items must match metadata-field filter (default dc.type=machineLearningModel) or owning collection must be in hf.api.exposed.collections; fail-closed by default
SHA computation: deterministic 40-hex SHA-1 from item UUID + last-modified + bitstream MD5s, enables client-side caching and resumable downloads
File download semantics: via BitstreamResource machinery (Range/206/304/412 support), UsageEvent/Matomo tracking, S3 direct-download redirect when enabled
Authorization: existing CLARIN license-gating and READ policies apply; anonymous access only for non-gated bitstreams (phase 1)
Configuration (default disabled):
Validation:
Problem description
Analysis
(Write here, if there is needed describe some specific problem. Erase it, when it is not needed.)
Problems
(Write here, if some unexpected problems occur during solving issues. Erase it, when it is not needed.)
Manual Testing (if applicable)
Copilot review
wiki
https://github.com/ufal/clarin-dspace/wiki/HuggingFace-compatible-API