Feature: Add DockerImage classes for docker.io, nvcr.io, and quay.io by georgiastuart · Pull Request #697 · singularityhub/singularity-hpc

georgiastuart · 2026-05-10T15:14:34Z

This is intended to resolve singularityhub/shpc-registry#480 and singularityhub/shpc-registry#481 . The driving motivations are rate limiting from dockerhub and odd output from NGC containers at times. The updated registry files in singularityhub/shpc-registry#483 use this singularity-hpc PR. This is a moderately sized PR, so I'm anticipating a couple rounds of iteration after the PR tests run.

Subclass Design

DockerHubImage, QuayDockerImage and NGCImage Subclass DockerImage and override the tags and digest methods. It's possible there's a more elegant way to simplify, but all these APIs have bespoke needs so I didn't try to unify further. All query metadata APIs rather than use crane.

NCG Compatibility

Querying the NCG API requires an API token. These are free to get for this purpose, but I kept ngcsdk as an optional dependency. If ngcsdk and the environment variable SHPC_NGC_API_KEY are present, shpc will use ngcsdk to update. Otherwise, it will fall back to crane and the normal DockerImage class.

Added CLI arguments for `update`

While cleaning up some mangled container.yaml files, I added two CLI arguments to update:

--max-tags=N: Overrides the default 5 tags to add
--purge: wipes the existing tags in the container.yaml file to start fresh.

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

georgiastuart · 2026-05-10T15:25:01Z

Shockingly, the tests passed the first time. Ready for review @vsoch !

vsoch

This looks great! A few comments, and then we should have a test for the registry classes. We will also want a strategy to clean up the current registry repository.

vsoch · 2026-05-10T15:36:12Z

+
+            if len(tags) == 0:
+                logger.error(
+                    f"The tag {tag} you provided is not known. Check that it and the container both exist."


Suggested for here would be to raise the ValueError with the message currently given to logger.error.

Refactored error handling a bit, but it still seems messy. Going to take a fresh look at it tomorrow!

vsoch · 2026-05-10T15:37:06Z

+                )
+                raise ValueError
+            new_tags = [x for x in tags if x.get("name")]
+            tags.extend(new_tags)


Sanity check here we will not have duplicates in this list. And what about order? I think different registries may do alphabetical vs. by date. If the user asks for some specific number of tags, we likely want to give them latest.

Question about this - I mainly took inspiration from the previous tag_quay method where it pulls all the tags and feeds it into the filtering, value extraction, and sorting functions in the update main function. DockerHub does appear to reliably return tags in order of latest update. Given that this class can handle a rate limit, is it better to continue pulling all the tags, or to cap it at the first X tags? I'm guessing we could comfortably restrict it to 5 pages or so and still be able to get the N most recent tags for even tag-spammy projects.

We should likely be conservative and cap at some X tags, especially if you are hitting a limit with requests.

vsoch · 2026-05-10T15:38:33Z

+                break
+        return tag_responses
+
+    def tags(self):


You are going in the right direction by using DockerHubImage for NGC, and it looks like there is still some redundancy in these classes (e.g., the tags filter for the sbom and similar). The main difference tends to be in the query tag API function. Could we put more logic in the DockerHubImage base class that the others inherit from with the shared logic?

Refactored to minimize redundancy. All class specialization now occurs in _query_tag_api. I marked the manifest and config methods to raise NotImplementedError in the subclasses. It appears that they aren't currently used elsewhere. I intend to implement, but throwing the error in for now just in case!

vsoch · 2026-05-10T15:40:17Z

    max_length (int) : the max number to return (latest)
    """
+
+    filtered_tags = [x for x in tags]


Sanity check - order and repeats?

vsoch · 2026-05-10T15:41:01Z

 __copyright__ = "Copyright 2021-2025, Vanessa Sochat"
 __license__ = "MPL 2.0"

-__version__ = "0.1.33"


Let's bump up to 0.2.0 for this merge. If you have other changes / PRs to get in we can encompass them with that release too.

Great, can do. I want to add gitlab.com and self hosted gitlab support in before merge. I'm also looking at a strategy to handle multiple "series" of tags, like "X.X.X-alpine" vs "X.X.X-trixie" or whatever.

Good thinking. 👍

georgiastuart · 2026-05-10T16:15:29Z

Thanks for the review!! I'll implement tests and the other suggestions (and check out the 2 line digest printing).

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

georgiastuart · 2026-05-10T22:48:03Z

To do before merge:

Add gitlab.com / self hosted support
Add tests for remote image types & error handling
(If feasible) look at "tag series" refactor
Force tag: digest yaml to print on one line

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

muffato · 2026-05-11T11:34:03Z

Hi @georgiastuart . I'd be happy to help with "Add gitlab.com / self hosted support". We started working on it with @vsoch in #585 but never got to complete it.

vsoch · 2026-05-11T13:46:52Z

Thank you @muffato !

georgiastuart added 15 commits May 9, 2026 17:58

Update schema for and / or

8d482ff

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Add and/or filtering and dockerhub image type

f2ab575

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Move image setup to top function

d6aefb6

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Add quaydockerimage

7101a42

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Add NGC support

76c594e

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Fix name mangling

ce5ee2a

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Fix when a manifest doesn't have a top-level digest

a0cd247

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Handle missing tags more gracefully

9d1ec69

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Lint

d25b546

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Document new update flags

6c4fd6c

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Add ngc requirement. Increment version

d0f797a

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Small fixes

31826ec

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Add info for ngc update

9910126

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Revert and/or filter

9dcd8e4

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Add check for ngc api key

4789a4c

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

georgiastuart mentioned this pull request May 10, 2026

Feature: Add DockerImage classes for docker.io, nvcr.io, and quay.io singularityhub/shpc-registry#483

Open

vsoch reviewed May 10, 2026

View reviewed changes

georgiastuart added 3 commits May 10, 2026 15:51

Refactor for cleaner subclasses

f4c2bb8

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Refactor exceptions

5826188

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Bump to 0.2.0

9b118a0

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Up line length to 120

f67c7a8

Signed-off-by: Georgia Stuart <georgia.stuart@gmail.com>

Uh oh!

Conversation

georgiastuart commented May 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Subclass Design

NCG Compatibility

Added CLI arguments for update

Uh oh!

georgiastuart commented May 10, 2026

Uh oh!

vsoch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

georgiastuart commented May 10, 2026

Uh oh!

georgiastuart commented May 10, 2026

Uh oh!

muffato commented May 11, 2026

Uh oh!

vsoch commented May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

georgiastuart commented May 10, 2026 •

edited

Loading

Added CLI arguments for `update`