Skip to content

Feat/new data #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
kisside wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

Feat/new data #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9df8737
feat: level 3
kisside Apr 18, 2026
661bc28
fix: embedding script
kisside Apr 18, 2026
c7aee40
fix: bump mlkit
kisside Apr 18, 2026
894a83b
feat: level 2
kisside Apr 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions configs/data/tiled/mou_2_224.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# @package _global_

dataset:
name: mou_2_224
mpp: 0.97
tile_extent: 224
level: 2
uris:
train: "mlflow-artifacts:/3/3f3ab8b587234175951e1a018015cef8/artifacts/mou_2_224_train"
val: "mlflow-artifacts:/3/3f3ab8b587234175951e1a018015cef8/artifacts/mou_2_224_val"
test: "mlflow-artifacts:/3/3f3ab8b587234175951e1a018015cef8/artifacts/mou_2_224_test"
11 changes: 11 additions & 0 deletions configs/data/tiled/mou_3_224.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# @package _global_

dataset:
name: mou_3_224
mpp: 1.94
tile_extent: 224
level: 3
uris:
train: "mlflow-artifacts:/3/e0f0170be75241d095446a33202edb95/artifacts/mou_3_224_train"
val: "mlflow-artifacts:/3/e0f0170be75241d095446a33202edb95/artifacts/mou_3_224_val"
test: "mlflow-artifacts:/3/e0f0170be75241d095446a33202edb95/artifacts/mou_3_224_test"
6 changes: 6 additions & 0 deletions configs/experiment/preprocessing/embeddings/mou_2_224.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# @package _global_

defaults:
- /data/tiled/mou_2_224
- /preprocessing/tile_encoder/virchow2
- _self_
6 changes: 6 additions & 0 deletions configs/experiment/preprocessing/embeddings/mou_3_2024.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# @package _global_

@gemini-code-assist gemini-code-assist Bot Apr 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The filename mou_3_2024.yaml appears to contain a typo. Based on the dataset name mou_3_224 and the companion file mou_2_224.yaml, it should likely be named mou_3_224.yaml.


defaults:
- /data/tiled/mou_3_224
- /preprocessing/tile_encoder/virchow2
- _self_
10 changes: 9 additions & 1 deletion preprocessing/embeddings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,14 @@ def save_embeddings(
df.to_parquet(embeddings_path, index=False, engine="pyarrow")


def resolve_dataset_uris(config: DictConfig) -> list[str]:
"""Resolve tiled dataset URIs from Hydra config."""
if config.dataset.get("uris") is not None:
return [str(uri) for uri in config.dataset.uris.values()]

raise ValueError("Embeddings preprocessing requires `dataset.uris`.")
Comment on lines +72 to +77

@gemini-code-assist gemini-code-assist Bot Apr 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current implementation of resolve_dataset_uris has two issues:

  1. It assumes config.dataset.uris is always a mapping (by calling .values()). If uris is provided as a list in the configuration, it will raise an AttributeError.
  2. It removes support for config.dataset.path, which breaks backward compatibility for existing configurations.

Consider a more robust implementation that handles both mapping and list types for uris, and falls back to path if uris is missing.

def resolve_dataset_uris(config: DictConfig) -> list[str]:
    """Resolve tiled dataset URIs from Hydra config."""
    dataset = config.get("dataset", {})
    if (uris := dataset.get("uris")) is not None:
        return [str(v) for v in (uris.values() if hasattr(uris, "values") else uris)]

    if (path := dataset.get("path")) is not None:
        return [str(path)]

    raise ValueError("Embeddings preprocessing requires `dataset.uris` or `dataset.path`.")



@with_cli_args(["+preprocessing=embeddings"])
@hydra.main(
config_path="../configs",
Expand All @@ -83,7 +91,7 @@ def main(config: DictConfig, logger: MLFlowLogger) -> None:
tile_encoder = tile_encoder.to(device)

dataset = SlideDataset(
uris=[config.dataset.path],
uris=resolve_dataset_uris(config),
transforms=A.Compose(
[A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))]
),
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ dependencies = [
"hydra-core>=1.3.2",
"lightning>=2.5.5",
"omegaconf>=2.3.0",
"rationai-mlkit",
"rationai-mlkit>=0.4.1",
"rationai-masks>=1.1.1",
"rationai-tiling>=1.1.1",
"ratiopath>=1.2.0",
Expand Down
4 changes: 2 additions & 2 deletions uv.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading