Fix: calibration refactoring and bug fixes

[JemmaLDaniel]

Summary

Ports calibration and dataset bug fixes from revisions, plus a refactor that de-duplicates peptide tokenisation, validity flags and match columns across data loaders.

Changes

Shared peptide metadata (data_loaders/utils.py)

• New cell/column helpers: normalize_peptide_cell, is_valid_peptide_tokens, as_token_list, finalize_peptide_metadata (pandas + polars).
• Ground-truth detection: has_ground_truth_sequence_labels -- a sequence column with only empty/null values is treated as unlabelled; column is dropped at load.
• Training mask: labelled_training_mask / require_labelled_rows for rows with valid_sequence=True.

Data loaders

• InstaNovo / MZTab: remove duplicated _evaluate_predictions / tokenization paths; call finalize_peptide_metadata instead.
• Winnow: backfill valid_prediction / valid_sequence when loading saved datasets that predate those columns.

Supervised paths exclude invalid ground truth

Rows still get correct=False and features, but do not enter fit curves when valid_sequence=False:

Consumer	Behaviour
ProbabilityCalibrator.fit	Fits scaler/classifier on labelled mask only
DatabaseGroundedFDRControl.fit	FDR curve from labelled rows only; imputes valid_sequence, num_matches, correct on all rows
resolve_diagnostics_labels	Sequence-derived labels restricted to valid rows; confidence aligned via shared index
filter_dataset (main.py)	Uses valid_prediction when present, else is_valid_peptide_tokens

Standalone bug fixes

Fix	File
save_metadata supports .parquet	calibration_dataset.py
Numpy token arrays in tokens_to_proforma	peptide.py
Skip all-empty sequence at spectrum load	utils.load_spectrum_data
iRT: unique-peptide / unique-RT checks, empty-metadata early exit, skip Koina when no valid spectra	retention_time.py
Mass error uses shared validity check	mass_error.py

Reviewer Notes

We dual-implement finalize_peptide_metadata because loaders are mid-migration: MZTab is polars through load/merge; InstaNovo and Winnow are still pandas at the merge boundary; CalibrationDataset is pandas end-to-end. Shared cell-level helpers guarantee identical behaviour; the polars/pandas wrappers are thin adapters so we don’t force premature conversion or block this PR on a full loader rewrite.

I would suggest merging #220 , #218 and #219 before this PR.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	0	0	100%
data_types.py	4	0	100%
calibration
__init__.py	0	0	100%
calibration_features.py	9	0	100%
calibrator.py	106	11	89%	70–71, 73, 108, 135–136, 138, 164, 169, 196–197
diagnostics.py	156	47	69%	74, 104, 108, 130, 192–207, 250–251, 255, 296, 298–313, 324–330
calibration/features
__init__.py	10	0	100%
base.py	8	0	100%
beam.py	47	0	100%
chimeric.py	82	1	98%	213
constants.py	9	0	100%
fragment_match.py	78	1	98%	203
mass_error.py	68	2	97%	17, 21
retention_time.py	160	14	91%	110, 113–114, 195, 202, 218, 269–271, 281, 284–285, 290–291
sequence.py	19	0	100%
token_score.py	37	1	97%	82
utils.py	261	3	98%	96, 368, 594
compat
__init__.py	0	0	100%
instanovo.py	10	6	40%	12, 14–15, 17, 24–25
datasets
__init__.py	0	0	100%
calibration_dataset.py	113	18	84%	155, 169, 171, 173, 180, 189, 202, 255, 257–258, 264–267, 269–272
interfaces.py	3	0	100%
psm_dataset.py	25	0	100%
datasets/data_loaders
__init__.py	5	0	100%
instanovo.py	106	16	84%	90, 93, 119, 142, 168–169, 172–174, 176–177, 179, 182–184, 191
mztab.py	179	32	82%	103, 106, 157, 161, 210–211, 223, 268, 271, 283–284, 296–298, 300–301, 303, 305, 311, 314–315, 322–324, 331, 333–336, 340, 472, 479
pointnovo.py	7	0	100%
utils.py	202	6	97%	14, 145–148, 216
winnow.py	44	4	90%	55–56, 101–102
fdr
__init__.py	0	0	100%
base.py	58	15	74%	81, 85–86, 91, 98–99, 105, 126, 129–130, 135, 137–138, 144, 186
database_grounded.py	25	0	100%
nonparametric.py	25	4	84%	62, 68–69, 72
scripts
__init__.py	0	0	100%
main.py	260	260	0%	8, 10–13, 16–20, 23–24, 26–28, 32, 39, 44, 47, 53, 55–56, 59, 68, 70–73, 75, 80, 87, 89–91, 93, 95–100, 103, 105–106, 111, 126, 129, 136–142, 145–146, 149, 162–164, 167, 170, 175, 177–179, 181, 183–184, 187–188, 191, 193–194, 196, 198, 200–201, 203, 206–207, 210–211, 214–215, 218–220, 222–225, 228–230, 232, 235, 249–251, 253, 255, 260, 262–264, 266–267, 269, 271–272, 274–276, 278, 280, 282–283, 287–290, 292–293, 295–296, 298–299, 301, 304, 318–320, 323, 326, 331, 333–335, 337–339, 341–342, 345–346, 349, 351–352, 354, 356, 358–359, 361, 364–365, 371–373, 375–378, 381–382, 385–386, 389–390, 393–394, 402–404, 408, 411, 415, 418, 424–426, 428–429, 436–437, 439, 441, 446, 448–450, 452–453, 456, 458–459, 461–464, 466–467, 469–470, 472–474, 480–481, 485–486, 489, 496, 501–502, 507–509, 512, 517, 527, 534, 536, 540, 542–543, 547–548, 551, 574, 587–588, 591, 613, 625–626, 629, 654, 667–668, 671, 686, 698–699, 702, 717, 729–730, 733, 745, 757–758, 761, 776, 788–789, 792, 801, 813–814
utils
__init__.py	4	0	100%
config_formatter.py	53	40	24%	29, 37–38, 40–42, 44, 55, 58–60, 62–63, 66–69, 72–74, 77–78, 80, 91, 102, 113, 127–128, 130–132, 145–147, 150, 153–154, 157–158, 160
config_path.py	76	5	93%	24–26, 117–118
peptide.py	25	1	96%	32
TOTAL	2274	487	78%

Tests	Skipped	Failures	Errors	Time
488	0 💤	0 ❌	0 🔥	39.251s ⏱️