PR: Backbone subsampling — stable sequence set across reruns

[filiperomero2]

Summary

Adds opt-in --backbone-from <prev_workdir> flag to flexpipe-run so a surveillance rerun can anchor itself to a previous build's subsampled strain list. The new subsample becomes the union of (stable backbone strains) + (freshly-selected new sequences), making results comparable across months-apart runs without re-subsampling from scratch.

Motivation

Without this, each run independently re-subsamples from the full available pool. Even older, previously-selected sequences can drop in or out of the tree between runs — conflating biological signal with sampling noise. For a lab running scheduled surveillance (e.g. RSV every 6 months), this makes "what's new since last time?" hard to answer reliably.

Mechanism

augur subsample is driven entirely by a YAML config. A sample set with only an include: path force-keeps every listed strain regardless of group_by / sequences_per_group caps. We exploit this:

1. _materialize_backbone() in run.py reads the strain column from <prev_workdir>/results/subsampled/metadata.tsv and writes it to <workdir>/config/backbone_strains.txt
2. cfg.subsampling.backbone_strains is set to that path before write_snakemake_config_overrides is called, so it propagates into the single resolved Snakemake configfile
3. The ingest Snakefile resolve_subsample_config rule passes it to flexpipe.config.resolve_subsample_config(), which injects samples.__backbone__: {include: <path>} into the resolved subsample YAML
4. augur subsample force-keeps the backbone strains on top of the normal subsample

Usage

# Initial build (e.g. June 2024)
flexpipe-run --config builds/rsv-a-brazil/config.yaml \
             --workdir /data/rsv-A --run-date 2024-06-01

# 2 years later — backbone anchors the 2024 selection
flexpipe-run --config builds/rsv-a-brazil/config.yaml \
             --workdir /data/rsv-B --run-date 2026-06-01 \
             --backbone-from /data/rsv-A

Edge cases

Scenario	Behaviour
--backbone-from omitted	Complete no-op — identical to current behaviour
Previous workdir missing or metadata absent	Warning logged, run proceeds without backbone
Empty backbone (header-only metadata)	Warning logged, run proceeds without backbone
--backbone-from == current workdir	Hard error, exit 2
Backbone strain withdrawn from source	Silently skipped by augur subsample (acceptable)
Backbone strain dropped by upstream QC / clade_filter	Cannot be force-kept — backbone retention is best-effort, bounded by the quality contract

Key design decisions

• Explicit CLI flag, not auto-detection — backbone is an operational decision per run, not a stable property of the build definition. Auto-detecting from the workdir risks silently anchoring to stale state on an ordinary rerun.
• No QC bypass — include applies within augur subsample, which runs after augur filter and clade_filter. Strains that newly fail QC are correctly excluded even if they were in the previous backbone.
• Runtime-only field — SubsamplingConfig.backbone_strains is never authored in builds/*/config.yaml (would bake a machine-specific absolute path into version control).
• Provenance — backbone_from path and backbone_strain_count are recorded in manifest.json.

Files changed

• flexpipe/paths.py — WorkdirPaths.backbone_strains property (config/backbone_strains.txt)
• flexpipe/config.py — SubsamplingConfig.backbone_strains: str | None; resolve_subsample_config() extended with backbone_strains param
• flexpipe/run.py — _materialize_backbone() helper; --backbone-from CLI arg; threading through run_pipeline → _run_pipeline_locked; manifest provenance
• ingest/Snakefile — resolve_subsample_config rule reads _sub.get("backbone_strains") and passes it through
• CLAUDE.md — usage example, semantics, and limitations documented

Tests

• 13 new unit tests covering _materialize_backbone (happy path, missing dir, empty metadata, self-reference, end-to-end run_pipeline propagation), resolve_subsample_config backbone injection, and config schema propagation to resolved YAML
• 1 new integration dry-run test confirming the ingest DAG plans cleanly when backbone_strains is set
• All 563 tests pass; 78.8% coverage (threshold 60%); ruff + black + mypy clean