Skip to content

ChemPropFeaturizer and PairwiseFeaturizer default n_jobs=4 is an odd default for DataLoader num_workers #558

Description

@smcolby

Problem

ChemPropFeaturizer (chemprop.py:121) and PairwiseFeaturizer (pairwise.py:99) both default to n_jobs=4, which is passed directly as num_workers to a PyTorch DataLoader.

# chemprop.py
n_jobs: int = 4  # → DataLoader(num_workers=self.n_jobs, ...)

# pairwise.py
n_jobs: int = 4  # → DataLoader(num_workers=self.n_jobs, ...)

The value 4 is arbitrary: it is too low to be "use all available cores" and too high to be "single-threaded". This can cause surprising behaviour:

  • On machines with fewer than 4 physical cores, spawning 4 workers adds overhead with no benefit.
  • On CI / restricted environments, spawning extra processes can fail or be slow.
  • For small molecule sets (e.g., unit/integration tests), multiprocessing overhead exceeds the compute savings.

Expected behaviour

The default should be either:

  • 1 — single-threaded, zero overhead, consistent cross-platform behaviour (recommended for a library default), or
  • -1 — use all available cores, consistent with the convention used elsewhere in the codebase (DescriptorFeaturizer, FingerprintFeaturizer)

The docstring for ChemPropFeaturizer.n_jobs already says "Number of parallel workers to use, by default 4" — the description is accurate but the default itself should be reconsidered.

Affected files

  • openadmet/models/features/chemprop.pyChemPropFeaturizer.n_jobs
  • openadmet/models/features/pairwise.pyPairwiseFeaturizer.n_jobs

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions