Skip to content

fix: replace HDF5 fill values with NaN before DataFrame construction#61

Merged
kaveenh merged 5 commits into
idtlab:developfrom
Aman-Cool:fix/hdf5-fillvalue-nan
Mar 10, 2026
Merged

fix: replace HDF5 fill values with NaN before DataFrame construction#61
kaveenh merged 5 commits into
idtlab:developfrom
Aman-Cool:fix/hdf5-fillvalue-nan

Conversation

@Aman-Cool

@Aman-Cool Aman-Cool commented Feb 25, 2026

Copy link
Copy Markdown
Contributor

Related Issues / Pull Requests

Closes #60

Description

hdf5_reader.py was loading raw HDF5 dataset values without replacing fill-value-encoded missing entries with NaN. This caused completeness.py to always report 100% completeness for HDF5 files where missing data is represented by a sentinel number like -9999.0 — which is standard practice in scientific datasets. This PR adds a six-line normalization step that checks obj.fillvalue after reading each dataset and replaces matching entries with np.nan before the DataFrame is constructed, so pd.isnull() correctly detects missingness downstream.

What changes are proposed in this pull request?

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality not to work as expected; for instance, examples in this repository must be updated too)
  • This change requires a documentation update

Checklist:

  • My code modifies existing public API, or introduces new public API, and I updated or wrote documentation
  • I have commented my code
  • My code requires documentation updates, and I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@Aman-Cool

Copy link
Copy Markdown
Contributor Author

Hey @jeanbez @kaveenh ,Was poking around the codebase and noticed that HDF5 files store missing data as fill values (like -9999.0) instead of NaN, so pd.isnull() never actually catches them and completeness always comes back as 100%. Small fix, just normalize the fill value to NaN right after reading the raw array.
Let me know if anything needs tweaking!

@Aman-Cool

Copy link
Copy Markdown
Contributor Author

@jeanbez ,Expanded the PR a bit, added _collect_fill_values() which pulls sentinels from four sources (_FillValue attribute, missing_value attribute, HDF5 native fill value, and caller-supplied values) and handles the ambiguous default-zero case with a warning instead of silently replacing. 15 tests added covering all six cases, all passing. Also updated the docs; completeness section now explains the fill value behaviour, file formats list is corrected, and the limitations page now mentions all five supported formats. Let me know if anything needs changing!

@kaveenh
kaveenh self-requested a review March 2, 2026 18:04
@kaveenh
kaveenh merged commit aa75577 into idtlab:develop Mar 10, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3:17 AM[Bug] HDF5 fill values silently reported as valid data — completeness score always 100% for fill-value-encoded missing values

2 participants