fix: replace HDF5 fill values with NaN before DataFrame construction#61
Conversation
|
Hey @jeanbez @kaveenh ,Was poking around the codebase and noticed that HDF5 files store missing data as fill values (like |
|
@jeanbez ,Expanded the PR a bit, added _collect_fill_values() which pulls sentinels from four sources (_FillValue attribute, missing_value attribute, HDF5 native fill value, and caller-supplied values) and handles the ambiguous default-zero case with a warning instead of silently replacing. 15 tests added covering all six cases, all passing. Also updated the docs; completeness section now explains the fill value behaviour, file formats list is corrected, and the limitations page now mentions all five supported formats. Let me know if anything needs changing! |
Related Issues / Pull Requests
Closes #60
Description
hdf5_reader.pywas loading raw HDF5 dataset values without replacing fill-value-encoded missing entries withNaN. This causedcompleteness.pyto always report 100% completeness for HDF5 files where missing data is represented by a sentinel number like-9999.0— which is standard practice in scientific datasets. This PR adds a six-line normalization step that checksobj.fillvalueafter reading each dataset and replaces matching entries withnp.nanbefore the DataFrame is constructed, sopd.isnull()correctly detects missingness downstream.What changes are proposed in this pull request?
Checklist: