Skip to content

Fix OneHotEncoder handle_missing='value' to encode missing values as zeros (fixes #400) - #485

Open
ranafaraz wants to merge 3 commits into
scikit-learn-contrib:masterfrom
ranafaraz:fix/onehot-handle-missing-value-400
Open

Fix OneHotEncoder handle_missing='value' to encode missing values as zeros (fixes #400)#485
ranafaraz wants to merge 3 commits into
scikit-learn-contrib:masterfrom
ranafaraz:fix/onehot-handle-missing-value-400

Conversation

@ranafaraz

Copy link
Copy Markdown

Summary

Fixes #400.

With handle_missing='value' (the default), a missing value present in the training data was given its own dummy column instead of being encoded as 0 in every dummy column. This contradicts the documented behaviour ("'value' will encode a missing value as 0 in every dummy column").

from category_encoders import OneHotEncoder
import pandas as pd

data = pd.DataFrame([('foo', 1), ('bar', 2), (None, 6)], columns=['c1', 'c2'])
OneHotEncoder(handle_missing='value').fit_transform(data)

Before this PR the None row produced an extra c1_3 column; after it, the missing row is 0 across c1_1/c1_2, matching the docs and the 'ignore' strategy.

Root cause

OneHotEncoder._fit maps handle_missing='value' to the ordinal strategy 'value'. When a NaN is present during fit, the internal OrdinalEncoder assigns it a positive code, which then survives the values[values > 0] filter in generate_mapping and becomes its own dummy column. (When NaN appears only at transform time it was already handled correctly via the base_df.loc[-2] = 0 row.)

Fix

Map the 'value' strategy to the ordinal 'return_nan' path (NaN -> -2), exactly as 'ignore' already does. The missing code is then dropped from the columns and encoded as all-zeros through the existing base_df.loc[-2] = 0 row. Other modes (indicator, return_nan, error, ignore) are unchanged.

Tests

  • Added test_handle_missing_value covering the issue's reproduction (both the transformed result and the mapping).
  • Full tests/test_one_hot.py passes locally (17 passed, 12 subtests) and ruff check is clean. No existing test needed changing — the inverse-transform round-trips still hold.

A note for maintainers

In the issue thread you mentioned the value vs ignore naming is a little ambiguous. This PR keeps both options and only makes value match its documented behaviour (so value and ignore now coincide for missing values). Happy to instead update the docs, deprecate/rename, or take whatever direction you prefer.

Prepared with AI assistance and verified locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Handle missing in one hot encoder

1 participant