Skip to content

GH-50515: [C++][Compute] Respect parent validity bitmap when casting nested structs with non-nullable fields#50546

Open
aaron-seq wants to merge 1 commit into
apache:mainfrom
aaron-seq:fix-50515-nested-struct-cast-masked-nulls
Open

GH-50515: [C++][Compute] Respect parent validity bitmap when casting nested structs with non-nullable fields#50546
aaron-seq wants to merge 1 commit into
apache:mainfrom
aaron-seq:fix-50515-nested-struct-cast-masked-nulls

Conversation

@aaron-seq

Copy link
Copy Markdown

Rationale for this change

When casting a nested struct whose inner field changes from nullable to
non-nullable, CastStruct::Exec currently calls
in_values->GetNullCount() > 0 without considering the parent struct's
validity bitmap. This causes a spurious ArrowInvalid error when the
child's physical nulls are fully masked by null parent entries.

Reported in #50515.

What changes are included in this PR?

In scalar_cast_nested.cc, replace the unconditional rejection with a
three-way check:

  1. Parent has no bitmap → child nulls are unmasked; reject.
  2. Child has no bitmap → element-wise check against parent validity.
  3. Both bitmaps present → use BinaryBitBlockCounter::NextAndNotWord()
    to detect parent-valid-AND-child-null positions without heap allocation.

Are these changes tested?

Five new C++ test cases in scalar_cast_test.cc:

  • StructNestedNullabilityAbsentParent — true violation, must reject
  • StructNestedNullabilityMasked — masked null, must succeed
  • StructNestedNullabilitySliced — offset correctness for both outcomes
  • StructNestedNullabilityAbsentChild — no nulls, must succeed
  • StructNestedNullabilityDeep — 3-level nesting with masked null

One new Python test in test_compute.py:

  • test_cast_struct_nested_nullability — end-to-end PyArrow validation
    covering success, rejection, and sliced array cases

Are there any user-facing changes?

Struct casts that were previously rejected with
ArrowInvalid: field '...' has nulls will now succeed when the nulls
are masked by parent-level nulls. This is a correctness fix, not a
behavior change — the previous behavior was incorrect per the Arrow
columnar format specification.

…ct cast

When casting a struct with a nullable->non-nullable field change,
CastStruct::Exec checks GetNullCount() on the child array without
considering the parent struct's validity bitmap. This rejects casts
where the child's physical nulls are fully masked by parent-level
nulls.

Intersect the parent and child validity bitmaps before deciding
whether unmasked nulls exist, using BinaryBitBlockCounter for the
common case and explicit fallbacks for absent bitmaps.
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #50515 has been automatically assigned in GitHub to PR creator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant