GH-50247: [C++] Reuse abstraction for null partitions in sorting functions by taepper · Pull Request #50248 · apache/arrow

taepper · 2026-06-25T01:48:42Z

Rationale for this change

@pitrou mentioned this as a follow-up in #46926

What changes are included in this PR?

Refactoring sorting methods to reuse the helper methods avoid maintaining two abstractions for null partitions. The new abstraction was very seamless to implement in most cases, but a few spots required some care

In particular, these functions were severly simlpified by the new abstraction:

MarkDuplicates: duplicate nulls and nans were detected by checking every single row for Null one additional time, after we already had (and discarded) the nullness information
GenericMergeImpl: merging of null-ranges involved repartitioning null and nan values in every merge invocation. Now, we track this distinction and do not need any merge function for null and nan blocks (unless we merge by multiple sort-keys, where we will merge according to the remaining sort keys)

Are these changes tested?

Yes, the compute test suite passes as before

Are there any user-facing changes?

No.

GitHub Issue: [C++] Reuse abstraction for null partitions in sorting functions #50247

github-actions · 2026-06-25T01:49:07Z

⚠️ GitHub issue #50247 has been automatically assigned in GitHub to PR creator.

pitrou

Thank you! This is excellent, and the simplification is very welcome. Just a couple minor comments below.

pitrou · 2026-06-25T08:35:09Z

Hmm, there are some regressions in the test suite (see CI runs).

Also this runtime assertion on Windows CI might give a clue:

27/98 Test  #49: arrow-compute-vector-sort-test ...............Exit code 0xc0000409
***Exception:   0.43 sec
[==========] Running 284 tests from 130 test suites.
[----------] Global test environment set-up.
[----------] 2 tests from TestNthToIndicesForReal/0, where TypeParam = arrow::FloatType
[ RUN      ] TestNthToIndicesForReal/0.NthToIndicesDoesNotProvideDefaultOptions
[       OK ] TestNthToIndicesForReal/0.NthToIndicesDoesNotProvideDefaultOptions (1 ms)
[ RUN      ] TestNthToIndicesForReal/0.Real
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\span(160) : Assertion failed: cannot compare incompatible span iterators

taepper · 2026-07-13T13:31:36Z

Thank you for the comments. I returned from vacation and fixed the windows errors. I also added some short-hands to the struct so we have less boilerplate in getting the begin/end pointers of the partition ranges.

pitrou

This is looking very good, just a couple additional comments.

pitrou · 2026-07-21T14:23:33Z

-  IndexType* operator()(IndexType* indices_begin, IndexType* indices_end,
-                        Predicate&& pred) {
-    return std::partition(indices_begin, indices_end, std::forward<Predicate>(pred));
+  template <typename Predicate>


Add a comment that this returns the right-hand partition?

pitrou · 2026-07-21T14:29:33Z

+  template <typename TargetIndexType>
+  GenericNullLikePartition<TargetIndexType> TranslateTo(
+      IndexType* indices_begin, TargetIndexType* target_indices_begin) const {
+    size_t non_null_offset = non_null_like_range.data() - indices_begin;


Can we DCHECK that indices_begin falls somewhere in the overall range?

pitrou · 2026-07-21T14:33:09Z

-        (nulls_end - indices_begin) + target_indices_begin,
-    };
+  static NullPartition NullsAtStart(std::span<uint64_t> indices,
+                                    std::span<uint64_t> non_null_tail) {


Note: it seems a bit weird to pass the null tail to NullsAtEnd and the non-null tail to NullsAtStart. At least add a comment?

pitrou · 2026-07-21T14:34:21Z

-        [&values, &offset](uint64_t ind) { return values.IsNull(ind - offset); });
-    return NullPartitionResult::NullsAtStart(indices_begin, indices_end, nulls_end);
+    auto non_null_tail = partitioner(indices, [&values, &offset](uint64_t ind) {
+      return values.IsNull(ind - offset);


ind - offset is mixed signed-unsigned arithmetic, which may error out on some compilers.

pitrou · 2026-07-21T14:47:29Z

-  void EmitIndices(const NullPartitionResult& p, const ArrayType& values, int64_t offset,
+  void EmitIndices(const NullLikePartition& p, const ArrayType& values, int64_t offset,
                   CounterType* counts) const {
    int64_t index = offset;


Can we DCHECK that the nan_range is empty?

taepper added 8 commits June 24, 2026 16:27

replace PartitionNulls by PartitionNullsAndNans

a4f9728

more helpers and migrations

5fe6eb8

spanify one

6225331

spanify another

650d857

spanify working

5a62018

more spans

36f348f

another span

7c77d23

replace Partition usage

c0fd6ec

taepper requested a review from pitrou as a code owner June 25, 2026 01:48

github-actions Bot added Component: C++ awaiting review Awaiting review labels Jun 25, 2026

pitrou reviewed Jun 25, 2026

View reviewed changes

Comment thread cpp/src/arrow/compute/kernels/vector_sort_internal.h Outdated

Comment thread cpp/src/arrow/compute/kernels/vector_sort_internal.h Outdated

Comment thread cpp/src/arrow/compute/kernels/vector_sort.cc Outdated

Comment thread cpp/src/arrow/compute/kernels/vector_sort.cc Outdated

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jun 25, 2026

taepper added 4 commits June 28, 2026 18:22

Change interface to PartitionResultByNullLikeness

b4aec27

consolidate more functions

a0d1e4a

change partition struct name to NullLikePartition

824bc4f

fix additional test and refactor more functions

91d6a60

taepper force-pushed the better-null-partitions branch from 54c3aca to 91d6a60 Compare June 28, 2026 16:22

taepper added 3 commits June 28, 2026 21:15

try workaround for passing windows ci

5333282

another simplification

b173cd4

reduce mixing of span iterators

410db4e

tadeja mentioned this pull request Jun 29, 2026

[C++] Simplify or remove arrow/util/functional.h #50250

Open

tadeja reviewed Jun 29, 2026

View reviewed changes

Comment thread cpp/src/arrow/compute/kernels/vector_sort.cc Outdated

kou changed the title ~~GH-50247: Reuse abstraction for null partitions in sorting functions~~ GH-50247: [C++] Reuse abstraction for null partitions in sorting functions Jul 2, 2026

taepper added 3 commits July 13, 2026 10:54

Merge remote-tracking branch 'origin/main' into better-null-partitions

b27698b

remove all uses of stl ranges algorithms

15b9099

clean-up

bd98d22

another mixed iterator occurence

a91e3a5

taepper force-pushed the better-null-partitions branch from 67fd062 to a91e3a5 Compare July 13, 2026 12:22

taepper added 3 commits July 13, 2026 15:00

add short-hand for range begins/ends in NullLikePartition

9e68728

replace another struct with NullLikePartition

eefc4d4

add drawing to std::rotate calls

0c097af

taepper requested a review from pitrou July 13, 2026 13:31

pitrou reviewed Jul 21, 2026

View reviewed changes

Uh oh!

Conversation

taepper commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pitrou commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

taepper commented Jul 13, 2026

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

taepper commented Jun 25, 2026 •

edited

Loading

pitrou commented Jun 25, 2026 •

edited

Loading