GH-35692: [C++][Parquet] Support to read fixed size list array with nulls by HuaHuaY · Pull Request #50271 · apache/arrow

HuaHuaY · 2026-06-27T07:14:26Z

Rationale for this change

Support to read fixed size list array with nulls.

What changes are included in this PR?

Modified FixedSizeListReader to add null slots to FixedSizeList.

Are these changes tested?

Yes.

Are there any user-facing changes?

User can read fixed size list array with nulls now.

This PR also fixes GH-35697.

GitHub Issue: [C++][Parquet] Null fixed-length lists cannot be read from parquet file #35692

pitrou · 2026-07-08T09:09:28Z

+                                int32_t expected_size) -> Status {
+      std::span<const int32_t> run_offsets(offsets + start,
+                                           static_cast<size_t>(length + 1));
+      const auto first_invalid_offset = std::ranges::adjacent_find(


I'm not sure C++20 ranges are available on all our CI platforms unfortunately (some of them require old compilers).

It looks like there are no compilation errors in the CI pipelines which are already run.

Let me enable the more annoying ones :)

The new CI tests have also been compiled, so I haven't changed the code yet.

@thisisnic Do you remember how to launch the R builds with the most outdated compilers? :)

Not off the top of my head, but there should be a recent-ish PR which fixed the last ones that broke it, and will have the CI job being run on it in the comments.

pitrou · 2026-07-08T09:15:21Z

@rok FYI :)

wgtmac · 2026-07-21T09:44:15Z

+          return Status::OK();
+        }
+        child_arrays.push_back(
+            ::arrow::MakeArray(data->child_data[0]->Slice(offsets[start], child_length)));


Could we avoid creating one Array per validity run? This could become expensive for wide fixed-size lists. Is it possible to use a single builder using AppendArraySlice() and AppendNulls() instead?

I don't think this is a good idea. First, the array is created via Slice, and the actual data isn't copied. Second, Concatenate handles different array types automatically, whereas doing it manually would be too much of a hassle.

My concern is the per-run object/allocation overhead rather than copying each valid slice: an alternating validity bitmap still creates O(number of runs) Array/ArrayData objects and O(number of null runs) null arrays before the final concatenation.

But I agree that my suggestion requires a lot of changes to handle different child types. I'm fine to keep it as-is by adding a TODO comment for a future improvement?

I have added a TODO comment in the latest commit.

wgtmac · 2026-07-21T09:54:22Z

Codex reminded me that code below is outdated after this fix and it seems that level_info.null_slot_usage is useless anymore.

arrow/cpp/src/parquet/level_conversion.cc

Lines 122 to 126 in 0077539

    
           if (output->null_count > 0 && level_info.null_slot_usage > 1) { 
        
             throw ParquetException( 
        
                 "Null values with null_slot_usage > 1 not supported." 
        
                 "(i.e. FixedSizeLists with null values are not supported)"); 
        
           }

HuaHuaY · 2026-07-21T11:01:56Z

Codex reminded me that code below is outdated after this fix and it seems that level_info.null_slot_usage is useless anymore.

This requires modifying the public methods in cpp/src/parquet/level_conversion.h, and I prefer opening a new pull request to address it.

pitrou · 2026-07-22T13:48:46Z

@HuaHuaY Are you waiting for another review here?

HuaHuaY · 2026-07-22T15:37:57Z

@HuaHuaY Are you waiting for another review here?

If there are no new comments, I think this PR is ok to be merged.

wgtmac

+1

wgtmac · 2026-07-27T02:40:49Z

I'll merge this tomorrow if no objection.

thisisnic · 2026-07-27T15:19:37Z

@github-actions crossbow submit -g r

thisisnic · 2026-07-27T15:20:35Z

I'll merge this tomorrow if no objection.

I've triggered the R builds just to make sure we don't have potential failures suggested in #50271 (comment)

HuaHuaY · 2026-07-27T15:23:01Z

I'll merge this tomorrow if no objection.

I've triggered the R builds just to make sure we don't have potential failures suggested in #50271 (comment)

It seems that no extra ci starts. I might be because @pitrou has added "CI: Extra R". R's CI has already been run.

github-actions · 2026-07-27T15:25:44Z

Revision: 06266f9

Submitted crossbow builds: ursacomputing/crossbow @ actions-d711fbc610

Task	Status
r-binary-packages
r-recheck-most
test-r-alpine-linux-cran
test-r-arrow-backwards-compatibility
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-fedora-clang
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-sanitizers
test-r-linux-valgrind
test-r-m1-san
test-r-macos-as-cran
test-r-offline-maximal
test-r-ubuntu-22.04
test-r-versions
test-r-wasm

HuaHuaY · 2026-07-27T15:26:41Z

Oh. Extra CI starts. It just takes a little while to start up.

HuaHuaY · 2026-07-27T15:33:26Z

The additional CI run revealed an issue. Arrow recently introduced simdjson but hasn't updated this CI configuration yet. https://github.com/ursacomputing/crossbow/actions/runs/30279773748/job/90022897470

HuaHuaY · 2026-07-28T02:46:15Z

I think these failures are unrelated to this PR. To be sure, I'll rebase and run it again.

HuaHuaY · 2026-07-28T02:48:01Z

@github-actions crossbow submit -g r

github-actions · 2026-07-28T02:48:37Z

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/30324136965

wgtmac · 2026-07-28T02:53:31Z

@github-actions crossbow submit -g r

github-actions · 2026-07-28T02:56:24Z

Revision: 6b0bd58

Submitted crossbow builds: ursacomputing/crossbow @ actions-523da37569

Task	Status
r-binary-packages
r-recheck-most
test-r-alpine-linux-cran
test-r-arrow-backwards-compatibility
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-fedora-clang
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-sanitizers
test-r-linux-valgrind
test-r-m1-san
test-r-macos-as-cran
test-r-offline-maximal
test-r-ubuntu-22.04
test-r-versions
test-r-wasm

wgtmac · 2026-07-28T03:54:58Z

I've checked that all CI failures are unrelated.

conbench-apache-arrow · 2026-07-28T10:57:19Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 5e698fe.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.

HuaHuaY requested review from pitrou and wgtmac as code owners June 27, 2026 07:14

github-actions Bot added Component: Parquet Component: C++ awaiting review Awaiting review labels Jun 27, 2026

pitrou reviewed Jul 8, 2026

View reviewed changes

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jul 8, 2026

pitrou added the CI: Extra: R Run extra R CI label Jul 8, 2026

HuaHuaY force-pushed the nested_fixed_size_list_reader branch from 915c757 to 2fc0d49 Compare July 9, 2026 09:42

github-actions Bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Jul 13, 2026

wgtmac reviewed Jul 21, 2026

View reviewed changes

HuaHuaY force-pushed the nested_fixed_size_list_reader branch from 2fc0d49 to c5cadcf Compare July 21, 2026 11:00

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jul 21, 2026

HuaHuaY force-pushed the nested_fixed_size_list_reader branch from c5cadcf to 3840ffc Compare July 22, 2026 16:09

wgtmac approved these changes Jul 22, 2026

View reviewed changes

Comment thread cpp/src/parquet/arrow/arrow_reader_writer_test.cc Outdated

wgtmac approved these changes Jul 24, 2026

View reviewed changes

HuaHuaY added 5 commits July 28, 2026 10:47

support to read fixed size list array with nulls

8338513

address review

fb4a9f7

address review

bbecb76

address review

f8cfd46

address review

6b0bd58

HuaHuaY force-pushed the nested_fixed_size_list_reader branch from 06266f9 to 6b0bd58 Compare July 28, 2026 02:47

wgtmac merged commit 5e698fe into apache:main Jul 28, 2026
57 of 61 checks passed

wgtmac removed the awaiting change review Awaiting change review label Jul 28, 2026

wgtmac mentioned this pull request Jul 28, 2026

[C++][Parquet] Null fixed-length lists cannot be read from parquet file #35692

Closed

HuaHuaY deleted the nested_fixed_size_list_reader branch July 28, 2026 03:56

HuaHuaY mentioned this pull request Jul 28, 2026

[C++][Parquet] Remove unused member null_slot_usage in struct LevelInfo #50678

Closed

Uh oh!

Conversation

HuaHuaY commented Jun 27, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuaHuaY Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pitrou commented Jul 8, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuaHuaY Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wgtmac commented Jul 21, 2026

Uh oh!

HuaHuaY commented Jul 21, 2026

Uh oh!

pitrou commented Jul 22, 2026

Uh oh!

HuaHuaY commented Jul 22, 2026

Uh oh!

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

wgtmac commented Jul 27, 2026

Uh oh!

thisisnic commented Jul 27, 2026

Uh oh!

thisisnic commented Jul 27, 2026

Uh oh!

HuaHuaY commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

HuaHuaY commented Jul 27, 2026

Uh oh!

HuaHuaY commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuaHuaY commented Jul 28, 2026

Uh oh!

HuaHuaY commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

wgtmac commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

wgtmac commented Jul 28, 2026

Uh oh!

Uh oh!

HuaHuaY commented Jun 27, 2026 •

edited by github-actions Bot

Loading

HuaHuaY Jul 9, 2026 •

edited

Loading

HuaHuaY Jul 22, 2026 •

edited

Loading

HuaHuaY commented Jul 27, 2026 •

edited

Loading

HuaHuaY commented Jul 27, 2026 •

edited

Loading