Track unknown logical type compatibility across Parquet implementations

[kevinjqliu]

Background

Parquet classifies new logical types as forward compatible:

"New logical types are considered forward compatible despite the loss of semantic meaning."

-- https://parquet.apache.org/docs/file-format/versions/#compatibility-categories

Readers that do not recognize a logical type should still read the underlying physical representation rather than reject the file.

This is increasingly important for VARIANT, GEOMETRY, GEOGRAPHY, and future logical types.

Related discussion:

• https://lists.apache.org/thread/flfzz94ftdrdop9d5b0o1hkqkprzj3l5
• An older parquet-java reader (1.15.2) failed to read a Parquet file containing the newer VARIANT logical type. The reader rejected the file instead of ignoring the unknown logical annotation and exposing the underlying physical representation.

Golden test files from parquet-testing

File	Purpose
unknown-logical-type.parquet	File containing an unknown logical type
case-001.parquet	File containing the VARIANT logical type. Readers that do not recognize the VARIANT logical type should treat it as an unknown logical type.

Expected behavior:

• File metadata and schema can be parsed.
• Underlying physical data can be read.
• Unsupported logical semantics are ignored.
• The reader does not fail solely because the logical type is unknown.

Compatibility status (updated as of July 20, 2026)

Implementation list from https://parquet.apache.org/docs/file-format/implementationstatus/

Implementation	Version tested	unknown-logical-type.parquet	case-001.parquet (VARIANT)	Issue / PR
parquet-java	1.17.0	❌ Unknown types: #2709 (open; no fix PR) [1] ✅	✅ VARIANT: #3072 (merged in 1.16.0)	❌ Older-reader VARIANT: #3633 (open) [2] ✅ Fixed in apache/parquet-java#3669, should be available in 1.18.x
Apache Arrow C++	25.0.0	✅ Unknown types: #41764 / #41765	✅ VARIANT: #45937, #50121, #50122, #50232	✅
Apache Arrow Rust (arrow-rs)	59.1.0	✅ Unknown types: #8776 / #8777	✅ VARIANT: #6736, #8408, #8480	✅
Apache Arrow Go	18.6.0 / main (f7bacb4)	❌ Unknown types: no Arrow Go issue/PR found; related upstream #41764 / #41765	✅ VARIANT: #310, #352, #433, #434, #455	❌
DuckDB	v1.5.4	✅ Unknown types: #16378	✅ VARIANT support: #18187, #18224, #18996	✅
Polars	1.42.1	❌ Unknown types: fails during footer metadata parsing; no exact issue/PR (related #20978)	❌ VARIANT: fails during footer metadata parsing; no exact issue/PR (related #20978, analogous #27893 / #27896)	❌
cuDF	26.06.01	✅ Unknown types: existing skip/fallback behavior documented in #22310 / 893daed	✅ VARIANT: #22312, #22310, #22416	✅
Hyparquet	1.26.2	✅ Unknown types: 8a98407 / 892c933	✅ VARIANT: #149, 6290e85, 45a7d5e	✅

[1] Fails during footer schema conversion, before projection or decoding. This is inconsistent with Parquet's forward-compatibility expectation for unknown logical annotations.
[2] Pre-1.16 versions do not understand VARIANT, so it behaves like an unknown logical type and fails during footer conversion, even if unprojected.

Completion criteria

• Record results and tested versions for each implementation.
• Open upstream issues for failures.
• Add regression tests using the shared parquet-testing files.
• Backport fixes where practical.

[kevinjqliu]

Repros

Parquet-java ❌

Setup, in the parquet-java folder:

git clone --branch apache-parquet-1.17.0 --depth 1 \
  https://github.com/apache/parquet-java.git
cd parquet-java

# My laptop has Thrift 0.23; this should not affect the result.
./mvnw -Plocal -pl parquet-cli -am -DskipTests \
  -Dthrift.version=0.23.0 \
  -Dformat.thrift.version=0.23.0 \
  package

CLI="$PWD/parquet-cli/target/parquet-cli-1.17.0-runtime.jar"

Run

java -jar "$CLI" version

for file in unknown-logical-type.parquet case-001.parquet; do
  echo "Testing: $file"
  java -jar "$CLI" schema --parquet "$file"
  java -jar "$CLI" cat -n 1 "$file"
done

parquet-mr version 1.17.0 (build fac0c746532e133beb928a7f6a7e57b510b477a1)
Testing: /Users/kevinliu/Downloads/unknown-logical-type.parquet
Unknown error
java.lang.NullPointerException: Cannot invoke "org.apache.parquet.format.LogicalType$_Fields.ordinal()" because the return value of "org.apache.parquet.format.LogicalType.getSetField()" is null
	at org.apache.parquet.format.converter.ParquetMetadataConverter.getLogicalTypeAnnotation(ParquetMetadataConverter.java:1320)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.buildChildren(ParquetMetadataConverter.java:2049)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetSchema(ParquetMetadataConverter.java:1997)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:1827)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:1786)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:649)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:578)
	at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:971)
	at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:961)
	at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:857)
	at org.apache.parquet.cli.commands.SchemaCommand.getParquetSchema(SchemaCommand.java:117)
	at org.apache.parquet.cli.commands.SchemaCommand.getSchema(SchemaCommand.java:102)
	at org.apache.parquet.cli.commands.SchemaCommand.run(SchemaCommand.java:82)
	at org.apache.parquet.cli.Main.run(Main.java:217)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.parquet.cli.Main.main(Main.java:245)
Unknown error
java.lang.NullPointerException: Cannot invoke "org.apache.parquet.format.LogicalType$_Fields.ordinal()" because the return value of "org.apache.parquet.format.LogicalType.getSetField()" is null
	at org.apache.parquet.format.converter.ParquetMetadataConverter.getLogicalTypeAnnotation(ParquetMetadataConverter.java:1320)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.buildChildren(ParquetMetadataConverter.java:2049)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetSchema(ParquetMetadataConverter.java:1997)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:1827)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:1786)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:649)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:578)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:570)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:564)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:508)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:490)
	at org.apache.parquet.cli.util.Schemas.fromParquet(Schemas.java:74)
	at org.apache.parquet.cli.BaseCommand.getAvroSchema(BaseCommand.java:545)
	at org.apache.parquet.cli.commands.CatCommand.runWithAvroSchema(CatCommand.java:81)
	at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:67)
	at org.apache.parquet.cli.Main.run(Main.java:217)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.parquet.cli.Main.main(Main.java:245)
Testing: /Users/kevinliu/Downloads/case-001.parquet
message table {
  required int32 id = 1;
  optional group var (VARIANT(1)) = 2 {
    required binary metadata;
    optional binary value;
    optional group typed_value (LIST) {
      repeated group list {
        required group element {
          optional binary value;
          optional binary typed_value (STRING);
        }
      }
    }
  }
}

Unknown error
java.lang.RuntimeException: Failed on record 0 in file /Users/kevinliu/Downloads/case-001.parquet
	at org.apache.parquet.cli.commands.CatCommand.runWithAvroSchema(CatCommand.java:101)
	at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:67)
	at org.apache.parquet.cli.Main.run(Main.java:217)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.parquet.cli.Main.main(Main.java:245)
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file file:/Users/kevinliu/Downloads/case-001.parquet
	at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:280)
	at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:136)
	at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:140)
	at org.apache.parquet.cli.BaseCommand$2$1.advance(BaseCommand.java:505)
	at org.apache.parquet.cli.BaseCommand$2$1.<init>(BaseCommand.java:486)
	at org.apache.parquet.cli.BaseCommand$2.iterator(BaseCommand.java:484)
	at org.apache.parquet.cli.commands.CatCommand.runWithAvroSchema(CatCommand.java:88)
	... 4 more
Caused by: java.lang.IndexOutOfBoundsException: Index -1 out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
	at java.base/java.util.Objects.checkIndex(Objects.java:361)
	at java.base/java.util.ArrayList.get(ArrayList.java:427)
	at org.apache.parquet.io.GroupColumnIO.getLast(GroupColumnIO.java:94)
	at org.apache.parquet.io.GroupColumnIO.getLast(GroupColumnIO.java:94)
	at org.apache.parquet.io.GroupColumnIO.getLast(GroupColumnIO.java:94)
	at org.apache.parquet.io.PrimitiveColumnIO.getLast(PrimitiveColumnIO.java:72)
	at org.apache.parquet.io.PrimitiveColumnIO.isLast(PrimitiveColumnIO.java:66)
	at org.apache.parquet.io.RecordReaderImplementation.<init>(RecordReaderImplementation.java:308)
	at org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:141)
	at org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:105)
	at org.apache.parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:186)
	at org.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:105)
	at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:156)
	at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:245)
	... 10 more

Apache Arrow C++ (via pyarrow) ✅

uv run --with pyarrow==25.0.0 python -c '
import pyarrow as pa
print(f"PyArrow {pa.__version__}")
print(f"Arrow C++ {pa.cpp_version}")
'

for file in unknown-logical-type.parquet case-001.parquet; do
  echo "Testing: $file"

  FILE="$file" uv run --with pyarrow==25.0.0 python - <<'PY'
import os
import pyarrow.parquet as pq

path = os.environ["FILE"]
parquet_file = pq.ParquetFile(path)

print("\nParquet schema:")
print(parquet_file.schema)

table = parquet_file.read()

print("\nArrow schema:")
print(table.schema)

print("\nData:")
print(table.to_pylist())

table.validate(full=True)
print("\nResult: OK")
PY
done

PyArrow 25.0.0
Arrow C++ 25.0.0
Testing: unknown-logical-type.parquet

Parquet schema:
<pyarrow._parquet.ParquetSchema object at 0x102a25800>
required group field_id=-1 schema {
  optional binary field_id=-1 column with known type (String);
  optional binary field_id=-1 column with unknown type (UNKNOWN);
}


Arrow schema:
column with known type: string
column with unknown type: binary
  -- field metadata --
  ARROW:extension:metadata: '{}'
  ARROW:extension:name: 'geoarrow.wkb'

Data:
[{'column with known type': 'known string 1', 'column with unknown type': b'unknown string 1'}, {'column with known type': 'known string 2', 'column with unknown type': b'unknown string 2'}, {'column with known type': 'known string 3', 'column with unknown type': b'unknown string 3'}]

Result: OK
Testing: case-001.parquet

Parquet schema:
<pyarrow._parquet.ParquetSchema object at 0x1034a0e80>
required group field_id=-1 table {
  required int32 field_id=1 id;
  optional group field_id=2 var (Variant(1)) {
    required binary field_id=-1 metadata;
    optional binary field_id=-1 value;
    optional group field_id=-1 typed_value (List) {
      repeated group field_id=-1 list {
        required group field_id=-1 element {
          optional binary field_id=-1 value;
          optional binary field_id=-1 typed_value (String);
        }
      }
    }
  }
}


Arrow schema:
id: int32 not null
  -- field metadata --
  PARQUET:field_id: '1'
var: struct<metadata: binary not null, value: binary, typed_value: list<element: struct<value: binary, ty (... 29 chars omitted)
  child 0, metadata: binary not null
  child 1, value: binary
  child 2, typed_value: list<element: struct<value: binary, typed_value: string> not null>
      child 0, element: struct<value: binary, typed_value: string> not null
          child 0, value: binary
          child 1, typed_value: string
  -- field metadata --
  PARQUET:field_id: '2'
-- schema metadata --
parquet.avro.schema: '{"type":"record","name":"table","fields":[{"name":"' + 442
writer.model.name: 'avro'

Data:
[{'id': 1, 'var': {'metadata': b'\x01\x00\x00', 'value': None, 'typed_value': [{'value': None, 'typed_value': 'comedy'}, {'value': None, 'typed_value': 'drama'}]}}]

Result: OK

Apache Arrow Rust (arrow-rs) ✅

cargo install parquet --version 59.1.0 --features cli --locked

parquet-read --version

for file in unknown-logical-type.parquet case-001.parquet; do
  echo "Testing: $file"
  parquet-schema "$file"
  parquet-read --json "$file"
done

parquet 59.1.0
Testing: unknown-logical-type.parquet
Metadata for file: unknown-logical-type.parquet

version: 2
num of rows: 3
created by: parquet-cpp-arrow version 20.0.0-SNAPSHOT
metadata:
  ARROW:schema: /////2ABAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAMAAAACAAIAAAABAAIAAAABAAAAAIAAADwAAAAGAAAAAAAEgAYAAgABgAHAAwAAAAQABQAEgAAAAAAAQQUAAAAtAAAAAgAAAAoAAAAAAAAABgAAABjb2x1bW4gd2l0aCB1bmtub3duIHR5cGUAAAAAAgAAAEwAAAAEAAAAxP///xwAAAAEAAAADAAAAGdlb2Fycm93LndrYgAAAAAUAAAAQVJST1c6ZXh0ZW5zaW9uOm5hbWUAAAAACAAMAAQACAAIAAAAEAAAAAQAAAACAAAAe30AABgAAABBUlJPVzpleHRlbnNpb246bWV0YWRhdGEAAAAAuP///xAAFAAIAAYABwAMAAAAEAAQAAAAAAABBRAAAAAsAAAABAAAAAAAAAAWAAAAY29sdW1uIHdpdGgga25vd24gdHlwZQAABAAEAAQAAAAAAAAA
message schema {
  OPTIONAL BYTE_ARRAY column with known type (STRING);
  OPTIONAL BYTE_ARRAY column with unknown type (_Unknown(2555));
}
{"column with known type":"known string 1","column with unknown type":"dW5rbm93biBzdHJpbmcgMQ=="}
{"column with known type":"known string 2","column with unknown type":"dW5rbm93biBzdHJpbmcgMg=="}
{"column with known type":"known string 3","column with unknown type":"dW5rbm93biBzdHJpbmcgMw=="}
Testing: case-001.parquet
Metadata for file: case-001.parquet

version: 1
num of rows: 1
created by: parquet-mr version 1.16.0-SNAPSHOT (build ee34713e4d906d61f95d2b09145945638b2e2296)
metadata:
  parquet.avro.schema: {"type":"record","name":"table","fields":[{"name":"id","type":"int"},{"name":"var","type":["null",{"type":"record","name":"var","fields":[{"name":"metadata","type":"bytes"},{"name":"value","type":["null","bytes"],"default":null},{"name":"typed_value","type":["null",{"type":"array","items":{"type":"record","name":"element","fields":[{"name":"value","type":["null","bytes"],"default":null},{"name":"typed_value","type":["null","string"],"default":null}]}}],"default":null}]}],"default":null}]}
  writer.model.name: avro
message table {
  REQUIRED INT32 id [1];
  OPTIONAL group var [2] (VARIANT(Some(1))) {
    REQUIRED BYTE_ARRAY metadata;
    OPTIONAL BYTE_ARRAY value;
    OPTIONAL group typed_value (LIST) {
      REPEATED group list {
        REQUIRED group element {
          OPTIONAL BYTE_ARRAY value;
          OPTIONAL BYTE_ARRAY typed_value (STRING);
        }
      }
    }
  }
}
{"id":1,"var":{"metadata":"AQAA","typed_value":[{"typed_value":"comedy","value":null},{"typed_value":"drama","value":null}],"value":null}}

Apache Arrow Go ❌

Setup:

ARROW_GO_VERSION=v18.6.0  # use main to test the development version

go version
go list -m "github.com/apache/arrow-go/v18@${ARROW_GO_VERSION}"

for file in unknown-logical-type.parquet case-001.parquet; do
  echo "Testing: $file"
  go run "github.com/apache/arrow-go/v18/parquet/cmd/parquet_reader@${ARROW_GO_VERSION}" \
    --no-metadata "$file"
done

Testing: unknown-logical-type.parquet
panic: invalid logical type

goroutine 1 [running]:
github.com/apache/arrow-go/v18/parquet/schema.getLogicalType(0x14000124000?)
	/Users/kevinliu/go/pkg/mod/github.com/apache/arrow-go/v18@v18.6.0/parquet/schema/logical_types.go:79 +0x344
github.com/apache/arrow-go/v18/parquet/schema.PrimitiveNodeFromThrift(0x14000114180)
	/Users/kevinliu/go/pkg/mod/github.com/apache/arrow-go/v18@v18.6.0/parquet/schema/node.go:273 +0x6c
github.com/apache/arrow-go/v18/parquet/schema.FromParquet.func1()
	/Users/kevinliu/go/pkg/mod/github.com/apache/arrow-go/v18@v18.6.0/parquet/schema/schema.go:94 +0xb4
github.com/apache/arrow-go/v18/parquet/schema.FromParquet.func1()
	/Users/kevinliu/go/pkg/mod/github.com/apache/arrow-go/v18@v18.6.0/parquet/schema/schema.go:99 +0x174
github.com/apache/arrow-go/v18/parquet/schema.FromParquet({0x14000118018?, 0x103a186c0?, 0x1400011a001?})
	/Users/kevinliu/go/pkg/mod/github.com/apache/arrow-go/v18@v18.6.0/parquet/schema/schema.go:109 +0xf8
github.com/apache/arrow-go/v18/parquet/metadata.(*FileMetaData).initSchema(0x14000136080)
	/Users/kevinliu/go/pkg/mod/github.com/apache/arrow-go/v18@v18.6.0/parquet/metadata/file.go:360 +0x2c
github.com/apache/arrow-go/v18/parquet/metadata.NewFileMetaData({0x14000120000, 0x354, 0x354}, {0x0, 0x0})
	/Users/kevinliu/go/pkg/mod/github.com/apache/arrow-go/v18@v18.6.0/parquet/metadata/file.go:316 +0x13c
github.com/apache/arrow-go/v18/parquet/file.(*Reader).parseMetaData(0x1400011a000)
	/Users/kevinliu/go/pkg/mod/github.com/apache/arrow-go/v18@v18.6.0/parquet/file/file_reader.go:194 +0x5c8
github.com/apache/arrow-go/v18/parquet/file.NewParquetReader({0x103a2a200, 0x1400010e030}, {0x0, 0x0, 0x10421cb01?})
	/Users/kevinliu/go/pkg/mod/github.com/apache/arrow-go/v18@v18.6.0/parquet/file/file_reader.go:124 +0x18c
github.com/apache/arrow-go/v18/parquet/file.OpenParquetFile({0x16ceeb203?, 0x140000306a0?}, 0xe0?, {0x0, 0x0, 0x0})
	/Users/kevinliu/go/pkg/mod/github.com/apache/arrow-go/v18@v18.6.0/parquet/file/file_reader.go:95 +0x7c
main.main()
	/Users/kevinliu/go/pkg/mod/github.com/apache/arrow-go/v18@v18.6.0/parquet/cmd/parquet_reader/main.go:148 +0x870
exit status 2
Testing: case-001.parquet
id                |metadata          |value             |value             |typed_value       |
1                 |01 00 00          |NULL              |NULL              |comedy            |
                  |                  |                  |NULL              |drama             |

Duckdb ✅

duckdb --version

for file in unknown-logical-type.parquet case-001.parquet; do
  echo "Testing: $file"
  duckdb -c "
    DESCRIBE SELECT * FROM read_parquet('$file');
    SELECT * FROM read_parquet('$file');
  "
done

v1.5.4 (Variegata) 08e34c447b
Testing: unknown-logical-type.parquet
┌──────────────────────────────────┐
│             Describe             │
│                                  │
│ column with known type   varchar │
│ column with unknown type blob    │
└──────────────────────────────────┘
┌────────────────────────┬──────────────────────────┐
│ column with known type │ column with unknown type │
│        varchar         │           blob           │
├────────────────────────┼──────────────────────────┤
│ known string 1         │ unknown string 1         │
│ known string 2         │ unknown string 2         │
│ known string 3         │ unknown string 3         │
└────────────────────────┴──────────────────────────┘
Testing: case-001.parquet
┌─────────────┐
│  Describe   │
│             │
│ id  integer │
│ var variant │
└─────────────┘
┌───────┬─────────────────┐
│  id   │       var       │
│ int32 │     variant     │
├───────┼─────────────────┤
│     1 │ [comedy, drama] │
└───────┴─────────────────┘

Polars ❌

uv run --no-project --with polars==1.42.1 \
  python -c "import polars as pl; print(pl.__version__)"

for file in unknown-logical-type.parquet case-001.parquet; do
  echo "Testing: $file"
  FILE="$file" uv run --no-project --with polars==1.42.1 python -c '
import os
import polars as pl

path = os.environ["FILE"]
print("Schema:", pl.read_parquet_schema(path))
print(pl.read_parquet(path))
'
done

1.42.1
Testing: unknown-logical-type.parquet
Traceback (most recent call last):
  File "<string>", line 6, in <module>
    print("Schema:", pl.read_parquet_schema(path))
                     ~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/kevinliu/.cache/uv/archive-v0/ayxxJlFFv9bnpc53rREUn/lib/python3.13/site-packages/polars/io/parquet/functions.py", line 388, in read_parquet_schema
    return scan_parquet(source).collect_schema()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/kevinliu/.cache/uv/archive-v0/ayxxJlFFv9bnpc53rREUn/lib/python3.13/site-packages/polars/lazyframe/frame.py", line 2800, in collect_schema
    return Schema(self._ldf.collect_schema(), check_dtypes=False)
                  ~~~~~~~~~~~~~~~~~~~~~~~~^^
polars.exceptions.ComputeError: parquet: File out of specification: LogicalType union has no variant set
Testing: case-001.parquet
Traceback (most recent call last):
  File "<string>", line 6, in <module>
    print("Schema:", pl.read_parquet_schema(path))
                     ~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/kevinliu/.cache/uv/archive-v0/ayxxJlFFv9bnpc53rREUn/lib/python3.13/site-packages/polars/io/parquet/functions.py", line 388, in read_parquet_schema
    return scan_parquet(source).collect_schema()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/kevinliu/.cache/uv/archive-v0/ayxxJlFFv9bnpc53rREUn/lib/python3.13/site-packages/polars/lazyframe/frame.py", line 2800, in collect_schema
    return Schema(self._ldf.collect_schema(), check_dtypes=False)
                  ~~~~~~~~~~~~~~~~~~~~~~~~^^
polars.exceptions.ComputeError: parquet: File out of specification: LogicalType union has no variant set

cuDF ✅

No repro: cuDF only provides Linux CUDA wheels and requires an NVIDIA GPU. It cannot run natively on macOS/Apple Silicon

Hyparquet ✅

Setup:

node --version
npm install --no-save hyparquet@1.26.2
npm list hyparquet --depth=0

for file in unknown-logical-type.parquet case-001.parquet; do
  echo "Testing: $file"

  FILE="$file" node --input-type=module -e '
    import {
      asyncBufferFromFile,
      parquetMetadataAsync,
      parquetReadObjects,
    } from "hyparquet";

    const file = await asyncBufferFromFile(process.env.FILE);
    const metadata = await parquetMetadataAsync(file);

    console.log("Schema:");
    console.dir(metadata.schema, { depth: null });

    console.log("Rows:");
    const rows = await parquetReadObjects({ file });
    console.dir(rows, { depth: null });
  '
done

v26.4.0

added 1 package in 335ms
/Users/kevinliu/Downloads
└── hyparquet@1.26.2 extraneous

Testing: unknown-logical-type.parquet
Schema:
[
  {
    type: undefined,
    type_length: undefined,
    repetition_type: 'REQUIRED',
    name: 'schema',
    num_children: 2,
    converted_type: undefined,
    scale: undefined,
    precision: undefined,
    field_id: undefined,
    logical_type: undefined
  },
  {
    type: 'BYTE_ARRAY',
    type_length: undefined,
    repetition_type: 'OPTIONAL',
    name: 'column with known type',
    num_children: undefined,
    converted_type: 'UTF8',
    scale: undefined,
    precision: undefined,
    field_id: undefined,
    logical_type: { type: 'STRING' }
  },
  {
    type: 'BYTE_ARRAY',
    type_length: undefined,
    repetition_type: 'OPTIONAL',
    name: 'column with unknown type',
    num_children: undefined,
    converted_type: undefined,
    scale: undefined,
    precision: undefined,
    field_id: undefined,
    logical_type: { field_2555: {} }
  }
]
Rows:
[
  {
    'column with known type': 'known string 1',
    'column with unknown type': 'unknown string 1'
  },
  {
    'column with known type': 'known string 2',
    'column with unknown type': 'unknown string 2'
  },
  {
    'column with known type': 'known string 3',
    'column with unknown type': 'unknown string 3'
  }
]
Testing: case-001.parquet
Schema:
[
  {
    type: undefined,
    type_length: undefined,
    repetition_type: undefined,
    name: 'table',
    num_children: 2,
    converted_type: undefined,
    scale: undefined,
    precision: undefined,
    field_id: undefined,
    logical_type: undefined
  },
  {
    type: 'INT32',
    type_length: undefined,
    repetition_type: 'REQUIRED',
    name: 'id',
    num_children: undefined,
    converted_type: undefined,
    scale: undefined,
    precision: undefined,
    field_id: 1,
    logical_type: undefined
  },
  {
    type: undefined,
    type_length: undefined,
    repetition_type: 'OPTIONAL',
    name: 'var',
    num_children: 3,
    converted_type: undefined,
    scale: undefined,
    precision: undefined,
    field_id: 2,
    logical_type: { type: 'VARIANT', specification_version: 1 }
  },
  {
    type: 'BYTE_ARRAY',
    type_length: undefined,
    repetition_type: 'REQUIRED',
    name: 'metadata',
    num_children: undefined,
    converted_type: undefined,
    scale: undefined,
    precision: undefined,
    field_id: undefined,
    logical_type: undefined
  },
  {
    type: 'BYTE_ARRAY',
    type_length: undefined,
    repetition_type: 'OPTIONAL',
    name: 'value',
    num_children: undefined,
    converted_type: undefined,
    scale: undefined,
    precision: undefined,
    field_id: undefined,
    logical_type: undefined
  },
  {
    type: undefined,
    type_length: undefined,
    repetition_type: 'OPTIONAL',
    name: 'typed_value',
    num_children: 1,
    converted_type: 'LIST',
    scale: undefined,
    precision: undefined,
    field_id: undefined,
    logical_type: { type: 'LIST' }
  },
  {
    type: undefined,
    type_length: undefined,
    repetition_type: 'REPEATED',
    name: 'list',
    num_children: 1,
    converted_type: undefined,
    scale: undefined,
    precision: undefined,
    field_id: undefined,
    logical_type: undefined
  },
  {
    type: undefined,
    type_length: undefined,
    repetition_type: 'REQUIRED',
    name: 'element',
    num_children: 2,
    converted_type: undefined,
    scale: undefined,
    precision: undefined,
    field_id: undefined,
    logical_type: undefined
  },
  {
    type: 'BYTE_ARRAY',
    type_length: undefined,
    repetition_type: 'OPTIONAL',
    name: 'value',
    num_children: undefined,
    converted_type: undefined,
    scale: undefined,
    precision: undefined,
    field_id: undefined,
    logical_type: undefined
  },
  {
    type: 'BYTE_ARRAY',
    type_length: undefined,
    repetition_type: 'OPTIONAL',
    name: 'typed_value',
    num_children: undefined,
    converted_type: 'UTF8',
    scale: undefined,
    precision: undefined,
    field_id: undefined,
    logical_type: { type: 'STRING' }
  }
]
Rows:
[ { id: 1, var: [ 'comedy', 'drama' ] } ]

[kevinjqliu]

The simplest way to reproduce this issue is with parquet-cli and https://github.com/apache/parquet-testing/blob/master/data/unknown-logical-type.parquet:

brew install parquet-cli
parquet --version
parquet schema ~/Downloads/unknown-logical-type.parquet

parquet-mr version 1.17.1 (build 78a8d3230eb4769db93de5f2f2e18363c04cae81)

Unknown error
java.lang.NullPointerException: Cannot invoke "org.apache.parquet.format.LogicalType$_Fields.ordinal()" because the return value of "org.apache.parquet.format.LogicalType.getSetField()" is null
	at org.apache.parquet.format.converter.ParquetMetadataConverter.getLogicalTypeAnnotation(ParquetMetadataConverter.java:1320)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.buildChildren(ParquetMetadataConverter.java:2049)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetSchema(ParquetMetadataConverter.java:1997)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:1827)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:1786)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:649)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:578)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:570)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:564)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:508)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:490)
	at org.apache.parquet.cli.util.Schemas.fromParquet(Schemas.java:74)
	at org.apache.parquet.cli.BaseCommand.getAvroSchema(BaseCommand.java:545)
	at org.apache.parquet.cli.commands.SchemaCommand.getSchema(SchemaCommand.java:104)
	at org.apache.parquet.cli.commands.SchemaCommand.run(SchemaCommand.java:82)
	at org.apache.parquet.cli.Main.run(Main.java:217)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.parquet.cli.Main.main(Main.java:245)

[kevinjqliu]

parquet-java is resolved in apache/parquet-java#3669 and should be part of the its 1.18.0 release