Fixed reading nullable nested (struct) columns from Parquet/Arrow

[zaleslaw]

Fixes #2041

Fix reading nullable nested (struct) columns from Parquet/Arrow

Problem it fixes

When reading a Parquet or Arrow file, a nullable nested column (an "optional group",
e.g. record { x, y }) was read incorrectly. In rows where the group is absent, the
DataFrame showed fake data — {x:0, y:0}, or values leaked from another row — and the
nested columns were typed as non-null.

Cause: Arrow stores a separate "is-null" bit for the struct itself, independent of its
child arrays. The reader ignored this bit and read the child values directly. Under a
null row those child values are undefined, so we got garbage.

What this PR does

• The Arrow reader now checks the struct's own null bit. When a group is absent in a
  row, all of its child cells become null and the child column types become nullable.
• It works at any depth: a null parent nulls every descendant field. A null struct
  element inside a list is handled the same way.
• Required groups (never null) are read exactly as before — no change.
• The bug lived in the shared Arrow conversion code, so the fix covers Parquet, Arrow
  IPC and Feather.

How it works (short)

A ColumnGroup cannot be null per row, so the null is pushed down into the child
columns. A small recursive helper injectNullsAt sets the child cells to null:

• value column → the cell becomes null (type widened to nullable);
• nested group → recurse into its children;
• list-of-struct (frame) → the cell becomes an empty frame (a frame cannot hold null).
  It dispatches on the column kind (public ColumnGroup / FrameColumn interfaces, no
  impl details) and returns a column unchanged when there is nothing to null, so no data
  is copied when there are no nulls.

Files

• arrowReadingImpl.kt — the fix: consult the struct validity buffer, injectNullsAt
  helper, and the same handling for null struct elements in readListVector; removed a
  piece of dead code.
• arrowReading.kt — KDoc note on readParquet about the behavior.
• docs/.../Parquet.md — updated (the old "Struct not supported" note was stale).
• Tests: new ArrowNullableStructTest.kt (all the nullable-struct cases), a shared
  arrowTestUtils.kt helper, the committed fixture
  nullable_nested_struct.parquet + TestFiles.md describing it.
• No public API change (apiCheck passes).

Tests

• In-memory Arrow IPC tests that put non-zero hidden values under a null parent, so
  a fix relying on zeros would be caught; one runs in NullabilityOptions.Checking.
• A null struct element inside a list.
• A "boss" case: a 3-level struct with value / list / list-of-struct / nested-group
  children and nulls at two levels at once.
• End-to-end Parquet: one test generates the data in code and one reads a committed
  fixture. The fixture is produced by our own code (Arrow DatasetFileWriter) and can
  be regenerated on any OS; DuckDB / the DataFrame writer can't produce this layout, so
  they are not used here.
• A test documenting the limitation below; plus Feather/IPC round-trip checks.
• Each test/fixture is documented with its pseudo-schema and a small data "head".

Known limitations (not fixed here, on purpose)

• A ColumnGroup cannot be null per row, so an absent group and a present group with
  all-null children look the same ({x:null, y:null}). The exact type {x:Int, y:Int}?
  from the issue would need a larger change in core.
• Writing a struct-level null back to Arrow is still not supported (the writer marks
  every struct row as present).

[Copilot]

Pull request overview

Fixes nullable Arrow/Parquet struct reads by propagating parent nulls into nested child columns.

Changes:

• Adds recursive null propagation for structs and list elements.
• Adds comprehensive IPC, Feather, and Parquet tests and fixtures.
• Documents nullable struct representation limitations.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
arrowReadingImpl.kt	Implements struct null propagation.
arrowReading.kt	Documents Parquet struct behavior.
ArrowNullableStructTest.kt	Adds nullable-struct tests.
arrowTestUtils.kt	Adds Arrow serialization helpers.
ArrowKtTest.kt	Reuses shared test helpers.
TestFiles.md	Documents test fixtures.
nullable_nested_struct.parquet	Adds regression fixture.
Parquet.md	Updates nested struct documentation.

Suppressed comments (5)

dataframe-arrow/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/ArrowKtTest.kt:57

• Ignore is unused after the fixture-regeneration test was placed in ArrowNullableStructTest; this fails the ktlint unused-import check.

import org.junit.Ignore

dataframe-arrow/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/ArrowKtTest.kt:67

• The refactor removes the last use of Channels, while Files and StandardCopyOption are newly added but unused. Remove all three imports to satisfy the ktlint unused-import check.

import java.nio.channels.Channels
import java.nio.file.Files
import java.nio.file.StandardCopyOption

dataframe-arrow/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/ArrowKtTest.kt:24

• The new helper removes this file's last uses of VectorSchemaRoot, ArrowFileWriter, and ArrowStreamWriter; the newly added vector imports are also unused. Remove all six stale imports so the ktlint unused-import check can pass.

import org.apache.arrow.vector.VarCharVector
import org.apache.arrow.vector.VectorSchemaRoot
import org.apache.arrow.vector.complex.ListVector
import org.apache.arrow.vector.complex.StructVector

dataframe-arrow/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/ArrowNullableStructTest.kt:29

• ByteArrayOutputStream is unused because the shared Arrow test utility owns the output stream. This import will fail the ktlint unused-import check.

import java.io.ByteArrayOutputStream

dataframe-arrow/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/ArrowNullableStructTest.kt:32

• Channels is unused after stream serialization was moved into arrowTestUtils.kt; remove it to satisfy ktlint.

import java.nio.channels.Channels

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.