Skip to content

JSON IO fix for null - #2056

Open
AndreiKingsley wants to merge 3 commits into
masterfrom
read_json_null_fix
Open

JSON IO fix for null#2056
AndreiKingsley wants to merge 3 commits into
masterfrom
read_json_null_fix

Conversation

@AndreiKingsley

@AndreiKingsley AndreiKingsley commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Fixes #2035
Fixes #2045
Fixes #2046
Fixes #2048

Now we parse null arrays as actually nulls, not empty lists.
Align with null writing as well.

See test updated tests for better understanding of new behavior.

But! There is still a kind of weird behavior:

DataFrame.readJsonStr( """[1,{"label":"record"},{"label":null}]""").toJson()

results into

[1,{"label":"record"},null]

due to our read implementation and array/value limitation. We need to completely rewrite it to prevent such data loss.
But I think this is totally acceptable at the moment — it’s an extremely rare occurrence, so I’ve documented it just in case


// A missing or `null` json array is read as `null`, so read it as an empty list
// when the schema requires the list to be non-nullable.
convertIf({ fromType, toSchema ->

@AndreiKingsley AndreiKingsley Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Jolanrensen with a new null json handling logic, openapi tests fails for schemas with non-nullalble List<T> type (that contains null that was parsed to empty lists with the old logic). Not sure wthether it's a problem from the current openapi schemas generation logic — maybe we need to align it with new null logic; this is the workaround suggested by AI. Please take a look to see if it is correct.

@AndreiKingsley
AndreiKingsley marked this pull request as ready for review September 2, 2026 11:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant