Skip to content

[spark] Cache blob field lookup in SparkInternalRow - #8943

Open
zhuxiangyi wants to merge 1 commit into
apache:masterfrom
zhuxiangyi:spark-cache-blob-field-lookup
Open

[spark] Cache blob field lookup in SparkInternalRow#8943
zhuxiangyi wants to merge 1 commit into
apache:masterfrom
zhuxiangyi:spark-cache-blob-field-lookup

Conversation

@zhuxiangyi

Copy link
Copy Markdown
Contributor

What is the purpose of the change

This PR optimizes SparkInternalRow.create(rowType) for very wide nested RowTypes.

When Spark reads a Paimon table containing a wide struct, for example a struct with tens of thousands of fields, nested struct conversion can repeatedly create SparkInternalRow wrappers. Before this change, each creation scanned all fields in the RowType to find BLOB fields.

In production, executor threads can spend a significant amount of time in the following stack:

org.apache.paimon.types.RowType.getFieldCount
org.apache.paimon.spark.data.SparkInternalRow$.blobFields
org.apache.paimon.spark.data.SparkInternalRow$.create
org.apache.paimon.spark.DataConverter.fromPaimon
org.apache.paimon.spark.AbstractSparkInternalRow.getStruct

This repeated scan is expensive for very wide structs.

Brief change log

  • Cache BLOB field lookup results in SparkInternalRow.
  • Use IdentityHashMap[RowType, Set[Int]] so cache lookup does not call RowType.hashCode or RowType.equals, which may also traverse wide schemas.
  • Add a regression test to verify that repeated SparkInternalRow.create(rowType) calls for the same RowType scan the fields only once.

Verifying this change

This change added tests and can be verified as follows:

mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -Pspark3 \
  -DfailIfNoTests=false \
  -DwildcardSuites=org.apache.paimon.spark.data.SparkInternalRowTest \
  -Dtest=none test

Result:

BUILD SUCCESS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant