Repository navigation
[FEATURE] Dedicated faster serialization when shuffle writing/reading #2596
Description
Code of Conduct
- I agree to follow this project's Code of Conduct
Search before asking
- I have searched in the issues and found no similar issues.
Describe the feature
After overlapping compression is applied, the majority of shuffle write times is the seralization. Is it possible to replace the kyro serialization with the fast implementation like apache fory? It looks it has a impressive benchmark result .
Motivation
No response
Describe the solution
No response
Additional context
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Activity
zuston commentedon Aug 28, 2025
cc @jerqi
roryqi commentedon Aug 28, 2025
There are some points that:
roryqi commentedon Aug 28, 2025
cc @zhengchenyu
zuston commentedon Aug 28, 2025
Yes. I will draft a simple impl to benchmark
[apache#2596] feat(spark): Introduce fory serializer