Issue #8080 : Support Hive-style partitioning in the Parquet File Output transform

[vbhanuchander-lang]

Closes #8080.

Brings Hive-style partitioned writes to the standard Parquet File Output transform, so the
layout that Native Spark mode can already produce no longer requires the Spark engine.

What it does

Configure one or more incoming fields as partition fields and the transform writes

/datalake/sales/year=2026/month=08/part-00-0000.parquet
/datalake/sales/year=2026/month=09/part-00-0001.parquet

instead of a single file set. With no partition fields configured the transform behaves exactly as
before — same single file, same name, same columns.

Design decisions worth a reviewer's attention

Partition columns are not written into the files. This matches DataFrameWriter.partitionBy,
which is what the issue asks for parity with, and it is what a reader expects when it recovers the
column from the folder name. Configuring every field as a partition field is rejected rather than
producing empty files.

Two things surfaced while testing that are worth calling out, because neither is obvious.

HopVfs percent-decodes the path it is given. Writing region=EU%2FWest produced
region=EU/West/ on disk — two folder levels — because VFS resolves the path as a URI. The Hive
convention still wants %2F in the folder name, so the partition key keeps its Hive-style form and
only the string handed to VFS has its % doubled to %25; VFS decodes that back to a literal %.
A literal % in a value is escaped as %25 for the same round-trip reason.

A per-run token in the file name. Without it, two runs in Append mode both wrote
part-00-0000.parquet and the second silently replaced the first — which is the opposite of
appending. Each run now contributes eight hex characters to the name.

Nulls and awkward values follow the Hive convention. A null or empty partition value becomes
__HIVE_DEFAULT_PARTITION__, the same sentinel Hive and Spark use, so the output stays readable by
them. Values are percent-escaped for the characters that would otherwise break the layout — most
importantly /, so region=EU/West produces one folder named region=EU%2FWest rather than two
levels.

Write modes (Existing data in the dialog) decide what happens to data already in a partition
folder, and only apply while partitioning:

Mode	Behaviour
Append (default)	Leave existing files, add new ones. Same as the pre-partitioning behaviour.
Overwrite partitions	Empty a partition folder before this run's first file goes into it. Untouched partitions are left alone.
Fail if exists	Refuse to write if the partition folder already exists.
Overwrite all	Empty the whole base folder once, before the first file is written.

The subtle one is Overwrite partitions: it must clear a folder once per run, not once per
file. A partition that gets closed and reopened (see below) would otherwise delete the rows the same
run had just written to it. There is a test for exactly that.

Bounded memory: Maximum open partitions (default 10). Every open Parquet writer buffers up to
a full row group — the row group size defaults to 256 MB — so one writer per distinct partition
value would run a pipeline out of memory on a wide key. Spark avoids this by sorting; a streaming
transform cannot, so instead the transform keeps at most N writers open and closes the
least-recently-written one when it needs another. Reopening a partition starts a new part-... file,
which is why a partition can end up with several files. The row-count split (Split into parts)
applies per partition file as well.

Structure

The partitioned path is additive: processRow branches on whether partition fields are configured,
and the existing single-file code is untouched on the other side of that branch. Building the Avro
and Parquet schema was extracted out of openNewFile() into buildSchema() so both paths share it
without the partitioned path rebuilding it per file.

Tests

ParquetOutputPartitionTest, 16 tests (the module is 47 in total, all passing), covering the path building and escaping, the field
exclusion and its three rejection cases, and end-to-end writes that assert the folder layout on disk
and read the Parquet files back:

• one folder per distinct value, and the partition column absent from the file's schema while both
  matching rows are present
• a value containing / stays one folder level
• each write mode, including Overwrite partitions leaving a sibling partition untouched, and not
  deleting what the same run just wrote (forced by setting the open-partition limit to 1)
• six distinct values with a limit of two open partitions still writes every row
• with no partition fields, still a single file with all three columns

./mvnw -pl plugins/tech/parquet -am test is green, and spotless:check passes.

One unrelated one-character fix

messages_en_US.properties had ParquetOutputDialog.RowGroupSize.Label Row group size with no =,
so that dialog label never resolved. Fixed while adding keys to the same file — happy to split it
out if you would rather it went separately.

[mattcasters]

Added a few smaller changes: lombok and trailing \ for windows.

[vbhanuchander-lang]

Thanks @mattcasters, and thanks for pushing the changes rather than sending them back — all three
are improvements.

The trailing backslash is a real miss on my part: partitionFolder only checked for /, so a base
path typed with a Windows separator would have produced a double separator before the first
name=value level. Good catch, and your test pins it.

On Lombok: I hand-wrote the accessors to match ParquetField next door, which still has them
expanded. Happy to follow the Lombok convention — worth a separate sweep for the older classes in
this package at some point, but not in this PR.

Pulled your commit and re-ran locally: the full module run is 48 tests, 0 failures
(17 in ParquetOutputPartitionTest now, with your new one), and Hop PR Build (Code) is green on
b6c5a79.

[mattcasters]

Thanks for the feedback @vbhanuchander-lang. I had already approved the PR, was just waiting for github to give the green light. I really appreciate the high quality PRs!