Add reversible predict-only (1/3) wavelet via Part 2 ATK signaling - #344
Add reversible predict-only (1/3) wavelet via Part 2 ATK signaling#344hmaarrfk wants to merge 3 commits into
Conversation
…rker <details><summary>Claude's draft</summary> Implements the "1/3 wavelet" discussed in issue aous72#261: the reversible 5/3 kernel with its update step nulled, so the low-pass subband of every decomposition holds the even-indexed samples of the previous resolution untouched. Decoding resolution level r of a losslessly-coded image yields exactly image[::2^r, ::2^r] -- no filtering, no values absent from the original -- while full-resolution decoding remains lossless. Intended for label/mask images where intermediate values are illegal. Encoder additions: - param_atk::init_rev13(): 2-step kernel (null update A=0,B=0,E=0 + 5/3 predict), ATK index 2. The null update is kept, rather than the step dropped, because with m_init = 0 the first synthesis step operates on the even-indexed subsequence, so a predict-only kernel needs an even number of steps. - param_atk::write(): ATK marker segment serialization (reversible, whole-sample symmetric, single-coefficient kernels). - public param_cod::set_wavelet_kern() / get_wavelet_kern(); write_headers emits the ATK segment and sets Rsiz (0x8000 | 0x20). - param_qcd::make_quant_steps and param_cap Ccap now route through is_reversible() so ATK-referenced kernels get correct QCD/CAP values (previously QCD was left uninitialized for wavelet_trans >= 2). Decode side is unchanged -- the existing generic ATK machinery already handles this kernel, so stock OpenJPH >= 0.30 decodes these codestreams bit-exactly (verified). New param_cod::is_predict_only() inspects the ATK lifting steps (never the file-local kernel index) so readers can reliably detect exact-subsampling codestreams from any encoder; verified against a Kakadu stream that uses ATK index 2 for a different kernel. All 86 ctest tests pass, including the Kakadu ATK/DFS decode tests. Resume this Claude session: ``` cd /home/mark/git/jp2k claude --resume 487c1e20-36f3-416f-8be1-437c25ed67fe ``` </details>
<details><summary>Claude's draft</summary> Adds tests/test_rev13.cpp (self-contained gtest, in-memory encode/decode, no reference codestreams needed), following the test_mixed_coc.cpp model: - LevelsAreExactSubsampling: full-resolution decode is lossless and every skipped-resolution decode equals image[::2^r, ::2^r], across even/odd dimensions including single-row and single-column images. - SignalingAndDetection: the main header carries an ATK marker segment, and a reader classifies the kernel from the ATK lifting steps (is_predict_only, is_reversible); Part 1 kernels classify as not predict-only and carry no ATK segment. - ImpulseAtOddPositionRev53LeaksRev13DoesNot: a handcrafted 8x8 image, zero except one impulse at (1, 1). The exact subsample image[::2, ::2] is all zeros; the test asserts the 5/3 kernel's update step leaks the impulse into its first resolution level (not the exact subsample) while the predict-only kernel returns exactly the all-zero subsample -- with both kernels lossless at full resolution. - Rev53RemainsLossless: guards the QCD/CAP code paths this feature reworked. All 90 ctest tests pass. Resume this Claude session: ``` cd /home/mark/git/jp2k claude --resume 487c1e20-36f3-416f-8be1-437c25ed67fe ``` </details>
…ernel <details><summary>Claude's draft</summary> Three conformance fixes found while checking this work against the freely-available 2002 edition of T.801: - T.801 Table A.27 sizes the Batk parameter by the coefficient type Coeff_Typ, but param_atk::read() consumed a fixed 16-bit field for all coefficient types (and the new write() mirrored that). Both now size Batk by Coeff_Typ. This matters for reading conformant streams that use 8-bit reversible kernel parameters. - T.801 Table A.2: Rsiz bit 5 (0x20) signals arbitrary decomposition styles and bit 7 (0x80) signals whole-sample symmetric kernels; param_siz::read() had the two assignments swapped, and the rev13 encoder used 0x20 where 0x80 is correct. Verified against a Kakadu-produced stream (Rsiz = 0xC0A0 for DFS + WS-ATK usage). - init_rev13 now uses 16-bit kernel coefficients (Coeff_Typ = 1, Satk = 0x5902, Latk = 17). With Coeff_Typ = 0 the previously written segment was 2 bytes longer than T.801 specifies (the Batk issue above); with Coeff_Typ = 1 the conformant layout is identical to what already-deployed OpenJPH releases parse, so the codestreams remain decodable by stock decoders (verified against the published ojph 0.9.1 wheels, which bundle stock OpenJPH). All 90 ctest tests pass. Resume this Claude session: ``` cd /home/mark/git/jp2k claude --resume 487c1e20-36f3-416f-8be1-437c25ed67fe ``` </details>
|
Pushed a conformance commit after checking the implementation against the freely-available 2002 edition of T.801:
All 90 ctest tests pass. |
|
Hi Mark, Thank you for PR and all the effort you put in. This requires careful attention. I need to find the time to think about it. More importantly, was this useful for your purpose? and how much is the improvement? Kind regards, |
It was! I added a test to "demonstrates" it by reading the lower levels and ensuring they are "bit exact" at the subsampled locations.
I actually ended up taking it a step further for my own work and implemented (with the help of AI of course) the rev12 transform. But the amount of changes there are insane, and I really don't to ask you to read through them.
Take all the time you need. I basically had to create my own PS. hmaarrfk#3 shows that you can get rid of much of your assembly written code for libhwy. Overall, my fork has about 2x the performance (barring me finding any bugs today) just because I sent AI loose on finding optimizations. PPS. hmaarrfk#3 thinks that T.801 conformance fixes found against the spec (Batk sized by Coeff_Typ; Rsiz bits 0x20/0x40/0x80 corrected). I haven't had time to look into it. But I think it found it when i asked it to build out an 16 bit codepath in addition to the 32bit and 64bit codepaths you have. Although I did use AI, i have read through this code and think that it is in the spirit of the rest of the codebase. It is "minimal" since the infrastructure to read through ATKs is already there, and mostly just adds the "write" path for this kind of tranform. |
Thank you for the discussion on the approach. This implements the 1/3 wavelet as a custom ATK, and reads the ATK to decode the image correctly.
Notably, we do NOT build a whole API to define the ATK, instead we define a OpenJPH specific wavelet that we can reference by name. Please let me know if you agree or disagree with this direction.
It seems to really help and your structure is really powerful that I think the changes are quite contained.
The comments are likely overly verbose, let me know if you want me to really trim them, but I think I mostly understand the changes.
Closes #261
Claude longer description
What this adds
A reversible predict-only ("1/3") wavelet: the 5/3 kernel with its update step nulled, signaled with a JPEG 2000 Part 2 ATK marker segment (index 2,
Rsiz |= 0x8000 | 0x20). With this kernel, the LL band of every decomposition holds the even-indexed samples of the previous resolution untouched, so decoding with r skipped resolutions returns exactlyimage[::2^r, ::2^r]— no filtering, no values absent from the original — while full-resolution decoding remains lossless. The intended use is AI-generated label/mask images (microscopy), where interpolated "in-between" values are illegal.param_atk::init_rev13()— the kernel. It keeps a null update step (A=0, B=0, E=0) rather than dropping it: withm_init = 0the first synthesis step operates on the even-indexed subsequence, so a predict-only kernel needs an even number of steps (a literal 1-step kernel corrupts reconstruction).param_atk::write()— ATK marker segment serialization (reversible, whole-sample symmetric, single-coefficient steps; byte layout mirrorsread()and was cross-checked against a Kakadu-generated ATK stream).param_cod::set_wavelet_kern()/get_wavelet_kern();write_headers()emits the ATK segment for kernels ≥ 2.param_qcd::make_quant_steps()andparam_capCcap now route throughis_reversible()so ATK-referenced kernels get correct QCD/CAP values (previously QCD was silently left uninitialized forwavelet_trans >= 2).param_cod::is_predict_only()— detects exact-subsampling kernels by inspecting the ATK lifting steps, never the kernel index (which is file-local in Part 2; your test suite'ssimple_dec_irv53_bhvhb_low_latency.jphuses index 2 for an irreversible 5/3, and correctly reportsfalse).What did not change
The decode path. Your existing generic ATK machinery handles this kernel as-is — we verified that stock OpenJPH (≥ 0.30) decodes these codestreams bit-exactly at every resolution level, so only encoders need this patch.
Testing
New self-contained library-API tests in
tests/test_rev13.cpp(in-memory encode/decode, no reference codestreams needed, modeled ontest_mixed_coc.cpp):image[::2^r, ::2^r]exactness at every level, for even/odd dimensions including single-row/column images;is_predict_only), with Part 1 kernels asserting the negative;All 90 ctest tests pass (including the Kakadu ATK/DFS decode tests); shared+static builds.
Happy to split, rename (e.g. the
rev13name), add tests in your preferred style, or adjust the API surface (e.g. decoupling the public kernel enum from the ATK index) — whatever makes review easiest.