Feat/read write direct - #88
Conversation
| * domain, or if the codec pipeline does not allow addressing inner chunks | ||
| */ | ||
| @Nullable | ||
| public ByteBuffer readInnerChunkDirect(long[] innerChunkCoords) throws ZarrException { |
There was a problem hiding this comment.
Maybe the arg could be a long[2][] addressing first the shard and then the inner chunk?
There was a problem hiding this comment.
I would disagree here. The recursive top to bottom descent already happens at other points in the program, and adressing each nested shard, would be a complete overkill here!
Array.readInnerChunkDirect -> Gives us back the shard that it is on
ShardingIndexedCodec -> gives us the mid-shards
while nestedcodec L2 -> gives us the leaf where it is, which gves us the complete recursive top to bottom descent, already.
Therefore no need for another parameter!
| * The shape of the smallest unit this pipeline encodes independently. For a sharded array this | ||
| * is the inner chunk shape of the sharding codec; otherwise it is the chunk shape itself. | ||
| */ | ||
| public int[] innerChunkShape() { |
There was a problem hiding this comment.
Although rarely used, it is spec-compliant to have multiple nested sharding codecs. A single innerChunkShape would not suffice.
There was a problem hiding this comment.
innerChunkShape() recurses through nested sharding codecs and returns the innermost
addressable shape readInnerChunkEncoded splits the coordinate per level with.
So arbitrary nesting depth works see testDirectInnerChunkReadNestedSharding
| * domain, or if the codec pipeline does not allow addressing inner chunks | ||
| */ | ||
| @Nullable | ||
| public ByteBuffer readInnerChunkDirect(long[] innerChunkCoords) throws ZarrException { |
There was a problem hiding this comment.
I think a writeInnerChunkDirect could also be interesting. It would read and parse (no decode) an existing shard, swap out the bytes for one provided inner chunk, and write out the shard again.
However, users would likely want to write multiple inner chunks for which this would be inefficient. Instead, this could also be a builder pattern, where you start a session for a shard (read and parse existing), then write multiple inner chunks to a buffer, and finally write (commit) the whole shard. Similar to what ShardingIndexedCodec.encode does internally.
There was a problem hiding this comment.
completly agree, editing multiple chunks directly was done pretty inefficient. Added functionality to directly write to multiple chunks at the same time!
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
adjusted write to be more efficient. ÌnnerChunkWriter.java is now doing any sort of inner chunk writing, collecting all needed chunks and reconstructing the shard + metadata by going through all of it iteratively. |
2 participants
implements #85 now, direct reads and writes for arrays.
These methods skip the codec pipeline for neccesary methods, avoiding the decode, encode round trip, which would imply a quality loss for lossy codecs.