Add PMTiles export - #12
Merged
Merged
Conversation
`tilefeed export <in.mbtiles> <out.pmtiles>` writes a PMTiles v3 archive: one file, served from S3/R2/any static host, read by clients with HTTP range requests. No tile server, no SQLite, no running process. Implemented from the v3 specification — 127-byte header, varint-column directories, Hilbert-ordered tile IDs — with the properties that make the format worth using: - identical tiles stored once (ocean and empty land collapse hard) - consecutive tile IDs sharing a blob collapse into one run entry - Hilbert ordering, so tiles near each other on the map are near each other in the file and a panning client reuses fetched ranges - leaf directories once the root would exceed the 16 KiB the spec allows, so a client can always fetch header and root in one request Validated against the reference implementation rather than only against itself: the Python `pmtiles` package reads an exported archive, parses the header and metadata, and returns all 29 tiles of the repo's fixture byte-identical to the source. A 60k-tile sparse archive exercising 15 leaf directories round-trips a 3000-tile sample identically, and every tile_id matches `zxy_to_tileid` across z0-z15. Export re-reads and verifies each archive before reporting success; a malformed one would otherwise only surface in a browser, somewhere else, later. `is_gzipped` moves to mvt.rs, which owns the gzip encoding, and is now shared with the tile server rather than duplicated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UDAsxXjaouazD2L4NhDxSb
The serious one: `tilefeed export a.mbtiles a.mbtiles` destroyed the source. Tiles are read before anything is written, so the output write truncated the still-open SQLite database, and verification then passed because it was reading the archive it had just written. Export now refuses an output that resolves to the input, and a test asserts the source survives. Also fixed: - **Compression was sampled from the first tile only.** A tilefeed MBTiles can hold both raw and gzipped tiles — Tippecanoe with `no_tile_compression` writes raw, incremental updates always gzip — so one header field was describing tiles it did not match, and clients would hand gzip bytes to an MVT parser. Archives are now uniformly gzipped, compressing raw tiles on the way in. - **Verification only checked the archive against itself.** It could not catch the coordinate-mapping bug class that this branch actually hit: such an archive is perfectly self-consistent and entirely wrong. Up to 64 tiles spread across the archive are now re-read and compared byte-for-byte with the source. - **An out-of-range tile_row panicked** in debug and wrapped silently in release, writing a mis-addressed archive. Coordinates outside their zoom's grid are rejected by name. Every arithmetic step on values read back out of an archive is checked, which a new test caught another instance of. - **The source was opened read-write**, flipping the journal to WAL and materializing Tippecanoe's view — so exporting a published artifact from read-only media failed outright. `open_read_only` avoids both, with an `immutable=1` fallback for a WAL database whose directory is not writable. Verified against a chmod-444 file in a chmod-555 directory. - **Memory held the archive two to three times over.** Tile blobs now stream through a staging file, and only coordinates are buffered for the Hilbert sort: a 164 MB archive of 40,000 tiles peaks at 28 MB. - `read_tile`/`verify_archive` took byte slices and indexed them with unvalidated header values. They are now `ArchiveReader`, generic over Read+Seek, so a malformed archive errors instead of panicking and nothing loads the whole file. - `get_tile_raw_tms` uses a cached statement; an export calls it once per tile. - The center falls back to the middle of the bounds rather than Null Island. - `test_zoom_bases_do_not_overlap` asserted an integer-division identity that held regardless of what `tile_id` returned. It now asks `tile_id` itself. - `serialize_directory` documents and debug-asserts its sortedness precondition. - Fixed the comment in `build_directories` describing the opposite of the code, and restored `encode_tile`'s doc comment, which the `is_gzipped` move displaced. - docs no longer claim the reference-implementation cross-check runs in CI; it is described as what it is, a manual check, with instructions to repeat it. Re-validated against the reference implementation after the rewrite: 29/29 tiles of the repo fixture and 2000 sampled tiles of a 40,000-tile archive read back byte-identical through the Python `pmtiles` package. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UDAsxXjaouazD2L4NhDxSb
CI runs `cargo clippy -- -D warnings`, which makes warnings errors; a plain `cargo clippy` does not, so this passed locally and failed there. Recorded the exact commands in CLAUDE.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UDAsxXjaouazD2L4NhDxSb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tilefeed export basemap.mbtiles basemap.pmtilesPMTiles packs a whole tileset into one file that clients read with HTTP range requests. Upload it to S3, R2, or any CDN and a map works with no tile server, no SQLite, no running process — the one distribution mode tilefeed couldn't do. It reads an MBTiles path directly, so it needs neither the database nor the config, composing like
inspectanddiff.What it implements
Written from the v3 spec — 127-byte header, varint-column directories, Hilbert-ordered tile IDs — with the properties that make the format worth using:
Export re-reads and verifies every archive before reporting success. A malformed one would otherwise surface in someone's browser, somewhere else, later.
Validation against the reference implementation
Self-consistency proves nothing here, so I checked against the Python
pmtilespackage (in an isolated venv):The repo's real Tippecanoe-built MBTiles:
A 60k-tile sparse archive, forcing 15 leaf directories:
Tile IDs — all 97 cases across z0–z15, including the spec's
(12, 3423, 1763) → 19078479, match the referencezxy_to_tileidexactly.A bug the tests caught
all_tile_coords()returns TMS rows and pairs withget_tile_raw_tms(), whileget_tile()takes XYZ and flips internally. My first version mixed them, so tiles were silently dropped. This is the hazard CLAUDE.md already warns about; I've added the specific pairing to that note.Tests
330 pass (up from 302); clippy
--all-targetsand fmt clean. 28 new: spec reference IDs, ID round-tripping, uniqueness and contiguity within a zoom, LEB128 varints, directory round-trip and the contiguous-offset shorthand, header byte offsets and rejection of bad magic/version/length, metadata flattening, leaf fallback and its reassembly, dedup and run collapsing end-to-end, empty input, and truncation detection.Limits (documented)
Export buffers the archive in memory — fine for a few GB, not tens; one level of leaf directories; a snapshot, so re-export after incremental updates.
🤖 Generated with Claude Code
https://claude.ai/code/session_01UDAsxXjaouazD2L4NhDxSb