Skip to content

feat(udp): optional per-endpoint ZSTD compression - #78

Open
MatejFranceskin wants to merge 1 commit into
upstream-rebasefrom
feat/zstd-compressed-endpoints
Open

MatejFranceskin wants to merge 1 commit into
upstream-rebasefrom
feat/zstd-compressed-endpoints

Conversation

@MatejFranceskin

@MatejFranceskin MatejFranceskin commented Jul 28, 2026

Copy link
Copy Markdown

Jira: SW-2443

Adds optional per-UDP-endpoint ZSTD compression, controlled by two new endpoint options:

ZstdCompression = true
ZstdDictionary  = /path/to.dict   # optional trained dictionary

It lets an endpoint compress its traffic to save bandwidth on a constrained link (e.g. a telemetry radio).

Behavior

  • Each outgoing datagram (a coalesced batch of framed MAVLink messages) is compressed into one standalone ZSTD frame at level 3, optionally with a trained dictionary.
  • Time-critical msgids (HEARTBEAT, SYSTEM_TIME, PING, MISSION_CURRENT, COMMAND_LONG, COMMAND_ACK, TIMESYNC, CURRENT_EVENT_SEQUENCE, REQUEST_EVENT) and datagrams that don't shrink are sent uncompressed.
  • The receiver auto-detects the 4-byte ZSTD magic per datagram and decompresses; plain MAVLink passes through unchanged (no negotiation; mixed streams decode fine).
  • Both ends must enable ZstdCompression and, if used, use the same ZstdDictionary.

Design

Compression sits only at the UDP syscall layer (compress before sendto in flush_pending_msgs, decompress after recvfrom in _read_msg); the shared MAVLink framer/CRC path and UART/TCP endpoints are untouched. UDP datagram atomicity means one datagram == one frame. RX_BUF_MAX_SIZE is raised to 8 KiB so a decompressed coalesced batch fits. The ZSTD library types are kept out of headers (opaque void* in ZstdCodec).

Includes tools/train-zstd-dict.sh to train a dictionary from captured MAVLink traffic, a default dictionary installed to <datadir>/mavlink-router/, and a README "ZSTD Compression" section.

Testing

  • Codec unit round-trip: 9/9 (round-trip, magic detection, skip-list, size-fallback, plain pass-through, corrupt-frame drop).
  • Two-instance loopback: all messages round-trip; wire carries ZSTD frames (~50% smaller) plus a plain frame for skip-listed messages.
  • Cross-built for arm64 and validated on target: two-instance loopback, 81/81 messages round-trip, ZSTD magic confirmed in captured payloads.

@MatejFranceskin
MatejFranceskin force-pushed the feat/zstd-compressed-endpoints branch from 64485ab to e77ac8d Compare July 28, 2026 05:34
@MatejFranceskin MatejFranceskin changed the title feat(udp): per-endpoint ZSTD compression for ARF-A radio endpoints feat(udp): optional per-endpoint ZSTD compression Jul 28, 2026
@MatejFranceskin
MatejFranceskin force-pushed the feat/zstd-compressed-endpoints branch 2 times, most recently from 36540aa to ed75cbf Compare July 28, 2026 05:46
Add two per-UDP-endpoint options, ZstdCompression (bool) and ZstdDictionary
(path), to compress an endpoint's traffic and save bandwidth on a constrained
link.

- Each outgoing datagram (a coalesced batch of framed MAVLink messages) is
  compressed into one standalone ZSTD frame at level 3, optionally with a
  trained dictionary.
- Time-critical message ids (HEARTBEAT, SYSTEM_TIME, PING, MISSION_CURRENT,
  COMMAND_LONG, COMMAND_ACK, TIMESYNC, CURRENT_EVENT_SEQUENCE, REQUEST_EVENT),
  and any datagram that does not shrink, are sent uncompressed.
- The receiver auto-detects the 4-byte ZSTD magic per datagram and decompresses,
  passing plain MAVLink through unchanged (no negotiation; mixed streams decode).

libzstd is an optional dependency (meson feature 'zstd', auto): when present the
feature is built, otherwise it is compiled out cleanly and an endpoint that
requests ZstdCompression fails at startup with a clear message.

Compression sits only at the UDP syscall layer (compress before sendto in
flush_pending_msgs, decompress after recvfrom in _read_msg); the shared MAVLink
framer/CRC path and UART/TCP endpoints are untouched. UDP datagram atomicity
means one datagram == one frame. RX_BUF_MAX_SIZE is raised to 8 KiB so a
decompressed coalesced batch fits.

The dictionary is provided per deployment (via the ZstdDictionary path); no
dictionary is bundled. tools/train-zstd-dict.sh trains one from captured MAVLink
traffic, and the README has a "ZSTD Compression" section.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@MatejFranceskin
MatejFranceskin force-pushed the feat/zstd-compressed-endpoints branch from ed75cbf to 0f6f9d3 Compare July 28, 2026 06:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant