Skip to content

Commit 70b069d

Browse files
Rewrite the README as a lean intro and move detail into docs/
The README had grown into a dense reference (caps, stricter validations, asymmetric encode/decode), which is unwelcoming to a first-time visitor trying to decide whether the package is relevant. Replace it with a short overview: what the package is, who should use it (and who should reach for a higher layer instead), a minimal example, and the place of this package in the four-package dqlite stack with links. Move the still-useful detail into docs/: thread-safety (single-owner codec objects) and the deliberate divergences from the C server and go-dqlite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 8f2dbef commit 70b069d

3 files changed

Lines changed: 117 additions & 78 deletions

File tree

‎README.md‎

Lines changed: 39 additions & 78 deletions
Original file line numberDiff line numberDiff line change
@@ -1,101 +1,62 @@
11
# dqlite-wire
22

3-
Pure Python wire protocol implementation for [dqlite](https://dqlite.io/), Canonical's distributed SQLite.
3+
Pure-Python codec for the [dqlite](https://dqlite.io/) wire protocol —
4+
encode and decode the messages a dqlite server speaks.
5+
6+
[dqlite](https://dqlite.io/) is Canonical's distributed SQLite, built on
7+
Raft. This package implements the bytes-on-the-wire layer: it turns
8+
request/response objects into frames and back, following the
9+
[official wire-protocol specification](https://canonical.com/dqlite/docs/reference/wire-protocol).
10+
It does no networking, pooling, or SQL — just framing.
11+
12+
## Is this the package you want?
13+
14+
Probably not, unless you are building a driver or doing wire-level work
15+
(a proxy, traffic capture/replay, a custom client). If you just want to
16+
run SQL against dqlite from Python, use one of the higher layers — see
17+
[The dqlite Python stack](#the-dqlite-python-stack) below.
418

519
## Installation
620

721
```bash
822
pip install dqlite-wire
923
```
1024

25+
Requires Python 3.13+.
26+
1127
## Usage
1228

1329
```python
1430
from dqlitewire import encode_message, decode_message
1531
from dqlitewire.messages import LeaderRequest
1632

17-
# Encode a message
33+
# Encode a request to bytes
1834
data = encode_message(LeaderRequest())
1935

20-
# Decode a message
36+
# Decode bytes back into a message object
2137
message = decode_message(data, is_request=True)
2238
```
2339

24-
## Thread-safety
25-
26-
`ReadBuffer`, `WriteBuffer`, `MessageEncoder`, and `MessageDecoder`
27-
are **not thread-safe**. Each instance must be owned by a single
28-
thread or a single asyncio coroutine at a time. This matches Go's
29-
`driver.Conn` contract from go-dqlite.
30-
31-
Concurrent use of a single instance from multiple threads produces
32-
**silent data corruption** — not exceptions. The `is_poisoned`
33-
mechanism catches torn state from signal delivery during
34-
single-owner execution, but it **cannot** detect lost-update races
35-
between concurrent threads. Fuzz testing reliably reproduces both
36-
duplicate message delivery and corrupt (garbage) message bytes with
37-
no exception surfacing.
38-
39-
If you need concurrent access, wrap every call site in an
40-
`asyncio.Lock` or `threading.Lock` at the layer that owns the
41-
socket and decoder.
42-
43-
## Protocol Reference
44-
45-
Based on the [dqlite wire protocol specification](https://canonical.com/dqlite/docs/reference/wire-protocol).
46-
47-
## Deliberate divergences from upstream
48-
49-
This library implements the dqlite wire protocol faithfully but adds a
50-
handful of defensive guards that the upstream C server and the
51-
canonical [go-dqlite](https://github.com/canonical/go-dqlite) client do
52-
not. They protect a Python client running in potentially adversarial
53-
network contexts and are all opt-out-able.
54-
55-
**Python-specific caps** (not present in C or Go; `None` disables):
56-
57-
- `DEFAULT_MAX_TOTAL_ROWS` (`MessageDecoder(max_total_rows=...)`, default
58-
10,000,000) — cap on rows accumulated across continuation frames for
59-
one query. Importable from `dqlitewire`.
60-
- `DEFAULT_MAX_CONTINUATION_FRAMES` (`MessageDecoder(max_continuation_frames=...)`,
61-
default 100,000) — cap on continuation frames for one query.
62-
Importable from `dqlitewire`.
63-
- `RowsResponse.DEFAULT_MAX_ROWS` (`MessageDecoder(max_rows=...)`,
64-
default 1,000,000) — per-frame row cap. Class-scoped, not exported
65-
at module level.
66-
- `ReadBuffer.DEFAULT_MAX_MESSAGE_SIZE` (`ReadBuffer(max_message_size=...)`,
67-
default 64 MiB) — envelope cap on a single frame. Class-scoped, not
68-
exported at module level.
69-
- `_MAX_PARAM_COUNT` (32,766 — matches SQLite's
70-
`SQLITE_MAX_VARIABLE_NUMBER`), `_MAX_COLUMN_COUNT` (2000 — SQLite's
71-
documented `SQLITE_MAX_COLUMN` compile-time default), `_MAX_FILE_COUNT` (100),
72-
`_MAX_NODE_COUNT` (10,000) — internal sanity bounds on decoded
73-
tuple / response sizes.
74-
75-
**Stricter-than-Go validations** (match the C server's intent):
76-
77-
- `decode_row_header` requires the full 8-byte marker (C defines
78-
`DQLITE_RESPONSE_ROWS_DONE = 0xff..ff` / `_PART = 0xee..ee`;
79-
go-dqlite checks only the first byte).
80-
- `encode_value(value, ValueType.BOOLEAN)` rejects arbitrary ints
81-
(accepts only `bool` or exactly `0`/`1`).
82-
- `FilesResponse.encode_body` rejects non-8-aligned file content (C's
83-
`dumpFile` asserts `len % 8 == 0`).
84-
- `encode_params_tuple` rejects `ValueType.UNIXTIME` outbound (C's
85-
`tuple_decoder__next` cannot decode it on the server side).
86-
- `StmtResponse` rejects a 16-byte body when `schema=1` (C's V1
87-
response is 24 bytes).
88-
89-
**Asymmetric encode/decode** (decoded for proxy / recorded-traffic
90-
round-trip; fresh construction rejected):
91-
92-
- `ClusterRequest` `format=0` — V0 response shape (id + address only).
93-
Decoded by `ClusterRequest.decode_body` for proxy / replay use. A
94-
*decoded* V0 frame carries the `_decoded` sentinel and re-encodes
95-
byte-identically, so capture-replay tooling can round-trip it through
96-
the dataclass. *Fresh* construction with `format=0` is rejected with
97-
`EncodeError`, because production senders always emit V1 (id + address
98-
+ role) and this client only decodes the V1 `ServersResponse`.
40+
## The dqlite Python stack
41+
42+
This is the lowest of four layered packages. Each builds on the one below:
43+
44+
| Package | Role |
45+
| --- | --- |
46+
| [sqlalchemy-dqlite](https://github.com/letsdiscodev/sqlalchemy-dqlite) | SQLAlchemy 2.0 dialect |
47+
| [dqlite-dbapi](https://github.com/letsdiscodev/python-dqlite-dbapi) | PEP 249 (DB-API 2.0) driver — sync & async |
48+
| [dqlite-client](https://github.com/letsdiscodev/python-dqlite-client) | Async wire client — pooling, leader discovery |
49+
| **dqlite-wire** — this package | Wire-protocol codec |
50+
51+
**Most applications should use [dqlite-dbapi](https://github.com/letsdiscodev/python-dqlite-dbapi)
52+
or [sqlalchemy-dqlite](https://github.com/letsdiscodev/sqlalchemy-dqlite).**
53+
54+
## Documentation
55+
56+
- [Thread-safety](docs/thread-safety.md) — codec objects are single-owner; read this before sharing one.
57+
- [Divergences from upstream](docs/divergences-from-upstream.md) — the
58+
defensive caps and stricter validations this codec adds on top of the C
59+
server and [go-dqlite](https://github.com/canonical/go-dqlite).
9960

10061
## Development
10162

‎docs/divergences-from-upstream.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Divergences from upstream
2+
3+
This library implements the dqlite wire protocol faithfully, but adds a
4+
handful of defensive guards that the upstream C server and the
5+
[go-dqlite](https://github.com/canonical/go-dqlite) client do not. They
6+
protect a Python client running in potentially adversarial network
7+
contexts. The caps are configurable (pass `None` to disable); the stricter
8+
validations match the C server's intent.
9+
10+
## Python-specific caps
11+
12+
Bounds on how much a single decode will allocate, so a hostile or buggy
13+
peer cannot exhaust memory. All are optional (`None` disables):
14+
15+
- `DEFAULT_MAX_TOTAL_ROWS` (`MessageDecoder(max_total_rows=...)`, default
16+
10,000,000) — rows accumulated across continuation frames for one query.
17+
- `DEFAULT_MAX_CONTINUATION_FRAMES`
18+
(`MessageDecoder(max_continuation_frames=...)`, default 100,000) —
19+
continuation frames for one query.
20+
- `RowsResponse.DEFAULT_MAX_ROWS` (`MessageDecoder(max_rows=...)`, default
21+
1,000,000) — per-frame row cap.
22+
- `ReadBuffer.DEFAULT_MAX_MESSAGE_SIZE` (`ReadBuffer(max_message_size=...)`,
23+
default 64 MiB) — envelope cap on a single frame.
24+
- Internal sanity bounds on decoded tuple/response sizes:
25+
`_MAX_PARAM_COUNT` (32,766 — SQLite's `SQLITE_MAX_VARIABLE_NUMBER`),
26+
`_MAX_COLUMN_COUNT` (2000 — SQLite's default `SQLITE_MAX_COLUMN`),
27+
`_MAX_FILE_COUNT` (100), `_MAX_NODE_COUNT` (10,000).
28+
29+
`DEFAULT_MAX_TOTAL_ROWS` and `DEFAULT_MAX_CONTINUATION_FRAMES` are
30+
importable from `dqlitewire`; the others are class-scoped.
31+
32+
## Stricter-than-Go validations
33+
34+
These match the C server's intent more closely than go-dqlite does:
35+
36+
- `decode_row_header` requires the full 8-byte end-of-rows marker (C defines
37+
`DQLITE_RESPONSE_ROWS_DONE = 0xff…ff` / `_PART = 0xee…ee`; go-dqlite checks
38+
only the first byte).
39+
- `encode_value(value, ValueType.BOOLEAN)` rejects arbitrary ints (accepts
40+
only `bool` or exactly `0`/`1`).
41+
- `FilesResponse.encode_body` rejects non-8-aligned file content (C's
42+
`dumpFile` asserts `len % 8 == 0`).
43+
- `encode_params_tuple` rejects `ValueType.UNIXTIME` outbound (the C server's
44+
tuple decoder cannot decode it).
45+
- `StmtResponse` rejects a 16-byte body when `schema=1` (C's V1 response is
46+
24 bytes).
47+
48+
## Asymmetric encode/decode
49+
50+
Some frames are decodable for proxy / recorded-traffic round-trips even
51+
though fresh construction of them is rejected:
52+
53+
- `ClusterRequest` with `format=0` (the V0 response shape: id + address
54+
only). `ClusterRequest.decode_body` decodes it for proxy/replay use, and a
55+
*decoded* V0 frame re-encodes byte-identically. *Fresh* construction with
56+
`format=0` is rejected with `EncodeError`, because production senders
57+
always emit V1 (id + address + role) and this client only decodes the V1
58+
`ServersResponse`.

‎docs/thread-safety.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Thread-safety
2+
3+
`ReadBuffer`, `WriteBuffer`, `MessageEncoder`, and `MessageDecoder` are
4+
**not thread-safe**. Each instance must be owned by a single thread or a
5+
single asyncio coroutine at a time. This matches the single-owner
6+
`driver.Conn` contract from [go-dqlite](https://github.com/canonical/go-dqlite).
7+
8+
Concurrent use of one instance from multiple threads produces **silent
9+
data corruption — not exceptions.** Fuzz testing reliably reproduces both
10+
duplicate message delivery and corrupt (garbage) message bytes with no
11+
error raised. The internal `is_poisoned` guard catches torn state from a
12+
signal delivered during single-owner execution, but it **cannot** detect
13+
lost-update races between concurrent threads.
14+
15+
If you need concurrent access, serialize it with an `asyncio.Lock` or
16+
`threading.Lock` at the layer that owns the socket and decoder. In
17+
practice the higher-level packages
18+
([dqlite-client](https://github.com/letsdiscodev/python-dqlite-client) and
19+
above) already do this for you — this constraint matters only if you drive
20+
the codec directly.

0 commit comments

Comments
 (0)