|
1 | 1 | # dqlite-wire |
2 | 2 |
|
3 | | -Pure Python wire protocol implementation for [dqlite](https://dqlite.io/), Canonical's distributed SQLite. |
| 3 | +Pure-Python codec for the [dqlite](https://dqlite.io/) wire protocol — |
| 4 | +encode and decode the messages a dqlite server speaks. |
| 5 | + |
| 6 | +[dqlite](https://dqlite.io/) is Canonical's distributed SQLite, built on |
| 7 | +Raft. This package implements the bytes-on-the-wire layer: it turns |
| 8 | +request/response objects into frames and back, following the |
| 9 | +[official wire-protocol specification](https://canonical.com/dqlite/docs/reference/wire-protocol). |
| 10 | +It does no networking, pooling, or SQL — just framing. |
| 11 | + |
| 12 | +## Is this the package you want? |
| 13 | + |
| 14 | +Probably not, unless you are building a driver or doing wire-level work |
| 15 | +(a proxy, traffic capture/replay, a custom client). If you just want to |
| 16 | +run SQL against dqlite from Python, use one of the higher layers — see |
| 17 | +[The dqlite Python stack](#the-dqlite-python-stack) below. |
4 | 18 |
|
5 | 19 | ## Installation |
6 | 20 |
|
7 | 21 | ```bash |
8 | 22 | pip install dqlite-wire |
9 | 23 | ``` |
10 | 24 |
|
| 25 | +Requires Python 3.13+. |
| 26 | + |
11 | 27 | ## Usage |
12 | 28 |
|
13 | 29 | ```python |
14 | 30 | from dqlitewire import encode_message, decode_message |
15 | 31 | from dqlitewire.messages import LeaderRequest |
16 | 32 |
|
17 | | -# Encode a message |
| 33 | +# Encode a request to bytes |
18 | 34 | data = encode_message(LeaderRequest()) |
19 | 35 |
|
20 | | -# Decode a message |
| 36 | +# Decode bytes back into a message object |
21 | 37 | message = decode_message(data, is_request=True) |
22 | 38 | ``` |
23 | 39 |
|
24 | | -## Thread-safety |
25 | | - |
26 | | -`ReadBuffer`, `WriteBuffer`, `MessageEncoder`, and `MessageDecoder` |
27 | | -are **not thread-safe**. Each instance must be owned by a single |
28 | | -thread or a single asyncio coroutine at a time. This matches Go's |
29 | | -`driver.Conn` contract from go-dqlite. |
30 | | - |
31 | | -Concurrent use of a single instance from multiple threads produces |
32 | | -**silent data corruption** — not exceptions. The `is_poisoned` |
33 | | -mechanism catches torn state from signal delivery during |
34 | | -single-owner execution, but it **cannot** detect lost-update races |
35 | | -between concurrent threads. Fuzz testing reliably reproduces both |
36 | | -duplicate message delivery and corrupt (garbage) message bytes with |
37 | | -no exception surfacing. |
38 | | - |
39 | | -If you need concurrent access, wrap every call site in an |
40 | | -`asyncio.Lock` or `threading.Lock` at the layer that owns the |
41 | | -socket and decoder. |
42 | | - |
43 | | -## Protocol Reference |
44 | | - |
45 | | -Based on the [dqlite wire protocol specification](https://canonical.com/dqlite/docs/reference/wire-protocol). |
46 | | - |
47 | | -## Deliberate divergences from upstream |
48 | | - |
49 | | -This library implements the dqlite wire protocol faithfully but adds a |
50 | | -handful of defensive guards that the upstream C server and the |
51 | | -canonical [go-dqlite](https://github.com/canonical/go-dqlite) client do |
52 | | -not. They protect a Python client running in potentially adversarial |
53 | | -network contexts and are all opt-out-able. |
54 | | - |
55 | | -**Python-specific caps** (not present in C or Go; `None` disables): |
56 | | - |
57 | | -- `DEFAULT_MAX_TOTAL_ROWS` (`MessageDecoder(max_total_rows=...)`, default |
58 | | - 10,000,000) — cap on rows accumulated across continuation frames for |
59 | | - one query. Importable from `dqlitewire`. |
60 | | -- `DEFAULT_MAX_CONTINUATION_FRAMES` (`MessageDecoder(max_continuation_frames=...)`, |
61 | | - default 100,000) — cap on continuation frames for one query. |
62 | | - Importable from `dqlitewire`. |
63 | | -- `RowsResponse.DEFAULT_MAX_ROWS` (`MessageDecoder(max_rows=...)`, |
64 | | - default 1,000,000) — per-frame row cap. Class-scoped, not exported |
65 | | - at module level. |
66 | | -- `ReadBuffer.DEFAULT_MAX_MESSAGE_SIZE` (`ReadBuffer(max_message_size=...)`, |
67 | | - default 64 MiB) — envelope cap on a single frame. Class-scoped, not |
68 | | - exported at module level. |
69 | | -- `_MAX_PARAM_COUNT` (32,766 — matches SQLite's |
70 | | - `SQLITE_MAX_VARIABLE_NUMBER`), `_MAX_COLUMN_COUNT` (2000 — SQLite's |
71 | | - documented `SQLITE_MAX_COLUMN` compile-time default), `_MAX_FILE_COUNT` (100), |
72 | | - `_MAX_NODE_COUNT` (10,000) — internal sanity bounds on decoded |
73 | | - tuple / response sizes. |
74 | | - |
75 | | -**Stricter-than-Go validations** (match the C server's intent): |
76 | | - |
77 | | -- `decode_row_header` requires the full 8-byte marker (C defines |
78 | | - `DQLITE_RESPONSE_ROWS_DONE = 0xff..ff` / `_PART = 0xee..ee`; |
79 | | - go-dqlite checks only the first byte). |
80 | | -- `encode_value(value, ValueType.BOOLEAN)` rejects arbitrary ints |
81 | | - (accepts only `bool` or exactly `0`/`1`). |
82 | | -- `FilesResponse.encode_body` rejects non-8-aligned file content (C's |
83 | | - `dumpFile` asserts `len % 8 == 0`). |
84 | | -- `encode_params_tuple` rejects `ValueType.UNIXTIME` outbound (C's |
85 | | - `tuple_decoder__next` cannot decode it on the server side). |
86 | | -- `StmtResponse` rejects a 16-byte body when `schema=1` (C's V1 |
87 | | - response is 24 bytes). |
88 | | - |
89 | | -**Asymmetric encode/decode** (decoded for proxy / recorded-traffic |
90 | | -round-trip; fresh construction rejected): |
91 | | - |
92 | | -- `ClusterRequest` `format=0` — V0 response shape (id + address only). |
93 | | - Decoded by `ClusterRequest.decode_body` for proxy / replay use. A |
94 | | - *decoded* V0 frame carries the `_decoded` sentinel and re-encodes |
95 | | - byte-identically, so capture-replay tooling can round-trip it through |
96 | | - the dataclass. *Fresh* construction with `format=0` is rejected with |
97 | | - `EncodeError`, because production senders always emit V1 (id + address |
98 | | - + role) and this client only decodes the V1 `ServersResponse`. |
| 40 | +## The dqlite Python stack |
| 41 | + |
| 42 | +This is the lowest of four layered packages. Each builds on the one below: |
| 43 | + |
| 44 | +| Package | Role | |
| 45 | +| --- | --- | |
| 46 | +| [sqlalchemy-dqlite](https://github.com/letsdiscodev/sqlalchemy-dqlite) | SQLAlchemy 2.0 dialect | |
| 47 | +| [dqlite-dbapi](https://github.com/letsdiscodev/python-dqlite-dbapi) | PEP 249 (DB-API 2.0) driver — sync & async | |
| 48 | +| [dqlite-client](https://github.com/letsdiscodev/python-dqlite-client) | Async wire client — pooling, leader discovery | |
| 49 | +| **dqlite-wire** — this package | Wire-protocol codec | |
| 50 | + |
| 51 | +**Most applications should use [dqlite-dbapi](https://github.com/letsdiscodev/python-dqlite-dbapi) |
| 52 | +or [sqlalchemy-dqlite](https://github.com/letsdiscodev/sqlalchemy-dqlite).** |
| 53 | + |
| 54 | +## Documentation |
| 55 | + |
| 56 | +- [Thread-safety](docs/thread-safety.md) — codec objects are single-owner; read this before sharing one. |
| 57 | +- [Divergences from upstream](docs/divergences-from-upstream.md) — the |
| 58 | + defensive caps and stricter validations this codec adds on top of the C |
| 59 | + server and [go-dqlite](https://github.com/canonical/go-dqlite). |
99 | 60 |
|
100 | 61 | ## Development |
101 | 62 |
|
|
0 commit comments