Skip to content

Commit 2a63822

Browse files
committed
Rewrite README in pageres style with API docs and promo banner
1 parent 8148648 commit 2a63822

3 files changed

Lines changed: 448 additions & 40 deletions

File tree

‎README.md‎

Lines changed: 213 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,32 @@
1-
# queuebridge
1+
# ![queuebridge](media/promo.svg)
22

3-
**Pass Pydantic models to `.delay()` / `.send()` / `enqueue_job()` — get models back from results.**
3+
[![PyPI version](https://img.shields.io/pypi/v/queuebridge.svg)](https://pypi.org/project/queuebridge/)
4+
[![Python](https://img.shields.io/pypi/pyversions/queuebridge.svg)](https://pypi.org/project/queuebridge/)
5+
[![CI](https://github.com/false200/queuebridge/actions/workflows/ci.yml/badge.svg)](https://github.com/false200/queuebridge/actions/workflows/ci.yml)
6+
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
47

5-
Bidirectional Pydantic typing for [Celery](https://docs.celeryq.dev/), [Dramatiq](https://dramatiq.io/), and [Arq](https://arq-docs.helpmanual.io/) with one shared wire codec.
8+
Bidirectional [Pydantic](https://docs.pydantic.dev/) serialization for [Celery](https://docs.celeryq.dev/), [Dramatiq](https://dramatiq.io/), and [Arq](https://arq-docs.helpmanual.io/). One shared wire codec — pass models on enqueue, get models back from results.
69

7-
## The problem
10+
Celery 5.5+ `pydantic=True` only validates on the worker. Callers still `model_dump()` before `.delay()`, and `.get()` returns a `dict`. Dramatiq chokes on models and UUIDs. Arq defaults to pickle. **queuebridge** fixes all three with a thin codec + backend adapters.
811

9-
Celery 5.5+ added `pydantic=True`, but it only validates on the **worker**:
10-
11-
- Callers must still `model_dump()` before `.delay()` — passing a model raises `TypeError: Object of type X is not JSON serializable` ([celery#9442](https://github.com/celery/celery/issues/9442))
12-
- `.get()` returns a `dict`, not your model
13-
14-
Dramatiq's default JSON encoder fails on models, UUIDs, and datetimes ([dramatiq#660](https://github.com/Bogdanp/dramatiq/issues/660)).
15-
16-
Arq defaults to pickle with no Pydantic story ([arq#497](https://github.com/python-arq/arq/issues/497)).
12+
## Install
1713

18-
```
19-
Producer Worker Client
20-
──────── ────── ──────
21-
.delay(model) ──X──> pydantic=True validates .get() → dict
22-
(model_dump() required) args on worker only
14+
```sh
15+
pip install queuebridge
2316
```
2417

25-
**queuebridge** fixes the producer side and client-side result decoding with a shared `__qb__` tagged wire format.
26-
27-
## Install
18+
Extras:
2819

29-
```bash
20+
```sh
3021
pip install queuebridge[celery] # Celery + Kombu
3122
pip install queuebridge[dramatiq] # Dramatiq
3223
pip install queuebridge[arq] # Arq + msgpack
33-
pip install queuebridge[all] # everything
24+
pip install queuebridge[all] # all backends
3425
```
3526

36-
## Quickstart
27+
Requires **Python 3.10+** and **Pydantic v2**.
28+
29+
## Usage
3730

3831
### Celery
3932

@@ -49,15 +42,10 @@ register_queuebridge(app)
4942
def process_order(order: OrderCreate) -> OrderResult:
5043
return OrderResult(id=order.id, status="processed")
5144

52-
# Enqueue with a model directly
5345
ar = process_order.delay(OrderCreate(id=1, sku="ABC"))
54-
55-
# Get a model back (not a dict)
5646
result = typed_result(ar, OrderResult).get(timeout=10)
5747
```
5848

59-
> **Note:** Celery cannot safely monkey-patch `AsyncResult.get()` globally. Use `typed_result()` for typed client results.
60-
6149
### Dramatiq
6250

6351
```python
@@ -98,9 +86,162 @@ class WorkerSettings:
9886
job_deserializer = deserialize
9987
```
10088

89+
## API
90+
91+
### `encode(value, *, tag_models=True)`
92+
93+
Recursively transform a Python value into a JSON-serializable structure.
94+
95+
#### value
96+
97+
*Required*
98+
Type: `Any`
99+
100+
The value to encode — Pydantic models, nested containers, `UUID`, `datetime`, `Decimal`, `Enum`, etc.
101+
102+
#### tag_models
103+
104+
Type: `boolean`
105+
Default: `true`
106+
107+
When `true`, `BaseModel` instances are wrapped in a `__qb__` envelope with a fully-qualified type name. When `false`, models are dumped with `model_dump(mode="json")` only.
108+
109+
```python
110+
from queuebridge import encode, decode
111+
from myapp.models import OrderCreate
112+
113+
wire = encode(OrderCreate(id=1, sku="ABC"))
114+
restored = decode(wire, OrderCreate)
115+
```
116+
117+
---
118+
119+
### `decode(value, hint=Any, *, strict=False)`
120+
121+
Recursively decode a wire value back to Python using an optional type hint.
122+
123+
#### value
124+
125+
*Required*
126+
Type: `Any`
127+
128+
Wire value — primitives, lists, dicts, or `__qb__` envelopes.
129+
130+
#### hint
131+
132+
Type: `Any`
133+
Default: `Any`
134+
135+
Type hint used for validation. `TypeAdapter(hint).validate_python()` is used when the hint is concrete.
136+
137+
#### strict
138+
139+
Type: `boolean`
140+
Default: `false`
141+
142+
When `true`, raise `QueuebridgeDecodeError` if the value cannot be decoded.
143+
144+
---
145+
146+
### `decode_wire(value)`
147+
148+
Recursively unwrap `__qb__` envelopes without type hints. Used internally by Dramatiq's decoder.
149+
150+
Type: `Any` → `Any`
151+
152+
---
153+
154+
### `register_queuebridge(app, *, strict=False)` — Celery
155+
156+
Register the `queuebridge-json` Kombu serializer on a Celery app. Idempotent — safe to call twice.
157+
158+
#### app
159+
160+
*Required*
161+
Type: `celery.Celery`
162+
163+
#### strict
164+
165+
Type: `boolean`
166+
Default: `false`
167+
168+
Reserved for future strict decode behavior.
169+
170+
Sets `task_serializer`, `result_serializer`, and `accept_content` on the app.
171+
172+
---
173+
174+
### `typed_result(async_result, return_type)` — Celery
175+
176+
Wrap a Celery `AsyncResult` so `.get()` returns a Pydantic model instead of a `dict`.
177+
178+
#### async_result
179+
180+
*Required*
181+
Type: `celery.result.AsyncResult`
182+
183+
#### return_type
184+
185+
*Required*
186+
Type: `type[T]`
187+
188+
Returns `TypedAsyncResult[T]` — proxies `.id`, `.state`, `.ready()`, etc.
189+
190+
> Celery cannot safely monkey-patch `AsyncResult.get()` globally. Use `typed_result()` on the client.
191+
192+
---
193+
194+
### `register_queuebridge(broker=None)` — Dramatiq
195+
196+
Install `QueuebridgeEncoder` via `dramatiq.set_encoder()`. Call once at process startup.
197+
198+
#### broker
199+
200+
Type: `dramatiq.Broker | None`
201+
Default: `None`
202+
203+
If provided, also calls `dramatiq.set_broker(broker)`.
204+
205+
---
206+
207+
### `get_serializer_pair()` — Arq
208+
209+
Returns `(serialize, deserialize)` callables for `job_serializer` / `job_deserializer`.
210+
211+
```python
212+
serialize, deserialize = get_serializer_pair()
213+
```
214+
215+
Uses **msgpack** over queuebridge-encoded dicts. Set on both `WorkerSettings` and `create_pool()`.
216+
217+
---
218+
219+
### `qb_task(fn)` — Arq
220+
221+
Decorator that decodes wire args/kwargs using function type hints before your async task runs.
222+
223+
Apply **outside** `@validate_call`:
224+
225+
```python
226+
@qb_task
227+
@validate_call
228+
async def process_order(ctx, order: OrderCreate) -> OrderResult:
229+
...
230+
```
231+
232+
---
233+
234+
### `typed_result(job, return_type)` — Arq
235+
236+
```python
237+
result = await typed_result(job, OrderResult)
238+
```
239+
240+
Decode the `job.result()` payload into a Pydantic model.
241+
101242
## Wire format
102243

103-
Non-JSON-native values are wrapped in a tagged envelope:
244+
Non-JSON-native values use a tagged envelope:
104245

105246
```json
106247
{
@@ -112,26 +253,58 @@ Non-JSON-native values are wrapped in a tagged envelope:
112253
}
113254
```
114255

115-
Decode uses function type hints (`TypeAdapter`) when tags are absent — a plain dict + `OrderCreate` hint still validates.
256+
| Python type | Encode | Decode |
257+
|-------------|--------|--------|
258+
| `BaseModel` | envelope + `model_dump(mode="json")` | `model_validate` or FQN import |
259+
| `UUID`, `datetime`, `Decimal`, `Enum` | tagged envelope | builtin dispatch |
260+
| `list`, `dict`, `set`, `tuple` | recurse | recurse via hint |
261+
| Primitives | pass through | pass through |
116262

117-
## Security
263+
A plain `dict` + `OrderCreate` hint still validates — tags are for ambiguity, not required when hints are known.
264+
265+
## Why not Celery `pydantic=True` alone?
118266

119-
Deserialization resolves types by fully-qualified name (`import_fqn`). **Only deserialize from brokers you trust.** Module allowlisting is planned for v0.2.
267+
```
268+
Producer Worker Client
269+
──────── ────── ──────
270+
.delay(model) ──X──> pydantic=True validates .get() → dict
271+
(model_dump() required) args on worker only
272+
```
273+
274+
- [celery#9442](https://github.com/celery/celery/issues/9442) — models not JSON-serializable on enqueue
275+
- [dramatiq#660](https://github.com/Bogdanp/dramatiq/issues/660) — no Pydantic support
276+
- [arq#497](https://github.com/python-arq/arq/issues/497) — pickle default, Pydantic requested
120277

121278
## Comparison
122279

123-
| Solution | Celery | Dramatiq | Arq | Bidirectional `.get()` |
124-
|----------|--------|----------|-----|------------------------|
280+
| Solution | Celery | Dramatiq | Arq | Typed `.get()` |
281+
|----------|--------|----------|-----|----------------|
125282
| Celery `pydantic=True` | worker only | — | — | no |
126283
| Blog / msgpack hacks | partial | partial | partial | varies |
127-
| **queuebridge** | yes | yes | yes | yes (`typed_result`) |
284+
| **queuebridge** | yes | yes | yes | yes |
285+
286+
## Security
287+
288+
Deserialization resolves types by fully-qualified name (`import_fqn`). **Only deserialize from brokers you trust.**
289+
290+
`ALLOWED_MODULE_PREFIXES` allowlisting is planned for v0.2.
291+
292+
## Examples
293+
294+
| Path | Description |
295+
|------|-------------|
296+
| [`examples/celery_fastapi/`](examples/celery_fastapi/) | FastAPI enqueue + typed result polling |
297+
| [`examples/dramatiq_example/`](examples/dramatiq_example/) | Dramatiq + `validate_call` |
298+
| [`examples/arq_example/`](examples/arq_example/) | Arq worker with custom serializers |
299+
| [`examples/smoke_test_complex.py`](examples/smoke_test_complex.py) | End-to-end smoke test (no Redis) |
300+
| [`pypi_verify/run_complex.py`](pypi_verify/run_complex.py) | PyPI install verification script |
128301

129-
## Roadmap
302+
## Related
130303

131-
- `allowed_modules` security filter on `register_queuebridge()`
132-
- Optional pickle extra
133-
- Chord / chain signature support
304+
- [Celery Pydantic docs](https://docs.celeryq.dev/en/stable/userguide/tasks.html#argument-validation-with-pydantic) — worker-only validation
305+
- [Arq custom serializers](https://arq-docs.helpmanual.io/#custom-job-serializers) — msgpack hook point
306+
- [Dramatiq encoders](https://dramatiq.io/advanced.html#custom-encoders) — `set_encoder()` extension point
134307

135308
## License
136309

137-
MIT
310+
MIT © [false200](https://github.com/false200)

‎media/promo.svg‎

Lines changed: 16 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)