Add Sphinx documentation and Read the Docs config

Author: false200

.github/workflows/ci.yml

@@ -36,3 +36,14 @@ jobs:
       - run: ruff check .
       - run: mypy src
       - run: pytest
+
+  docs:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+      - run: pip install -e ".[all,docs]"
+      - run: sphinx-build -b html docs docs/_build/html

.gitignore

@@ -32,6 +32,7 @@ ENV/
 htmlcov/
 .mypy_cache/
 .ruff_cache/
+docs/_build/
 
 # IDE
 .idea/

.readthedocs.yaml

@@ -0,0 +1,20 @@
+# Read the Docs configuration file
+# https://docs.readthedocs.io/en/stable/config-file/v2.html
+
+version: 2
+
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+
+sphinx:
+  configuration: docs/conf.py
+
+python:
+  install:
+    - method: pip
+      path: .
+      extra_requirements:
+        - docs
+        - all

README.md

@@ -26,6 +26,8 @@ pip install queuebridge[all]        # all backends
 
 Requires **Python 3.10+** and **Pydantic v2**.
 
+**Documentation:** https://queuebridge.readthedocs.io
+
 ## Usage
 
 ### Celery

docs/Makefile

@@ -0,0 +1,14 @@
+# Minimal makefile for Sphinx documentation
+
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS)
+
+.PHONY: help Makefile
+
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS)

docs/_static/.gitkeep

@@ -0,0 +1,71 @@
+API reference
+=============
+
+This section is generated from the package source. For narrative guides, see the tutorials.
+
+Core
+----
+
+.. automodule:: queuebridge
+   :members:
+   :undoc-members:
+
+Codec
+-----
+
+.. automodule:: queuebridge.codec
+   :members:
+   :undoc-members:
+
+Types and exceptions
+--------------------
+
+.. automodule:: queuebridge.types
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Hints
+-----
+
+.. automodule:: queuebridge.hints
+   :members:
+   :undoc-members:
+
+Celery
+------
+
+.. automodule:: queuebridge.celery
+   :members:
+   :undoc-members:
+
+.. automodule:: queuebridge.celery.install
+   :members:
+
+.. automodule:: queuebridge.celery.result
+   :members:
+   :show-inheritance:
+
+Dramatiq
+--------
+
+.. automodule:: queuebridge.dramatiq
+   :members:
+   :undoc-members:
+
+.. automodule:: queuebridge.dramatiq.encoder
+   :members:
+   :show-inheritance:
+
+Arq
+---
+
+.. automodule:: queuebridge.arq
+   :members:
+   :undoc-members:
+
+.. automodule:: queuebridge.arq.install
+   :members:
+
+.. automodule:: queuebridge.arq.serializer
+   :members:

docs/api/index.rst

@@ -0,0 +1,67 @@
+How it works
+============
+
+queuebridge has two layers:
+
+1. **Codec** (``queuebridge.codec``): encode/decode Python values to a JSON-safe wire format.
+2. **Backend adapters**: plug the codec into Celery, Dramatiq, or Arq.
+
+.. code-block:: text
+
+   Your app                queuebridge              Task queue
+   --------                -----------              ----------
+   OrderCreate model  -->  encode()           -->   JSON / msgpack on wire
+                           __qb__ tags
+
+   Worker task        <--  decode + hints    <--   wire dict
+   receives model
+
+The flow (Celery example)
+-------------------------
+
+1. **Producer**: ``process_order.delay(OrderCreate(...))``
+2. **Serializer**: Kombu calls ``queuebridge.encode()`` on the message body.
+3. **Broker**: JSON bytes travel over Redis/RabbitMQ.
+4. **Worker**: Celery ``pydantic=True`` validates kwargs dicts into models.
+5. **Return**: Worker returns ``OrderResult``; Celery dumps to dict; codec encodes for result backend.
+6. **Client**: ``typed_result(ar, OrderResult).get()`` decodes to a model.
+
+What each backend does
+----------------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Backend
+     - Encode hook
+     - Decode hook
+   * - Celery
+     - Kombu ``queuebridge-json`` serializer
+     - Worker ``pydantic=True``; client uses ``typed_result``
+   * - Dramatiq
+     - ``QueuebridgeEncoder.encode()``
+     - ``QueuebridgeEncoder.decode()`` + ``validate_call``
+   * - Arq
+     - msgpack + ``encode()``
+     - ``@qb_task`` + ``validate_call``
+
+Type hints matter
+-----------------
+
+Decoding uses your function annotations via ``typing.get_type_hints`` and Pydantic's ``TypeAdapter``.
+
+If the wire value is a plain dict and your parameter is ``order: OrderCreate``, queuebridge calls ``OrderCreate.model_validate(dict)``.
+
+If the wire value has a ``__qb__`` tag, the tag's type name is used to reconstruct the object.
+
+See :doc:`wire-format` for tag details.
+
+When you do not need tags
+-------------------------
+
+If you always pass type hints and use plain ``model_dump()`` dicts, decoding still works. Tags help when:
+
+* The type is ambiguous (``Union``, ``Any``)
+* You have nested UUID/datetime outside a model
+* You decode without a concrete hint

docs/concepts/how-it-works.rst

@@ -0,0 +1,61 @@
+Why not Celery pydantic=True alone?
+====================================
+
+Celery 5.5+ supports ``@app.task(pydantic=True)``. That is useful, but it only covers **half** the story.
+
+What pydantic=True does
+-----------------------
+
+On the **worker**, Celery:
+
+* Converts kwargs dicts into Pydantic models using your annotations
+* Dumps return models to dicts via ``model_dump()`` for the result backend
+
+What it does not do
+-------------------
+
+On the **producer** (your API, CLI, or script calling ``.delay()``):
+
+* You must still serialize models yourself
+* Passing ``OrderCreate(...)`` directly raises ``TypeError: not JSON serializable`
+
+On the **client** (calling ``.get()``):
+
+* You receive a **dict**, not your return model
+
+Official docs quote
+-------------------
+
+From the `Celery task documentation <https://docs.celeryq.dev/en/stable/userguide/tasks.html#argument-validation-with-pydantic>`_:
+
+   Argument validation only covers arguments/return values on the task side.
+   You still have serialize arguments yourself when invoking a task with
+   delay() or apply_async().
+
+Community reports
+-----------------
+
+* `celery#9442 <https://github.com/celery/celery/issues/9442>`_ - models not serializable on enqueue
+* `dramatiq#660 <https://github.com/Bogdanp/dramatiq/issues/660>`_ - JSON encoder fails on models
+* `arq#497 <https://github.com/python-arq/arq/issues/497>`_ - request for native Pydantic support
+
+What queuebridge adds
+---------------------
+
+.. list-table::
+   :header-rows: 1
+
+   * - Stage
+     - Celery alone
+     - With queuebridge
+   * - ``.delay(model)``
+     - Fails or needs ``model_dump()``
+     - Works
+   * - Worker receives
+     - Model (with pydantic=True)
+     - Model (with pydantic=True)
+   * - ``.get()`` returns
+     - ``dict``
+     - ``dict`` (use ``typed_result`` for model)
+
+queuebridge does not replace Celery's worker validation. It complements it by fixing producer serialization and offering client-side typed results.

docs/concepts/why-not-celery-pydantic.rst

@@ -0,0 +1,98 @@
+Wire format
+===========
+
+queuebridge uses a **JSON-first** wire format with optional **type tags** for values that lose type information in JSON.
+
+The ``__qb__`` envelope
+-----------------------
+
+Non-primitive values that need type information are wrapped like this:
+
+.. code-block:: json
+
+   {
+     "__qb__": {
+       "t": "myapp.models.OrderCreate",
+       "v": 1,
+       "d": {"id": 1, "sku": "ABC"}
+     }
+   }
+
+.. list-table::
+   :header-rows: 1
+
+   * - Field
+     - Meaning
+   * - ``t``
+     - Fully-qualified type name (module + class)
+   * - ``v``
+     - Wire format version (always ``1`` for now)
+   * - ``d``
+     - JSON-safe payload
+
+Constants (Python)
+------------------
+
+.. code-block:: python
+
+   from queuebridge.types import QB_TAG, QB_VERSION
+
+   QB_TAG == "__qb__"
+   QB_VERSION == 1
+
+Types and their wire representation
+-----------------------------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 25 35 40
+
+   * - Python type
+     - On the wire
+     - Decoded via
+   * - ``BaseModel``
+     - ``__qb__`` envelope, ``d`` = ``model_dump(mode="json")``
+     - ``model_validate`` or FQN import
+   * - ``UUID``
+     - ``__qb__`` with ``t: uuid.UUID``, ``d: "<uuid string>"``
+     - ``UUID(d)``
+   * - ``datetime``, ``date``, ``time``
+     - ``__qb__`` with ISO string in ``d``
+     - ``fromisoformat``
+   * - ``Decimal``
+     - ``__qb__`` with string in ``d``
+     - ``Decimal(d)``
+   * - ``Enum``
+     - ``__qb__`` with enum FQN and value
+     - Enum class lookup
+   * - ``list``, ``tuple``, ``set``
+     - JSON list (elements encoded recursively)
+     - Hint-driven (``list[T]``, etc.)
+   * - ``dict``
+     - JSON object (keys coerced to str)
+     - Hint-driven
+   * - ``str``, ``int``, ``float``, ``bool``, ``null``
+     - Pass through unchanged
+     - Pass through
+
+Encode and decode API
+---------------------
+
+.. code-block:: python
+
+   from queuebridge import encode, decode
+
+   wire = encode(my_model)
+   restored = decode(wire, OrderCreate)
+
+``decode_wire()`` unwraps all ``__qb__`` envelopes recursively without hints. Dramatiq uses this on incoming messages.
+
+Celery note
+-----------
+
+The Celery serializer uses ``encode(..., tag_models=False)`` for compatibility with ``pydantic=True``. Models become plain dicts on the wire; UUID/datetime inside ``model_dump(mode="json")`` are already strings.
+
+Arq note
+--------
+
+Job dicts are msgpack-encoded **after** ``encode()``, so the binary blob contains queuebridge-tagged JSON-compatible structures.