Bringing up a feed

From nothing to a feed that publishes, records and can be audited afterwards.

This is the path, in order, with the decision at each step and where the work lands. It spans four places: this repository (the libraries), the venue's own repository (the adapter and the binary), the feed specification (edge-feed-spec), and the infrastructure repositories (addresses, hosts, rollout). Read GLOSSARY.md first if datagram, channel, era and port role are not already fixed terms for you.

First: which of two jobs is this?

They share almost nothing, and starting the wrong one costs weeks.

	A venue on an existing feed	A new feed type
Example	a second exchange on Top-of-Book	a feed with its own datagram magic and message table
Wire format	already specified and implemented	has to be specified, then implemented and cross-verified
Codec crate	none — reuse `dz-edge-tob` / `-mbp` / `-refdata`	a new one under `rust/codec`
Subscriber	already exists	a new parser, and usually a book-builder
Golden vectors	already exist	new ones, asserted by both languages
Where the work is	the venue's repository, mostly	this repository, mostly

Most work is the left-hand column. If that is you, skip to The publisher. The right-hand column comes first only when no existing feed can carry what the venue produces.

A new feed type

Do these in order. Each step exists because the next one cannot be verified without it.

Specify it in edge-feed-spec and get that merged. The message table, the field offsets, the port roles, the sequencing rules, the reset reasons. Nothing here may decide any of it, and a publisher that decided one of these for itself is the defect class this whole design exists to prevent.
Write golden vectors into testdata/golden/. They are the cross-language contract: the Rust encoder and the Go decoder each assert against the bytes, and neither is checked against the other. Agreement between our own encoder and decoder proves nothing.
Add the codec crate under rust/codec, implementing dz_edge_core::Feed. Two constants carry the specification's own decisions: MAGIC, and CARRIES — the message table transcribed, which is what makes DatagramBuilder::push refuse a Type ID the feed does not list rather than emit a datagram no conformant subscriber can act on.
Teach the lowering to build those messages, in rust/publisher/dz-publisher-lowering. This is the only place events become wire messages, and it is deliberately the only place: Per-Instrument Seq, the flags byte and fixed-point scaling live here so that no venue can spell any of them differently.
Add the normalized events the feed needs, if the boundary cannot express them yet — in rust/adapter/dz-adapter-core. Adding a variant here is a deliberate widening of what every venue may say, so it wants the same review as a spec change.
Write the Go parser under go/, and a book-builder if the feed has book state. The parser is not optional: it is how a publisher is proven to be readable by something that was not written alongside it, and every end-to-end exercise in this repository reads its output.
Add the feed to the recorder's spec list so a recorder can be configured for it. Nothing in the record path decodes a datagram, so this is naming and accounting rather than parsing.

The publisher

A venue owns two things: its upstream protocol, and its own book state machine. It owns nothing else. Everything a specification already decided — the Instrument ID, the Source ID, the Channel ID, the sequence numbers, the scaling, the Update Flags byte, the Action, the datagram and its 1,232-byte cap — belongs to the crates here and is not expressible through the boundary. Not by convention: there is no parameter to pass one through.

See rust/adapter/README.md for the trait and rust/publisher/README.md for what sits behind it.

1. Implement the adapter, in the venue's repository

One type implementing dz_adapter_core::Adapter:

Method	What the venue answers
`message_types`	the upstream message names it will report, for one metric label
`poll_listings`	the instruments it knows about, as `InstrumentSpec`s
`on_connected`	the subscription it wants sent, after every connect — reconnects included, which is what makes a silently lost subscription come back
`poll_upstream`	whatever is still outstanding on a connection that is already open — optional, and what a venue needs when its instrument set changes without a reconnect
`on_payload`	one upstream payload, mapped onto normalized events
`on_disconnected`	the session ended, and why
`snapshot`	the book it holds for one instrument, and how deep that book goes — depth feeds only

Three rules that are easy to get wrong and expensive to get wrong:

An instrument admitted mid-session is not subscribed by admitting it. poll_listings offers instruments to the publisher; it says nothing to the venue. A publisher whose venue lists a new instrument during a session will publish that instrument's definition, count it in the manifest, and receive nothing for it — the feed is healthy and the instrument is silent. If the venue's set can change without a reconnect, implement poll_upstream and write there what has not been written on that connection. Unlike re-offering a listing, this write is not deduplicated: what it queues reaches the venue every time it is queued.
Update Flags states presence, not change. A side's updated and gone bits are mutually exclusive. This is the opposite of what the field's name suggests, and it is what conformant publishers do. The boundary expresses a side as Gone or Present, so a venue cannot state the combination the wire has no encoding for.
A quantity of zero is a removal and nothing else. The Action is derived from the quantity plus a presence hint; a venue does not choose it.
Depth Bound of zero claims the complete book. A depth adapter's snapshot returns a DepthBound, and Complete is a positive claim a subscriber is entitled to sum into available liquidity. An adapter that writes the top N of its book returns DepthBound::levels(n); one that writes all of it returns Complete, and an empty book is legitimately complete. There is no default, because the value a default would pick is the strongest claim on the feed.

Keep the adapter's own book state machine and reuse the venue's existing decoder if it has one. An adapter that folds the book a second way is validating itself.

2. Depend on these crates

A tag in this repository is the release; there is nothing on crates.io. See RELEASING.md for the whole of it, including how to cut the next one, but the two lines that matter here:

[workspace.dependencies]
dz-adapter-core = { git = "https://github.com/malbeclabs/edge-multicast-ref", tag = "v0.6.0" }
dz-publisher-runtime = { git = "https://github.com/malbeclabs/edge-multicast-ref", tag = "v0.6.0" }
dz-ingress-core = { git = "https://github.com/malbeclabs/edge-multicast-ref", tag = "v0.6.0", features = ["websocket"] }
# or, for a venue whose feed arrives over a session:
dz-ingress-fix = { git = "https://github.com/malbeclabs/edge-multicast-ref", tag = "v0.6.0" }

Every crate from the same tag, pinned in one place. Two tags in one graph means two copies of dz-edge-core, and then Scalar from one is not Scalar from the other — a type mismatch between two types with the same name, which is among the least legible errors cargo produces. The transport marker features are not optional decoration either: [ingress] kind refuses a transport the binary does not link.

3. Write the binary

It is a registry and nothing else — the one thing the runtime cannot know is which adapters a binary contains:

fn main() -> std::process::ExitCode {
    dz_publisher_runtime::run(
        AdapterRegistry::new()
            .with("a-venue-tob", |cx| Ok(Venue::single(
                Box::new(VenueAdapter::new(cx)?),
                venue_input(cx)?,
            ))),
    )
}

A venue with several upstreams for one feed builds one Input per cx.sources() entry and returns them together — see Several sources for one feed.

run() owns configuration loading, the guards, the signals, the metrics, the egress, the reference data and the teardown. There is no default adapter and no fallback: a [adapter] kind this binary did not register is a startup error naming every token it did, because what is in this binary is the question an operator cannot answer from the config file in front of them.

The command line is the configuration path — bare, after --config, or joined to it as --config=<path> — plus --version/-V and --help/-h, which are answered wherever on the line they are written rather than in first position only. The arguments are read in order and the first decisive one answers: an option the command-line reader does not know is refused by name rather than opened as a file, so a misspelled flag fails as the flag it is; two configuration files are refused naming both rather than one of them being dropped; an option left without its value is named as the option, in either spelling; and a flag that publishes nothing is answered from the flag itself, so --version reached before a misspelling prints the version and --version reached after one reports the misspelling. --help prints the accepted forms on stdout and exits 0.

--version writes one line to stdout and exits 0: the version, alone. That is the string a deployment compares against the version it pinned, and the tag with the v removed, so the comparison is an equality rather than a parse. A version handed over with nothing in it, or with a line break inside it, is reported as unknown: one line that fails every comparison is an answer, and two lines are not a version. run() answers with the runtime's own version. A venue that releases on its own stream answers with its own — the whole of the change is which function main calls:

fn main() -> std::process::ExitCode {
    dz_publisher_runtime::run_with_version(env!("CARGO_PKG_VERSION"), registry())
}

The same string reaches dz_publisher_build_info{version}, so what a scrape reports and what the binary answers cannot disagree about which build is deployed. The commit and the toolchain are on that gauge and deliberately not on the line: a consumer of --version compares one string against one pin, and a second field on the line is a field position for it to parse. A build handed no version reports unknown, which compares equal to no pin anybody holds.

The transport comes from rust/ingress. [ingress] kind names one of a closed set — websocket, fix, multicast, poll, filetail, uds — and the ones that are built are:

websocket, for a venue that carries its book on a subscription. A venue that authenticates its upgrade passes a header provider, which runs on every connect attempt — a venue signing a fresh timestamp and headers computed once at startup gives a publisher that connects and can then never reconnect.
fix, for a venue whose feed arrives over a session: the transport owns the framing, the outbound sequence, the heartbeat cadence, the logout and the close, and the venue composes only the fields of its own logon. TLS is on with no key anywhere to turn it off, and both halves of the endpoint are checked at load rather than at the first connect. It is the rest of this section.
poll, for a venue whose instrument catalogue is a request rather than a subscription: the endpoint is asked on poll_interval, the body arrives as a payload, and no code in the venue's binary holds a timer, a backoff or a failure count. An unchanged response is liveness and not a payload, so the idle guard still fires on a catalogue that has stopped changing. https needs the crate's tls feature and is refused at load without it, and a credential goes on the query string rather than in a userinfo section — nothing sends one, so http://user:secret@host/catalogue is refused at load instead of going out unauthenticated and answering 401 to a document that looks right.

The other three — multicast, filetail, uds — are named by the set and have no crate behind them yet. They are listed because an operator who has misspelled a transport needs to be told the whole set rather than the part this build happens to carry, which is the difference between no such transport and not built with it; naming one in a document is the startup failure below and not a feed. rust/ingress is the list of what exists.

A transport whose crate is not linked is refused at startup, so the binary depends on dz-ingress-core with the marker feature for the transports it means to allow.

A venue on the session transport writes its logon where it writes its subscriptions. [ingress] kind = "fix" resolves to dz-ingress-fix, which owns the framing, the outbound sequence, the heartbeat cadence, the logout and the close — everything except the one part that is genuinely the venue's, which is the logon's own fields. Those are composed in on_connected, beside the subscriptions, as a body of tag=value fields with no 8, 9, 34, 52, 141 or 10 in it: the transport frames it, numbers it 1, states the reset flag on it, and reads the heartbeat interval out of it rather than from any key. Write the logon first and the subscriptions after it — that order is the only way to express it, and a subscription written first is refused with nothing sent. A connect at which the adapter writes no logon is refused naming on_connected, because a transport that logged on with a body it composed itself would be signing for the venue.

That refusal is a fatal error, and whether a fatal error ends the process is that connection's role. On the one connection most publishers run it is exactly the startup failure it reads as, and the same on a role = "primary" or role = "upstream-partition" connection. On a role = "comparison" connection it is that connection's driver dropped and named on stderr, with its connection_state left at 0 and the publisher carrying on — so an adapter with a logon bug on a comparison connection gives a process that looks healthy while one upstream never connects at all. Read Several sources for one feed before relying on a startup failure to tell you, and watch dz_publisher_ingress_connection_state per connection rather than the process being up.

A logon whose signature covers 34 or 52 cannot be composed here at all, and that is a limitation and not a position. The list above refuses both tags, and what makes them unstatable is more than the refusal: the sending time is stamped when the transport frames the message, the sequence belongs to the session, and the transport takes both after on_connected has returned — so nothing on that path can tell an adapter what either value is about to be. A venue whose scheme signs a canonical string over SendingTime or MsgSeqNum — SendingTime | MsgType | MsgSeqNum | SenderCompID | TargetCompID into RawData is what several ask for — therefore composes a signature the venue refuses, however correct the key and the canonical form in the adapter are. What that looks like in production is the part worth knowing before you meet it: a publisher that connects, logs on, is refused and reconnects for the life of the process, with dz_publisher_ingress_connect_failures_total{reason="unauthorized"} climbing, dz_publisher_ingress_connection_state at 0, and nothing in any log line that looks like a defect — the transport is behaving exactly as written and the venue is behaving exactly as documented. So read what your venue's logon signs before porting a build_logon into on_connected. dz-ingress-fix's own crate documentation holds the canonical statement of this, the seam it does and does not reach, and why the narrower change that would close it has ordering to settle rather than a parameter to add.

An instrument admitted mid-session reaches a subscription through poll_upstream, which is what a session transport needs and a re-subscribing one hides: subscriptions live on the session, so without it the instrument waits for a reconnect while the feed looks healthy. Write only what is outstanding there — nothing on that path is deduplicated.

The session's own keys are [source.upstream.session], a subtable of the venue's own upstream:

[source.upstream.session]
endpoint = "203.0.113.10:9443"        # host:port, no scheme
server_name = "session.example.com"   # what the certificate is verified against

TLS is on and there is no key to turn it off anywhere but a loopback endpoint. Both halves of that endpoint are checked at load: endpoint for being host:port, and the name the certificate is verified against — server_name, or the endpoint's own host where the key is absent — for being a name a certificate can name at all. A publisher that starts and then cannot connect is the shape worth avoiding, and on a role = "comparison" source it is a process that looks healthy with one upstream that never comes up. The outbound sequence resets at every logon and nothing is persisted, because a resend delivers deltas whose value has expired while the publisher's own snapshot recovery is a sequence-correct repair; a document asking for continuity with persist_sequence = true is refused at load, naming the key. A venue that requires continuity is one this transport does not serve.

4. Prove it offline before you point it at anything

[adapter.replay] substitutes recorded upstream bytes for the transport. The adapter cannot tell the difference, which is the property that makes the exercise worth anything: the real config document, the real registry, the real adapter, the real lowering, real sockets, real teardown.

Lay recorded payloads out in name order, point [adapter.replay] path at them, and read the other end with this repository's own Go parser — asserting values, not datagram counts. Every end-to-end exercise here does exactly that, and each one has found bugs that no unit test did.

A replay drives one connection, however many [[source]] blocks the document declares. A fixture directory is one recording and nothing in it says which connection a payload arrived on, so one replaying input replaces every enabled block and takes the primary's name. Replaying the directory once per block would publish every payload as many times as there are enabled blocks, and a race between two copies of one recording is not a race. So a replay is a single-upstream exercise by construction, and a publisher with several enabled blocks has to plan its offline proof around that:

Every payload reaches the adapter on the primary's connection, so nothing that turns on which connection delivered a payload runs offline — the primary's own entry is exercised and nothing beside it is: a mapping from a connection to the part of the upstream subscribed there, a first-copy-wins discard across redundant paths — the second copy never arrives, so the discard never runs — or on_connected and on_disconnected for any block but the primary's. The per-conn state failure in Several sources for one feed is exactly the failure a replay cannot reproduce.
Metric pre-creation reads the input list after the substitution, so the two families labelled by connection — dz_publisher_ingress_connection_state and dz_publisher_ingress_messages_total — come up with one connection value in a replay run rather than one per enabled block. This is the quiet one: the run publishes and nothing names the narrowing, and a venue checking that every series renders at startup gets a pass covering one connection out of however many it enabled. That check belongs against a live run.
What a replay does settle for several blocks is the document and the composition: the roles resolve, the venue's main builds one Input per enabled block, and the enabled set and the built set are held equal — so an enabled block the binary does not build is a startup failure offline too. A block carrying enabled = false is skipped before either set is formed, so it is declared, is not built, and is not a startup failure. Those transports are built and never connected, and a line on stderr names how many payloads are being replayed, from where, and under which connection.

For the single upstream source the mechanism was built for, that bounds nothing: the recorded bytes drive the whole of run() — the config document, the registry, the adapter, the lowering, the sockets and the teardown — which is the strongest offline proof this runtime offers.

A replay that reaches the end of its recording exits non-zero, so the exit code is not the thing to gate on. The payloads run out, the driver reads that as a connection that closed and reconnects, and the second connect is refused — a replay that reconnected would fold the same deltas into the book twice. That refusal is fatal, the replaying input carries the primary's connection, and the first primary to give up ends the process through the consistency guard, which is a failing exit code. rust/publisher/dz-publisher-runtime/examples/replay.sh runs the publisher accordingly. What an offline proof asserts on is what came out of the sockets — the records a subscriber decoded, read with this repository's own Go parser — together with the line stderr carries per replayed payload.

5. Configuration

One document, checked at startup, with unknown keys refused — so a misspelled section is a startup failure rather than a default nobody noticed.

venue = "a-venue"

[egress]
pin = "192.0.2.10"      # the source address to send from, not discovered
ttl = 1                 # required; no default. 1 is the attached segment only

[[feed]]
spec = "top-of-book"
# shard = "alpha"           # optional; absent is the default shard
channel_id = 0
source_id = 1
multicast_group = "233.252.0.10"
mktdata_port = 41000
refdata_port = 41001
# snapshot_port = 41002     # depth feeds only
# snapshot_cycle = "5s"     # depth feeds only: one full rotation of the book
heartbeat_interval = "1s"
definition_cycle = "30s"
manifest_cadence = "5s"
idle_guard = "30s"

[refdata]
state_dir = "/var/lib/a-venue-publisher"
# forget_delisted_after = "168h"   # absent keeps every Instrument ID for good
# id_allocation = "derived"        # absent is "sequential"; see below before setting it

[refdata.selection]
bootstrap_top_n = 64
max_published = 256
warn_published_above = 200

[ingress]
kind = "websocket"

[adapter]
kind = "a-venue-tob"

[adapter.upstream]
# the venue's own keys, deserialized by the venue's own code

[adapter.credentials]
# paths, never secrets

pin is not optional in practice. An egress that resolves its source address off the default route sends from the wrong interface the moment the feed lives on a tunnel — and the IGMP report leaves by the wrong path too, so the symptom is silence that reads as a clean feed.
ttl has no default, and a document that omits it is refused at load. State the hop count the group's path actually takes; if the subscribers are on the attached segment, that value is 1. It is required because a wrong hop count is invisible from every direction an operator can look: a locally attached subscriber receives, so a check on the publisher's own host passes; every datagram is sent successfully, so nothing in the egress series moves, because the kernel accepted each one and a router discarded it; and a subscriber that never joined has nothing to number, so gap detection reports nothing either. ttl = 0 is refused as well: it is not a smaller hop count than 1 but a datagram the kernel accepts and no interface ever carries.
source_id and channel_id are identity on the wire. Two publishers sharing a Source ID on one group are indistinguishable to a subscriber's gap detection. Two [[feed]] blocks in one document sharing a channel_id are refused at load naming both, rather than started: the metric families keyed on channel_id cannot tell two channel instances apart, so the pair would pre-create one set of series and both would write to it with nothing saying so.
shard is optional, and an operator needs it only when one process carries several channels of one specification. Absent means the default shard, which is what a publisher with one channel per specification has always been, so a document that never names one keeps meaning what it always meant. Blocks sharing a shard carry one published set — one Instrument Count, one Manifest Seq, one definition cycle — while each block stays its own channel instance, with its own sequence series, Reset Count and era. The venue's adapter admits an instrument by shard name and still cannot name the Channel ID that shard resolves to; the mapping between the two is this document's. Every shard carries a block for every specification any shard carries, and a document leaving one out is refused at load naming the shard and the specification it has no block for: an instrument admitted to a shard with no top-of-book block would have quotes that reach no wire.

A name is one lowercase path component of at most sixty-four bytes, because it becomes one. Spelling the default shard's own token is refused rather than taken as a synonym: two spellings of one shard would be two era files. And the default shard keeps <spec>.era under [refdata] state_dir rather than gaining a shard component, so adding the key to a document for the first time renames no state a running deployment reads — a renamed era file reads as no era file, and a publisher that has been on era 7 for months would restart on era 1 and announce nothing.
definition_cycle and idle_guard are one answer per publisher, even though they are written per feed. One reference-data registry serves every feed, because Instrument ID identity can only be one thing, and the idle guard measures one publisher's silence. Two enabled feeds stating different values is a startup error naming both, rather than the first block's answer quietly winning.
forget_delisted_after is what bounds the state record, and it has a cost. Each new instrument appends one line to [refdata] state_dir, and a delisted one keeps its entry so that a relisting gets its own Instrument ID back. For a venue that lists short-lived instruments, the entries have no end unless this key is set. Set, an entry is dropped by the first compaction written after the instrument has gone that long unpublished, and a symbol the venue relists after it has been dropped is minted a new Instrument ID: a subscriber holding the old one sees an instrument end and a different one begin. A relisting before that compaction keeps its own ID. State it longer than any gap after which the venue relists a symbol it delisted. No Instrument ID is ever re-issued, whatever it is set to. Anything under a second is refused, and a fraction of a second above that is rounded up. A compaction the disk has no room for leaves the record as it is, and the publisher goes on appending to it and serving every Instrument ID in it.
id_allocation = "derived" is for a channel published from more than one host. Under the default, "sequential", an Instrument ID is the next number in the order this process first saw each instrument, so two hosts carrying one channel agree only while they see every listing in the same order. Under "derived", a symbol the record does not hold gets the CRC-32 of its Symbol, at or above a floor that is the record's next_id when the key was first set. Two hosts then agree on every ID without sharing anything. A derived ID that is zero, below the floor, or already held is declined, not moved, and the publisher names the symbol, and the holder if there is one, in a log line. One divergence is left: two hosts that first admit a colliding pair in opposite orders hold the ID under different symbols. It needs a CRC-32 collision, and is reported on both hosts rather than prevented. Three rules come with it:
- Every host of the channel starts from the same record, or all of them start from none. Seed a new host in two steps. First switch the host it is seeded from to "derived", with a restart. Then copy that host's instruments.state into the new host's state_dir before the new host first starts: not the era file and not writer.lock. The copy can be taken while the source runs. A copy of a host still minting sequentially goes stale with its next mint, which is why the switch comes first. A host started cold derives IDs for symbols a seeded one holds below its floor, and disagrees on every one.
- It forgets nothing. It is refused together with forget_delisted_after.
- There is no way back. A record written under "derived" is refused by a later "sequential" start.
A depth feed with no snapshot_cycle cannot be joined mid-session. It still emits the recovery snapshots a reset obliges, but a subscriber that arrives after the deltas started has nothing to build a book from — a level update states the resting quantity at a price, so nothing later corrects a missing start. The publisher says so at startup; both shipped publishers run a periodic snapshot, both at five seconds.
Addresses in examples are documentation ranges (RFC 5737 and MCAST-TEST-NET). scripts/check-public-repo-rules.sh enforces that for code in this repository, and it is the same habit worth keeping in a config template: an address in an example is copied into production sooner or later.

Several sources for one feed

A venue often carries the same book twice by different paths — a websocket and a FIX session, a local socket and a remote stream, two validators of one chain. They are not the same stream: conflation differs, per-connection sequencing differs, and each arrives at its own moment. So which one publishes is a decision, and [[source]] is where it is stated:

[[source]]
name = "ws"                 # the `connection` metric label, from the file
ingress = "websocket"
role = "primary"            # this one publishes

[source.upstream]
# the venue's own keys, for this source

[[source]]
name = "fix"
ingress = "fix"
role = "comparison"         # connected, driven, counted — for the race

[source.upstream]

Exactly one primary, publisher-wide, and it is a startup error otherwise. Two primaries are two publishers' worth of events on the channel instances they reach: the Sequence Number series is per channel instance, so a subscriber's gap detection reads the two interleaved as its own losses and cannot tell which. None is a publisher whose data has no path to the wire at all, heartbeating channels it never fills.

Publisher-wide and not per feed, and that is not a simplification. A per-feed rule would be a statement about routing this runtime does not do: every source's payloads reach one adapter, the adapter emits events, and no event carries the source it came from — so nothing here can confine one source's data to one feed. There is no carries key for the same reason. A key that reads as a partition while nothing partitions is worse than no key: it made two primaries with disjoint declarations resolve cleanly while both upstreams' events landed on one channel instance.
An upstream that arrives partitioned declares one primary and one upstream-partition per further connection. A partitioned upstream is one whose connections each carry instruments no other connection carries — a venue that splits its subscriptions across sessions, or admits only so many instruments per connection. Losing one of those loses that subset entirely, so role = "upstream-partition" says one thing about a live run: its fatal error ends the process, exactly as the primary's does.
```
[[source]]
name = "ws-a"
ingress = "websocket"
role = "primary"              # the one the published set is defined against

[[source]]
name = "ws-b"
ingress = "websocket"
role = "upstream-partition"   # carries what `ws-a` does not
```
Declare that second connection comparison instead — the only other role that is not a primary — and a fatal error on it is a driver dropped, a gauge at 0, and a publisher that carries on serving the instruments it still gets: half the published set goes stale while the process reports itself healthy, and the idle guard sees nothing because the surviving connections keep the aggregate busy.

Which connection is nominated primary is the operator's choice, and the one-primary rule counts it and nothing else — so a partitioned upstream is one primary however many connections it arrives on. The role states no routing: it names no instruments, no feeds and no shards, every payload from it reaches the one adapter exactly as the primary's does, and the adapter is what knows which instruments arrive where. It is not a carries key under another name, and what makes it statable is that it describes nothing but which failures are fatal and, at load, whether two of these blocks may state one credential.

What it covers is a fatal error, and that is narrower than "the partition stopped working". A driver returns only on IngressError::Fatal, which is any non-retryable connect, send or receive operation on that connection — a configuration fault found at connect most of all, and equally a message the transport cannot carry at all; Connect and Ended are the retryable cases, retried for ever with no attempt limit. So a partition whose session the venue stops accepting mid-run, or whose host stops resolving after start, reconnects at the backoff ceiling and the process does not end: its instruments go stale, its connection_state flaps or sits at 0, and the venue-wide idle guard stays quiet because the other connections keep the aggregate busy. Watch dz_publisher_ingress_connection_state and dz_publisher_ingress_reconnects_total per connection, and alert on a partition's gauge at 0 the same way you would on a single-connection publisher's.

One venue account across the partitions is statable, which is the shape most of them arrive in. The shared-credential rule below refuses two enabled blocks when one's whole credentials table appears in the other's, and this is the one pair of roles it is not asked about: a primary beside an upstream-partition, or two upstream-partition blocks, may write the same key_path and the document loads. A venue that permits several sessions per credential and hands out one credential leaves no other way to write it, and the refusal's advice — give each block its own credential, or disable one of them — would have left you copying one key file to as many paths as you have connections.

What the exemption does not do is promise the venue agrees. Whether one credential may hold several sessions is the venue's answer, not this file's: if it permits one and you write one, the second logon evicts the first, and the symptom is dz_publisher_ingress_reconnects_total climbing on two connections together with dz_publisher_ingress_connection_state alternating between them. Give each connection its own credential when the venue issues them, and read the exemption as the file getting out of your way rather than as a statement about the venue.

A comparison block is not exempt, in either direction, and neither is a second primary. A block in either of those roles carries the whole book, so a second logon of it on one credential is the copy-paste the rule exists to refuse. A venue whose connections need no credential writes none, and several blocks with no credentials table are not two logons with one credential.
The transport is named once. Either [ingress] kind for a publisher with one source, or [[source]] ingress per source. Both is refused: a key read only when another is absent is a key an operator cannot reason about. A document that names it per source need not write [ingress] at all.
A polled source states its own endpoint and cadence. dz-ingress-poll reads endpoint and poll_interval out of that source's own [source.upstream] block — a transport's own keys are the transport crate's to parse, the same rule that keeps [ingress] in one crate — so the venue's main deserializes that block and builds the transport from it. It is an interval and not a cycle: definition_cycle and snapshot_cycle are one pass over a set divided by the set's size, and one tick here is one request with no set to divide by. A polled source that omits it does not parse, because a transport with no cadence polls in a loop or never. There is no default and there is a floor: 50ms is the shortest cadence the transport runs, which refuses the unit slip — "1ms" where "1m" was meant is one character, and a catalogue asked a thousand times a second is how a publisher's address gets blocked rather than merely being wrong.

Keep poll_interval well under [ingress] idle_timeout. The driver hands a receive what is left of the idle guard, so a cadence longer than the guard spends that budget and reports idle before the next request is due: the driver ends the connection with timeout, reconnects, polls once, and does it again. poll_interval = "60s" under idle_timeout = "30s" gives one payload per window with connection_state flapping and reconnects_total{reason="timeout"} climbing, on a document that reads as correct. Nothing refuses it at load — the transport cannot see [ingress], and [ingress] does not know a source is polled — so this is the rule that has to be read rather than enforced. An unchanged catalogue is liveness and does not reset the guard, which is deliberate, so the guard has to be long enough for a poll that answers.
The name in the file is the metric label. dz_publisher_ingress_* carries connection, pre-created at 0 for every declared source, so a second upstream that never came up is a series sitting at zero rather than no series at all. A name with leading or trailing whitespace is refused rather than trimmed — "ws" and "ws " would be two series a dashboard cannot tell apart. Under [adapter.replay] that pre-creation follows the one replaying input instead, so connection takes one value whatever the document declares — a narrower label, not a single series, since dz_publisher_ingress_messages_total keeps a child per message_type under it: see Prove it offline before you point it at anything.
One session per source, and per nothing else. One driver is opened per enabled [[source]], so a publisher carrying sixty-two channel instances of one feed specification over one source opens one upstream connection: a shard partitions the published set and says nothing about how many upstream connections exist. That matters on a session transport, because a venue may permit one session per credential and answer a second logon by evicting the first.

So two enabled sources are refused at startup, naming both blocks, when one block's whole credentials table also appears in the other's. That is the copy-paste failure: a second block with a new endpoint and the credential nobody changed — equal tables where it was copied and left alone, and one table inside the other where a passphrase_path was added afterwards and the key_path beside it was not.

Containment, and not any key the two happen to agree on: the keys under credentials are the venue adapter's, so nothing in the runtime can tell an identity path from a trust root, and two accounts each holding their own key_path while pointing one ca_path at the same bundle are an ordinary deployment. The question the check can answer without reading a key's name is whether anybody edited one — a table that disagrees with another on a key they both write is one somebody edited, and a table lying wholly inside another is one nobody finished editing.

What the check cannot see is two different paths holding the same account — nothing here can know that, and its symptom is both sources reconnecting in step, with dz_publisher_ingress_connection_state alternating between them. The venue's own logon refusal is the authority.

The one pair of roles this is not asked about is a primary beside an upstream-partition, or two upstream-partition blocks: those are several sessions of one venue account by definition, and one credential between them loads. Every pair involving a comparison is refused as above, and a second primary is refused by the one-primary rule, so the copy-paste is intact — a block duplicated and left alone carries the role it was copied from, and reaching the exemption costs you the one line copy-paste is defined by not editing. See the upstream-partition bullet above for the limit of it.
Absent [[source]] is one source, named by the transport the venue builds. Every document written before the array existed still means exactly that, including that its fatal errors end the process.

The runtime does not merge two sources — the venue does. Every source reaches one adapter, and each payload carries the connection that delivered it, so the adapter decides which of two prices is current and when to fail over. That is the same rule as the book state machine, for the same reason: it follows the venue's microstructure, and one shipped publisher already reconciles two validator streams this way with a reorder window and a grace fallback.

Two consequences an operator has to know before configuring a second source.

role is not a gate on what reaches the wire. The runtime cannot keep a comparison connection off it, because the adapter emits events and no event carries the connection it arrived on. What it buys is the name in the file, the startup check above, what an analysis tier reads to know which side of a race is which — and one runtime behaviour: whether a fatal error on that connection ends the process. A driver returns only on a fatal error, and those are the non-retryable connect, send and receive operations: an invalid endpoint, a missing credential path, an unsupported scheme, a message the transport cannot carry at all. A fatal error on the primary ends the process, and so does one on an upstream-partition, because what either carries the published set depends on. A fatal error on a comparison does not: a mistyped URL on one is that connection's driver dropped and named, with its connection_state left at 0 — the alert for exactly this case — and the primary carrying on.

Nothing retries that source, and a restart is what does. Some of those causes are only fatal for one attempt — a credential path that does not exist yet, under late secret injection, is the plain one — and before this the process exited and both sources came back together. So the trade has a cost, and it is this: a fault that used to clear on a restart the process took itself now needs one somebody takes, and the gauge sitting at 0 is the only thing that says so. Watch dz_publisher_ingress_connection_state == 0 per connection, not just the process being up.

An adapter serving two sources must key its per-connection state by conn. One adapter object receives every source's on_connected and on_disconnected, so an adapter that keeps one upstream sequence cursor, one authentication token, or one "have I subscribed yet" flag is correct with one source and silently wrong with two: a comparison connection flaps, the primary's cursor is cleared, and the primary's next payload is read as a discontinuity — which an adapter that answers discontinuities with a reset turns into an InstrumentReset and a recovery snapshot on the live wire, from a connection that publishes nothing. Migration is one config block and one line in the venue's main with the adapter untouched, which is exactly why this is worth reading first — and an offline run is not what catches it, because a replay drives one connection, the primary's, and no other block's on_connected ever runs. See Prove it offline before you point it at anything.

The recorder

The recorder is agnostic to the feed, to the venue, and to whether it is reading a socket or a file, and nothing in its record path decodes a datagram — a message a decoder would reject is exactly the message the archive must hold, because the evidence needed to diagnose that bug is what the bug destroys. So recording a new feed is configuration, not code.

See rust/recorder/README.md.

site = "a-site"
recorder = "recorder-1"
env = "prod"

[[feed]]
spec = "top-of-book"
multicast_group = "233.252.0.10"
interface = "dz0"          # the device, not the address, in AF_PACKET mode
mktdata_port = 41000
refdata_port = 41001
expected_sources = ["192.0.2.10"]
expected_channel_ids = [0]

[capture]
mode = "afpacket"          # the network's copy, not one socket's
buffer = "64MiB"

[archive]
# rotation, compression, staging budget, where objects land

[health]
[metrics]

Three things to know before the first run:

afpacket needs CAP_NET_RAW and an Ethernet capture device. The parse reads a 14-byte Ethernet header, so a handle on any other datalink is refused at open rather than drained — a tunnel with no link layer of its own opens on bare IP, and without the refusal every frame fails the parse, nothing is archived, and the recorder reports itself healthy against a live feed. Point the feed at a device that has an Ethernet link layer, or run the recorder in socket mode, which records on any device and declares its synthesised headers as synthesised. Socket mode is recorder-wide, so it applies to every feed in that configuration.
expected_sources and expected_channel_ids gate counting and alerting, never the archive. A wrongly recorded datagram is filterable afterwards on its source address; a wrongly dropped one is gone.
The staging budget is enforced, and it evicts. A recorder that blocked on a full disk would stall its drain thread and turn an object-storage outage into a feed-loss incident. It gives up history instead, counts what it gave up, and never gives up the segment it is publishing.

Pointing a feed at a mode

The configuration above is archive mode: dz-recorder writes objects and dz-recorder-load derives rows from them. It is what a host recording a production feed for evidence runs, and the [archive] staging_dir and completed_dir in that file are what select it — there is no flag:

dz-recorder --config /etc/dz-recorder/recorder.toml

Neither arrangement is a default and no flag names either. Each is selected by the resource only it can run on, and the four cases are all the cases there are:

`[archive]` directories	`--inline-config`	What runs
stated	not given	archive mode
not stated	given	inline mode
stated	given	refused, naming the key and the file
not stated	not given	refused, naming both

So a host cannot drift between the two: changing arrangement takes an edit that positively states the new one, and every half-finished edit is a non-zero exit code before a socket is bound. Deleting the [archive] section to stop keeping bytes for an afternoon reaches the fourth row and refuses — it does not become inline mode. And nothing in any unit file, pipeline or runbook needs an edit for this: an archive host selects its arrangement by saying what it already said.

Inline mode is the other arrangement — one process that captures the feed, derives its rows and loads them, keeping no datagrams. It is for bringing a feed up, for a host that was never going to keep the bytes, and for one deploy unit. What it gives up is not small: a conformance rule written next month has nothing to run against, a row cannot be re-derived, nothing verified the bytes the rows came from, and no market data rows are derived at all — event, instrument and book_top come from a codec walk that archive mode runs in the loader. Read the two modes before putting a host in it.

Two files, and neither has a password key. The feed above stays in the recorder's own file — group, ports, expected_sources, site and recorder — because that file's config_hash is written into every row as provenance and a database endpoint is not part of what a recorder does. Inline mode's own file carries the window bound, the ring, the spool and its budget, the ledger and the destination, and is passed beside it:

dz-recorder --config /etc/dz-recorder/recorder.toml \
            --inline-config /etc/dz-recorder/inline.toml

# /etc/dz-recorder/inline.toml — see rust/recorder/dz-recorder/inline.example.toml
[inline]
window_bytes    = "16MiB"
window_interval = "10s"
ring_datagrams  = 8192
spool_dir       = "/var/lib/dz-recorder-inline/spool"
spool_max       = "8GiB"
ledger          = "/var/lib/dz-recorder-inline/ledger.jsonl"

[clickhouse]
endpoint = "http://192.0.2.20:8123"
database = "recorder"
user     = "dz_loader"

The [archive] section comes out. Nothing writes an object here, so a configuration stating both this file and a staging_dir or completed_dir is a contradiction rather than a mode, and is refused at startup naming the key and the file. Ignoring it quietly is how a host is believed to be keeping bytes for a year that it never kept for a second.
A configuration stating neither an archive nor this file is refused too, naming both. Inline mode needs a spool directory, a ledger and a destination, archive mode needs its two directories, and there is no defensible value to invent for any of the five: a recorder that guessed a destination would load rows into a database nobody chose. Silence is not a mode.
No [[market_data]] entry is accepted, and one is refused by name rather than ignored. event, instrument and book_top come from a codec walk, and nothing in the record path decodes a datagram — the rule that makes the transport grains trustworthy, since a message a decoder would reject still carries the sequence number whose absence is the finding. A feed whose market data rows are wanted runs archive mode, and the loader derives them.
spool_dir must exist and be writable by the service user, and ledger must not be inside it. Both are refused at startup, the second for the reason the loader's ledger may not live inside its objects directory: a file the budget cannot classify is a file eviction cannot reach.
The password comes from DZ_LOADER_CLICKHOUSE_PASSWORD_FILE — a systemd credential, readable by the service user alone — or from DZ_LOADER_CLICKHOUSE_PASSWORD, and from nowhere else. That is where the loader's already comes from; inline mode invents no second mechanism.
The build carries the mode by default. inline is a default feature, because the arrangement is a property of a host's configuration and the released asset is one asset for the fleet — a build carrying one arrangement would have to be matched to configurations at deploy time. Add --features tls for an https destination: that endpoint is refused rather than silently downgraded to plain HTTP with the password on the wire. A binary built with --no-default-features is the record-only one and can only be in archive mode, so it refuses a configuration that selects inline mode by the feature's name.

--check is what the deployment pipeline runs before it restarts anything, in either mode, and it is an ExecStartPre in the units:

dz-recorder --config /etc/dz-recorder/recorder.toml --check
dz-recorder --config /etc/dz-recorder/recorder.toml \
            --inline-config /etc/dz-recorder/inline.toml --check

It validates both files, plans the feeds — every refusal archive mode makes about a group, a port role or an interface is made here too — prints which arrangement is running and what it keeps, and asks the destination for SELECT 1. The arrangement is the first line of its output in both modes, so a deployment pipeline can read which one it has just deployed rather than infer it from which keys were echoed back. It is also where a configuration stating both arrangements, or neither, is caught: run as an ExecStartPre, an unclear arrangement costs a failed pre-check rather than a recorder that will not start. Nothing is bound, nothing is created and nothing is joined, and the spool and the ledger are not touched, so it is safe against a host that is already recording. A gate that passed without reaching the destination would let a pipeline restart a recorder that cannot write.

Alert on the age of the oldest unposted window, and never on the eviction counter. A full spool budget evicts on every window at steady state by design, so the counter rises whether or not anything is wrong; one window older than the eviction horizon is history already gone, and no re-run recovers it. This is the same rule as dz_loader_oldest_unloaded_age_seconds in archive mode, over windows instead of objects. Size spool_max against the column-store outage the host intends to survive.

Auditing what was recorded

The point of keeping the bytes is answering did the publisher send what the spec says it must, and did it arrive? after the fact. The analysis tier reads an archive back and, for a feed whose publisher is built on these crates, can re-lower the venue's own recorded events offline and join them to the recorded multicast on (Instrument ID, Per-Instrument Seq) — a diff rather than a guess. Two limits worth knowing before relying on it: a factor the publisher applied that the wire does not carry cannot be recovered offline, and the join needs the recorded reference data as well as the market data.

The infrastructure repositories

This repository holds no addresses, no hosts and no inventory. What follows is the shape of the work and the order that matters; the values live in the infrastructure repositories, and each of those owns its own review.

Allocate the multicast group and ports. One group per feed, one port per port role, from the range your fleet owns. Two feeds sharing a group is the mistake that produces a subscriber decoding another feed's magic; write the allocation down where the next person will look for it, not only in the config that consumes it.
Decide the publisher's egress interface and its source address, and set pin to it. On a topology where the feed leaves over a tunnel, this is the difference between a live feed and silence.
Provision the host: the binary, a service unit, the state directory, the credentials as files with an owner and a mode, and — for a recorder — CAP_NET_RAW, the capture device and the staging disk. Publisher and recorder are separate concerns and usually separate hosts.
Pin the version. Deploy a version, not a branch. A pin that says which build is running is what makes an incident answerable; a role that installs latest is a fleet nobody can describe.
Roll subscribers before publishers whenever the wire changes. A subscriber implementing an older schema discards a datagram it cannot read rather than misreading it, so the symptom of the wrong order is silence, not errors — and silence is what a healthy feed also looks like from the publisher's side.
Apply the schema before rolling a recorder that writes a new column. The loader posts FORMAT JSONEachRow with the field names, and ClickHouse's input_format_skip_unknown_fields defaults to 1: at that default an insert naming a column the table does not have is accepted, and the field is dropped. A binary rolled ahead of its migration would load rows whose new column holds a value nobody wrote, and a view that excludes that value — which is what a view over a hash column has to do, since there is no honest default for a hash — would read every one of those rows as a row nobody recorded. The loader therefore posts every insert with that setting at 0, so the wrong order is a refused batch and a loud error naming the field and the objects, rather than the silence the rule above warns about. The objects stay unloaded and load by themselves once the migration is applied, which is why the rule is still the rule: a load that stops is a feed with a hole in it until somebody applies the file. Rolling a recorder back is unaffected — an older binary omits a field rather than sending an unknown one, which is a different setting the loader does not touch. The recorder's migrations are in rust/recorder/dz-recorder-clickhouse/db/clickhouse, numbered and applied in order, and each says what it costs a deployment that applies it late.
Canary one publisher, then walk the rest. Restarting an era is visible to every subscriber of that feed: a Reset Count change is a reset, and a subscriber is expected to drop its book and re-bootstrap. Doing that to a whole fleet at once is a decision, not a side effect of a deploy.
Scrape the metrics before you need them. The normative dz_publisher_* and dz_recorder_* sets are in rust/publisher/dz-publisher-metrics. The one series to alert on first is the connection-state gauge sitting at zero: a publisher whose upstream never came up looks identical, from the outside, to a market that is quiet.

Checklists

A venue on an existing feed

A new feed type, additionally

specified in edge-feed-spec and merged there first
golden vectors in testdata/golden/, asserted by both languages
codec crate with MAGIC and CARRIES transcribed from the message table
the lowering builds every message the feed carries
a Go parser exists, and a book-builder if the feed has book state
the feed is nameable in a recorder configuration

Navigation Menu

Repository navigation

BRINGING-UP-A-FEED.md

BRINGING-UP-A-FEED.md

Bringing up a feed

First: which of two jobs is this?

A new feed type

The publisher

1. Implement the adapter, in the venue's repository

2. Depend on these crates

3. Write the binary

4. Prove it offline before you point it at anything

5. Configuration

Several sources for one feed

The recorder

Pointing a feed at a mode

Auditing what was recorded

The infrastructure repositories

Checklists

FilesExpand file tree

BRINGING-UP-A-FEED.md

Latest commit

History

BRINGING-UP-A-FEED.md

File metadata and controls

Bringing up a feed

First: which of two jobs is this?

A new feed type

The publisher

1. Implement the adapter, in the venue's repository

2. Depend on these crates

3. Write the binary

4. Prove it offline before you point it at anything

5. Configuration

Several sources for one feed

The recorder

Pointing a feed at a mode

Auditing what was recorded

The infrastructure repositories

Checklists