|
| 1 | +--- |
| 2 | +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. |
| 3 | +# SPDX-License-Identifier: Apache-2.0 |
| 4 | +title: Standalone Selection Service |
| 5 | +subtitle: Select workers and account for reservations without forwarding inference requests |
| 6 | +--- |
| 7 | + |
| 8 | +## Overview |
| 9 | + |
| 10 | +The standalone selection service (`python -m dynamo.select_service`) exposes the |
| 11 | +KV router's worker selection and active-load accounting over HTTP. It does not |
| 12 | +forward model requests or own response streams. External runtimes such as Ray |
| 13 | +register their worker catalog, request a selection, contact the selected worker, |
| 14 | +and report the reservation lifecycle. |
| 15 | + |
| 16 | +The service combines: |
| 17 | + |
| 18 | +- KV overlap indexing from worker ZMQ events. |
| 19 | +- KV-aware and load-aware worker selection. |
| 20 | +- Explicit or atomic selection and reservation. |
| 21 | +- Best-effort active-load synchronization between selector replicas. |
| 22 | +- Startup KV index recovery from another selector or standalone indexer. |
| 23 | + |
| 24 | +## Build And Launch |
| 25 | + |
| 26 | +Build the Python bindings with the `select-service` feature: |
| 27 | + |
| 28 | +```bash |
| 29 | +cd lib/bindings/python |
| 30 | +VIRTUAL_ENV=../../../.venv ../../../.venv/bin/maturin develop --uv --features select-service |
| 31 | +``` |
| 32 | + |
| 33 | +Launch the service from the repository root: |
| 34 | + |
| 35 | +```bash |
| 36 | +.venv/bin/python -m dynamo.select_service --port 8092 |
| 37 | +``` |
| 38 | + |
| 39 | +The service binds to `0.0.0.0` and does not provide authentication. Run it on a |
| 40 | +trusted internal network or place it behind an appropriate network policy. |
| 41 | + |
| 42 | +### CLI |
| 43 | + |
| 44 | +| Flag | Default | Description | |
| 45 | +|------|---------|-------------| |
| 46 | +| `--port` | `8092` | HTTP server port. | |
| 47 | +| `--threads` | `4` | KV indexer worker threads. | |
| 48 | +| `--indexer-peers` | none | Comma-separated HTTP URLs used for startup KV recovery through `/dump`. | |
| 49 | +| `--replica-sync-port` | none | Local ZMQ PUB port for active-load lifecycle events. The selector binds `tcp://*:<port>` internally. | |
| 50 | +| `--replica-sync-peers` | none | Comma-separated ZMQ PUB endpoints for selector peers. Requires `--replica-sync-port`. | |
| 51 | + |
| 52 | +Router scheduling behavior continues to use the standard Dynamo router |
| 53 | +environment configuration. |
| 54 | + |
| 55 | +## Worker Registration |
| 56 | + |
| 57 | +Every selector replica must receive the same worker catalog before it serves |
| 58 | +selection traffic. Replica traffic never creates workers. |
| 59 | + |
| 60 | +```http |
| 61 | +POST /workers |
| 62 | +Content-Type: application/json |
| 63 | +
|
| 64 | +{ |
| 65 | + "worker_id": 1, |
| 66 | + "model_name": "model", |
| 67 | + "tenant_id": "default", |
| 68 | + "endpoint": "http://worker:8000", |
| 69 | + "block_size": 16, |
| 70 | + "data_parallel_start_rank": 0, |
| 71 | + "data_parallel_size": 2, |
| 72 | + "kv_events_endpoints": { |
| 73 | + "0": "tcp://worker:5557", |
| 74 | + "1": "tcp://worker:5558" |
| 75 | + }, |
| 76 | + "replay_endpoint": "tcp://worker:5560" |
| 77 | +} |
| 78 | +``` |
| 79 | + |
| 80 | +`POST /workers` returns `201`. `PATCH /workers/{worker_id}` updates supplied |
| 81 | +fields, `DELETE /workers/{worker_id}` removes the worker, and `GET /workers` |
| 82 | +lists catalog state. `model_name` and `tenant_id` scope all selection, indexer, |
| 83 | +and load state; both default to `"default"` when omitted. |
| 84 | + |
| 85 | +`GET /health` is process liveness. `GET /ready` returns `200` only after at |
| 86 | +least one worker is schedulable, otherwise `503` with lifecycle details. |
| 87 | + |
| 88 | +## Selection API |
| 89 | + |
| 90 | +### `POST /select` |
| 91 | + |
| 92 | +Select a worker without booking active load: |
| 93 | + |
| 94 | +```json |
| 95 | +{ |
| 96 | + "selection_id": "select-123", |
| 97 | + "model_name": "model", |
| 98 | + "tenant_id": "default", |
| 99 | + "block_hashes": [11, 12, 13, 14, 15, 16, 17, 18], |
| 100 | + "sequence_hashes": [21, 22, 23, 24, 25, 26, 27, 28], |
| 101 | + "isl_tokens": 512 |
| 102 | +} |
| 103 | +``` |
| 104 | + |
| 105 | +### `POST /select_and_reserve` |
| 106 | + |
| 107 | +Select and atomically book load in the receiving selector process. Supply a |
| 108 | +globally unique `reservation_id`, or allow the service to generate one: |
| 109 | + |
| 110 | +```json |
| 111 | +{ |
| 112 | + "selection_id": "select-123", |
| 113 | + "reservation_id": "request-123", |
| 114 | + "model_name": "model", |
| 115 | + "tenant_id": "default", |
| 116 | + "block_hashes": [11, 12, 13, 14, 15, 16, 17, 18], |
| 117 | + "sequence_hashes": [21, 22, 23, 24, 25, 26, 27, 28], |
| 118 | + "isl_tokens": 512 |
| 119 | +} |
| 120 | +``` |
| 121 | + |
| 122 | +Both endpoints return the same selection shape: |
| 123 | + |
| 124 | +```json |
| 125 | +{ |
| 126 | + "selection_id": "select-123", |
| 127 | + "reservation_id": "request-123", |
| 128 | + "model_name": "model", |
| 129 | + "tenant_id": "default", |
| 130 | + "worker_id": 1, |
| 131 | + "dp_rank": 0, |
| 132 | + "endpoint": "http://worker:8000", |
| 133 | + "block_size": 16, |
| 134 | + "overlap": { |
| 135 | + "longest_matched": 128, |
| 136 | + "gpu": 64, |
| 137 | + "dp": {"0": 64, "1": 32}, |
| 138 | + "cpu": 96, |
| 139 | + "disk": 128 |
| 140 | + }, |
| 141 | + "effective_prefill_tokens": 384 |
| 142 | +} |
| 143 | +``` |
| 144 | + |
| 145 | +`selection_id` and `reservation_id` are omitted when absent. All `overlap` |
| 146 | +values are matched token counts. `gpu`, `cpu`, and `disk` use the cumulative |
| 147 | +Mooncake tier semantics documented in the standalone indexer's |
| 148 | +[per-instance tier breakdown](standalone-indexer.md#per-instance-tier-breakdown). |
| 149 | +A zero-overlap response includes the selected `dp_rank` with value `0`. |
| 150 | + |
| 151 | +The overlap summary is raw observability. `effective_prefill_tokens` is the |
| 152 | +authoritative weighted prefill-load value computed by the same cache-credit |
| 153 | +formula used for scheduler booking. It is not derived from `longest_matched`. |
| 154 | + |
| 155 | +The previous public fields `cached_tokens` and `effective_overlap_blocks` have |
| 156 | +been removed. Their values remain internal scheduler inputs. |
| 157 | + |
| 158 | +## Ray Select-Then-Reserve Flow |
| 159 | + |
| 160 | +Ray can keep model invocation separate from selector admission: |
| 161 | + |
| 162 | +1. Call `POST /select`. |
| 163 | +2. Send the request to the returned `endpoint` and `dp_rank`. |
| 164 | +3. Call `POST /reservations` with a globally unique reservation ID, selected |
| 165 | + worker identity, the same prompt representation, and the returned |
| 166 | + `effective_prefill_tokens`. |
| 167 | +4. Report prefill completion and request completion through the lifecycle API. |
| 168 | + |
| 169 | +```http |
| 170 | +POST /reservations |
| 171 | +Content-Type: application/json |
| 172 | +
|
| 173 | +{ |
| 174 | + "reservation_id": "request-123", |
| 175 | + "model_name": "model", |
| 176 | + "tenant_id": "default", |
| 177 | + "worker_id": 1, |
| 178 | + "dp_rank": 0, |
| 179 | + "sequence_hashes": [21, 22, 23, 24, 25, 26, 27, 28], |
| 180 | + "isl_tokens": 512, |
| 181 | + "effective_prefill_tokens": 384 |
| 182 | +} |
| 183 | +``` |
| 184 | + |
| 185 | +When supplied, `effective_prefill_tokens` is authoritative and directly enables |
| 186 | +prefill-load tracking. It must not exceed the normalized input sequence length. |
| 187 | +When omitted, existing router configuration controls prefill tracking. The |
| 188 | +reservation API does not accept or derive accounting from overlap fields. |
| 189 | + |
| 190 | +## Reservation Lifecycle |
| 191 | + |
| 192 | +```http |
| 193 | +POST /reservations/{reservation_id}/prefill_complete |
| 194 | +POST /reservations/{reservation_id}/output_block |
| 195 | +DELETE /reservations/{reservation_id} |
| 196 | +``` |
| 197 | + |
| 198 | +`prefill_complete` clears active prefill load. `output_block` updates only the |
| 199 | +receiving selector's local decode-block accounting and accepts an optional |
| 200 | +`decay_fraction` in `[0.0, 1.0]`. `DELETE` frees the reservation. |
| 201 | + |
| 202 | +**NOTE:** Output-block updates are intentionally not replica-synchronized. |
| 203 | +They can occur at high frequency, and broadcasting them would consume |
| 204 | +disproportionate network bandwidth. |
| 205 | + |
| 206 | +## Peer Planes |
| 207 | + |
| 208 | +The selector has two independent peer configurations: |
| 209 | + |
| 210 | +| Plane | Transport | Flags | Purpose | |
| 211 | +|-------|-----------|-------|---------| |
| 212 | +| Indexer recovery | HTTP | `--indexer-peers` | Fetch a compatible `/dump` during startup and replay KV events into the local indexer. | |
| 213 | +| Replica synchronization | ZMQ | `--replica-sync-port`, `--replica-sync-peers` | Share admission, prefill-complete, and free events by model and tenant. | |
| 214 | + |
| 215 | +Example: |
| 216 | + |
| 217 | +```bash |
| 218 | +.venv/bin/python -m dynamo.select_service \ |
| 219 | + --port 8092 \ |
| 220 | + --indexer-peers http://selector-b:8092 \ |
| 221 | + --replica-sync-port 9092 \ |
| 222 | + --replica-sync-peers 'tcp://selector-b:9092' |
| 223 | +``` |
| 224 | + |
| 225 | +Configure the reverse peer direction on selector B for bidirectional lifecycle |
| 226 | +synchronization. `GET /dump` exposes the selector's current indexer snapshot in |
| 227 | +the same recovery format as the standalone indexer. |
| 228 | + |
| 229 | +Replica-sync peers may also be changed without restarting the selector: |
| 230 | + |
| 231 | +```http |
| 232 | +POST /replica_sync/register_peer |
| 233 | +Content-Type: application/json |
| 234 | +
|
| 235 | +{"endpoint":"tcp://selector-b:9092"} |
| 236 | +``` |
| 237 | + |
| 238 | +The same body is accepted by `POST /replica_sync/deregister_peer`. |
| 239 | +`GET /replica_sync/peers` returns the sorted configured endpoints. Dynamic |
| 240 | +membership is in-memory; after restart, only peers supplied through |
| 241 | +`--replica-sync-peers` are restored. These routes only manage live ZMQ |
| 242 | +replica-sync peers. They do not alter the HTTP indexer-recovery peers. |
| 243 | + |
| 244 | +## Consistency Invariants |
| 245 | + |
| 246 | +- Replica synchronization is bounded and best-effort. Delays, reordering, |
| 247 | + dropped events, and temporary active-load divergence are accepted. |
| 248 | +- There is no sequencing, acknowledgement, replay, backpressure, or |
| 249 | + resynchronization for replica lifecycle events. |
| 250 | +- Unknown worker, model, tenant, DP-rank, and block-size events are dropped. |
| 251 | + Register the same worker catalog on every selector before routing traffic. |
| 252 | +- Admission, prefill-complete, and free are synchronized. Output-block growth |
| 253 | + remains local to avoid excessive network bandwidth. |
| 254 | +- Startup recovery waits for recovered events to be submitted to the indexer, |
| 255 | + not for complete processing. Early selections may temporarily miss recovered |
| 256 | + KV state. |
| 257 | +- `/select` followed by `/reservations` provides eventual, not atomic, |
| 258 | + cross-replica admission. Use `/select_and_reserve` for atomic local booking. |
| 259 | +- Reservation IDs must be globally unique. Existing conflict and retry |
| 260 | + semantics are unchanged; no idempotency ledger is added. |
| 261 | + |
| 262 | +## Inspection APIs |
| 263 | + |
| 264 | +- `GET /loads` returns active-load snapshots, optionally filtered by |
| 265 | + `model_name` and `tenant_id`. |
| 266 | +- `POST /potential_loads` estimates worker load for a prompt without selection. |
| 267 | +- `POST /overlap_scores` returns per-worker/per-rank tiered overlap rows. |
| 268 | +- `GET /dump` returns the compatible indexer recovery snapshot. |
0 commit comments