Configure and instrument simulation observability

[anth-volk]

Fixes #685

Summary

• own one policyengine-observability runtime in each simulation entry, gateway, and executor process
• sample 100% of traces in the API v1 simulation path
• use X-PolicyEngine-Observability-Id as the only HTTP transport for observability_id
• create the identifier only when a submission endpoint accepts work, and restore the persisted value when a status endpoint reads existing work
• keep health, version, invalid, and missing-job requests free of fabricated calculation identifiers
• persist the identifier with annual, budget-window, and Stage 12 functional state
• pass validated observability context to Modal functions as a separate keyword-only observability_context argument
• keep calculation payloads free of observability context and identifiers
• preserve the same identifier when budget-window and Stage 12 coordinators dispatch child simulations
• define all supported span names in the shared stage registry
• accept explicitly supplied safe scalar attributes in local logs and spans without maintaining an exhaustive simulation-owned list
• transport only job_id, observability_id, and simulation_id through Modal observability context and retain separate bounded metric labels
• retain submission_claim_id as cache-ownership metadata that is separate from diagnostic correlation
• accept the legacy _telemetry.process_id field only as a temporary alias for submission_claim_id, with an explicit removal comment
• preserve application responses and simulation results when runtime telemetry operations fail
• document the identifier registry, HTTP lifecycle, Modal transport, persistence, trace boundaries, attribute handling, and failure behavior for AI tools and operators

Deployment dependencies

1. Preserve asynchronous observability context policyengine-observability#32 is merged, version 3.0.1 is published, and this PR requires that version in every affected project and library lockfile.
2. Add Stage 12 observability identifier schema policyengine-api#3852 is merged; its nullable Stage 12 observability_id migration must be applied before Stage 12 writes that field.
3. Deploy this PR before Instrument and provision API v1 observability policyengine-api#3850. Stop older Modal workers during deployment because their function signatures do not accept the separate observability_context argument.
4. Deploy Instrument and provision API v1 observability policyengine-api#3850 after the simulation services are running this version.

The Modal signature change is intentionally breaking. This deployment does not call older Modal functions and does not maintain compatibility for the former payload representation.

Verification

• Ruff formatting and lint checks passed for the changed Python files.
• 11 simulation observability tests passed earlier in this PR; the 7 focused runtime configuration tests passed again with the published policyengine-observability 3.0.1 package after removing the local attribute list.
• 69 gateway endpoint tests passed.
• 96 executor tests passed.
• 13 simulation entry tests passed.
• A focused gateway-to-parent-to-child-to-poll test verifies that one UUID crosses every asynchronous boundary without entering the calculation payload.
• The branch rebased cleanly on the current main branch.

The Household API and UK Chat are outside this change.

[anth-volk]

The rollout dependencies are now explicit:

1. Merge Add Stage 12 observability identifier schema policyengine-api#3852 and apply its nullable Stage 12 observability_id migration.
2. Deploy this PR.
3. Deploy Instrument and provision API v1 observability policyengine-api#3850.

Commit 192cdca adds compatibility with the identifiers emitted by the currently deployed API, rejects conflicting old and new identifiers, and requires the executor deployment to finish before the gateway deployment begins. It also fixes the stale executor tests that failed in the previous CI run. The focused regression checks pass locally: 7 observability, 45 request contract, 19 deployment definition, and 53 executor tests.

[anth-volk]

Rebased this PR onto current main (6884241) after PolicyEngine/policyengine-api#3852 merged.

• Preserved the new Stage 12 deployment completion checks added on main.
• No additional simulation contract change was required for #3852: the PR still accepts run_id as observability_id and process_id as submission_claim_id during the deployment and rollback period, while emitting the new response fields.
• Confirmed the Stage 12 persistence model still stores the nullable 36-character observability_id expected by the merged API schema.

Checks run locally:

• Ruff formatting: 123 source files passed.
• Lock validation: all seven affected package lock files passed.
• Observability library: 23 passed.
• Simulation contract compatibility: 53 passed.
• Stage 12 persistence: 6 passed.
• Simulation entry: 96 passed.
• Simulation gateway: 195 passed.
• Simulation executor: 219 passed.
• Shared FastAPI observability: 11 passed.

The rebased head is adf5b89.

[anth-volk]

Commit 3e5114e establishes one canonical observability_id path.

• HTTP requests and responses use X-PolicyEngine-Observability-Id.
• The gateway persists that identifier with asynchronous job metadata and restores it on poll responses.
• Modal calls receive the identifier and trace data only through validated _observability_context.
• _telemetry retains submission_claim_id and other simulation metadata but contains no observability identifier.
• Legacy body identity fields are accepted and discarded during deployment compatibility; they cannot replace the header value.
• Public request and response JSON schemas no longer expose observability_id.

Deployment order: database precursor API #3852, this PR, then API v1 #3850. Within this deployment, update the executor before the gateway. Roll back API v1 #3850 before rolling this PR back.

The full repository test command passed. After the final compatibility adjustment, the focused observability, contract, and gateway suites also passed with 11, 49, and 69 tests respectively.

[anth-volk]

Follow-up commit 9113e1b removes the remaining Stage 12 fallback path:

• the HTTP request middleware's validated identifier is now required by the Stage 12 backend
• Stage 12 removes any identifier found in captured runtime context before inserting the explicit request identifier
• the coordinator no longer reapplies the identifier from the persisted parent record

The database field remains available for durable lookup and diagnostics. Runtime propagation into Modal now occurs only through _observability_context.

Verification: 67 simulation-entry tests and 29 Stage 12 executor tests passed together; the focused backend suite passed again with 13 tests, and Ruff formatting and lint checks passed.

[anth-volk]

Implemented the latest review fixes in 8940fc4:

• The shared FastAPI setup now installs an inner correlation layer after request tracing begins, so the canonical observability_id is attached to the active request span and request.completed record.
• This fixes both the Cloud Run simulation entry service and the Modal routing service, which use the same middleware ordering.
• Backend identity context is now attached while the entry service request runtime is active.
• Added regression tests that read the active runtime context from inside a request.

Verification:

• Full repository unit suite passed: 583 executor tests, 145 entry-service tests, 212 routing-service tests, 102 contract tests, 28 shared-observability tests, and 10 Stage 12 persistence tests (plus documented skips/deselections).
• Focused Ruff checks for all changed files passed.

The repository-wide make check command still reports existing lint findings in untouched modules and cannot find pyright after the test-only dependency sync. The changed files pass their focused Ruff checks.

[anth-volk]

Implemented temporary input compatibility in c402b59.

• Incoming telemetry now maps legacy process_id to canonical submission_claim_id.
• A canonical submission_claim_id takes precedence when both fields are present.
• run_id remains ignored, and the HTTP header remains the only source of observability_id.
• The compatibility code contains an explicit comment requiring removal after the API v1 deployment and rollback period.

Verification:

• The observability, contract, and gateway focused tests passed (12, 49, and 69 tests).
• Ruff formatting and lint checks passed for the changed files.

[anth-volk]

Implemented the canonical asynchronous correlation design in commits 47da9f5 through 91fa773:

• observability_id travels in the HTTP header and in a separate keyword-only Modal context argument.
• Calculation payloads no longer contain observability context.
• Submission endpoints create or preserve the identifier; polling restores it from functional state.
• Budget-window children and Stage 12 simulations preserve the parent identifier.
• The identifier and stage registries are documented in the AI-facing engineering guidance.

Focused simulation observability, gateway, executor, and entry tests passed. The remaining prerequisite is the 3.0.1 package release from PolicyEngine/policyengine-observability#32, followed by dependency and lockfile updates in this PR.