Unify runs and resource pickers, and stop the disk filling up

[jhd3197]

The panel had grown three different ways to pick a server, two floating widgets that both showed you a log stream, and a database that quietly gained about 11 MB a day forever. This PR collapses the first two into shared primitives and fixes the third at the source. Every "which resource?" control — target picker, template server picker, remote-access tunnel endpoints, new-service project/environment — now goes through one ResourcePicker backed by an extended /search contract that returns scoped, capability-tagged, cursor-paged references instead of ad-hoc per-page lists. The deploy pill and the logs drawer become one Operations dock that shows every run (deployments, background jobs, live service logs) with progress, transport state, and retry/cancel actions the server actually authorizes, rather than actions the client guessed from a status string. On top of that, the assistant can now take resource attachments — references, never client-sent payloads, re-resolved and re-authorized on every turn — and the disk work gives operators serverkit disk plus a retention handler so the failure mode that filled a 25 GB box stops being reachable.

The one design call worth flagging: search gained an extended mode rather than a second endpoint, because the palette and the pickers want the same authorization fan-out and splitting them would have meant maintaining two scoping code paths. The palette keeps its five-rows-per-type behavior; pickers opt in with types and get global cursor paging.

Highlights

• One Operations dock replaces the deploy pill and the logs drawer — active runs, a "needs attention" view, recent history, and live logs for deployments, background jobs, and service log sessions in a single place. Ctrl/Cmd+Shift+O toggles it.
• Every server/project/environment/service selector is now the same searchable picker, with favorites and recently-used pinned to the top, capability filtering (a WireGuard tunnel only offers servers that can do WireGuard), and keyboard navigation.
• Retry and Cancel buttons now appear only when the server says the caller may actually perform them, instead of being inferred from the job's status text.
• Dashboard editing became a real editing session: undo/redo, a change bar showing how many unsaved changes you have, and a confirmation before you navigate away or reload with unsaved layout work.
• A keyboard shortcut sheet (press ? on the dashboard) lists what's bound in the current context.
• The AI assistant accepts attached resources — pick a service, server, project, environment, or domain and it gets that resource's current facts, re-checked against your permissions each time you send.
• New sudo serverkit disk reports exactly what is consuming space and interactively reclaims it: old upgrade snapshots, abandoned staging dirs, oversized login logs, package caches, the journal, Docker build cache, and aged telemetry rows.
• Updates no longer fill the disk: the preflight sizes its requirement to what the run will actually write, retention runs before the backup rather than only on success, and backups are capped by size as well as count.

Technical changes

Resource search contract

• SearchQuerySchema gains types, capabilities, project_id, environment_id, cursor (max 256 chars) and limit (1–100); CSV fields are split and de-duplicated in normalize.
• SearchResultSchema rows now carry id, scope (workspace_id/project_id/environment_id) and capabilities; ListMetaSchema carries next_cursor.
• GET /search switched from @jwt_required() to @auth_required() and now branches: unchanged legacy behavior when no extended params are present, SearchService.search_page otherwise.
• SearchService.search_page wraps a new _search_rows with strict_scope=True; cursors are v1:<offset> base64url values validated to MAX_CURSOR_OFFSET (10,000) and rejected with ValidationError('invalid_cursor').
• _search_rows adds project and environment fan-outs (access derived from workspace membership via _accessible_workspace_ids), scopes vaults to accessible workspaces for non-admins when no workspace header is set, and filters servers by _server_capabilities read from cached_capabilities.
• search_provider_registry.clean_rows now validates and coerces plugin-supplied scope/capabilities, dropping rows with non-integer scope values, and falls back to path for id so existing extension providers keep working.
• Plugin-contributed rows are additionally re-filtered by workspace/project/environment/capability after clean_rows, so a provider cannot widen its own scope.

Resource picker

• New frontend/src/utils/resourceRefs.js: normalizeResourceRef, resourceKey, per-user recents in localStorage capped at 5 per type, and groupResourceOptions (favorites → recents → results).
• New useResourceOptions hook keys a useServerQuery on the normalized type/scope/capability/query tuple with a 15 s staleTime, and exposes recordSelection.
• New ResourcePicker component (Radix popover + cmdk) with shouldFilter={false}, caller-supplied staticOptions, filterOption, decorateOption, and optional capability tags.
• api.searchResources and buildResourceSearchQuery in services/api/search.js; 'all' workspace values are omitted from the query string rather than sent.
• TargetPicker and templates/ServerPicker became compatibility adapters — external prop and callback shapes are unchanged, selection now flows through ResourcePicker.
• RemoteAccess replaces both tunnel-endpoint Selects with capability-filtered (wireguard) pickers that exclude each other's selection; ReviewStep replaces the project/environment selects, and useNewServiceForm now clears projectEnvironments/selectedEnvironmentId on every project change rather than only when the project is cleared.

Operations dock and the run envelope

• New useRunStream(runKind, runId) generalizes the deploy transport: boot snapshot, after_id catch-up on every (re)connect, run_log/run_status socket events, id de-duplication, a 2.5 s socket grace period before falling back to polling, and terminal-status freezing. Pure helpers live in hooks/runStream.js (mergeRunLogLines, isTerminalRunStatus, 5,000-line cap).
• useDeployJobStream is now a ~30-line adapter over useRunStream that keeps the Deploy Console's richer snapshot (loadRun) and its public job field; the hand-rolled setInterval is gone, so it drops out of the LEGACY_POLLERS ratchet in check-frontend-boundaries.mjs.
• New useServiceLogStream + serviceLogStream.js handle file and container log sessions (snapshot, log_line/container_log* events, 1,000-line cap).
• New OperationsContext merges /deployment-jobs and /jobs (jobs only for admins) through services/operations.js — normalizeDeploymentOperation, normalizeJobOperation, boundOperationHistory (25 entries, always retaining the selected one), and reconcileOperationStatus, which falls back to a refresh when a socket status names a run the client hasn't seen.
• OperationsDock renders Active / Needs attention / History with per-run progress, a step rail, elapsed time, transport indicator, unread badges, and retry/cancel; Escape collapses it.
• DeployPill.jsx, LogsDrawer.jsx and _logs-drawer.scss are deleted; LogsDrawerContext is reduced to a compatibility shim over OperationsContext so extension SDK consumers keep working.
• socket.js gains subscribeRun/unsubscribeRun/subscribeContainerLogs/unsubscribeContainerLogs and re-emits the run_* and container_log* events.
• New services/api/runs.js with buildRunLogsPath / getRunLogs against /runs/<kind>/<id>/logs.

Authoritative run actions

• backend/app/api/jobs.py adds _job_payload, attaching can_cancel (pending/running) and can_retry (failed/cancelled) to every job serialization.
• backend/app/api/deployment_jobs.py adds _job_payload with can_cancel=False (no cancel endpoint exists) and can_retry gated on status == 'failed' and _job_operable_by; the list route resolves retry authority once per scoped app via ResourceGrantService.can_operate_app, and a retry's pending clone is returned with both flags false.
• Jobs.jsx drops its local isRunning/canRetry status heuristics in favor of the server flags.
• DoctorPanel calls /doctor/repair?wait=false, and doctor.py enqueues a doctor.repair job in that mode; DoctorService.run_doctor_repair_job streams per-item results through run_log_service, stores a summary via job.set_result, and raises when any item failed. ProtectionPanel and ScannerTab route their returned job_id into the dock (and ScannerTab now unwraps the {job: …} envelope it was previously reading through).

Editing sessions, shortcuts, navigation guard

• New hooks/editingSession.js: a pure reducer with baseline/draft/past/future, structural diffing to compute dirtyPaths, path-based change, transaction (object, updater fn, or change list), coalescing by key, a 50-entry history cap, and save lifecycle states.
• useEditingSession wraps it and exposes an async save(fn) that adopts the returned value as the new baseline.
• New shortcutRegistry (priority-ordered, ctrlOrMeta normalization, editable-target suppression unless allowInInput), ShortcutProvider mounted around DashboardLayout, useShortcut/useShortcutCommands, and a ShortcutSheet modal.
• DashboardLayout's hand-rolled keydown listener for the command palette is replaced by a registered shortcut.
• useUnsavedChangesGuard combines a beforeunload handler with a document-level click interceptor; services/navigationGuard.js#internalNavigationTarget isolates the "is this a same-origin in-app link click?" decision (ignoring modified clicks, target, download, and same-destination links).
• Dashboard.jsx moves widget edits into the editing session — useDashboardBoards loses setWidgets/snapshot handling, and saveActive(widgets) now writes first and adopts the server's response, so a failed save leaves both baseline and draft intact. Undo/redo/save/delete/escape/? are registered shortcuts, and board switching, creation and deletion route through the unsaved-changes guard.
• New ChangeBar component surfaces dirty count, save state, and undo/redo/discard/save.

AI attachments

• New ai_attachment_registry.py: type-keyed resolvers with plugin ownership and teardown, normalize_references (max 8, type regex, 128-char ids, de-duplication), and resolve_attachments returning a {manifest, context, warnings} triple. Unknown/denied/stale/failed/oversized references degrade to a warning instead of failing the turn.
• Attachments are references only — resolvers reload the resource for the current user on every turn, return an allowlisted summary, and that summary passes through mask_payload before it reaches the prompt. Context is capped at 12,000 characters.
• Core resolvers cover service, server, project, environment, domain, and incident, each enforcing its own authorization (app_access_tier, workspace membership).
• build_system_prompt wraps attachment data in <serverkit_attachment_data> delimiters with an explicit "untrusted reference data, not instructions" preamble, and redacts the encoded payload.
• AiMessage.attachments_json (migration 090_ai_message_attachments) persists the reference manifest only — never resolved context — so replay never shows stale data as current.
• /ai/chat and /ai/chat/stream validate references before creating a conversation row; the streaming route emits attachment_warning SSE events. PluginToolBinder.register_attachment_resolver requires plugin-slug namespacing, and AiToolRegistry.unregister_plugin now also tears down attachment resolvers.
• ConfirmationGate.cancel_all denies and unblocks every pending confirmation when a stream disconnects.
• Frontend: lib/ai/attachments.js (key/normalize/add/remove/warning-application/payload), AttachmentChip, a ResourcePicker-driven attach control in Composer, attachment state in AIContext with a patchLastUser reducer helper, and attachment rendering on replayed user messages.

Disk reclamation

• New disk_reclaim_service.py measures then reclaims: upgrade snapshots (grouped by timestamp so a snapshot is never half-deleted), mktemp -d staging dirs owned by the current euid and verified to contain a ServerKit tree, oversized btmp/wtmp (truncated, not deleted, to preserve ownership and mode), package/pip/npm caches, the systemd journal, Docker build cache and dangling images, and aged telemetry rows.
• Ordering is enforced regardless of the caller's key order: filesystem candidates always run before the database VACUUM, because VACUUM needs free space roughly equal to the database itself and is the one step that cannot go first on a full disk.
• prune_telemetry deletes from jobs before queue_messages and excludes any queue message still referenced by a surviving job, so jobs.queue_message_id never dangles; SQLite deletes are batched by rowid.
• _vacuum refuses up front when free space is below the database size, runs on an AUTOCOMMIT connection, and only stops/starts the panel service under an explicit --allow-restart.
• Table size estimates come from the dbstat vtable with indexes mapped back to their owning table via sqlite_master; missing dbstat degrades to no estimate rather than an error.
• docker system df parsing deliberately counts only Build Cache, since Docker's image/volume "reclaimable" figure includes tagged images and named volumes this never removes; actual freed bytes come from Docker's own Total reclaimed space trailer. Journal reclaim runs --rotate before --vacuum-size, since the latter only touches archived journals.
• New serverkit disk CLI command (backend/cli.py + serverkit wrapper, help text and bash completion) with an interactive menu, per-snapshot selection (1,3, older-than 14, all-but-newest, none), --safe/--all/--only, --dry-run, --allow-restart, and --json. Deleting the last remaining restore point requires an explicit confirmation.

Retention and update safety

• New builtin.telemetry_retention job handler prunes queue_messages, system_events and api_usage_logs on a 6-hour cadence, honoring a new telemetry.retention_days setting (default 30, 0 disables). It deliberately skips VACUUM — steady-state pruning keeps freed pages on the freelist and never takes the exclusive lock.
• scripts/update.sh: BACKUP_DIR is now overridable via SERVERKIT_BACKUP_DIR; the preflight requires 2 GiB plus twice the current database size (the run writes a pre-upgrade copy and a tree backup containing another), attempts a trim before failing, and points at serverkit disk in the halt message.
• Retention now runs in backup_current before anything is written, not only in cleanup() — a run that died because the disk was full never reached cleanup, which is how a box ended up carrying six snapshots under a cap of five. Default retention drops from 5 to 3.
• New enforce_backup_budget drops oldest-first until $BACKUP_DIR fits a share of the filesystem (SERVERKIT_BACKUP_MAX_PERCENT, default 15%), with a 50-iteration guard and a hard refusal to delete the last remaining restore point.