diff --git a/docs/architecture/assessment/README.md b/docs/architecture/assessment/README.md new file mode 100644 index 000000000..88d9568a4 --- /dev/null +++ b/docs/architecture/assessment/README.md @@ -0,0 +1,61 @@ +# AI Assessments — Getting Started + +**An assessment uses an LLM to grade your items against a rubric and gives you back a structured result** (scores, reasoning, feedback) for every item — not free text, but a fixed JSON shape you choose. + +You give Kaapi two things: + +1. A **config** — your rubric (the grading instructions), the model to use, and the exact result shape you want back. +2. Your **items** — the rows you want graded (text and/or image/PDF URLs). + +Kaapi grades every item and delivers the results to your **webhook**. + +--- + +## The whole flow in three steps + +| Step | You do | Kaapi does | +|---|---|---| +| **1. Create a config** | Save an `ASSESSMENT` config once (`POST /configs`) | Stores it, versioned | +| **2. Submit items** | `POST /assessments` with your rows + a `callback_url` | Returns an `assessment_id`, starts grading in the background | +| **3. Get results** | Wait for the webhook | POSTs the finished results to your `callback_url` | + +You never poll or wait on the request — submitting returns immediately, and the results arrive later at your webhook. + +![BATCH assessment flow](assets/batch-flow.png) + +**BATCH is fully batched.** Both stages run as provider **batch jobs** — the +pre-filters run as a batch, and the assessment runs as a batch. Results are +delivered to your **webhook** when everything completes (no polling). + +--- + +## Two methods (Kaapi picks for you) + +You never set a "mode". Kaapi looks at your input and decides: + +| Method | When | Input shape | Status | +|---|---|---|---| +| **BATCH** | Many items at once | `data` is a list of rows | ✅ Available | +| **RESPONSE** | A single item, fast | a single `query` | 🚧 WIP (returns `501` today) | + +This guide covers **BATCH**, the method that is live. + +--- + +## Supported models + +Pick the provider per config (and per pre-filter): + +| Provider | Value in config | Status | +|---|---|---| +| OpenAI | `openai` | ✅ | +| Google (AI Studio / Gemini) | `google` | ✅ | +| Anthropic (Claude) | `anthropic` | ✅ | +| Google Cloud / Vertex | — | 🚧 WIP | + +--- + +## Where to go next + +1. **[Configuration and versioning](configuration-and-versioning.md)** — build your rubric, choose the model, define the result shape, and manage versions. +2. **[API contract](api-contract.md)** — request/response fields, types, status values, and error codes. diff --git a/docs/architecture/assessment/api-contract.md b/docs/architecture/assessment/api-contract.md new file mode 100644 index 000000000..95ebe58e9 --- /dev/null +++ b/docs/architecture/assessment/api-contract.md @@ -0,0 +1,177 @@ +# API Contract — `POST /assessments` + +Precise request and response shapes for the BATCH assessment API. For a +walkthrough with context, see the [overview](README.md). + +Everything is delivered by **webhook** — there is no status or result poll +endpoint. RESPONSE-shaped input returns `501` (WIP). + +**Sample input / output JSON files:** +https://drive.google.com/drive/folders/1BCaauUuXr9DaZTWI-_-x101SDT4ktwp5?usp=share_link + +--- + +## Request + +`POST /assessments` + +| Field | Type | Required | Notes | +|---|---|---|---| +| `config` | object | ✅ | which saved config version to run | +| `config.id` | UUID | ✅ | config id (must be tagged `ASSESSMENT`) | +| `config.version` | int ≥ 1 | ✅ | config version to pin | +| `input` | object | ✅ | a `data` list ⇒ BATCH; a bare `query` ⇒ RESPONSE (501) | +| `input.query` | string (non-empty) | ✅ | template; `{column}` placeholders filled per row | +| `input.data` | array (≥ 1) | ✅ | rows; each row is a flat `{ column: string }` object | +| `callback_url` | URL (**HTTPS**) | ✅ | webhook the result is POSTed to | +| `request_metadata` | object | optional | echoed back unchanged in the result | + +Rules: + +- **Strict input** — no extra keys are allowed on `input`; a body carrying both + `data` and `attachments` is rejected. +- **Rows match the config's `input_schema`** — every declared column present, no + undeclared columns, `image`/`pdf` values must be URLs. Otherwise `422`. +- **`callback_url`** must be HTTPS and public (private/loopback hosts are rejected). + +```json +{ + "config": { "id": "a9015dbf-…", "version": 1 }, + "input": { + "query": "Grade {answer_sheet} for submission {submission_id}.", + "data": [ + { "submission_id": "s1", "answer_sheet": "https://cdn.example.com/s1.jpg" } + ] + }, + "callback_url": "https://your-app.example.com/webhooks/assessment", + "request_metadata": { "batch": "class7-term1" } +} +``` + +### Building the batch input + +The `input` object is built from your configuration's `input_schema`: + +1. **One object per item** goes in `input.data`. Each object's keys are the column + names declared in the config's `input_schema`, and the values are strings. +2. **Attachment columns** (`image` / `pdf`) take a URL string; text columns take + plain text. +3. **`input.query`** is a template. Any `{column}` placeholder is replaced with + that row's value at grading time, so one template applies to every row. +4. **Match the schema exactly** — every declared column present, no extra columns. + +Example: for `input_schema = { submission_id: text, answer_sheet: image(url) }`, +each row is `{ "submission_id": "...", "answer_sheet": "https://..." }` and the +`query` can reference `{submission_id}` and `{answer_sheet}`. + +--- + +## Response — submit acknowledgement (`200`) + +Returned immediately; contains no results. Wrapped in the standard envelope +`{ success, data, error, metadata }`. + +| Field (`data`) | Type | Notes | +|---|---|---| +| `assessment_id` | UUID | correlate with the webhook | +| `status` | enum | `PROCESSING` on accept | +| `message` | string | human-readable | +| `inserted_at` / `updated_at` | timestamp | ISO-8601 | + +```json +{ + "success": true, + "data": { + "assessment_id": "8a2a7bc1-…", + "status": "PROCESSING", + "message": "Your assessment is being processed", + "inserted_at": "2026-08-12T10:15:30Z", + "updated_at": "2026-08-12T10:15:30Z" + }, + "error": null, + "metadata": null +} +``` + +--- + +## Webhook — the result (POST to `callback_url`) + +Delivered once, on completion. + +| Field | Type | Notes | +|---|---|---| +| `assessment_id` | UUID | matches the ack | +| `status` | enum | terminal (see below) | +| `data` | object | the `AssessmentBatchResult` (BATCH) | +| `request_metadata` | object \| null | echoed from the request | + +`data` (`AssessmentBatchResult`): + +| Field | Type | Notes | +|---|---|---| +| `total_items` | int | number of input rows | +| `counts.assessed` | int | rows graded | +| `counts.filtered` | int | rows gated out by a pre-filter | +| `counts.errors` | int | rows with an error | +| `items` | array | one `AssessmentResult` per input row, in order | + +`items[]` (`AssessmentResult`): + +| Field | Type | Notes | +|---|---|---| +| `output.assessment` | object \| string \| null | your `json_output_schema` filled in; string for free-text; `null` if gated out / failed | +| `output.pre_filter.topic_relevance` | `{verdict: bool, reasoning: string}` \| null | null if not configured | +| `output.pre_filter.duplicate_detection` | `{verdict: bool, reasoning: string}` \| null | null if not configured | +| `error` | string \| null | per-row error | + +```json +{ + "assessment_id": "8a2a7bc1-…", + "status": "COMPLETED", + "data": { + "total_items": 2, + "counts": { "assessed": 1, "filtered": 1, "errors": 0 }, + "items": [ + { + "output": { + "assessment": { "score": 20, "feedback": "…" }, + "pre_filter": { "topic_relevance": { "verdict": true, "reasoning": "…" } } + }, + "error": null + }, + { + "output": { + "assessment": null, + "pre_filter": { "topic_relevance": { "verdict": false, "reasoning": "off-topic" } } + }, + "error": null + } + ] + }, + "request_metadata": { "batch": "class7-term1" } +} +``` + +--- + +## Status values + +| Status | Meaning | +|---|---| +| `PENDING` | accepted, not started | +| `PROCESSING` | grading in progress (the ack status) | +| `COMPLETED` | all rows graded, no errors | +| `COMPLETED_WITH_ERRORS` | finished, some rows errored | +| `FAILED` | the run failed | + +`status` lives on the envelope only — it is never duplicated inside `data`. + +## Error codes (at submit) + +| Code | When | +|---|---| +| `422` | invalid body, or a row doesn't match `input_schema`, or a non-HTTPS/private `callback_url` | +| `404` | config id not found | +| `501` | RESPONSE-shaped input (single `query`) — WIP | +| `503` | failed to dispatch for processing (retry) | diff --git a/docs/architecture/assessment/assets/batch-flow.png b/docs/architecture/assessment/assets/batch-flow.png new file mode 100644 index 000000000..a625f4ff6 Binary files /dev/null and b/docs/architecture/assessment/assets/batch-flow.png differ diff --git a/docs/architecture/assessment/assets/response-flow.png b/docs/architecture/assessment/assets/response-flow.png new file mode 100644 index 000000000..6e8880ce2 Binary files /dev/null and b/docs/architecture/assessment/assets/response-flow.png differ diff --git a/docs/architecture/assessment/configuration-and-versioning.md b/docs/architecture/assessment/configuration-and-versioning.md new file mode 100644 index 000000000..ea52af81b --- /dev/null +++ b/docs/architecture/assessment/configuration-and-versioning.md @@ -0,0 +1,197 @@ +# Configuration and Versioning + +A **configuration** is your saved grading setup: the rubric, the model, the input +columns, and the structured result shape. You create it once, and every +assessment references it. Configurations are **versioned** so you can iterate +safely — each change adds a new version and never disturbs results that already +ran. + +This guide explains each component with examples. For an exhaustive field-by-field +table (types, allowed values, defaults), see the +[config field reference](config-field-reference.md). + +**Sample configuration JSON files (OpenAI, Claude, Gemini):** +https://drive.google.com/drive/folders/1BCaauUuXr9DaZTWI-_-x101SDT4ktwp5?usp=share_link + +--- + +## The tag + +Every configuration carries a **tag** that fixes its shape. An assessment +configuration must use: + +``` +tag = ASSESSMENT +``` + +- The tag is chosen at creation and **cannot be changed** afterwards. +- It drives validation: a config tagged `ASSESSMENT` must match the assessment + shape below, and a wrong shape is rejected with `422`. + +--- + +## The shape (`config_blob`) + +An `ASSESSMENT` config has two parts: + +```json +{ + "name": "answer-sheet grader", + "tag": "ASSESSMENT", + "config_blob": { + "pre_filters": { "...": "optional checks before grading" }, + "assessment": { "...": "the grading call (required)" } + } +} +``` + +- **`assessment`** (required) — the grading call. +- **`pre_filters`** (optional) — quick checks that run before grading. + +--- + +## Component 1 — `assessment` (required) + +The grading call. + +| Field | What it is | +|---|---| +| `provider` | `openai`, `google`, or `anthropic` | +| `type` | always `"text"` | +| `params.model` | the model id, e.g. `gpt-4o` | +| `params.instructions` | the **rubric** — how to grade (system prompt) | +| `params.input_schema` | **required** — the columns you will submit, and their types | +| `params.json_output_schema` | the **result shape** returned per item | + +```json +"assessment": { + "provider": "openai", + "type": "text", + "params": { + "model": "gpt-4o", + "instructions": "Grade each section out of 25 against the rubric and give brief feedback.", + "input_schema": { "...": "see Component 2" }, + "json_output_schema": { "...": "see Component 3" } + } +} +``` + +--- + +## Component 2 — `input_schema` (the columns) + +`input_schema` declares the columns every submitted row will have. It is +**mandatory** and must list at least one column; every column must declare a +`type`. + +| `type` | Value in a row | +|---|---| +| `text` | any text | +| `image` | an image URL | +| `pdf` | a PDF URL | + +Attachment columns (`image` / `pdf`) take a URL — add `"format": "url"`. + +```json +"input_schema": { + "submission_id": { "type": "text" }, + "answer_sheet": { "type": "image", "format": "url" } +} +``` + +Every column declared here must be present in every submitted row (see the +[API contract](api-contract.md)). + +--- + +## Component 3 — `json_output_schema` (the result shape) + +A JSON Schema `object` describing exactly what comes back per item. Whatever you +define is what the model fills in — so you get structured scores/feedback, not +free text. + +```json +"json_output_schema": { + "type": "object", + "properties": { + "score": { "type": "integer" }, + "feedback": { "type": "string" } + }, + "required": ["score", "feedback"] +} +``` + +--- + +## Component 4 — `pre_filters` (optional) + +Checks that run before grading. Each pre-filter is its own small LLM call with +its own model and criteria. + +| Pre-filter | Purpose | Default on a "no" | +|---|---|---| +| `topic_relevance` | Is the item on-topic / worth grading? | Skips grading for that item (`stop_on_fail: true`) | +| `duplicate_detection` (WIP) | Is the item a duplicate? | Records only, still grades (`stop_on_fail: false`) | + +Each pre-filter carries: + +| Field | What it is | +|---|---| +| `provider` | `openai` / `google` / `anthropic` (defaults to `openai`) | +| `params.model` | the model for this check (a cheaper one is fine) | +| `params.instructions` | **required** — the criteria for the check | +| `stop_on_fail` | `true` = a "no" skips grading for that item; `false` = record only | +| `knowledge_base_id` | *(duplicate_detection only)* the corpus to compare against | + +```json +"pre_filters": { + "topic_relevance": { + "provider": "openai", + "params": { + "model": "gpt-4o-mini", + "instructions": "Accept only a photo of a hand-drawn answer sheet; reject blank or off-topic images." + }, + "stop_on_fail": true + } +} +``` + +An item that fails a gate still appears in the results (with an empty grading +output and its pre-filter verdict), so you can see why it was skipped. + +--- + +## Full example + +A grader on OpenAI — a `topic_relevance` gate on `gpt-4o-mini`, the assessment on +`gpt-4o`: +[`config_openai.json`](https://github.com/ProjectTech4DevAI/kaapi-backend/blob/feat/doc-assessment-architecture/z_assessment_test/tap_test/config_openai.json) + +--- + +## Versioning + +Configurations are versioned. You never overwrite one — you add a version. + +| Action | Endpoint | +|---|---| +| Create a configuration | `POST /configs` (with `tag = ASSESSMENT`) | +| Add a new version | `POST /configs/{config_id}/versions` | +| List all versions | `GET /configs/{config_id}/versions` | +| Get a specific version | `GET /configs/{config_id}/versions/{version_number}` | + +How it works: + +- **Create** a configuration once; it starts at version 1. +- **Iterate** by adding versions — change the rubric, swap the model, adjust the + result shape. Each save is a new version; earlier versions remain intact. +- **The tag stays fixed** across versions, so every version keeps the assessment + shape. +- **An assessment pins the exact version it ran with** (`config_id` + + `config_version`). Editing the configuration later never changes a result that + has already completed — reproducible by design. +- **Track** versions with the list/get endpoints and choose which + `config_version` to submit. + +You reference the chosen `config_id` and `config_version` when you submit an +assessment — see the [API contract](api-contract.md). diff --git a/docs/architecture/kaapi-ai-assessment-ARCHITECTURE.md b/docs/architecture/kaapi-ai-assessment-ARCHITECTURE.md new file mode 100644 index 000000000..55155ca86 --- /dev/null +++ b/docs/architecture/kaapi-ai-assessment-ARCHITECTURE.md @@ -0,0 +1,264 @@ +# Kaapi AI Assessments — Architecture Overview + +> **New here?** Start with the user guides in [`assessment/`](assessment/README.md) +> (onboarding, configs, running an assessment). This document is the technical +> architecture of the **API-client** assessment surface. + +## Purpose + +The **AI Assessments** API-client lets a caller grade items with an LLM against a +saved rubric and receive a **structured result** (a caller-defined JSON object) +per item. It is a pure API surface — submit a request, get results delivered to a +**webhook**. There is no console/UI flow here. + +The method is **inferred from the input shape**, never passed as a flag: + +| Method | Input | Status | +|---|---|---| +| **BATCH** | a `data` list of rows | ✅ shipped | +| **RESPONSE** | a single `query` | 🚧 WIP — returns `501` | + +This document describes the **BATCH** path (the one that is built) and notes the +RESPONSE path as WIP. Grading runs on a **provider Batch API** — OpenAI, Google +AI Studio (Gemini), or Anthropic. Google Cloud / Vertex is WIP. + +Key properties: + +- **Webhook-only delivery.** The request carries a required `callback_url`; the + finished result is POSTed there. There is no status or result poll endpoint. +- **One config, one execution.** A request pins one `config_id + config_version` + (tag `ASSESSMENT`) and creates one parent `assessment` + one `assessment_run` + (the execution). +- **A staged pipeline, one provider batch per stage.** Optional gate/pass-through + pre-filters run first, then the assessment stage grades only the rows that + passed every gate. +- **Runtime state lives in one JSONB bag.** All pipeline state sits on + `assessment_run.execution` (a `BatchRunState`); no dedicated columns. +- **Self-driving Celery loop.** A Celery task advances the pipeline one tick at a + time and re-enqueues itself until the run is terminal — no external cron. + +--- + +## 1. The high-level view + +```mermaid +sequenceDiagram + autonumber + participant C as Client + participant S as POST /assessments + participant W as Celery (run_assessment_api_batch) + participant P as Provider Batch API + participant Hook as Client webhook + + C->>S: config + input(rows) + callback_url + S->>S: validate config + callback_url + rows + S-->>C: 200 ack (assessment_id, PROCESSING) + S->>W: enqueue first tick + loop one stage per tick, self-re-enqueuing + W->>P: submit current stage's batch + W->>P: (next tick) poll batch + P-->>W: completed → parse verdicts / results + end + W->>Hook: POST final AssessmentBatchResult +``` + +The server does no slow work: it validates and registers the request, then the +Celery task submits a batch, exits, and re-enqueues itself to poll later. When the +last stage completes, the result is delivered to the webhook. + +See the [BATCH flow diagram](assessment/assets/batch-flow.png) for the +decision-level view. + +--- + +## 2. Component map + +``` +backend/app/ +├── api/routes/assessment/ +│ └── api.py ★ POST /assessments (method inferred; RESPONSE → 501) +│ +├── services/assessment/api/ +│ ├── submission.py ★ submit(): validate config + callback_url + rows, +│ │ persist assessment + execution, seed the bag, dispatch +│ ├── batch.py ★ the staged pipeline driver: +│ │ build_pipeline · run_batch_stage (one tick) · +│ │ _submit_stage · _poll_outcome · _advance_or_finalize · +│ │ _finalize · _fail · parse_batch_results +│ ├── results.py build_result → AssessmentBatchResult (one item per row) +│ └── callbacks.py deliver(): POST the result via the SSRF-guarded send_callback +│ +├── crud/assessment/ +│ └── api.py create_assessment · create_execution · save_execution_state · +│ set_execution_batch_job · update_status · list_executions +│ +├── core/batch/ shared provider batch infra +│ └── openai.py · gemini.py · anthropic.py the three batch providers +│ +├── celery/tasks/job_execution.py run_assessment_api_batch (self-re-enqueues each tick) +│ +└── models/ + ├── assessment/assessment_api.py request/response models + BatchRunState (the bag) + ├── assessment/assessment.py Assessment · AssessmentRun tables · enums + └── config/assessment_blob.py AssessmentConfigBlob (pre_filters + assessment) +``` + +`★` = read first: [submission.py](../../backend/app/services/assessment/api/submission.py) +is the entry point; [batch.py](../../backend/app/services/assessment/api/batch.py) +is the pipeline engine. + +--- + +## 3. Data model + +```mermaid +flowchart TD + Cfg["config (tag=ASSESSMENT)\nconfig_version.config_blob"] + A["assessment (parent)\nmethod=BATCH · status"] + R["assessment_run (execution)\nconfig pin + execution bag (JSONB)"] + BJ["batch_job\none per stage"] + + Cfg -->|config_id + version| R + A --> R + R -->|stage_batches| BJ +``` + +- **`assessment`** — one submission: `method`, aggregate `status`, org/project. For + BATCH there is exactly one child execution. +- **`assessment_run`** — the execution: the config pin (`config_id + config_version`) + and the **execution bag** on `execution` (JSONB). +- **The execution bag** (`BatchRunState`, + [assessment_api.py](../../backend/app/models/assessment/assessment_api.py)) holds + all runtime state: `pipeline`, current `stage` + `stage_status`, `stage_batches` + (stage → batch id), `stage_output_urls`, per-stage `verdicts` and `counters`, + the per-row `gate_passed` flags, `provider` / `model`, `input_schema`, + `callback_url`, and `request_metadata`. Idempotent redelivery is keyed off + `stage_status`. + +The schema (UUID assessment id, method/status enums, the `execution` bag column) +is set up in migration `078_refactor_assessment_tables`. + +--- + +## 4. The config + +A run resolves one `ASSESSMENT`-tagged config into an `AssessmentConfigBlob` +([assessment_blob.py](../../backend/app/models/config/assessment_blob.py)): + +- **`assessment`** (required) — `provider` (`openai` / `google` / `anthropic`), + `type: text`, and `params`: `model`, `instructions` (the rubric), a **mandatory** + typed `input_schema` (the submission columns), and an optional + `json_output_schema` (the structured result shape). +- **`pre_filters`** (optional) — `topic_relevance` and/or `duplicate_detection`. + Each is its own LLM call with its own `provider` + `params` (criteria live in + `params.instructions`, **mandatory**) and a `stop_on_fail` flag. + +See **[configuration-and-versioning.md](assessment/configuration-and-versioning.md)** for the authoring guide. + +--- + +## 5. The staged pipeline + +`build_pipeline` ([batch.py](../../backend/app/services/assessment/api/batch.py)) +compiles the config's pre-filters into an ordered stage list, by kind +(`GATE pre-filters → PASS_THROUGH pre-filters → ASSESSMENT`), and rows flow +through it like this: + +```mermaid +flowchart LR + Rows["all rows"] --> Gate{"GATE stage\n(e.g. topic relevance)"} + Gate -->|verdict pass| PT["PASS_THROUGH stage\n(e.g. duplicate detection)\nannotate, drop nothing"] + Gate -->|verdict fail| Gated["gate_passed = false"] + PT --> Assess["ASSESSMENT stage\ngrade gate-passed rows only"] + Assess --> Res["AssessmentBatchResult\none item per row"] + Gated -->|assessment = null\n+ pre-filter verdicts| Res +``` + +- **GATE** (`stop_on_fail: true`, e.g. topic relevance) — runs on every row; a + failing verdict marks that row `gate_passed = false`. +- **PASS_THROUGH** (`stop_on_fail: false`, e.g. duplicate detection) — runs on + every row, records a verdict, drops nothing. +- **ASSESSMENT** — always last; batches **only** `gate_passed` rows. Gate-failed + rows carry `assessment: null` plus their pre-filter verdicts into the result. + +### One tick = `run_batch_stage` + +Each Celery invocation runs one tick and returns `{"requeue": bool}`: + +```mermaid +flowchart TD + Start["run_batch_stage"] --> Res["resolve blob (guarded → _fail)"] + Res --> St{"stage_status?"} + St -->|PENDING| Sub["submit stage batch"] + Sub -->|submitted| RQ["requeue = true"] + Sub -->|empty subset| Adv["_advance_or_finalize"] + St -->|PROCESSING| Poll["poll batch"] + Poll -->|processing| RQ + Poll -->|failed| Fail["_fail → webhook"] + Poll -->|completed| Rec["record results"] --> Adv + Adv -->|more stages| RQ + Adv -->|last stage| Fin["_finalize → webhook"] +``` + +- **Submit** a `PENDING` stage's batch, then requeue to poll it next tick. +- **Poll** a `PROCESSING` stage; on completion, record verdicts/results and + `_advance_or_finalize` to the next stage — or `_finalize` if it was the last. +- **Empty subset** (all rows gated out): the stage submits no batch and + `_advance_or_finalize` moves on — which, for the last stage, finalizes and fires + the webhook (no livelock). + +The task [`run_assessment_api_batch`](../../backend/app/celery/tasks/job_execution.py) +re-enqueues itself at `POLL_COUNTDOWN_SECONDS` whenever `requeue` is true, and +stops once the run is terminal. + +--- + +## 6. Results & delivery + +- `build_result` ([results.py](../../backend/app/services/assessment/api/results.py)) + assembles an `AssessmentBatchResult`: `total_items`, `counts` + (`assessed` / `filtered` / `errors`), and one `AssessmentResult` per row — + `{ output: { assessment, pre_filter }, error }`. Gate-failed rows are included + with `assessment: null`. +- `_finalize` sets the terminal status and calls + `deliver` ([callbacks.py](../../backend/app/services/assessment/api/callbacks.py)), + which POSTs an `AssessmentCallback` (`{ assessment_id, status, data, + request_metadata }`) to the client `callback_url` via the shared + **SSRF-guarded, HMAC-signed** `send_callback`. + +The result body carries no per-call `metadata` block — only the graded outputs, +pre-filter verdicts, and counts. + +--- + +## 7. Failure modes + +| Concern | Behaviour | +|---|---| +| **Async model** | Provider Batch APIs do all model work; Celery only builds/submits and polls. | +| **Webhook-only** | Results are delivered to `callback_url`; there is no poll endpoint. `callback_url` is validated (HTTPS + SSRF guard) at submit, so a bad URL is rejected `422` up front. | +| **All rows gated out** | The assessment stage submits no batch and the run finalizes with an all-gated result (still delivered). | +| **Non-transient tick error** | A bad/deleted config version or a provider/credential/network error during submit routes through `_fail` → status `FAILED` + a failure webhook. | +| **Transient poll error** | A provider/network hiccup while polling just retries next tick — a running batch is never failed for a transient error. | +| **Idempotent redelivery** | State is keyed off `stage_status`, so a duplicate Celery delivery re-polls or re-submits the same stage safely. | +| **Per-row validation** | Rows are validated against `input_schema` at submit; a missing/extra column or a non-URL attachment fails `422`, naming the row. | + +--- + +## 8. RESPONSE method 🚧 (WIP) + +A single-item, low-latency path (one `query` → one `AssessmentResult`, optionally +via an LLM-chain when pre-filters are configured) is planned. It is not built — +the route returns `501` for RESPONSE-shaped input. See the +[planned RESPONSE flow](assessment/assets/response-flow.png). + +--- + +## Related + +- **[assessment/README.md](assessment/README.md)** — the user-facing getting-started + guide, config authoring, and how to run an assessment. +- `kaapi-llm-call-ARCHITECTURE.md` — the production single-call endpoint whose + versioned config store backs assessment configs (`tag=ASSESSMENT`). +- `kaapi-knowledge-base-ARCHITECTURE.md` — File Search vector stores used by the + duplicate-detection pre-filter (WIP).