feat(vit-gpt2): add CPU image-captioning recipes

[ssss141414]

Summary

Adds current-main CPU support for nlpconnect/vit-gpt2-image-captioning, a ViT/GPT-2 image-captioning composite, with FP32 and materially realized FP16 encoder/decoder recipes. The L2 contribution generalizes task normalization, multimodal processor composition, and bounded caption evaluation; the final tester reached L3 PASS with full coverage of the committed CPU FP32/FP16 component tuples. examples/recipes/README.md remains unchanged.

Model metadata

What the model does

An image-captioning encoder-decoder model that transforms one RGB image into an English natural-language caption: a ViT encoder produces visual patch states and a cross-attentive GPT-2 causal decoder generates caption tokens.

• Evidence: the pinned model card declares image-to-text/image-captioning and provides captioning examples; its pinned config declares VisionEncoderDecoderModel with nested ViT and GPT-2 components; real-image PyTorch and split-ONNX generation both produced English captions. Confidence: verified.

Primary user stories

• A user supplies a photograph to obtain a concise English description for image understanding, indexing, or accessibility workflows. Evidence: the pinned model card widget and sample pipeline use photographs as input and generated_text captions as output. Confidence: verified.

Supported tasks

• image-to-text on the checkpoint, Transformers, Optimum ONNX, and WinML surfaces. Evidence: the checkpoint pipeline tag is image-to-text; Optimum registers vision-encoder-decoder for image-to-text and image-to-text-with-past; WinML registers the image-to-text composite and emits image-feature-extraction plus text2text-generation components. Confidence: verified.
• image-text-to-text on the Transformers surface. Evidence: current TasksManager auto-detection returns image-text-to-text, while the prior WinML ONNX export boundary rejected that alias. Confidence: verified.

Model architecture

VisionEncoderDecoderModel
|-- ViT encoder (224x224 RGB, patch 16, 197 tokens, hidden 768)
|   |-- Patch projection + CLS/position embeddings
|   |-- Encoder block x 12
|   |   |-- Self-attention (12 heads, head dim 64)
|   |   `-- MLP (768 -> 3072 -> 768) + residual/LayerNorm
|   `-- encoder_hidden_states [1,197,768]
`-- GPT-2 LM decoder (vocab 50,257, max positions 1,024)
    |-- Token + position embeddings
    |-- Decoder block x 12
    |   |-- Causal self-attention + static KV cache (12 heads, head dim 64)
    |   |-- Cross-attention over encoder_hidden_states
    |   `-- MLP (768 -> 3072 -> 768) + residual/LayerNorm
    `-- LM head -> logits [1,1,50257] -> greedy generation

• Source/confidence: pinned nested ViT/GPT-2 config dimensions, Transformers VisionEncoderDecoderModel, WinML VisionDecoderWrapper, and final encoder/decoder ONNX graph contracts and hierarchy tags. The matching 768-wide encoder and decoder require no projection module. Confidence: verified.

Validation and support evidence

Baseline

The full current-main baseline was rerun at e564a6375d6cd2b596fb3d21d918f07824b349e0 using WinML 0.2.0 package metadata (0.3.0 in the source pyproject.toml), Python 3.11.9, Transformers 5.14.1, Optimum 2.1.0, ONNX 1.18.0, and ONNX Runtime Windows ML 1.24.5.202604171637.

• Recipe-free inspect/build floor: FAIL-CONTRACT / FAIL. Auto-detection returned image-text-to-text; WinML had no matching exporter composite, and build exited 2 after 27.214 s. This established a capability floor, not a reason to stop at L0.
• Explicit image-to-text config: PASS, emitting exactly an image-feature-extraction encoder and text2text-generation decoder. Diagnostic component builds passed in 79.696 s and 149.209 s with no semantic CLI overrides.
• FP32 component perf: encoder p50 89.304 ms, 11.26 samples/s, +29.24 MB RSS; decoder p50 43.656 ms, 22.82 samples/s, +112.95 MB RSS. Composite perf was CLI-blocked because component build folders lacked a top-level architecture config.
• Numeric floor: encoder cosine 0.9999999999990252 (max abs 4.2617e-06), decoder-logit cosine 0.9999999999999869 (max abs 7.62939e-05), and 24 present KV outputs. PyTorch generated a herd of giraffes and zebras grazing in a field; ONNX generated a herd of giraffes standing in a field; exact tokens differed but both were semantically coherent.
• Eval floor: FAIL before sample processing. AutoProcessor resolved to a tokenizer-only TokenizersBackend, not a ProcessorMixin, so no metric was emitted.
• Optimum probe: VENDOR+OVERRIDE. Vendor tasks were image-to-text and image-to-text-with-past; WinML additionally exposed feature-extraction and text2text-generation for the composite.

Goal

• Effort: L2: implement the narrow class-wide task/evaluator support path, not checkpoint hardcoding.
• Goal ceiling: L3: march through exact builds, component perf plus assembled generation, PyTorch/ONNX numeric parity, and one bounded final-SHA FP32 CPU functional smoke.
• Outcome: L2: ship task-family code, regression coverage, four exact CPU recipes, and learned model/methodology findings.

Success required encoder and decoder builds for CPU FP32 and FP16; material FP16 realization; complete named decoder inputs, 12 KV layers, and 24 present outputs; perf for every artifact; tensor/KV/generation parity reporting; one deterministic 1-2-row COCO smoke with all fan-out dimensions capped; and component- plus op-level static analysis for every artifact. The charter was not re-issued and the ceiling was not downgraded.

Outcome

L2 shipped; L3 PASS; coverage: full. All committed CPU FP32/FP16 encoder and decoder tuples passed L0 and L1, L2 passed for both precisions, and exactly one bounded L3 functional smoke passed. There are no deferred tuples and no final blockers.

The PR contains four recipe paths and exactly these six source/test paths:

• src/winml/modelkit/loader/resolution.py
• src/winml/modelkit/inference/pipeline.py
• src/winml/modelkit/eval/image_to_text_evaluator.py
• tests/unit/loader/test_detect_task_and_class.py
• tests/unit/inference/test_pipeline.py
• tests/unit/eval/test_image_to_text_evaluator.py

The four model findings vision-encoder-decoder-005 through vision-encoder-decoder-008 capture processor/recipe/KV contracts, FP32/FP16 realization and perf, numeric parity plus coherent generation divergence, and static component/EP findings. They are maintained separately in the draft Lane A skill PR #217 at commit 0bef530285b785641504038232b03b0be45c3009. That PR also records methodology finding _meta-109; methodology was updated with paired tester/reviewer guidance for deterministic static-provider analysis without changing this PR's tester verdicts.

Per-EP/device/precision results and Functional smoke Eval

Tier	EP / device	Precision	Component	Verdict	Build	Mean	p50	Throughput	RSS delta	VRAM delta
L0/L1	CPUExecutionProvider / cpu	fp32	encoder	PASS	126.8433614 s	90.321 ms	90.467 ms	11.07 samples/s	+29.28 MB	0 MB
L0/L1	CPUExecutionProvider / cpu	fp32	decoder	PASS	152.1119709 s	42.468 ms	42.506 ms	23.55 samples/s	+113.43 MB	0 MB
L0/L1	CPUExecutionProvider / cpu	fp16	encoder	PASS	100.6639445 s	111.104 ms	109.902 ms	9.00 samples/s	+37.01 MB	0 MB
L0/L1	CPUExecutionProvider / cpu	fp16	decoder	PASS	182.4544937 s	46.098 ms	45.812 ms	21.69 samples/s	+126.54 MB	0 MB

All four ONNX artifacts passed checker validation at IR 8 / opset 17. FP32 initializers were encoder {FLOAT: 200, INT64: 6} and decoder {FLOAT: 254, INT64: 43, BOOL: 1}; FP16 initializers were encoder {FLOAT16: 200, INT64: 6} and decoder {FLOAT16: 254, INT64: 43, BOOL: 1}. Decoder artifacts retained 12 past-key/value layers and 24 present outputs. Schema-v2 model precision was fp32/fp16; benchmark precision was auto for FP32 and fp16 for FP16.

Assembled greedy generation also passed: FP32 took 2.8438051999546587 s (3.868056785385786 tokens/s) and FP16 took 3.172113100066781 s (3.4677199875907396 tokens/s). Component latencies above are not summed or mislabeled as end-to-end caption latency.

L2 numeric parity: PASS. FP32 encoder cosine was 0.9999999999987601, logits cosine 0.9999999999999883, and minimum present-KV cosine 0.9999999999997096. FP16 encoder cosine was 0.999999639303887, logits cosine 0.9999999921532873, and minimum present-KV cosine 0.9999997401972208. In both precisions, generation first diverged at token index 8: PyTorch selected token 5671 ( monitor) and produced a desk with a computer and a monitor ; WinML selected token 5118 ( chair) and produced a desk with a computer and a chair . Token sequences were not exact; both captions were coherent, and the divergence is reported rather than hidden.

Functional smoke Eval: PASS. Exactly one final-candidate FP32 CPU run used LIME-DATA/COCO-Caption2017 revision b14474455722b8c56eb7cdeb3f6be76fca1bba7a, streaming train, deterministic first two rows with no shuffle. It processed 2 and skipped 0. Caps were one candidate prompt, one beam, one frame/crop, generation length 32, and five references per image. Input schema, reference-label semantics, and generated-caption semantics were verified. It emitted CIDEr 0.8427 and CER 0.6316. This single bounded run is end-to-end operability evidence only, not representative accuracy or benchmark quality. The former blocker was tokenizer-only AutoProcessor; generalized image-processor/tokenizer composition removed it.

Delta

Recipe paths:

• examples/recipes/nlpconnect_vit-gpt2-image-captioning/cpu/cpu/image-to-text_fp32_config_encoder.json
• examples/recipes/nlpconnect_vit-gpt2-image-captioning/cpu/cpu/image-to-text_fp32_config_decoder.json
• examples/recipes/nlpconnect_vit-gpt2-image-captioning/cpu/cpu/image-to-text_fp16_config_encoder.json
• examples/recipes/nlpconnect_vit-gpt2-image-captioning/cpu/cpu/image-to-text_fp16_config_decoder.json

Both FP32 recipes are identical to current auto-config. The FP16 recipes are identical except /quant changes from null to the complete current generated FP16 block (mode=fp16, fp16_keep_io_types=true) for the required precision contract. Shared invariants remain: eager Transformers attention, no eval block, no SROIE reference, and the decoder's 12 past-key/value layers plus 24 present outputs. Recipe invocation used no recipe-owned semantic CLI overrides; FP16 used only the current --precision fp16 selector. No-recipe inspect/config/build/eval acceptance passed, reducibility is consistent with the charter, and examples/recipes/README.md is untouched.

Bug fix explanation

1. Symptom and trigger: recipe-free inspect/build auto-detected image-text-to-text, which WinML could not export as the registered image-captioning composite; Eval then failed before processing because AutoProcessor supplied only tokenizer behavior.
2. Root cause: the Transformers-facing alias was crossing the WinML export boundary without consulting registered composite support, and the shared pipeline assumed one AutoProcessor object always carried image preprocessing, tokenization, decoding, and decoder-start behavior.
3. Changed symbols and mechanism: resolve_task normalizes auto-detected image-text-to-text to image-to-text only for model types with a registered WinML composite. _ImageToTextProcessor, _load_image_to_text_processor, _pipeline_component_kwargs, and create_pipeline preserve complete ProcessorMixin objects or compose AutoImageProcessor with the tokenizer-only result, validate required capabilities, and use decoder_start_token_id for empty caption prompts. WinMLImageToTextEvaluator.compute bounds multi-reference caption generation to 32 new tokens and one beam while preserving scalar OCR behavior.
4. General rule: decisions derive from registered composite metadata and processor capabilities, not the checkpoint ID. This supports the class of registered image-captioning encoder-decoder models and fails closed when required image/text/decode capabilities are absent.
5. Compatibility and blast radius: explicit tasks and unrelated unregistered multimodal model types are unchanged; the HF pipeline name remains image-text-to-text; WinML's public composite remains image-to-text; complete processors are returned unchanged; non-empty prompts preserve tokenizer output; scalar OCR calls and CER semantics remain unchanged. Intentional changes are alias normalization for registered composites and bounded caption smoke generation.
6. Regression evidence: focused loader/inference/eval tests: 95 passed; Ruff: all checks passed; mypy: no issues in 435 source files; analyze partition: 1526 passed, 45 skipped; models/loader/datasets/export: 1526 passed, 6 skipped, 2 xfailed; optim: 711 passed, 16 skipped, 1 xfailed; commands/config/build/compiler/session/eval: 3564 passed, 9 skipped; remaining core/ONNX/cache/utils/inspect/regression/CLI: 869 passed, 2 skipped, 1 deselected. All live PR checks are successful.

There is no stale scripts/e2e_eval change in this contribution.

Analyze summary - component level and op level

PASS: complete component- and op-level static rule analysis was recovered for all four final artifacts. This is static compatibility analysis, not runtime execution or accelerator validation.

Component-level summary

Artifact	Architecture regions	Mapping	Confidence	Actionable EP findings
fp32 encoder	patch embedding; 12 encoder/self-attention blocks; graph support	246 mapped, 120 partial, 0 unmapped	Actual final ONNX scopes + frozen tensor boundaries; explicit graph-support partial region	None
fp16 encoder	patch embedding; 12 encoder/self-attention blocks; graph support	246 mapped, 122 partial, 0 unmapped	Actual final ONNX scopes + frozen tensor boundaries; explicit graph-support partial region	None
fp32 decoder	embeddings; 12 self-/cross-attention/MLP blocks; KV cache; LM head; graph support	764 mapped, 13 partial, 0 unmapped	Actual final ONNX scopes + frozen tensor boundaries; explicit graph-support partial region	OpenVINO CPU/GPU/NPU: Add unsupported, ScatterND/Where unknown. QNN GPU: Concat/Gather partial, LessOrEqual unsupported, ScatterND/Where unknown. QNN NPU: Gather partial, ScatterND/Where unknown. NvTensorRTRTX GPU: ScatterND/Where unknown.
fp16 decoder	embeddings; 12 self-/cross-attention/MLP blocks; KV cache; LM head; graph support	764 mapped, 63 partial, 0 unmapped	Actual final ONNX scopes + frozen tensor boundaries; explicit graph-support partial region	Same actionable OpenVINO/QNN/NvTensorRTRTX classifications as FP32 decoder.

Graph-support nodes outside repeated module scopes remain an explicit partial semantic region. runtime.generation is runtime-only; no ONNX node mapping is fabricated for it.

Op-level summary

Artifact	Graph	Dominant ops	EP roll-up
fp32 encoder	366 ops / 11 types	Reshape 121; Gemm 72; Transpose 49; Add 37; LayerNormalization 25; MatMul 24	Six rule-backed classifications; no actionable partial/unsupported findings.
fp16 encoder	368 ops / 12 types	Reshape 121; Gemm 72; Transpose 49; Add 37; LayerNormalization 25; MatMul 24	Six rule-backed classifications; no actionable partial/unsupported findings.
fp32 decoder	777 ops / 22 types	Reshape 219; Transpose 108; Add 86; Gemm 84; Mul 72; MatMul 49; LayerNormalization 37; Split 24	OpenVINO/QNN/NvTensorRTRTX findings as above.
fp16 decoder	827 ops / 22 types	Reshape 219; Transpose 108; Add 86; Gemm 84; Mul 72; Cast 51; MatMul 49; LayerNormalization 37	OpenVINO/QNN/NvTensorRTRTX findings as above.

Rule-less EP groups CPU, CUDA, MIGraphX, TensorRT, and DML have all operator types unknown; this is not a runtime failure claim.

Reproduce commands

$OUT='temp/vit-gpt2-image-captioning-repro'
winml build -c examples/recipes/nlpconnect_vit-gpt2-image-captioning/cpu/cpu/image-to-text_fp32_config_encoder.json -m nlpconnect/vit-gpt2-image-captioning -o $OUT/fp32-encoder
winml build -c examples/recipes/nlpconnect_vit-gpt2-image-captioning/cpu/cpu/image-to-text_fp32_config_decoder.json -m nlpconnect/vit-gpt2-image-captioning -o $OUT/fp32-decoder
winml build -c examples/recipes/nlpconnect_vit-gpt2-image-captioning/cpu/cpu/image-to-text_fp16_config_encoder.json -m nlpconnect/vit-gpt2-image-captioning -o $OUT/fp16-encoder --precision fp16
winml build -c examples/recipes/nlpconnect_vit-gpt2-image-captioning/cpu/cpu/image-to-text_fp16_config_decoder.json -m nlpconnect/vit-gpt2-image-captioning -o $OUT/fp16-decoder --precision fp16
winml perf -m $OUT/fp32-encoder/model.onnx --device cpu --ep cpu --iterations 10 --warmup 2 --memory --format json -o $OUT/perf-fp32-encoder.json
winml perf -m $OUT/fp32-decoder/model.onnx --device cpu --ep cpu --iterations 10 --warmup 2 --memory --format json -o $OUT/perf-fp32-decoder.json
winml eval -m encoder=$OUT/fp32-encoder/model.onnx -m decoder=$OUT/fp32-decoder/model.onnx --model-id nlpconnect/vit-gpt2-image-captioning --task image-to-text --dataset LIME-DATA/COCO-Caption2017 --dataset-revision b14474455722b8c56eb7cdeb3f6be76fca1bba7a --split train --samples 2 --no-shuffle --streaming --column input_column=image --column label_column=answer --ep cpu --device cpu -o $OUT/eval.json
$env:WINMLCLI_RULES_DIR='<populated-rules-root>'
winml analyze --model $OUT/fp32-encoder/model.onnx --ep openvino --device cpu --no-check-optim --no-run-unknown-op --output $OUT/analyze-fp32-encoder-openvino-cpu.json

[ssss141414]

Independent pre-PR review: APPROVE. Scope is coherent: the nested-config JSON fix is generic and regression-tested; all four CPU recipes parse and retain the auto-config export/optimizer/loader contract; genuine FP16 is evidenced by initializer dtypes; encoder and decoder pass L0-L3 in both CPU precisions; and examples/recipes/README.md is untouched. Coverage annotation: partial because this host exposes CPU runtime only. All 26 required EP/device/precision tuples are nevertheless enumerated, and static analysis is not represented as accelerator runtime validation.

[ssss141414]

REQUEST_CHANGES

I independently reviewed PR #1122 at head 17b8002018d29991ce579e7012fa1ed2d09e0809, PR base/merge-base/parent e564a6375d6cd2b596fb3d21d918f07824b349e0, from a clean detached checkout. The implementation and frozen evidence are otherwise sound, but the current-main gate is no longer satisfied.

Blocking issue

• Planner / tester / explainer: origin/main is now 19e9ba81cbde31c937e5fffa12f8f4292d7eddf9, not the charter, PR-base, and final-candidate evidence SHA e564a6375d6cd2b596fb3d21d918f07824b349e0. The complete old-to-current delta has 19 files and includes pyproject.toml, uv.lock, src/winml/modelkit/commands/perf.py, src/winml/modelkit/session/session.py, src/winml/modelkit/onnx/external_data.py, and compiler paths. Those overlap dependencies plus the measured build/perf/session/external-data surface, so this cannot be accepted as NO-IMPACT. Refresh/rebase to current main, issue a dependency-aware PARTIAL-RERUN or FULL-RERUN attestation, rerun every invalidated downstream stage into fresh immutable roots, and refresh final-SHA body/check evidence. Preserve the existing measurements as executed on e564a637; do not relabel them as current-main runs.

Reviewed scope and engineering result

The live exact-head diff contains 10 files (the supplied six-file expectation is stale): three shared source files, three regression files, and four CPU recipes. Task normalization is gated by registered composite support; processor composition is capability-based; caption caps apply only to multi-reference rows; explicit/unregistered task behavior and OCR scalar semantics are preserved. No shared-source model-ID branch exists. All four recipes have the correct target paths, eager attention, exact encoder/decoder identities, fp32 quant: null, fp16 mode: fp16 with keep-IO, and 24 past/24 present KV tensors per decoder. examples/recipes/README.md, scripts/e2e_eval, skill files, and scratch files are absent from the diff.

The three threads resolved by removal are substantiated: scripts/e2e_eval/run_eval.py and scripts/e2e_eval/utils/recipes.py are byte-identical to the PR base, and no equivalent quantized-precision, mixed-layout precedence, or EP-only nested-discovery change survives.

Independent reruns

• uv run --no-sync ruff check src/ tests/: PASS.
• uv run --no-sync mypy -p winml.modelkit: PASS, 435 source files.
• Focused loader/pipeline/evaluator regression: 95 passed.
• Discriminating task-normalization/processor-composition selection: 6 passed.
• CI partitions: analyze 1526 passed/45 skipped; models 1526 passed/6 skipped/2 xfailed; optim 711 passed/16 skipped/1 xfailed; commands 3564 passed/9 skipped; remaining 869 passed/2 skipped/1 deselected.

All used the existing locked venv with --no-sync, no installation, and PYTHONPATH pinned to the reviewer checkout. I did not run Eval.

Frozen L0-L3 and Analyze evidence

• L0: four exact encoder/decoder x fp32/fp16 artifacts, IR 8/opset 17, correct named static I/O, co-located external data, and material FLOAT16 realization. Schema-v2 fp16 is corroborated by both model and benchmark precision fields.
• L1: P50/throughput/RSS are fp32 encoder 90.467 ms / 11.07 s^-1 / 29.28 MB, fp32 decoder 42.506 / 23.55 / 113.43, fp16 encoder 109.902 / 9.00 / 37.01, fp16 decoder 45.812 / 21.69 / 126.54.
• L2: encoder, logits, and all 24 present-KV outputs have the reported high cosines; both precisions retain the honest coherent token-8 divergence (monitor vs chair) under assembled generation.
• L3: exactly one final-SHA FP32 CPU functional smoke, pinned COCO revision, 2 rows, one beam/crop/prompt, max 32 new tokens, processed 2/skipped 0, CIDEr 0.8427, CER 0.6316; operability only.
• Analyze: four artifacts have complete component/op mappings with zero unmapped nodes and six rule-backed static target classifications each. The interrupted monolithic --check-optim attempt emitted no JSON and is not acceptance evidence; 24 target scans plus four CPU diagnostics used --no-check-optim and retained parseable partial-success JSON where applicable.

GitHub and handoff state

Immediately before publication: PR is OPEN/DRAFT with model-scale-by-skill; head/base are unchanged; all 9 checks are COMPLETED/SUCCESS; 12 line comments, 1 prior conversation comment, 12 submitted reviews, and 6 review threads were enumerated; 0 threads are open. Lane A PR #217 is OPEN/DRAFT/labeled at exact commit 0bef530285b785641504038232b03b0be45c3009, with its single check successful and the expected four files/findings (vision-encoder-decoder-005..008, _meta-109).

Handoff hashes verified: charter f89cdc74...50ecf, producer d10c7769...0c12, tester verdict 577a8106...143d, learner 202dad45...89a4, explainer 39d6ebf7...83f7, thread-resolution 1f16681a...4c20; sealed-manifest verification found no mismatches.