[provider-openai] Retry transient list_models() failures during model discovery

[danshapiro]

Summary

Component: microsoft/amplifier-module-provider-openai, whose Issues are disabled.

OpenAIProvider.list_models() currently calls self.client.models.list() directly. The provider intentionally initializes AsyncOpenAI with SDK retries disabled and configures its own retry policy—five retries by default—which completion requests already use.

Consequently, a transient /v1/models failure receives only one attempt during model discovery. When routing resolves a glob candidate before the model cache has been populated, the downstream resolver catches and logs the exception and skips that candidate. The failure is nonfatal, but it can demote the intended routing choice and produces a noisy traceback.

Observed behavior

A transient HTTP 500 from model listing was observed while resolving a model glob with an empty routing cache:

• configured provider retries: 5;
• model-list calls: 1;
• routing contained the exception;
• the failing glob candidate was skipped;
• the synthetic resolver result was empty (None).

A deterministic, self-contained, network-free reproduction confirms the one-call behavior on the current provider implementation:

import asyncio
from types import SimpleNamespace
from unittest.mock import AsyncMock

import httpx
import openai
from amplifier_module_provider_openai import OpenAIProvider


def test_current_list_models_gets_only_one_attempt():
    transient_500 = openai.APIStatusError(
        "synthetic transient failure",
        response=httpx.Response(
            500,
            request=httpx.Request(
                "GET", "https://api.openai.com/v1/models"
            ),
        ),
        body=None,
    )
    models_list = AsyncMock(side_effect=transient_500)
    fake_client = SimpleNamespace(
        models=SimpleNamespace(list=models_list)
    )
    provider = OpenAIProvider(
        config={"max_retries": 5},
        client=fake_client,
    )

    assert provider.config["max_retries"] == 5

    try:
        asyncio.run(provider.list_models())
    except openai.APIStatusError:
        pass
    else:
        raise AssertionError("expected the synthetic HTTP 500")

    assert models_list.await_count == 1


if __name__ == "__main__":
    test_current_list_models_gets_only_one_attempt()

The reproduction above executes without network access and observes exactly one models.list() call.

The routing cache helps only after a successful model listing; it cannot protect the first discovery attempt. After an empty result, routing can continue evaluating configured candidate or role fallbacks as control flow, but this evidence does not show that any fallback successfully resolved.

Expected behavior

• Transient model-list failures should use the provider's existing retry/backoff policy before reaching routing.
• Standard Retry-After and Azure x-ms-retry-after-ms headers should retain the same semantics as completion requests.
• A server-directed delay above configured max_delay should retain the existing fail-fast behavior.
• Retries should emit the established PROVIDER_RETRY event.
• Non-retryable and exhausted failures should continue to propagate as typed provider errors.
• Routing should retain its existing candidate/role fallback evaluation and successful-result caching behavior.
• SDK retries should remain disabled so there is only one retry policy.

Impact

This affects cold model discovery and any later discovery attempt made before a successful result populates the cache. A transient provider failure can silently remove or demote a glob candidate even though the same failure during completion would be retried.

The resolver's exception containment keeps this path nonfatal. Routing may continue evaluating configured candidate or role fallbacks, but the observed and synthetic evidence showed no resolved model for the failing glob candidate.

Proposed direction

Implement the fix in the OpenAI provider rather than changing routing:

1. Protect list_models() with the existing provider-owned RetryConfig and retry/backoff utility.
2. Translate SDK exceptions to the same typed kernel errors used by completion requests.
3. Reuse or exactly preserve existing 429 behavior:
   • standard Retry-After;
   • x-ms-retry-after-ms fallback;
   • standard-header precedence;
   • retry_after propagation;
   • fail-fast when the delay exceeds max_delay.
4. Emit the existing provider retry hook event for each retry.
5. Leave routing fallback evaluation and cache policy unchanged.

Regression tests

Please cover:

• success on the first attempt;
• transient HTTP 500 followed by success;
• persistent 5xx exhaustion;
• non-retryable 401 without retry;
• 429 with standard Retry-After;
• 429 with x-ms-retry-after-ms;
• standard-header precedence when both are present;
• server delay above max_delay;
• PROVIDER_RETRY event name and payload;
• routing integration showing that transient failure is retried before a glob candidate is skipped;
• existing candidate/role fallback evaluation after retries are exhausted.

Related work

At head 9407928, inspected on 2026-08-13, microsoft/amplifier-module-provider-openai#61 was open and directionally addressed transient 5xx retry.

At that inspected head, its model-list 429 branch did not preserve the completion path's Retry-After/maximum-delay behavior, and its new retry-event callback lacked direct test coverage. Later changes to PR #61 may supersede that assessment. Updating and merging that PR could resolve this issue.

microsoft/amplifier#369 is related but distinct: it concerns repeated successful list_models() calls and caching/single-flight behavior, not retrying transient failures.