fix(server): bound resident model memory with LRU eviction and idle TTL sweeper

[rkfshakti]

Context

Multi-voice / multi-model servers leak GPU memory until OOM: ModelProvider (mlx_audio/server.py) caches every loaded model in an unbounded dict and nothing ever evicts it. Three voice switches in the repro add ~35GB of permanently-resident memory; DELETE /v1/models works but nothing calls it automatically.

Closes #835

Description

Adds two complementary, configurable eviction policies to ModelProvider:

• LRU eviction — loading beyond --max-resident-models (default 1) evicts the least-recently-used model and calls mx.clear_cache() to release its weights.
• Idle TTL — --model-idle-ttl-seconds (default 0 = off) unloads models unused for that long via a background sweeper started/stopped by the FastAPI lifespan.

Both are also settable via MLX_AUDIO_MAX_RESIDENT_MODELS and MLX_AUDIO_MODEL_IDLE_TTL_SECONDS; CLI flags take precedence.

Changes in the codebase

• ModelProvider now tracks per-model last-use timestamps, guards its dict with a threading.Lock (safe against FastAPI threadpool workers racing the event-loop sweeper), and exposes _evict / _evict_least_recently_used / start_sweeper / stop_sweeper.
• app_lifespan starts/stops the sweeper so the task is cancelled on shutdown.
• main() gains --max-resident-models and --model-idle-ttl-seconds; applied directly to the import-time provider instance.
• Docs: server options table + model-management section updated.

Changes outside the codebase

None.

Additional information

• Existing behavior is preserved by default: with max_resident_models=1 and TTL 0, a single-model server behaves exactly as before (its one model is never evicted).
• Eviction only happens on a new load above capacity (LRU) or after the TTL window (sweeper), so the actively-used model is never dropped mid-request.

Checklist

• Tests added/updated — 7 new unit tests (LRU eviction, capacity clamping, remove_model memory release, idle sweeper, TTL-disabled no-op)
• Documentation updated
• Issue referenced (Closes ModelProvider (mlx_audio.server) never evicts loaded models — unbounded memory growth with multiple voices #835)

[rkfshakti]

Friendly ping on this one — it fixes the ModelProvider memory leak reported in #835 (LRU eviction + idle TTL sweeper), and all CI is green on the current head. It's been quiet for about two weeks; is there anything else you'd like changed, or could someone take a look when convenient? Happy to rebase on latest main if that helps.

[Lazarus-931]

hi @rkfshakti, i appreciate the pr!

I think this can be better designed, specifically _evict(), I think adding gc.collect() before mx.clear_cache() might solve the leak issue :)

[Lazarus-931]

i think dropping the reference does not free the weights, since nn.Module has reference cycles

[Lazarus-931]

universally, we do small weights for testing.

these tests don't really reflect that your change fixes the leak since this is not even calling nn.Module

I'd say tests can be improved as well

[rkfshakti]

Thanks @Lazarus-931 — agreed, and applied. _evict() now calls gc.collect() before mx.clear_cache() (f4a4a89) so cyclical model/processor references are collected first; otherwise those buffers stay alive and clear_cache() can't reclaim them.

While adding the regression test for the ordering, the extra gc.collect() overhead exposed a timing race in the idle-sweeper test (the second sweep could land past the TTL for a freshly loaded model), so I extracted the stale-reaping into a deterministic evict_stale(now) method and rewrote that test against a fake monotonic clock. Also rebased onto current main (was 94 commits behind) — conflict-free.

8/8 model-provider tests green locally. Re-review when you have a moment. 🙏

[Lazarus-931]

small fix, otherwise gtg :)

[Lazarus-931]

shouldn't max-resident-models when 1, cause stt and tts adapters to load through same model_provider via _load_model_for_inference, so we'd have, which we don't want a server handling both.

i think it needs to be 0 by default

[rkfshakti]

Thanks @Lazarus-931 — applied in dc2bd44. --max-resident-models now defaults to 0 (unbounded, LRU disabled) so a server mixing STT and TTS adapters won't evict each other's model from the shared provider; setting >= 1 opts into the LRU bound. The clamp-to-1 test was replaced with an unbounded-set regression test, and docs/guides/web-ui-api-server.md explains the mixed-adapter rationale. All 8 memory/eviction tests green locally.

[github-actions]

⚠️ The pre-commit checks failed in CI.

Run the following from the repository root, then commit and push any changes:

python -m pip install pre-commit
pre-commit run --all-files

The project's pre-commit configuration includes Black and the other required formatters.