OpenResponses Memproxy is an educational infrastructure component of the EDU AI LAB. The parent project is localai-isnot-cheap.
It is a small, Docker-only OpenResponses-compatible cache/proxy for exploring inference routing, response flows, observability, and resource tradeoffs. It is lab-only: it is not a general-purpose product, hosted service, durable store, or full OpenAI API replacement.
- Accepts a focused subset of the OpenAI Responses API.
- Routes model requests to configured OpenAI-compatible upstream providers.
- Keeps response flows in a weighted in-memory cache.
- Exposes request/store statistics, metrics, and optional OpenTelemetry traces.
The API and behavior are documented in docs/README.md.
The intended deployment is Docker Compose; setup, environment variables,
resource limits, health checks, routing aliases, and container troubleshooting
are in docker/README.md.
The response store is ephemeral. Process restarts, container replacement, and out-of-memory termination can remove active response flows.
This project follows the OpenResponses standard, an open specification for interoperable model response APIs. The standard and its reference implementation are developed in the openresponses/openresponses repository.
This project began from Masaic's Kotlin-based OpenResponses implementation and now uses AgC as its base. It reuses the response lifecycle, input-item routes, streaming, telemetry, and management endpoints while keeping them focused on the OpenResponses API.
EDU AI LAB tailors the AgC foundation into a small educational cache/proxy for
studying inference infrastructure and resource tradeoffs. It adds an ephemeral
in-memory response cache and exposes GET /stats so cache, request, and JVM
behavior can be inspected without a UI. The goal is to make the cost and
capacity implications of different resource choices visible during experiments.
The EDU AI LAB expands the original implementation with an in-memory response cache for exploring response flows and inference resource tradeoffs.
Memory usage is controlled through:
- A weighted Caffeine cache with an approximately 2 GiB cache limit.
- A 3 GiB JVM heap inside a Docker container limited to 4 GiB of memory.
- An 80% soft-threshold cleanup and a one-minute cleanup interval.
- A ten-minute idle-flow timeout.
Cache data is ephemeral and may be removed by eviction, cleanup, process
restart, container replacement, or out-of-memory termination. See
docs/README.md for detailed cache behavior and
docker/README.md for Docker resource configuration.
NOTE: Neither OpenAI Responses nor OpenResponses explicitly defines TTL guidance, and no established best practice was found. When the internal flow TTL expires, the complete response chain is purged.
Licensed under the Apache License, Version 2.0.