Adaptive Disaggregated Inference on a Role-free Fleet!
SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
Open-source Python control plane for LLM inference: launch, route, scale, and observe disaggregated serving on infrastructure you control.
A lightweight LLM inference runtime, evolving toward heterogeneous, stage-disaggregated serving.
A policy layer above transport for KV movement, workload-aware admissibility, and explainable routing in disaggregated inference.
RPC distributed prefill + slot-state PD disaggregation PoC for llama.cpp
A fault-tolerant LLM routing system that decouples inference from AWS Bedrock by routing prefill and decode tasks through SQS and ensuring zero-downtime scaling with a graceful drain sidecar.
Research fork of LLMServingSim 2.0 investigating Head-of-Line (HoL) blocking mitigation in disaggregated LLM serving via novel scheduling algorithms.
Mini-Dynamo: disaggregated prefill/decode LLM inference platform with KV-cache-aware routing and observability (simulated, no GPU needed)
From-scratch LLM inference engine for Qwen2.5/Qwen3: paged attention with prefix caching, continuous batching, a Triton decode kernel, CUDA-graph decode, FlashAttention prefill, speculative decoding, prefill/decode disaggregation, and cross-model KV transfer that escalates Qwen3-1.7B → 4B without re-prefilling.
To associate your repository with the disaggregated-inference topic, visit your repo's landing page and select "manage topics."