llm-ai-gateway

A Django-based AI gateway service that provides a controlled and standardized interface for interacting with Groq LLM APIs. This service acts as an abstraction layer between backend systems and the LLM provider, ensuring consistent request validation, response formatting, and error handling.

---

Overview

This project implements an LLM Gateway Microservice designed to:

• Validate and sanitize incoming prompts
• Support conversational context via message history
• Abstract direct interaction with the Groq API
• Standardize responses for upstream services
• Provide structured logging and error handling

Instead of exposing the LLM provider directly, this service ensures that all interactions go through a controlled, observable, and maintainable interface.

---

Architecture Role

This service sits between the backend (llm-ai-server) and the LLM provider:

Client / Backend → llm-ai-gateway → Groq API

Responsibilities:

• Input validation (DRF serializers)
• Message transformation (OpenAI-compatible format)
• LLM interaction via service layer
• Centralized exception handling
• Request tracing via X-Request-ID
• Rate limiting and abuse protection

---

Tech Stack

• Django
• Django REST Framework
• Groq API (OpenAI-compatible)
• Docker
• colorlog (structured logging)

---

Key Features

1. Service Layer Abstraction

• LLMService encapsulates all LLM interactions
• Decouples API layer from provider implementation

2. Input Validation

• Strict validation via serializers
• Enforces prompt constraints and message structure

3. Conversation Support

• Accepts conversation history
• Converts messages into OpenAI-compatible format

4. Centralized Error Handling

• Custom exception handler for:

  • validation errors
  • provider failures
  • unexpected server errors

• Consistent error response format

5. Observability

• Structured logging with:

  • request IDs
  • token usage
  • response metadata

• Colored logs for readability

6. Rate Limiting

• Anonymous request throttling (100/hour)
• Prevents abuse of public endpoint

---

API Endpoints

Generate Response

POST /api/v1/prompt/

Request

{
  "prompt": "Explain hashmap in Java",
  "conversation_history": [
    {
      "role": "user",
      "content": "What is a map?"
    }
  ]
}

Response

{
  "prompt": "Explain hashmap in Java",
  "result": "A HashMap in Java is a data structure..."
}

---

Health Check

GET /api/v1/health/

Response

{
  "status": "ok"
}

---

Error Handling

All errors follow a consistent structure:

{
  "error": "Error message",
  "request_id": "optional-request-id"
}

Error Types:

• 400 → Invalid input (validation errors)
• 502 → LLM provider failure
• 500 → Internal server error

---

Environment Configuration

Create a .env file based on .env.example:

GROQ_API_KEY=your_api_key_here
GROQ_DEFAULT_MODEL=llama-3.1-8b-instant
GROQ_DEFAULT_TEMPERATURE=0.7
GROQ_DEFAULT_MAX_TOKENS=1024
GROQ_SYSTEM_PROMPT=You are a helpful assistant.

DJANGO_PORT=8000

---

Running Locally

1. Clone repository

git clone <repo-url>
cd llm-ai-gateway

2. Setup environment

cp .env.example .env
pip install -r requirements.txt

3. Run server

python manage.py runserver

---

Running with Docker

docker-compose up --build

---

Design Decisions

Why a Gateway Layer?

Directly exposing LLM providers creates:

• tight coupling
• inconsistent responses
• lack of control over usage

This service introduces:

• a stable contract
• centralized validation
• observability and logging

---

Why Service-Oriented Structure?

Separating views → serializers → services ensures:

• maintainability
• testability
• scalability for future providers (OpenAI, Anthropic, etc.)

---

Future Improvements

• Authentication & API keys
• Streaming responses (token streaming)
• Caching frequent prompts
• Multi-provider support
• Request persistence (audit/logging DB)

---

Related Services

• llm-ai-server → Spring Boot backend
• llm-ai-client → Angular frontend