Calibrate Backend

Backend for Calibrate, a framework for evaluating AI agents which let you move from slow, manual testing to a fast, automated, and repeatable testing process for your entire agent stack.

Installation

Install dependencies using uv:

uv sync --frozen

For local development with tests:

uv sync --frozen --group dev

Running locally

Create src/.env.

cp src/.env.example src/.env

See ENV.md for what each variable means.

Start the development server:

cd src
uv run uvicorn main:app --reload

The app will be available at: http://localhost:8000

API documentation: http://localhost:8000/docs

Contributing

Related repositories:

• Calibrate frontend
• Calibrate CLI

Reference docs:

• Architecture diagram

After cloning the repo, enable the project's git hooks so the pre-commit test runner fires on commits to main:

git config core.hooksPath .githooks

Every contributor needs to run it once.

Local database

For local development, TablePlus is a handy GUI for opening and inspecting the SQLite database.

Running tests

From the repository root (not src/):

pytest tests

Citation

If you use Calibrate in your work, please cite it:

@software{calibrate2026,
  author = {Dalmia, Aman and Doshi, Jigar},
  title  = {Calibrate: An open-source framework for evaluating AI agents},
  year   = {2026},
  url    = {https://calibrate.artpark.ai},
  note   = {Source code: \url{https://github.com/ARTPARK-SAHAI-ORG}}
}

License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.