Skip to content
BranchesTags

Repository files navigation

TOONDB

TOONDB is a lightweight document database engine written in Go, with a production-style HTTP API and durable on-disk storage.

Repository:

TOONDB is designed for:

  • single-tenant containers
  • multi-tenant SaaS deployments (isolated by tenant)
  • app-friendly access over HTTP Basic Auth

What TOONDB Means

  • Document database: stores JSON-like documents (map[string]any) inside named collections.
  • Schema validation: each collection can enforce field types, required fields, and strict mode.
  • Indexed lookups: equality filters on indexed fields use a hash index for faster reads.
  • Durability: writes are persisted with WAL (write-ahead log) and recovered on restart.
  • TOON export: collections can be dumped as TOON text for compact LLM-friendly consumption.

TOON Format (Short Definition)

TOON (Token-Oriented Object Notation) is a compact, human-readable encoding of the JSON data model. It uses indentation for nested structure and a tabular style for uniform arrays to reduce token cost while preserving data fidelity.

TOONDB stores data internally and can return collection dumps in TOON through GET /v1/collections/{collection}/dump.

For TOON format specification, tooling, and ecosystem, see the official project:

Token Efficiency (TOON Benchmarks)

The section below summarizes published TOON benchmark results from the official TOON project.

Benchmark setup (official):

  • Two tracks are used for fair comparison.
  • Mixed-Structure Track: nested or semi-uniform data (TOON, JSON, YAML, XML; CSV excluded).
  • Flat-Only Track: flat tabular datasets (CSV, TOON, JSON, YAML, XML).
  • Retrieval accuracy is evaluated with 209 questions across 4 models.

Efficiency ranking (accuracy per 1K tokens; higher is better):

TOON           ████████████████████   27.7 acc%/1K tok  | 76.4% acc | 2,759 tokens
JSON compact   █████████████████░░░   23.7 acc%/1K tok  | 73.7% acc | 3,104 tokens
YAML           ██████████████░░░░░░   19.9 acc%/1K tok  | 74.5% acc | 3,749 tokens
JSON           ████████████░░░░░░░░   16.4 acc%/1K tok  | 75.0% acc | 4,587 tokens
XML            ██████████░░░░░░░░░░   13.8 acc%/1K tok  | 72.1% acc | 5,221 tokens

Key takeaway:

  • TOON is reported at 76.4% accuracy vs JSON 75.0%, while using 39.9% fewer tokens in that benchmark summary.

Token totals by benchmark track:

Mixed-Structure total
TOON                ████████████████░░░░   227,830 tokens
vs JSON             (−21.9%)               291,711 tokens
vs YAML             (−5.7%)                241,474 tokens
vs XML              (−31.0%)               330,206 tokens
vs JSON compact     (+14.7%)               198,546 tokens

Flat-Only total
CSV                 ███████████████████░    63,997 tokens
TOON                ████████████████████    67,778 tokens (+5.9% vs CSV)

CSV note:

  • CSV is excluded from the mixed ranking because it only supports 109 of 209 questions (flat-only cases).
Dataset Catalog (expand)
Dataset Rows Structure CSV Support Eligibility
Uniform employee records 100 uniform Yes 100%
E-commerce orders with nested structures 50 nested No 33%
Time-series analytics data 60 uniform Yes 100%
Top 100 GitHub repositories 100 uniform Yes 100%
Semi-uniform event logs 75 semi-uniform No 50%
Deeply nested configuration 11 deep No 0%
Valid complete dataset (control) 20 uniform Yes 100%
Array truncated: 3 rows removed 17 uniform Yes 100%
Extra rows beyond declared length 23 uniform Yes 100%
Inconsistent field count (row width mismatch) 20 uniform Yes 100%
Missing required fields 20 uniform Yes 100%

Structure classes:

  • uniform: same fields across objects with primitive values
  • semi-uniform: mixed uniform and non-uniform objects
  • nested: nested arrays/objects
  • deep: deeply nested with minimal tabular eligibility
Per-Model Accuracy (expand) 
claude-haiku-4-5-20251001
-> TOON           59.8% (125/209)
  JSON           57.4% (120/209)
  YAML           56.0% (117/209)
  XML            55.5% (116/209)
  JSON compact   55.0% (115/209)
  CSV            50.5% (55/109)

gemini-3-flash-preview
  XML            98.1% (205/209)
  JSON           97.1% (203/209)
  YAML           97.1% (203/209)
-> TOON           96.7% (202/209)
  JSON compact   96.7% (202/209)
  CSV            96.3% (105/109)

gpt-5-nano
-> TOON           90.9% (190/209)
  JSON compact   90.9% (190/209)
  JSON           89.0% (186/209)
  CSV            89.0% (97/109)
  YAML           87.1% (182/209)
  XML            80.9% (169/209)

grok-4-1-fast-non-reasoning
-> TOON           58.4% (122/209)
  YAML           57.9% (121/209)
  JSON           56.5% (118/209)
  XML            54.1% (113/209)
  JSON compact   52.2% (109/209)
  CSV            51.4% (56/109)
Performance by Question Type and Example Dataset Breakdown (expand)

Question type performance:

Question Type TOON JSON YAML JSON compact XML CSV
Field retrieval 99.6% 99.3% 98.5% 98.5% 98.9% 100.0%
Aggregation 61.9% 61.9% 59.9% 58.3% 54.4% 50.9%
Filtering 56.8% 53.1% 56.3% 55.2% 51.6% 50.9%
Structure awareness 89.0% 87.0% 84.0% 84.0% 81.0% 85.9%
Structural validation 70.0% 60.0% 60.0% 55.0% 85.0% 80.0%

Example dataset highlights:

Dataset TOON Accuracy TOON Tokens Note
Uniform employee records 73.2% 2,498 Close to CSV, much smaller than JSON/YAML/XML
E-commerce nested orders 82.3% 7,458 Best accuracy in this set
Time-series analytics 78.3% 1,553 Better accuracy than JSON/YAML/XML in this set
Top 100 GitHub repos 66.7% 8,779 Slightly above CSV accuracy
Semi-uniform event logs 65.0% 5,819 JSON compact smaller in this set
Deep nested config 94.8% 655 Best accuracy in this set

Sources:

Notes:

  • These figures are maintained by the TOON project and may change as datasets/models evolve.
  • Token counts are reported in TOON docs using GPT-5 o200k_base tokenizer (gpt-tokenizer).

Features

  • Document CRUD with collection-level schema
  • Indexed equality filtering (eq) with hash indexes
  • Numeric comparisons (gt, lt, gte, lte)
  • Tenant-scoped storage under tenants/<tenant>
  • HTTP API plus bridge-style action endpoint
  • Docker-ready deployment

Quick Start (Docker)

Run from repository root:

docker compose up -d --build

Or run directly in one line:

docker run -d --name toondb -p 6767:6767 -e TOONDB_TENANT=acme -e TOONDB_USERNAME=myuser -e TOONDB_PASSWORD=mypassword -v toondb_data:/var/lib/toondb/data-go amitdey13/toondb

Default endpoint:

  • http://localhost:6767

Default credentials in docker-compose.yml:

  • username: replace_me_user
  • password: replace_me_password

Health check:

$u = "replace_me_user"
$p = "replace_me_password"
$auth = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("$u`:$p"))
curl -H "Authorization: Basic $auth" http://127.0.0.1:6767/health

Run Without Docker

go run ./cmd/toondb-service -host 127.0.0.1 -port 6767 -tenant acme -data-dir ./data-go

On startup, the service prints JSON containing URL, username, password, and tenant.

API Overview

All endpoints require HTTP Basic Auth.

Core endpoints:

  • GET /health
  • POST /v1/collections/{collection}/create
  • POST /v1/collections/{collection}/schema/update
  • POST /v1/collections/{collection}/insert
  • POST /v1/collections/{collection}/find
  • POST /v1/collections/{collection}/update
  • POST /v1/collections/{collection}/delete
  • GET /v1/collections/{collection}/dump

Bridge endpoint:

  • POST /v1/bridge

Bridge actions:

  • create_collection
  • insert
  • update
  • update_schema
  • find
  • delete
  • dump
  • shutdown

API Payload Definitions

Schema object (create and schema/update):

{
  "schema": {
    "fields": {
      "id": "number",
      "name": "string",
      "active": "boolean",
      "meta": "object",
      "tags": "array"
    },
    "indexed_fields": ["id", "name"],
    "required_fields": ["id", "name"],
    "strict": true
  }
}

Supported field types:

  • string
  • number
  • boolean
  • array
  • object

Filter object (find, update, delete):

{
  "filter": {
    "conditions": [
      { "field": "id", "op": "eq", "value": 101 },
      { "field": "score", "op": "gte", "value": 80 }
    ]
  }
}

Supported operators:

  • eq
  • gt
  • lt
  • gte
  • lte

Insert payload:

{
  "doc": {
    "id": 101,
    "name": "Ada",
    "active": true
  }
}

Update payload:

{
  "filter": {
    "conditions": [{ "field": "id", "op": "eq", "value": 101 }]
  },
  "patch": {
    "active": false
  }
}

Configuration

Environment variables:

  • TOONDB_HOST (default: 0.0.0.0)
  • TOONDB_PORT (default: 6767)
  • PORT (fallback for platforms that provide PORT)
  • TOONDB_DATA_DIR (default: ./data-go)
  • TOONDB_TENANT (default: default)
  • TOONDB_USERNAME (optional; generated when empty)
  • TOONDB_PASSWORD (optional; generated when empty)

If username or password is missing, TOONDB generates credentials at startup.

Storage Layout

<dataDir>/tenants/<tenantId>/
  collections.meta.json
  <collection>.toon
  <collection>.idx
  <collection>.wal

File roles:

  • .toon: collection records (TOON-serialized)
  • .wal: write-ahead log for crash recovery
  • .idx: persisted hash index
  • collections.meta.json: schemas and collection metadata

JavaScript SDK

npm install toondb
import { ToonDBClient } from "toondb";

const client = new ToonDBClient({
  baseUrl: "https://your-instance.toondb.io",
  username: "your-user",
  password: "your-pass"
});

Benchmarking and Testing

Contributing

See CONTRIBUTING.md.

License

This project is licensed under the Server Side Public License v1 (SSPL-1.0). See LICENSE for the full text.

Credits

About

The database built for agent memory

toondb.lythe.ai

Topics

agentagentsdatabasetoontoondb

Resources

Readme
License

Contributing

Contributing
Activity
Custom properties

Stars

3 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages