mare_mark

Reproducible benchmarking, statistical comparison, tuning and self-contained reports for MoonBit.

mare_mark is an experiment harness, not a stopwatch wrapper. It validates every implementation against an oracle before timing it, measures in calibrated batches with a balanced, seeded order, keeps every raw observation in an append-only JSONL record, decides with paired robust statistics and a seeded bootstrap, and renders reports that need no server. A result can be replayed, re-analysed and audited from its record.

Installation

moon add Luna-Flow/mare_mark@0.4.0

Requires MoonBit with moonc 0.10 or later.

Example

import {
  "Luna-Flow/mare_mark/model",
  "Luna-Flow/mare_mark/env_detect",
  "Luna-Flow/mare_mark/event",
  "Luna-Flow/mare_mark/runner",
  "Luna-Flow/mare_mark/report",
  "moonbitlang/async",
}

async test "loop versus closed form" {
  let looped = @runner.Implementation::stateless("loop", "1", (n : Int) => {
    let mut total = 0
    for i in 0..<n {
      total += i
    }
    @model.OperationResult::completed(total, ())
  })
  let formula = @runner.Implementation::stateless("formula", "1", (n : Int) => {
    @model.OperationResult::completed(n * (n - 1) / 2, ())
  })
  let plan = @runner.single_step("triangle", [1000, 100000])
    .with_immutable_input(context => context.dataset_key.scale, n => n.to_string())
    .compare([looped, formula])
    .against_equal(n => n * (n - 1) / 2, (expected, actual) => expected == actual)
    .compile()
    .unwrap()
  let record = @event.JsonlSink::new()
  let environment = @env_detect.detect(compiler_flags="release", concurrency=1)
    .snapshot
  let summary = @runner.run(
    plan,
    @runner.RunContext::new(environment, record.as_sink(), 42UL, @runner.ProtocolPreset::QuickCheck.validated()),
  )
  inspect(summary.passed_count, content="4")
  let html = @report.html(@report.document_from_jsonl(record.to_jsonl(), target="native").unwrap())
  inspect(html.has_prefix("<!doctype html>"), content="true")
}

@env_detect.detect fills the environment snapshot from the running process (target, OS, hostname, CPU and cores, time, git revision, a fresh run id); every field can be overridden by a labelled argument, and what it cannot detect is "unknown" and listed in undetected. The example measures sequentially, so it passes concurrency=1. Both implementations are validated on both scales before timing; the JSONL record holds every observation, and @stats.compare_paired turns the paired confirmatory blocks into a decision. The command-line tool renders records and replays failures. report pairs the confirmatory observations of each implementation with a baseline block by block, applies the recorded outlier policy and threshold, and shows the seeded-bootstrap decision of every comparison in a "Comparisons" table, in the HTML and on the terminal:

moon run src/cli --target native -- report --baseline scalar testdata/report/compare.jsonl report.html
moon run src/cli --target native -- replay testdata/replay/sample.jsonl --dry-run

Packages

Package	Role
model	shared vocabulary: versions, protocols, environments, outcomes, events, decisions
env_detect	environment snapshots detected from the running process
generator	seed derivation and input fingerprints
fixture	input lifecycle and setup timing
experiment	oracles, shrinking, crossover analysis
runner	validation, warmup, calibration, balanced blocks, raw events
event	sinks and the JSONL record
ir_sink	short constructors for the common sinks
stats	summaries, paired comparisons, bootstrap intervals, outlier views
ir_model	Plot IR
report	JSONL to JSON, SVG and self-contained HTML
tune	tuning policy: scores, practical ties, Pareto fronts, seeded subsets
tune_gemm	worked tuning domain: blocked matrix multiplication
cli	the mare-mark executable: report and guarded replay

All packages build on every target. runner.run is asynchronous and runs on native, JS and wasm; subprocess workers and replay need native. env_detect.detect probes the host fully on native, partly on Node.js, and falls back to environment variables elsewhere.

Documentation

The manual is published at https://lunaflow.cn/en/mare_mark/ in English, Chinese and Japanese. Its source is doc/manual/index.md: an API page, a tutorial and a design page for every package, plus getting started, architecture and verification.

Contributing

Run moon fmt, moon info, moon check --target all, and moon test --target native and --target js before a pull request; the verification guide lists the full matrix and the smoke tests. Use Conventional Commits. pkg.generated.mbti files are generated: update them with moon info, never by hand, and update the API pages when they change. Agent-specific notes are in AGENTS.md.

License

Apache-2.0. See LICENSE.