Interop idea: EvalPort export/import for Prompt/Sample/Run data

[adhabnr-ux]

Hi, first — really nice engine. The Prompt/Sample/Run/PromptExecution split, pinned runs, and the cost/token accounting in PromptExecution#cost are a clean model for this problem, and having it live as a mountable Rails engine instead of a separate SaaS is a genuinely useful design choice.

I maintain EvalPort (Apache 2.0), an interchange format for portable LLM eval data — TestCase / Grader / Result / ResultSet as plain JSON, so eval data isn't locked into one tool. It's not a runner or a grading engine itself; it's a schema plus a Python/TS SDK, with 37 adapter packages already merged for various eval tools (e.g. deepeval-openeval-adapter, ragas-openeval-adapter).

Opening this as a genuine "would this be useful" question, not a PR — I don't know your roadmap or whether interop is a priority right now.

Why I think the mapping is fairly natural here:

• RubyLLM::Evals::Sample (variables, eval_type, expected_output) maps onto EvalPort's TestCase — the input/expected-output pair.
• eval_type in this repo — exact, contains, regex, human_judge, llm_judge (from app/models/ruby_llm/evals/sample.rb) — maps onto EvalPort's Grader types, which cover the same exact/contains/regex/human/model-judge spread.
• RubyLLM::Evals::PromptExecution (with passed, message, input/output/thinking token counts, cost) maps onto EvalPort's Result.
• A Run (with its accuracy, cost, judge_cost, total_cost methods) maps onto a ResultSet — the aggregate over one run's executions.

Since Ruby isn't one of EvalPort's SDK languages (Python/TS only today), the natural boundary would be JSON serialization rather than a native Ruby adapter package under adapters/ — e.g. something like:

# sketch — not a PR, just illustrating the shape
def prompt_execution_to_evalport_result(execution)
  {
    schema: "evalport.result/v1",
    test_case: {
      input: execution.variables,
      expected_output: execution.expected_output
    },
    grader: { type: execution.eval_type }, # exact | contains | regex | human_judge | llm_judge
    output: execution.message,
    passed: execution.passed,
    cost: execution.cost,
    tokens: {
      input: execution.input,
      output: execution.output,
      thinking: execution.thinking
    }
  }
end

def run_to_evalport_result_set(run)
  {
    schema: "evalport.result_set/v1",
    accuracy: run.accuracy,
    total_cost: run.total_cost,
    results: run.prompt_executions.map { |e| prompt_execution_to_evalport_result(e) }
  }
end

That would let a Run's results round-trip through EvalPort's format, which would be useful if you ever wanted to compare a run pinned here against results from a different tool, or feed this engine's output into something else that speaks EvalPort JSON.

Happy to sketch a fuller export module or a small doc page if there's interest — otherwise no worries at all, feel free to close this if it's out of scope.

— Sahi, independent contributor (not affiliated with this project)