Eval Scorers

When an AI worker completes a task, how do you know whether the work was any good? Eval scorers provide structured quality assessment of agent work — each scorer examines a completed run and produces a numeric score with optional reasoning.

What eval scorers are

An eval scorer is a defined method for assessing some aspect of an agent run's quality. Each scorer produces a score between 0 and 1, where 0 is the worst possible outcome and 1 is the best. Scorers can be applied to any agent run, and multiple scorers can assess the same run from different angles.

Each scorer definition includes:

Field	Purpose
Scorer code	A unique identifier (e.g. CI_PASS_RATE)
Name	A human-readable display name
Scorer type	How the evaluation is performed (see below)
Role scope	Which role this scorer applies to — or all roles if unset
Configuration	Type-specific settings (prompt template, function name, thresholds)

Scorers are tenant-scoped and can be customised per tenant. Raytio ships with a set of built-in scorers that cover common quality dimensions.

Scorer types

There are three types of eval scorer, each suited to different kinds of assessment:

Rule-based scorers

Rule-based scorers use deterministic logic to evaluate a run. They check specific, measurable outcomes — did the CI pipeline pass? Did the agent add tests? These scorers are fast, cheap, and fully predictable.

A rule-based scorer specifies a scoring function — a named function that examines the run's artifacts and metadata to produce a score. The function has access to the run record, its artifacts, and external data (such as CI pipeline status).

Model-graded scorers

Model-graded scorers use an AI model to evaluate the work — essentially asking a second AI to review the first AI's output. They are well-suited to subjective assessments that are hard to capture with rules, such as "did the commit messages follow conventions?" or "does the merge request address what the issue asked for?"

A model-graded scorer specifies an eval prompt — a prompt template that is filled with the run's context (diff, MR description, work item description) and sent to an evaluation model. The eval model returns a score and reasoning.

tip

Model-graded scorers can use a different model than the one that performed the work. This enables cross-model validation — for example, evaluating a Claude run with GPT-4o or vice versa. Using a different model for evaluation reduces the risk of systematic blind spots.

Statistical scorers

Statistical scorers compute aggregate metrics across multiple runs rather than evaluating a single run in isolation. They answer questions like "what is this worker's average success rate?" or "how has commit quality trended over the last month?"

Built-in scorers

Raytio ships with four built-in scorers that cover the most common quality dimensions:

Scorer code	Type	What it measures
CI_PASS_RATE	RULE_BASED	Did the merge request pipeline pass on the first push?
COMMIT_QUALITY	MODEL_GRADED	Are commit messages clear, is the diff appropriately scoped, and does the change follow conventions?
ISSUE_ADHERENCE	MODEL_GRADED	Does the merge request address what the work item actually asked for?
TEST_COVERAGE_DELTA	RULE_BASED	Did the agent add or modify tests as part of the change?

CI_PASS_RATE

A rule-based scorer that checks whether the CI pipeline passed on the first push. This is a strong signal of code quality — an agent that consistently produces work that passes CI on the first attempt is more reliable than one that requires fix-up commits.

• Score 1.0 — pipeline passed on first push
• Score 0.0 — pipeline failed

COMMIT_QUALITY

A model-graded scorer that reviews the agent's commits and diff. The evaluation prompt examines:

• Whether commit messages are descriptive and follow conventions
• Whether the diff is appropriately scoped (not too broad, not too narrow)
• Whether the changes are clean and well-structured

ISSUE_ADHERENCE

A model-graded scorer that compares the agent's output against the original work item. The evaluation prompt checks:

• Whether the merge request addresses the requirements in the work item
• Whether any requirements were missed
• Whether the agent introduced changes beyond what was asked for

TEST_COVERAGE_DELTA

A rule-based scorer that checks whether the agent added or modified test files as part of the change. For work that should include tests (features, bug fixes), this is a basic hygiene check.

• Score 1.0 — tests were added or modified
• Score 0.0 — no test changes were made

How evals run

Evaluations run asynchronously after a successful agent run. When a run completes with status SUCCESS and produces a merge request, the system creates eval tasks as child work items:

Each eval task is a child work item with a finish-to-start dependency on the original work item. This means:

• Evals do not block the original work — the merge request is created immediately
• Evals get their own tracking, retry logic, and status visibility through PPM
• Eval tasks are assigned to a dedicated EVAL_SCORER role, which can have its own worker definition and model configuration

The decoupled design means that if an eval fails (for example, the eval model is temporarily unavailable), it can be retried independently without re-running the original work.

Cross-model validation

The EVAL_SCORER role can be configured with a different model than the one used for execution. This is a deliberate design choice: if a Claude worker produces code and a GPT-4o evaluator reviews it (or vice versa), systematic biases in either model are less likely to go undetected.

To configure cross-model evaluation, set the EVAL_SCORER worker definition's model preferences to use a different provider or model than your execution workers.

Eval results

Each eval produces a result record linked to both the agent run and the scorer:

Field	What it contains
Score	A value between 0.0000 and 1.0000
Reasoning	An explanation of the score (especially for model-graded)
Evaluated at	When the evaluation was performed
Metadata	Raw eval data — model used, token count, additional detail

Over time, eval results build up a dataset that lets you answer questions like:

• "What is the average CI pass rate for the backend role?"
• "Which model produces the highest issue adherence scores?"
• "Is commit quality improving or declining over time?"

This data feeds into the observability and reporting capabilities described in Cost Management.