Evals

Run dataset and CI evaluations against prompts, agents, tools or workflows. Hosted targets make network requests and incur provider charges; “dataset evaluation” does not mean model-free or offline execution.

Start Here

Before running the local commands below, configure a local workspace so .actspan.env disables portal reporting. Export your OpenAI key before live calls:

export OPENAI_API_KEY="<your-openai-api-key>"

Run one smoke case after the target succeeds normally:

cd "$HOME/actspan/actspan-examples/openai/data_enrichment"
actspan eval run eval_summarize_data_enrichment.yaml \
  --mode smoke \
  --fail-on-constraints \
  --report-json /tmp/actspan-eval.json

Expected result: the summary reports total cases, pass rate, must-pass failures, constraint status, failed case IDs, average cost, and the detailed case-record location.

A nonzero exit caused by a failed constraint is an evaluation result, not automatically a CLI defect. Read the failed checks before changing the runner.

Use the public examples archive for the command above. It contains the complete summary suite and fixed dataset. The field synopses below are reference fragments, not additional files in that archive. Create their workspace and dataset before running them.

Syntax (core fields)

id: <eval_id>                     # required
description: "..."                # optional
prompt:                           # inline prompt OR target
  provider: openai
  model: gpt-4.1-mini
  template: |
    ...
# or:
target:
  file: ./workspace.yaml
  kind: agent | prompt | tool | workflow
  name: <ref_name>

dataset:                          # required
  from_file: ./datasets/basic_math.yaml
  format: yaml|jsonl
  id_key: case_id
  input_key: question
  expected_substring_key: expected

input_map: { question: question } # map dataset fields into template/target

metric: ... | checks: { rules: [...] }  # at least one required

variants: []                      # optional sweeps
params: {}                        # optional model params applied to each run
recommend:                        # optional recommend config
  search: { method: grid, params: { temperature: { values: [0.0, 0.3] } } }
  trials: 1

Field notes:

• id required; used as eval_id in run outputs.
• Use either prompt (inline) or target (reference to an existing prompt, agent, tool or workflow).
• If target is omitted and an adjacent workspace.yaml contains summarize_agent, the CLI auto-selects that agent as the default eval target.
• dataset: inline rows or file reference; supports key remapping.
• metric or checks must be present.
• variants: optional overrides to sweep params/template.
• recommend: optional search settings for eval recommend.
• params: optional model params applied to each run (temperature, max_output_tokens, etc.).
• input_map: map dataset fields into template/target variables.
• constraints: optional hard constraints for recommend/run gating.
• suggest: use actspan eval suggest on the --runs-out artifact to generate isolated improvement suggestions.
• eval commands default to --state-mode minimal, which suppresses runtime state/history/cache/telemetry and writes default eval artifacts under .actspan/evals/<eval_id>/.
• --state-mode off writes nothing to disk and is incompatible with --runs-out, --report-json, --lock-out, and --write-yaml-patch.
• --no-state is a legacy alias for --state-mode off.
• --mode smoke trims the dataset and grid for quick, low-cost smoke runs.
• --max-concurrency N runs eval cases in parallel. The default is 4, and progress plus ETA are emitted to stderr during runtime eval commands.
• --resume-from PATH resumes a runtime eval from an existing runs.jsonl artifact. When resuming in place, new rows append incrementally as cases finish.

Common params (examples)

• temperature: 0–2; lower settings can reduce sampling variability but do not guarantee deterministic results. Good defaults: 0–0.3 for classify, 0.2–0.5 for summarize.
• max_output_tokens: cap verbosity; keeps costs bounded.
• top_p, frequency_penalty, presence_penalty: standard OpenAI sampling knobs.
• Any OpenAI request param supported by your provider adapter can be placed in params.

Workflow evaluation from the public archive

The public refund directory includes eval_refunds.yaml with this target:

target:
  file: ./workspace.yaml
  kind: workflow
  name: refund_request

This is an excerpt: run the complete downloaded file with its matching workspace, policies, synthetic fixtures and fresh ledger as described in the refund walkthrough. Evaluations can write sandbox payments. A stage or prompt evaluation does not establish full workflow coverage.

Reference example: create sample_eval.yaml

The following self-contained illustrative dataset makes live OpenAI requests. It is not an additional file shipped in the public examples archive.

id: math_sample
dataset:
  - { id: answer42, question: "What is 6 times 7?", expected_substring: "42" }
prompt:
  provider: openai
  model: gpt-4.1-mini
  template: |
    You are a calculator. Respond with ONLY the numeric answer.
    Question: {{ question }}
input_map: { question: question }
checks:
  rules:
    - id: must_contain_42
      type: must_include_any
      phrases: ["42"]
recommend:
  search:
    method: grid
    params:
      temperature: { values: [0.0, 0.3] }
  trials: 1

Comprehensive example (inline dataset + metric + checks + variants):

id: classify_eval
description: "Classify short questions into qa vs fact_finding"
prompt:
  provider: openai
  model: gpt-4.1-mini
  template: |
    You are a classifier. Return JSON: {"routing": "qa|fact_finding", "confidence": 0-1}.
    Question: {{ question }}
params:
  temperature: 0
input_map: { question: input }
dataset:
  - { id: case1, input: "What is 2+2?", expected_substring: "qa" }
  - { id: case2, input: "Tell me about my account", expected_substring: "fact_finding" }
metric:
  type: contains_substring
  field: output_text
checks:
  rules:
    - id: must_output_json
      type: regex_must_match
      regex: "^\\{.*routing.*\\}$"
variants:
  - id: temp_sweep
    overrides:
      params:
        temperature: 0.2
constraints:
  - metric: pass_rate
    op: ">="
    value: 0.9

Reference fragment for your own declared classify_agent and separately created JSONL dataset:

id: agent_eval
target:
  file: ./workspace.yaml
  kind: agent
  name: classify_agent
dataset:
  from_file: ./datasets/classify_cases.jsonl
  format: jsonl
  id_key: id
  input_key: input
  expected_substring_key: expected
params:
  temperature: 0.1
metric:
  type: contains_substring
  field: output_text
checks:
  rules:
    - id: must_return_json
      type: regex_must_match
      regex: "^\\{.*routing.*\\}$"

Tool source evidence

tool_called proves that a tool invocation was observed. It does not prove that a retrieval or hosted search returned inspectable source evidence. Use tool_sources_present when the workflow requires actual source URLs:

checks:
  rules:
    - id: web_search_completed
      type: tool_called
      name: web_search
      ok: true
    - id: web_search_returned_sources
      type: tool_sources_present
      name: web_search
      min_count: 1
      hard_fail: true

The source check counts unique returned URLs. A reported source count without the underlying source records does not pass the check. Passing this rule proves that inspectable source evidence exists; it does not by itself prove that every generated claim is supported by a specific source.

For prompt and agent targets, feedback retries and JSON repair retain the actual tool traces from earlier attempts. Attempt metadata identifies the response that produced each trace; the final raw provider response still describes the call that produced the final output. Usage totals include the returned attempts. A repair call with no tools cannot erase an earlier MCP call or manufacture evidence for a tool that was never called. Schema-valid repair alone does not establish that repaired values are supported by the retrieved sources.

Inline dataset with variants (sweep temperature):

id: inline_sweep
dataset:
  - { id: a, input: "Paris?", expected_substring: "qa" }
  - { id: b, input: "Tell me about your systems", expected_substring: "fact_finding" }
prompt:
  provider: openai
  model: gpt-4.1-mini
  template: "Classify: {{ input }}"
input_map: { input: input }
metric:
  type: contains_substring
  field: output_text
variants:
  - id: temp0
    overrides: { params: { temperature: 0.0 } }
  - id: temp03
    overrides: { params: { temperature: 0.3 } }

Recommend sweeping multiple params:

id: sweep_multi
prompt:
  provider: openai
  model: gpt-4.1-mini
  template: "Answer briefly: {{ question }}"
input_map: { question: input }
dataset:
  - { id: x, input: "What is 2+2?", expected_substring: "4" }
metric: { type: contains_substring, field: output_text }
recommend:
  search:
    method: grid
    params:
      temperature: { values: [0.0, 0.3] }
      max_output_tokens: { values: [64, 128] }
  trials: 1

This grid has 2 × 2 = 4 configurations. Planned model executions are configurations × dataset cases × trials; provider retries, tools and output repair can add requests. With one case and one trial, this example plans four executions. Review the grid and pricing before launching it.

Notes:

• You can sweep any provider param your adapter supports (e.g., top_p, penalties, logprobs, tool_choice, etc.). Each key under params produces a cartesian product of values.
• Keep the grid reasonable; large grids can be expensive. Use trials > 1 for repeated sampling per config if desired.

Inputs vs. Outputs in evals

• Inputs to the model come from dataset rows, mapped via input_map into your prompt/target variables.
• Outputs produced by actspan eval run|sweep|recommend default to .actspan/evals/<eval_id>/runs.jsonl, .actspan/evals/<eval_id>/report.json, and for recommend, .actspan/evals/<eval_id>/lock.yaml.
• --runs-out, --report-json, --summary-json, and --lock-out override those defaults where supported.
• actspan eval run --report-json ... writes a compact machine report by default. Full raw case evidence stays in runs.jsonl; pass --report-detail full only when you need one forensic single-file artifact.
• actspan eval run --summary-json ... writes a small human-review report with metrics, failed cases, compact per-case summaries, tool evidence, and links to raw evidence.
• You can also save outputs elsewhere by passing explicit paths. Each JSONL line typically includes:
  • eval_id, case_id (dataset id), variant_id (if any)
  • input (mapped fields), prompt_name/target info
  • response fields: output_text, output_parsed (if JSON), usage (tokens/cost), model
  • metric result or checks results (pass/fail, reasons)
  • constraints_ok and constraint_reasons (if constraints are used)
  • status and error (if the call failed)
• You can post-process these outputs for reporting or feed them into prompts in follow-up runs (e.g., use output_parsed to debug prompt quality).
• You can also feed the runs artifact into actspan eval suggest to generate isolated suggestions about what to change next.
• Suggestions are validation artifacts, not automatic edits. The intended loop is: suggest -> candidate revision -> compare/gate -> human decision.
• actspan should validate a suggestion before trusting it, but it should not auto-apply that suggestion to source or release state.
• Hosted-tool targets such as web_search emit a warning because they can be slow/expensive. Use --mode smoke or smaller datasets/grids for those evals.
• actspan eval prune --eval-id <id> removes .actspan/evals/<id>/ for a completed eval run.
• actspan cache prune --older-than 1d and actspan state prune --older-than 7d clean up old local artifacts.

Example:

actspan eval run sample_eval.yaml

actspan eval recommend sample_eval.yaml \
  --runs-out .actspan/evals/math_sample/recommend.runs.jsonl \
  --report-json eval_reports/sample_recommend.json \
  --lock-out eval_locks/candidate.lock.yaml

actspan eval suggest .actspan/evals/math_sample/recommend.runs.jsonl \
  --report-json eval_reports/sample.suggestions.json

actspan eval compare sample_eval.yaml \
  eval_locks/baseline.lock.yaml \
  eval_locks/candidate.lock.yaml \
  --runs-out eval_runs/sample_compare.jsonl \
  --report-json eval_reports/sample_compare.json

actspan eval dataset add-case eval_datasets/regressions.yaml \
  --input "Where is my order?" \
  --expected-substring "tracking" \
  --must-pass \
  --slice regressions

actspan eval dataset add-from-runs eval_datasets/regressions.yaml \
  .actspan/evals/math_sample/runs.jsonl \
  --slice regressions

actspan eval grader calibrate eval_reports/sample.report.json labels/human_labels.yaml \
  --report-json eval_reports/sample.calibration.json

actspan eval gate eval_reports/sample_compare.json --format json

Resume a long-running eval from the same artifact:

actspan eval recommend sample_eval.yaml \
  --runs-out .actspan/evals/sample_eval/runs.jsonl \
  --resume-from .actspan/evals/sample_eval/runs.jsonl \
  --report-json .actspan/evals/sample_eval/report.json

Canonical objects emitted today:

• actspan.eval.dataset
• actspan.eval.grader_version
• actspan.eval.case_result
• actspan.eval.experiment
• actspan.eval.suggestion
• actspan.eval.grader_calibration
• actspan.eval.release_gate
• actspan.eval.review_report

What suggest is for:

• identify clustered failure patterns
• surface recurring client prompt shapes
• create isolated suggestions before starting an experiment
• preserve debug refs back to the sampled failed cases

What suggest is not for:

• deciding the winner between baseline and candidate revisions
• promoting changes directly without experiment/eval validation

What compare is for:

• running the same eval spec against baseline and candidate lockfiles
• generating a stable experiment artifact
• showing deltas, regressions, and a promotion-oriented recommendation

What compare is not for:

• pairwise side-by-side LLM judging
• asking one judge prompt to pick a winner between baseline and candidate outputs
• subjective preference tasks such as "which answer sounds better?" without explicit per-output scoring rules

What dataset is for:

• turning manual regressions into reusable dataset rows
• importing failed eval runs into regression slices
• preserving enough source metadata to trace why a case exists later

What gate is for:

• turning compare or run evidence into a stable release decision
• marking a candidate as promotable, review, or blocked
• preserving the baseline/candidate evidence bundle that drove the decision

What grader calibrate is for:

• checking grader agreement against human-labeled cases
• measuring false positives, false negatives, disagreement rate, and score error
• producing a reusable calibration artifact tied to the current grader_version

Example run output line (abridged):

{
  "eval_id": "classify_eval",
  "case_id": "case1",
  "variant_id": "temp0",
  "input": { "input": "What is 2+2?" },
  "response": {
    "output_text": "{\"routing\":\"qa\",\"confidence\":0.98}",
    "output_parsed": { "routing": "qa", "confidence": 0.98 },
    "usage": { "input_tokens": 42, "output_tokens": 18, "total_tokens": 60 },
    "model": "gpt-4.1-mini-2025-04-14"
  },
  "metric": { "type": "contains_substring", "score": 1.0, "passed": true },
  "checks": [
    { "id": "must_return_json", "passed": true }
  ],
  "constraints_ok": true,
  "status": "success"
}

The remaining compare, dataset and calibration commands are reference examples. Create the named lockfiles, datasets, reports and human labels first; those paths are not supplied by the public archive. Do not run a release gate on a missing or mismatched report.

Validation

• actspan validate sample_eval.yaml --schema eval

Common errors

• Missing id or dataset fields.
• Providing both prompt and target (use only one).
• Checks/metric absent.
• Dataset path wrong or format not set to yaml/jsonl correctly.