Actspan Docs Quickstart
DocsGuides

Make one configuration change, validate it, and evaluate its effect.

On this page

Change and evaluate a workspace

Change one instruction against fixed cases, save a baseline report, then compare the candidate's checks and output. A passing candidate alone does not show whether the change improved the behavior you care about.

Before you begin

Download the public CLI and examples. Use CLI 0.2.0 and public2, with your OpenAI key exported. Evaluations make model calls and incur charges. The summary suite uses fixed research records and does not rerun web search.

cd "$HOME/actspan/actspan-examples/openai/data_enrichment"
export ACTSPAN_PORTAL_DISABLED=1
actspan init workspace.yaml --no-env-file --no-portal
actspan validate workspace.yaml

Adjust the extraction path if needed. Git is optional: the public archive has ordinary files, not a Git repository.

Save a baseline

Create a new comparison directory and preserve the workspace:

COMPARISON_DIR=$(mktemp -d "$PWD/comparison.XXXXXX")
cp workspace.yaml "$COMPARISON_DIR/workspace.before.yaml"
actspan eval run eval_summarize_data_enrichment.yaml \
  --mode smoke --fail-on-constraints \
  --report-json "$COMPARISON_DIR/baseline.json"

Read the baseline case, its output and failed checks before editing. The desired behavior is a shorter summary that preserves the original topic, has three to six bullets and includes a takeaway. Smoke mode uses a small subset; use the same full suite for both versions before a release decision.

Make one change

Find summarize_prompt in workspace.yaml and ask for shorter bullets. Leave the model, output schema, evaluation data and checks unchanged. Compare files:

diff -u "$COMPARISON_DIR/workspace.before.yaml" workspace.yaml

diff exits 0 when identical, 1 for differences, and greater than 1 for an error. Expected differences are not failed validation. Run the next commands separately if your shell stops on any nonzero exit.

actspan validate workspace.yaml
actspan preview workspace.yaml

Review only the intended instruction change. Resolve validation errors before making further provider calls.

Evaluate the candidate

actspan eval run eval_summarize_data_enrichment.yaml \
  --mode smoke --fail-on-constraints \
  --report-json "$COMPARISON_DIR/candidate.json"

Compare case IDs, individual checks, output length and the actual bullets in the two reports. Confirm the same model and data were selected. A live model can vary with unchanged settings, so one shorter result does not establish a repeatable improvement. Add a direct length criterion if shorter bullets are a release requirement and retain representative allowed cases.

Keep the before/after files and detailed case records printed by the evaluator. Do not weaken checks merely to obtain a pass. See evaluation for full datasets and constraints.

Configuration and managed resources

Providers, prompts, agents, workflows and policies are distinct pieces. Read core concepts before moving between them. apply reconciles supported long-lived resources; run executes a target. This example has no managed resources to provision.

Next steps