Experiments & Trials

An experiment is a systematic run over an agent graph. You bind it to a graph and (optionally) a dataset, define a parameter space, and let a sampler explore that space. Each parameter configuration becomes one trial, and each trial runs the graph once per dataset item.

Use an experiment when you want to compare configurations rigorously — sweep models, prompts, and hyperparameters, score the outputs, and see which settings win.

Building an experiment

The ExperimentBuilder walks you through binding the graph and dataset and defining the space.

1. Bind a graph. Choose the agent graph the experiment runs.

2. Bind a dataset (optional). Each trial runs the graph once per dataset item; without a dataset, a trial is a single run.

3. Define the parameter space. For each tunable field, pick a mode (see below). Parameters include model and prompt pickers as well as numeric/graph fields.

4. Choose a sampler to explore the space (grid, random, or TPE).

5. Preview, then run.

Parameter modes

Each field in the space uses one of four modes:

Mode	Meaning
fixed	One constant value for every trial
values	An explicit list of values to try
range	A numeric range the sampler draws from
off	The field is excluded from the sweep

Samplers

The sampler decides which configurations from the space become trials:

Sampler	Strategy
Grid search	Every combination in the space
Random	Random configurations from the space
TPE (Bayesian)	Tree-structured Parzen Estimator — optimizes toward better scores

The score, and a judge as the score

The score metric is the number a trial is ranked by. It comes from what the graph’s Eval nodes leave on each row — pick one of their numeric keys — and it applies to every sampler, not only TPE: a grid sweep with a score is how two versions get a number to compare.

When the graph has no Eval node, or when there is no reference answer to compare against, switch on the judge: a model of your choice scores every row after it runs. It reads the question, the output, and the row’s own columns — {{answer}}, {{rating}}, {{comment}} or any other — so a dataset made from real answers and the thumbs people gave them is a judging rubric on its own. The judge leaves llm_judge in the row’s eval, in [0, 1], with its reason next to it, and llm_judge is then offered as the score metric.

The judge’s model and prompt are pinned to the experiment: a judge that changes makes two trials incomparable. Change the judge and you have a new experiment. Its tokens are counted per row (llm_judge_tokens); they are not part of the trial’s cost_usd, which is what the graph itself spent.

Preview before a full run

Use Preview to launch a small number of sample trials before committing to the full sweep. This catches prompt/graph mistakes and gives you a feel for cost and output shape before you spend on the entire space.

Running & live progress

A running experiment streams progress over SSE. You will see events as work proceeds:

• TrialStarted — a trial began
• TrialCompleted — a trial finished (with its score and metrics)
• Finished — the experiment is done

What a trial records

One experiment produces N trials — one per parameter configuration. Each trial tracks:

Field	Description
param_overrides	The exact configuration for this trial
score	Primary score from the graph’s Eval node
eval metrics	Full metric set from the Eval node
token counts	Tokens consumed across the run
cost	Estimated cost
error	Any failure for the trial

Tip

Scores and metrics come from the graph’s Eval node. If your graph has no Eval node, trials run but carry no score. See the node reference.

Reading results & plots

The Results view aggregates across trials. ExperimentPlots visualize how score and metrics move with each parameter, so you can spot the best configurations and trade-offs (e.g. score vs. cost).

Trial inspector

Open the TrialInspectorDrawer on any trial to drill into a single run. It shows the node-level execution timeline and the run events, so you can see exactly what each node did, where time and tokens went, and where a failure occurred.

Append outputs to a dataset

From the results, the AppendFromTrialModal appends a trial’s outputs back into a dataset as new rows — useful for curating generations into gold data or feeding the next round of work.