Evaluating and Improving Agents at Scale

1. The eval gap problem

Most teams shipping AI products measure the wrong thing. They run a benchmark, see a higher number, and ship — only to discover that their users are no happier than before. This is the eval gap: the disconnect between the scores you can measure cheaply and the outcomes that actually matter.

Michele Catasta, Replit’s President and Head of AI, frames it bluntly: a benchmark score is only useful if it correlates with user satisfaction. Most don’t, or only weakly. Why?

• Benchmarks are narrow. Academic evals reward the right token at the right position. Real users care about whether the app they asked for actually runs.
• They don’t cover your population. A model that scores 90% on a general coding benchmark may be mediocre at exactly the tasks your users bring.
• Intermediate metrics mislead. Measuring whether the agent called the right tools, or produced syntactically valid code, says nothing about whether the end result works.

The practical consequence: a team can spend weeks upgrading to a newer model, run their existing evals, declare victory, deploy — and ship a regression. Or they can be so afraid of this failure mode that they never upgrade at all.

Replit’s solution is a two-pillar evaluation system that forces both “does it work in isolation?” and “is it better for our actual users?” to be answered before any model change ships.

2. Pillar one — VibeBench and offline evaluation

VibeBench (open-sourced at vibebench.ai) is Replit’s answer to the first part of the eval gap: building a benchmark that is both automated and predictive of user satisfaction.

What makes a good offline eval

The key design insight is that the eval should test the same thing the user experiences. For a coding agent, users don’t read the source code — they run the app. VibeBench evaluates accordingly:

1. The model receives a real-world PRD (product requirement document) — tasks like “build a todo app with authentication” or “create a REST API with pagination.”
2. The model generates a full application.
3. The application is deployed into a sandbox.
4. Browser automation runs against the live sandbox to check whether it actually works.
5. A score is produced.

Twenty such PRDs form the benchmark suite. They were chosen from real Replit user requests, which is what gives VibeBench its correlation with production quality — the distribution of tasks matches what real users ask.

Why end-to-end matters

Measuring at the end of the pipeline captures everything: hallucinated imports, broken routing, auth that doesn’t persist sessions, UI that crashes on click. No intermediate metric sees all of these at once. The browser doesn’t care why the app is broken; it just reports that it is.

This also means VibeBench can run entirely unattended. No human needs to read code, launch a dev server, or click through the app. A nightly CI job can run all 20 PRDs against a candidate model and produce a score before anyone arrives at the office.

3. Pillar two — online evaluation

An offline eval tells you whether a change is better in isolation. It cannot tell you whether it is better for your specific users in your specific product context. That requires online evaluation: putting the change in front of real users and measuring what happens.

Replit uses two techniques together.

AB testing

Route a fraction of production traffic to the new model version; keep the rest on the current version. Measure outcomes — task completion, user satisfaction signals, error rates — and determine whether the new model is genuinely better for this population.

AB tests are the gold standard for “does this change help our users?” but they are expensive: you need real users, real time, and a statistically significant sample. You cannot run them 100× a night. That is exactly why offline evals exist: to filter out the obviously-bad changes before you spend the time on an AB test.

Trace clustering

AB tests tell you aggregate outcomes. They do not tell you why things go wrong. Trace clustering fills that gap.

The idea: take thousands of production interaction traces (the full sequence of what the agent did and what the user asked), and group them by similarity. Patterns emerge that are invisible in aggregate metrics:

• A cluster of traces where users ask for “login with Google” always ends in an error
• A cluster where multi-file refactors stall partway through
• A cluster where the agent loops on a specific library version conflict

These recurring failure patterns are the raw material for improvement. Without clustering, a single failure looks like noise. At scale, it becomes a signal you can act on.

4. The Telescope pipeline

VibeBench and AB testing answer the question “is this better?” Telescope is the system that asks and answers that question continuously, at scale, without waiting for an engineer to initiate the cycle.

The pipeline has four stages:

Stage 1 — Discover. Telescope continuously clusters production traces. When a cluster grows large enough to be statistically meaningful — a recurring failure pattern — it surfaces as a candidate for improvement.

Stage 2 — Create. A coding agent (Claude) takes the failure pattern as input and generates a PR with a proposed fix. The agent has access to the codebase, the failure traces, and any relevant context. No human writes this code.

Stage 3 — Evaluate. The PR is tested against VibeBench. If the benchmark score clearly improves, the change is a candidate for merging. If the result is controversial — marginal improvement, or a tradeoff between different task types — the change moves to an AB test with real users.

Stage 4 — Ship or iterate. If evaluation passes (benchmark or AB test), the PR is merged. If it fails, the results are fed back as additional context and the cycle restarts.

The numbers

Catasta shares a striking metric: 90% of Replit’s AI engineering work is now aided by agents running inside Telescope or similar pipelines. The system doesn’t mean there are fewer engineers — it means each engineer’s judgment is applied at the highest-leverage point: deciding which experiments to run and what the results mean, rather than writing the boilerplate code for each experiment.

5. What humans still do

A common misreading of a system like Telescope is that it removes humans from the loop. It doesn’t — it changes where humans are in the loop.

The parts Telescope automates are the ones that don’t require taste:

• Clustering traces to find recurring patterns
• Writing code to fix a well-specified problem
• Running an eval suite and comparing scores

The parts that remain firmly human are the ones that do require taste:

AB test decisions. Aggregate metrics rarely tell an unambiguous story. A new model might be better at long tasks but worse at short ones. It might improve success rate but increase latency. A human decides which tradeoff to accept based on product strategy, not benchmark arithmetic.

Hypothesis formation. Trace clusters show you what is failing. They don’t tell you why, or which of ten possible fixes is worth trying first. That inference requires understanding the product, the users, and the model’s known failure modes.

Designing the next experiment. The most valuable engineering work is choosing the right question to ask. What should the next eval measure? Which failure pattern should Telescope attack next? These are creative decisions that shape the whole pipeline’s output.

eval_cases = [
    {
        "input": "Build a REST endpoint that returns paginated results",
        "pass_if": lambda output: check_endpoint_works(output),
    },
]

def run_eval(model_fn):
    results = [case["pass_if"](model_fn(case["input"])) for case in eval_cases]
    return sum(results) / len(results)

python run_eval.py --model current > baseline.json

import json, time

def log_trace(input, output, success):
    record = {
        "ts": time.time(),
        "input": input,
        "output": output,
        "success": success,
    }
    with open("traces.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")

python -c "
import json
failures = [json.loads(l) for l in open('traces.jsonl') if not json.loads(l)['success']]
for f in failures[:30]:
    print(f['input'][:80])
"

python run_eval.py --model current  > current.json
python run_eval.py --model candidate > candidate.json
python compare_evals.py current.json candidate.json

Where to go next

• Watch the original talk by Michele Catasta — the live demo of Telescope in action is worth seeing.
• Explore VibeBench to see how Replit’s benchmark is structured and consider adapting it for your domain.
• Continue with Prompting for Agents to understand how to design the agent behavior that your evals will measure.