Before We Ship: How Anthropic's Red Teams Test Claude Models

1. The Storm Before the Launch

Before any Claude model reaches the public, something specific happens: a small group of partner companies gains early access, and they immediately set about breaking it. The framing from one of these partners is vivid: “We know a storm’s ahead, but there’s something exciting about a storm because it’s all hands on deck.” Getting a new model is not a quiet event for these teams — it triggers a rapid mobilization.

The experience of receiving a new model is described as moving at “the speed of light” and requiring an immediate pivot from whatever the team was previously working on. “We have something new, let’s figure out what it’s like” captures the orientation: a combination of urgency and genuine curiosity. The grounding of their existing work has changed with the new model’s arrival, and the first job is to understand the nature of that change.

This energy is significant for what it reveals about the pre-release partnership dynamic. These are not passive testers running a checklist. They are active explorers trying to understand what the new model can and cannot do — and their findings feed directly back into what actually ships. When they discover that something works better or worse than expected, that information shapes the model before it reaches the broader market. The loop between testing partner and model team is tight and consequential.

2. Automated Evals First: The Standard Operating Procedure

When partners receive a new model, the very first action is consistent: start the automated evals. “The very first thing is we will start automated evals just so that they start running in the background.” The evals run in the background because they take time, and the team wants results as quickly as possible while simultaneously doing manual exploration.

Automated evals are the foundation of systematic model assessment because they produce comparable, reproducible results across model versions. If your eval suite runs on the previous model version, you know your baseline. When you run the same suite on the new model, any improvement or regression is immediately visible and attributable. A partner describes seeing a testing agent success rate increase by approximately 20% from one model to the next — a concrete, measurable improvement that did not require subjective judgment to identify.

The evals also reveal something more forward-looking: failing evals are a signal of what the next model generation will improve. “Things that don’t work today are the best sign for, here’s what the next models are going to be way better at.” This reframes failure in evaluation as information rather than disappointment. A persistent failure mode that suddenly starts working with a new model is a confirmation that something important has changed. “Seeing evals that have never worked start working and then start working consistently, this model is going to be something special.”

3. Capability Jumps: When Evals Leap Forward

The qualitative description of experiencing a major model capability improvement is striking in its specificity. One partner describes a transition: “It went from this agent can sometimes answer questions, sometimes get stuck, to, oh, my God, it is answering every question quickly and accurately.” This is not a gradual improvement. It is a phase transition — a qualitative change in the character of what the agent can do.

These capability jumps are what distinguish major model releases from incremental updates. A 5% improvement on a benchmark is meaningful but continuous. A jump from “sometimes works” to “reliably works” changes what you can actually build on top of the model. The products that become possible are different. The use cases that are worth pursuing shift. A partner building an agentic product on top of Claude experiences this as their product suddenly becoming much more valuable — the same code, the same architecture, the same system prompt — because the underlying capability has changed.

This is the practical meaning of “the grounding has changed” when a new model arrives. It is not just that the model scores better on standardized benchmarks. It is that the set of problems you would be willing to attempt with an agent has expanded. Tasks that were previously not worth pursuing because the failure rate was too high become viable. Use cases that seemed futuristic become deliverable now.

4. What It Feels Like to Build at the Frontier

The partners asked to characterize building at the frontier in a single word or phrase produce a revealing range: “Dazzling — it can be blinding at times. Just the brightness, opportunity, excitement.” “Compounding — we get the latest tools, which leads to our customers getting a better product, which leads to us getting better products.” “You have a big wave under you that is changing the way your user is working and changing the way you are working. And you have to keep your balance. And you know there are bigger waves coming.”

These are not marketing phrases. They reflect a genuine epistemic state: excitement at what is possible, responsibility for getting it right, and a strong awareness that the current state is not the final state. The wave metaphor is particularly apt: a surfer on a big wave is experiencing something exhilarating and requires skill and focus to ride, but they are not controlling the wave — they are responding to it. And they know the ocean has bigger waves.

The relationship with Anthropic that partners describe is notably close: “It feels like I have a conversation with you almost every other day. The engineers on the team, I feel like, are almost on the same team. It’s less like we’re just buying something from you, and more like we build with you.” This is consistent across the various partner companies that appear in this series — the partnership model is collaborative and tight, not transactional. That closeness is part of what makes pre-release testing valuable: the feedback flows easily in both directions because the trust is established.

🧠 Check your understanding

5 questions · your answers are saved in this browser only

1. 1. What is the first thing partner companies do when they receive a new Claude model?

   Schedule a meeting with Anthropic to review the release notes Start automated evals running in the background immediately Write a new system prompt to take advantage of new capabilities Compare the model against competitor models on public benchmarks

   Partners consistently describe running automated evals as the immediate first step — 'just so that they start running in the background' — while simultaneously doing manual exploration to understand the new model.

2. 2. How do partners reframe consistently failing evals?

   As bugs in their test suite that need to be fixed As capabilities the model will never achieve and should be removed As the best signal for what the next model generation will significantly improve As areas where human judgment will always outperform AI

   One partner explicitly says 'things that don't work today are the best sign for, here's what the next models are going to be way better at' — reframing failure as forward-looking signal rather than current limitation.

3. 3. What does a partner mean when they describe a model capability jump as going from 'sometimes answers questions' to 'answering every question quickly and accurately'?

   The model's response speed increased significantly A qualitative phase transition occurred — not gradual improvement but a change in the character of what the agent can reliably do The model gained access to real-time information The model's context window expanded to handle more questions simultaneously

   This describes a qualitative capability jump — a transition from unreliable to reliable performance — which changes what products are worth building and what use cases become viable. It is distinct from gradual benchmark improvement.

4. 4. How do partners characterize the nature of their relationship with Anthropic?

   A standard enterprise software vendor relationship with SLA guarantees Less like buying something and more like building with them — close, collaborative, almost on the same team A competitive dynamic where both parties are building similar products A contractual research relationship with formal deliverable milestones

   Partners consistently describe the relationship as collaborative rather than transactional: 'It's less like we're just buying something from you, and more like we build with you.' This closeness enables fast feedback loops during pre-release testing.

5. 5. What does the 'compounding' description of frontier AI development mean?

   AI capabilities are growing at a compound annual growth rate Interest compounds over time as AI products generate more revenue Better models enable better products, which attract better customers, which inform even better models — a self-reinforcing cycle Technical debt compounds as AI systems become more complex

   'We get the latest tools, which leads to our customers getting a better product, which leads to us getting better products.' This describes a positive feedback loop where each improvement compounds into the next.

Reset quiz

🛠️ Build it yourself

Follow these exact steps to reproduce it yourself · estimated time: ~45 min

Prerequisites

• Access to Claude API
• An existing agent or workflow you can test against multiple configurations

Step 1 — Design three “pipe dream” evals

Identify three tasks that your agent currently fails at consistently — things you would love it to be able to do but that currently produce unsatisfactory results. Write them as formal test cases with clear pass/fail criteria. These are your forward-looking evals.

Step 2 — Establish your baseline

Run your current agent on a full eval suite (including the three failing pipe-dream cases) and record every score. This is your baseline. Note the date and model version.

Step 3 — Simulate a model upgrade

When a new Claude model becomes available, run the same eval suite on the new model without changing anything else in your configuration. Record the scores. What changed? Did any of your “pipe dream” evals start passing?

Step 4 — Identify the phase transitions

Compare the two result sets and look for qualitative changes — cases where the model went from “sometimes correct” to “consistently correct.” These are your signal that something fundamental improved. Focus your product development attention here.

Step 5 — Feed insights back

Write a brief summary of what improved, what regressed, and what still does not work. Share this with your model provider if possible. This is the feedback loop that shapes future model development — your observations have value beyond your own product.

Step 6 — Update your roadmap

Based on the pattern of what improved with the new model, make a prediction about what the next model generation will significantly improve. Adjust your product roadmap to anticipate those improvements rather than waiting to discover them after launch.