The most useful word in Lenny's newsletter this week was not the name of either AI model. It was "blind."
Lenny Rachitsky ran a blind taste test between two frontier models, Opus 5.5 and GPT-6 Sol. Same tasks, outputs judged without knowing which model produced which. Whatever you think of the winner, the method is the part worth stealing.
Because growth teams almost never judge anything blind.
We grade with the names on
Think about how a normal week of decisions actually gets made.
We review creative knowing which agency made it and which one the CMO likes. We read experiment results knowing which variant was our idea. We compare channels knowing which one we fought to get budget for last quarter. We assess a new tool knowing who championed buying it.
Then we call the outcome data-driven.
The data is real. The numbers are not fabricated. But the reading of those numbers happens in a head that already knows which answer it would prefer, and that head is remarkably good at finding the interpretation it wants. Not through dishonesty. Through the ordinary, constant pressure of already caring about the result.
Blind evaluation exists in medicine, in wine tasting, in hiring for orchestras, for one reason: the people involved discovered that knowing the source changes the verdict, even when everyone is trying hard to be fair. There is no reason to think growth teams are immune to something that affects doctors and sommeliers.
I am as guilty of this as anyone
I have caught myself reading a flat test result as "promising" because I designed the treatment, and reading the same shape of result on a colleague's test as "inconclusive."
Same curve. Same confidence interval. Different word. The only variable that changed was whose idea it was.
What makes this so hard to catch is that both readings felt reasonable at the time. "Promising" is a defensible description of a flat result with a slight upward lean. So is "inconclusive." Neither is a lie. The bias lives in which defensible word you reach for, and you reach for it before you have noticed you are choosing.
Once I started looking for it, I saw it everywhere, including in the rooms I was running:
- The variant the team was excited about gets a second week "to be sure." The one nobody liked gets called on day five.
- The channel someone argued for gets judged on a generous attribution window. The one they argued against gets judged on a strict one.
- The creative from the expensive agency gets a "let's give it more spend before we decide." The in-house one gets switched off.
Nobody in any of those moments thinks they are being unfair. That is precisely the problem.
The bias isn't in the data. It's in which defensible word you reach for, and you reach for it before you notice you're choosing.
A few cheap ways to take the names off
None of this needs a new tool. It needs a handful of habits that remove the information your judgement should not be using.
Decide the metric and the kill threshold before launch, in writing
Most bad calls happen after someone sees the numbers and picks the metric that flatters them.
Every experiment produces several metrics. Conversion went flat, but average order value ticked up. Retention dipped, but engagement in week one rose. If you decide what counts as success after the results are in, you will almost always find a metric that tells the story you wanted. That is not analysis. It is shopping.
Write down, before launch, the one metric that decides the test and the result at which you would kill it. Put it somewhere the whole team can see. Then hold yourself to it when the numbers come back, especially when you do not like them.
Relabel the variants in the readout
Call them A and B. Randomise which is which. The person presenting the results should not be able to tell the room which one is the "new" version.
This feels almost silly to do, and that is part of why it works. The moment a team knows which variant is the challenger, a story starts forming around it. "New" carries hope. "Control" carries the weight of the status quo. Strip the labels, look at the numbers, decide. Then reveal.
Review creative with the source hidden
When you compare creative, remove the agency name, the designer, and any hint of who championed which concept. Judge the work as work.
You will be surprised how often the in-house option wins once nobody knows it was in-house. You may also be surprised in the other direction. Either way, you learn something you could not have learned with the names attached.
Have someone who did not build the test write the conclusion
The person who designed an experiment is the worst-placed person to summarise it. They know what they hoped for, and the summary will lean toward it, however carefully they write.
Hand the one-line conclusion to someone with no stake in the result. Ideally someone who would be mildly happy to see it fail. Their summary will be blunter, and it will be closer to the truth.
What this actually costs
On paper, blind evaluation costs almost nothing. A rule written in a document. A relabelled slide. A colleague's ten minutes.
In practice, it costs something much harder: the willingness to find out that your favourite idea lost.
That is the real reason most teams do not grade blind. Not because the method is complicated, but because the method works. It removes the cushion that lets a team keep believing in the thing it already decided to believe in. A team that grades blind will kill more of its own darlings, and some of those darlings belong to senior people.
I have come to think this is one of the clearest tests of whether a growth team is actually experimental, or just runs experiments. Running experiments is a process. Being experimental is a willingness to be wrong in public about things you care about. The first is easy to set up. The second is a culture, and you can see which one a team has by how it reads a result it did not want.
The model makers are grading blind
There is something slightly funny about this moment. The companies building the most sophisticated systems in technology are having them evaluated blind, by people who deliberately strip away the brand to see the work.
Meanwhile, in most growth orgs, the readout still opens with the name of whoever had the idea.
The bar here is not high. You do not need a research team or an evaluation framework. You need to decide what winning looks like before you look, hide the labels while you judge, and let someone without a horse in the race write down what happened.
The model makers are grading their products blind.
Your experiment readouts can manage the same.