What to do when experiments fail

Hi there 👋,

Let me paint the scene for you.

I’m sitting with a new client, discussing the possibilities for their brand, and then the inevitable happens…

Client: Meta ads don’t work for us.

Daphne: Okay… why do you think this?

Daphne waits for an explanation and their experience in testing various approaches. She wonders if they feel it isn’t a relevant channel for their business model.

Maybe their cost of acquisition is too high? Maybe their customers aren’t on those platforms? Maybe it’s not suitable for their products?

Client: We tested it once. It didn’t work. It’s too expensive.

Daphne: *screams internally*

I’ve seen this happen too often.

Brands ran one campaign that was too expensive and drew the conclusion that this channel was not to be used again.

They miss out on countless opportunities because of one bad campaign.

To kill or not to kill

Before I end up on an FBI watchlist, let me clarify that I’m talking about experiments.

I am all for killing things that don’t work (only experiments, agents!).

If anything, I’ve found that companies don’t do that enough. But sometimes, we’re too quick to assume an experiment is simply a failure.

It’s tempting to only focus on wins, but there is much potential in failed experiments. 

Some failures are actually future wins, once you implement your learnings.

To be honest I hate the word failure, I’d rather call those experiments learnings.

Some of my biggest wins - I’m talking huge value proposition improvements or landing pages - were total flops to begin with.

Why do we move on so quickly from our ‘failures’?

1. In growth, we follow the data. If the data says no, we move on.

2. Stakeholder pressure. It SUCKS to talk about failures. I remember the stress of explaining a loss to stakeholders. It’s hard to convince them to let you continue something that hasn’t worked.

So when do we stick with a failure? And how do we convince people that it isn’t a loss just yet?

Should I stay or should I go?

This decision stems from knowing with confidence what did and didn’t work. Dive deeper than the label of ‘failure’.

Maybe you didn’t define a clear hypothesis or choose an accurate metric for success.

To determine whether the experiment could be improved, create 2-3 theories of what didn’t work.

If we circle back to the case of the Meta ads, imagine that your results show a major issue is a low CTR leading to a high CPC.

These might be the problem areas:

• The copy didn’t resonate as well as we expected

• The creatives didn’t capture their attention

• The targeting was off

There could have been other reasons, but let’s start with these.

Choose the likeliest option and design a follow-up test. And slowly, rule out any other causes for the failure.

Imagine if all you had to do was rewrite the copy differently, perhaps add a CTA or shorten it, and you’d see a huge increase in CTR. Imagine if you gave up too early to see this success.

How do I get other people on board?

My clients claimed meta ads didn’t work for them. They had evidence of this, at least in their eyes.

So how could I combat this and get approval?

1. Manage expectations upfront - highlight that this is a normal part of experimenting and multiple tests are needed to be confident.

2. Get buy-in for a focus area - share insights on why you’re focusing on this area.

3. Learn to speak their language - don’t rely on your own language of data, but translate it for them.

Want more tips for convincing stakeholders? Check out the complete article, which contains 5 more tips!

Recommendation

In every edition of Growth Waves, I also share a related book, individual or newsletter to check out related to the week's topic.

I once spoke to Ethan Garr, co-host of the Breakout Growth Podcast, about the importance of getting pre-mortems before running experiments.

We both felt that we often run experiments that we can’t draw any conclusions from simply because we ran the experiment wrong.

Ethan had the idea of writing your end report before you run the experiment - like ending your novel before you’ve even started it.

Check out Ethan’s article about it here.

I love this as it prevents pointless experiments, saving you time, money and grey hairs.

Some failed experiments are experiments that never should have been run in the first place.

How you run experiments matters, it is the whole reason I created a course on how to run and track experiments better.

I could talk endlessly about this subject, and that’s why I’ve expanded on it in this article and will also be speaking on this topic at How to Web 2023.

They say you regret the things you don’t do more than the things you do.

So don’t not run an experiment for fear of failure, see it as a learning.

That doesn’t mean you should stay and repeat the exact same thing, but that you should try to always understand the root cause.

Daphne