Is this an experiment or an optimisation?

Hi there 👋,

Today, we’re going right back to the beginning to lay the foundation correctly. 

Many of us think we know what an experiment is. A client will tell me that they perform A/B test subject lines for emails.

But there is no concrete difference between the two subject lines - just alternative phrasing - and they don’t use the findings to change anything.

This is not an experiment. This is just two different subject lines.

You can’t expect to grow your company if you’re not experimenting and optimising, especially if you don’t even recognise the difference between those two things.

What’s an experiment?

Experiments are a tool to measure and learn what performs better.

We are making a change and need to know what works best for it.

Particularly for startups, this might be a big change, and you have to be selective in those.

There is a risk to the change, no matter how confident we are.

Examples of experiments

Here are some examples of experiments:

1. A new email flow

2. A new ad campaign

3. Adding a new block on a landing page

4. A new pricing model

5. Targeting a different audience

6. A different value proposition

What is an optimisation?

But what are those other changes then, if not experiments?

Often these are optimisations. This is the process of adapting what you have to be more effective or get new results.

What do optimisations look like?

• Changing a targeting setting in a running Meta ads campaign, e.g. excluding a certain audience or broadening the age range

• Trying a different type of subject line in the weekly newsletter

• Adding negative keywords to improve a Google Ads Search Campaign

• Shortening some copy on the website

• Added a new FAQ question to the website

This doesn’t mean we don’t measure the performance of those areas, but rather that we don’t track it in our weekly experiments sheet and don’t define a hypothesis or measure of success.

We don’t compare it to a benchmark or a variant to decide if we should continue with that change.

When should you choose an optimisation or an experiment?

Let’s give it a go with that first example: changing a targeting setting in a running Meta ads campaign.

Scenario 1: 

This is a huge campaign with a £20,000 monthly ad budget, and improving ad performance is the key focus of the quarter.

So you would classify the change as an experiment, monitoring the performance the week before and after the change, or using the Meta ads experiment feature to run two variants.

Scenario 2: 

This is a small campaign with a £600 monthly ad budget. It is running fine, and your focus right now is on retention.

You’ve noticed some overlap with another campaign, so you’ve decided to classify it as an optimisation and just change it.

Questions to ask yourself

This might still seem like definition picking, but this is one of the challenging aspects of running experiments.

If you aren’t sure whether it should be an experiment or optimisation, ask yourself the following questions:

1. Is this a key focus area for this quarter?

2. Is there a big risk or reward involved in this change?

3. Can we measure the impact of this change?

If the answer is yes to all three of these questions, you’ll want to define it as an experiment.

If not, it’s an optimisation.

Both of these things will help your brand grow, but knowing the difference allows you to correctly allocate the amount of time and budget that will go towards it.

In a startup, both of those things are extremely valuable. So always take a moment to consider what you’re working on, and how it fits into your goals.

Recommendation

In every edition of Growth Waves, I also share a related book, individual or newsletter to check out related to the week's topic.

Would you like to learn more about experiment and optimisations?

Check out my course on Growth Experiment Tracking, where we cover the full experiment process and how to create a bulletproof plan for growth.

There’s a feature to ask questions throughout, so you can get personalised advice for your brand.

The course is split into a warm-up and 5 different stages - you can get all this content, plus access for your team members, for just £145!

The difference between experiments and optimisations can seem like a tom-ay-to tomato distinction.

I agree, the line is blurry between optimisations and experiments. 

But their slight differences are still important, and differentiating them allows you to keep focused and on track to growth.

So you say tomato, I’ll say tom-ay-to, and we’ll both know the difference between experiments and optimisations.

Daphne