6 reasons you should run an experiment first

Hi there 👋,

When I’ve been working with clients for a while, the same question starts to pop up.

It can feel like Whac-A-Mole, the moment I get it down in one place, it pops up from another angle.

That mole, or question, is, “Should this be an experiment?”

They’ve been introduced to the intoxicating world of experimenting, and they want more.

• Every issue is an experiment waiting to happen.

• Every drop in results is an experiment is an excuse for an experiment.

• Every unexplored avenue is an experiment in disguise.

They have their experiment glasses on, and they see nothing else.

So we reach a point where I have to create some boundaries on what is worth experimenting, and what isn’t.

Stop! Before you consider an experiment…

There are two requirements you need to fulfil before you can even consider experimenting, which are as follows:

1. Does experimenting justify the extra investment? For example, is there a positive ROI?

2. Is there enough traffic to test?

The first requirement focuses on whether the potential lift of experiments winning justifies the extra time and resources required to experiment.

Phrased differently: does the possible lift exceed the costs of running experiments?

The costs of running experiments are not only the tooling and costs of ads, but also the cost of you and your team’s time.

It’s easy to forget that your time is a resource as well, and unfortunately, a finite one!

The second requirement is more quantitative:

Is there enough traffic to test?

You might want to experiment, but you can’t do this without enough traffic.

There are still ways to test on low-traffic setups or alternative solutions.

For example, testing before/after, driving additional traffic via ads, and testing on micro-conversions vs end conversions.

So when should you go full-speed ahead on that experiment?

Here are six reasons why you might choose to experiment first:

1. Big investment. Changes that require a significant investment are worth testing on a smaller scale first. Try to figure out the smallest way you could test a substantial change before going full-on.

2. Low certainty of what will work. When you are not sure what will work or what is needed, experiments are a great way to find this out. It could be that an area is completely new for your organisation or that you’re still looking for consistent patterns that drive growth.

3. Big potential impact. Sometimes you have a lot of confidence in an experiment, and the potential impact is very high. In this instance, you might want to run it as an experiment. This decision is more tactical; you want to know the exact impact.

4. Trade-offs/Limited resources. When you’re very limited in certain resources, such as development time, the extra effort of experiments is worth investing in. You have to be very critical in where you do and don’t invest your time.

5. Variables. Many growth actions have variables and options. Does version 1 or version 2 of the onboarding email convert better? The way to understand which variable works better is through experimentation. It also decreases your risk.

6. Messaging. Even if you base the copy precisely on what your customers say, messaging is still tricky. You often don't know what phrasing or value proposition will work best with which target audience.

Recommendation

In every edition of Growth Waves, I also share a related book, individual or newsletter to check out related to the week's topic.

This is a topic where I struggle to find recommendations, so I went and made my own!

I had worked with countless startups on improving their experimentation process, and I kept seeing the same mistakes and getting identical questions.

So I compiled everything I know into my course, Growth Experiment Tracking which I released a few weeks ago.

The responses so far have been great, with the general sentiment being that it has made it so much easier to get into running experiments consistently and creating structure.

This course walks you through my 5 stage process to bring more focus and structure to your startup. It includes my experiment tracking Google Sheet, and the chance to ask questions throughout.

Check it out if you’re curious about improving your experimentation process!

We all go through this experiment-happy phase. When I first started in growth, I also saw potential experiments everywhere I looked.

But by following these six guidelines for experiments, you can easily separate the opportunities from the resource drainers.

They’ll be your personal Whac-A-Mole machine to answer that question once and for all.

Daphne

P.S. A bit of a crazy week as I was featured on two podcasts. I’m so proud of both episodes as I shared openly a lot of the messy and tough reality of growing startups.

The first was on the In Growth We Trust Podcast and talks about contradictory opinions about growth from the downsides to growing in public to why B Corp might not be worth it.

The second was about getting your voice & ideas heard in growth on the Delivering Value podcast. Definitely check out both and let me know what you think.