Coastal Media Brand

Continuous optimization is a process in product management that should not be overlooked. Acting as a feedback loop, it’s a way to continue to test and improve your product based on what you learn.

In this article, we’ll learn about continuous optimization and how it’s essential for maximizing product success.


Table of contents


What is continuous optimization?

Continuous optimization is the practice of analyzing your experiments, assessing how they perform, and identifying what to do next. This often leads to iterations of experiments. Continuous optimization allows you to maximize value as the final part of a well-performing experimentation process.

The goal of continuous optimization is to assess the impact of the tests you ran and create capacity to keep working through the hypothesis. This involves running multiple experiments, one after the other, to maximize the learnings and results you see from one hypothesis.

Continuous optimization is the end step in a product experimentation process — once you have worked through each experiment, you are feeding all of your learnings back to the earlier stages in the process to help you to establish new insights, hypotheses, and tactics to really leverage your success.

Why is continuous optimization important?

Continuous optimization allows you to take real learnings — from the large sample size that you have experimented with — and analyze the behavior. Making tweaks to leverage the gains on a successful hypothesis can lead to further positive results, and learning from what didn’t work and tweaking your approach is important too.

When I worked at Booking.com, we experimented with everything. We didn’t just experiment once, but a successful experiment was far more likely to be the fourth or fifth iteration. Luckily for us, experimentation helped us to learn to get there.

Why don’t all teams use continuous optimization?

Continuous optimization is a step that many organizations leave out — usually due to friction points. Examples of friction points include if the lifecycle from idea to live product is too long or if you don’t have the right mechanisms for easy reporting.

However, a ton of value lies in this step of the process. This means that companies that do implement this step are far more likely to see successful results ahead of competitors.

Far too often, teams operate in this way:

How Most Teams Operate Graphic

They only generate one solution for each of their hypotheses.

They test this one tactic and hope that it works. If this tactic doesn’t work, they have probably already moved on to the next piece of “planned” work. They don’t have the capacity to revisit or change their approach.

In this case, they’re not really getting or maximizing the benefits of following a testing process. They don’t have the ability to complete the loop and learn from this:

Continuous Optimization Complete LoopContinuous Optimization Complete Loop

There are a few common reasons why teams don’t complete the loop. In essence, it’s because it does take more effort:

  • The time from idea to live takes too long, and so the team have already moved on to other work
  • There’s no easy access to data to analyze experiments in detail
  • There’s no capacity built in to analyze experiments and assess how to optimize going forward
  • They are more focussed on outputs than outcomes. They’re more focused on doing things rather than doing the right thing

How to effectively implement continuous optimization

The key to effectively implementing continuous optimization is to have multiple tactics to test for a hypothesis and that you are able to analyze the data to assess what to do.

The approach should look more like this:

How To Effectively Implement Continuous OptimizationHow To Effectively Implement Continuous Optimization

Teams should have multiple hypotheses to test to achieve their objective and multiple tactics identified that could achieve the hypothesis.

In this set up, you might:

  • Test Tactic 1
  • Receive an inconclusive result
  • Look into the analysis and realize that the tactic provided some benefits and isn’t completely invalidating the hypothesis

In this case, it’s likely that the hypothesis could still be true — but you either need to try testing one of the other tactics or iterate on what you’ve learned to optimize Tactic 1.

How to decide whether to iterate on a tactic or test a new tactic

The reasons that you might want to iterate on Tactic 1 as opposed to moving to your next tactic would be:

  • If you noticed a bug in your experiment process that impacted the test — you can fix and test again
  • If the tactic performed well in some markets but might have caused harm for others. You can identify why and iterate
  • If it looked like the tactic had some positive impact, such as increased page progression, but had a negative impact elsewhere. You might have ideas that could mitigate this

If none of those are true but you still believe that your hypothesis could be true, then you might want to try experimenting with another tactic.

Key signs to look out for when analyzing your experiment

Was the experiment positive, negative, or insignificant?

Positive impact

If the experiment is positive, then great! Switch it on and analyze how you can extend this behavior. Can you take the concept to other areas of the site? Can you dial up this concept and interactions even more?

Negative impact

If an experiment has less than 10 percent confidence, it is usually pretty safe to say it is having a negative impact. This is an amazing learning, as you know that this area is something that impacts customer behavior.

You could then try a different tactic to see if the failure of the experiment was due to the tactic being poorly executed or if the hypothesis could drive positive behavior itself.

If you can’t get this to succeed, then you could conclude that your hypothesis was invalid but look to explore alternative hypotheses. This area of your product is still important!

Insignificant impact

If the experiment is insignificant, analyze whether there were any key changes to customer behavior or if you perhaps harmed the customer experience further down the line. If you can see some positive signs of hope, then I would suggest tweaking your variant or trying a new one based on what you learnt.

How to make continuous optimization easy

The key to achieving all of this with ease is that your tactics aren’t too big. You need to be trying to experiment with the smallest deliverable that will allow you to test your hypothesis. Then you will know whether to invest more time in this space.

To make it easier to carry out continuous optimization, you want to make sure you’re set up for success. This includes:

  • Testing small deliverables to reduce your time to live
  • Have experiment reports set up that are easy to access and make it easy to analyze results
  • Have a flexible roadmap that has time baked in for you to continuously optimize — ideally by using a hypothesis-led roadmap instead of a feature-led roadmap

Conclusion

Continuous optimization really can help you to drive higher experimentation success, resulting in more value delivered. For any business that wants to have an experimentation culture and really test and learn, then it is an essential part of the process.

There are a few key principles to get right to allow you to achieve continuous optimization with ease, such as testing small deliverables and having your data set up to make it easy to analyze experiments.

Once you have these set up, then you will easily be able to keep learning and growing from your experimentation practice and see the results that you desire.

Featured image source: IconScout

LogRocket generates product insights that lead to meaningful action

LogRocket identifies friction points in the user experience so you can make informed decisions about product and design changes that must happen to hit your goals.

With LogRocket, you can understand the scope of the issues affecting your product and prioritize the changes that need to be made. LogRocket simplifies workflows by allowing Engineering and Design teams to work from the same data as you, eliminating any confusion about what needs to be done.

Get your teams on the same page — try today.

Evie Brockwell
Coach & consultant. Working with teams worldwide to make them really really good at product.

Coastal Media Brand

© 2024 Coastal Media Brand. All rights Reserved.