Something increasingly top of mind for me now that I’m focused on product growth, and seems obvious, but I’ve seen at every company I’ve worked at: Good experiment design – and documentation of that design – is critical.

An experiment can be made or broken by the incredibly simple but sneakily nuanced decision of when a user enters the test.

Let’s take a simple example: You want to test whether users are more likely to click on one of two variations of a CTA on some page in your product. But the user needs to get to that page first.

If a user enters the test upon logging in, your test will probably result in a sample ratio mismatch. Not everyone who logs in sees the page in question, but those users who don’t will probably register as a “non-conversion.” and thus throw off your test results.

Can’t trust those results. You want users to only enter the test upon viewing the page with the CTA in question.

This may be obvious upon reading this, but it might not be obvious to the person tasked with implementing the test design (whether that’s an engineer, another teammate using a 3rd party testing tool, or even yourself in the moment).

So document it clearly, alongside everything that’s critical for the test design. It’ll only help!

Here’s a tangentially related gif that I thought was kinda funny: