In the first few days of an A/B test, where different visitors see two versions of a page, the results tend to swing up and down radically. One version may look like a clear winner on Monday and fall behind by Friday. Stopping the test early, while one version happens to be ahead, can lead you to keep a change that doesn’t really help.
Before you start, decide how many visitors each version needs (a free sample size calculator can tell you), and let the test run until it reaches that number. A back of the napkin calculation is to wait for at least 25% of your monthly traffic to hit a page before making a decision. So if you get 1,000 visitors to that page per month, wait until at least 250 people have seen the new version. Lower-traffic sites may need several weeks per test since they don’t have enough data to accurately know what future visitors will respond to.
