Dynamic A/B testing: get to know your customers through email

by Webpower Europe in Blog

A/B Test: More knowledge about your customers

Compared to the complex dynamics you can find when advertising in social networks, or the “full funnel” approaches to e-commerce, email might feel slightly less sophisticated. Email, mostly, is regarded as a simple push channel: we send messages to our customers, and we hope they like ‘em.

Obviously, in reality most of us are not that naïve: we often look at the data that our email campaigns generate: we can see that last week’s email had more clicks (proportionally), than this weeks email. And, we can wonder why last week’s content was more appealing.

Most of us are also quite familiar with the most rudimentary next step in learning from email. Besides just looking at our data over time, we can start actively experimenting. We can do AB tests: we can send out a version of our email (we will call it “A”) to 10% of our customers, and we send out version “B” to another 10%. Next, we look at the data and send the remaining 80% of our customers the best converting email.

Eyeballing our data, and AB testing, both teach us something about our content. This is knowledge we can use in subsequent interactions. In this blog post we will explore a few novel methods to learn from email that make email as exciting a channel as the other ones. If we treat email not just as a single batch (which actually we weren’t really in the AB testing case: we made a split between the test 20% and the remainder 80%) but rather as sequential interactions with customers, we can learn much more.

Dynamic A/B testing

The basic intuition here is simple: within the AB test, during the test period, there is a ½ probability of receiving email “A”, and a ½ probability of receiving “B”. After the test, if the conversion for “A” was the highest, these probabilities change to 1 and 0 respectively. However, by stretching our sends over time, we could also change these probabilities smoothly: when we have no data the probabilities are ½ and ½, but when “A” seems to perform better (based on a small trial) we change the probabilities to (say) 2/3 and 1/3. Doing such a dynamic AB test is guaranteed to outperform a static one. Exactly how the probabilities should change is a bit tricky, but we have that covered..

The nice thing about this dynamic AB testing is not only that it outperforms your static one, it also allows for adding new options into the test all the time. So, in an automatically generated “abandoned shopping cart” email, you can add new versions all the time. You don’t do single “yes” or “no” tests, but rather you improve constantly.


Treating emails as sequential interactions with customers opens up more possibilities then dynamic AB testing. For example, how would you determine the best price of a new online service that you are introducing? You know that if the price is too high, likely not many people will purchase the service. However, if you price too low, you won’t make any money. So, where is that optimum?

Instead of paying a market research agency to conduct costly analysis of your potential customers, why not use your active email base to learn the best price? You can sequentially offer, in emails, new prices to customers. For each customer you see the response, and you compute the revenue. Effective algorithms to do so exist, and we are currently testing this method of pricing in comparison to alternatives (such as the market research agency). We don’t have this available right now, but we will in the future. Likely, we will call it dynamic pricing.


If stretching out emails over time allows you to learn from sequential interactions, image what you could learn from sequential interactions with individual customers. If your weekly newsletter targets the same recipient group all the time, you can actively learn, based on responses to these newsletters, what makes your customers tick.

Through repeated emails, we can for example learn the product preferences of your customers. What movies does she like? Which music does she like? And, does she like road biking? We can actively test these questions in emails, and build a profile of your customer.

But, we can also go beyond mere product preferences. We can learn which arguments work for which customer: does she like discounts? Or, does she like popular products? Merely by treating email as a means to sequentially learn about customers, we can build psychological profiles of individuals that allow you to target better. This, we have available for you right now, both for email as well as web.

Knowledge back to your platform

So, by thinking of email not as a “batch” and “push” channel, but rather as an interactive channel which allows you to learn about customers, email can be as exciting as the fancy social network stuff. Actually, with email, you know who you are talking to, and this allows you to gain more knowledge then is often the case in the other channels. We are always looking for more ways to learn from email.

Email is pretty cool, isn’t it?

Share this article