Anyone got any good strategies for A/B testing with the Play Framework?

Question

I'm interested in good strategies for A/B or split testing with the Play Framework.

The obvious choice would be to use Google Website Optimizer, however I'd prefer not to for several reasons:

• Only allows you to test stuff in the presentation layer
• Difficult to test based on things like revenue (rather than binary events like clicks or conversions)
• Have to mess up templates with Javascript code
• Tests must be discrete, whereas I like to be constantly testing new variations

Any ideas?

Answer 1

If you want the controller to decide which version to use, you can do this (in Scala at least) by selecting between two different templates, like this:

if (mode == "A")
  html.showA(product)
else
  html.showB(product)

Doing this in the controller has the advantage of being able to track which user received what version of the page, and thus showing him the same page upon each subsequent log in.

Edit: Forgot to mention you then need two different templates, showA.scala.html and showB.scala.html in this case.

Answer 2

although I think Google Website Optimizer would be a good choice, I assume you could build your own system via:

• A @Before method which generates a random value for the session (a test seed let's say) or a value based on a parameter you decide (revenue, etc)
• Some logic around the value of the seed in the business layer
• Tags for the UI part, in which the tag shows some text or another (or image, or...) depending on the value of the seed
• A table in the database to store information linked to a session (to trace the movements of the user as he comes into your site)
• Some reporting over the data in the table (I can only recommend Jasper or Birt, haven't work with others)

But it will be quite a lot of work.

Honestly I truly believe using existing systems (Google Website) is a better idea. Think that all the time you spend setting your environment for your custom A/B testing is time you are not developing your product. And although A/B is good to increase conversion, you will need many iterations over it to find the best choice, so the most time you spend on other things, the more money you are potentially loosing.

In fact, from your complains on Google's tool:

• I have to say that the main point of A/B testing is the presentation layer, as that's what the user sees and what drives their actions. They don't care if your implementation of a merge sort is nicer, they care that your logo is cute.
• Whatever you do, you will clutter your templates. That or you will have to create many "clones" of a page, one for each version you want to show. Either way, it's not "nice" code.
• You need to runt he tests for a while to have a valid number of samples before being able to decide something. That means you can't change the options every day (unless your site has thousands of users daily) or you won't have statistically significant data

So from your 4 points, I believe that 3 are debatable. I would advise to first use some existing tool to increase the performance. Once you have the best conversion possible then you can think on doing some extra custom work based on other parameters.

---

Answer 3

1. firebase remote config (it is free) as AB test options provider. it allows you to define A/B... groups in the percentage
2. in your project create class to get config values and provide interface // for experiment 1 fun isButtonColorBlue(): Boolean fun textButton(): String return values depends from your Remote Config
3. one advice to implement A/B variants in your code. To keep code stability and do not affect unit tests (you probably wrote), create the child class for the class where you are going to make changes, and override the method you are going to change according to the AB test. Then replace definition in your code to use new class version. In that case, all tests for the old class are in safe.
4. To get AB test result use any Analytics SDK. When you get remote config and define the group for user you need to update user info (for Google mFirebaseAnalytics.setUserProperty("experiment_button", "A"))