"i only speak liquid" #78: Developing for A/B Tests - Quick, Ugly, Out the Door

Hey everyone,

This is Elisa’s 2nd edit of “i_only_speak_liquid”:

Elisa is a 14-year e-commerce vet who’s worked with a bunch of DTC brands as a freelancer, full-time employee, and agency dev.

She’s passionate about taking old things and making them better, so she loves working with brands with chaotic old codebases to build new features and improve site speed.

Ofc: She’s an expert on Storetasker 😉 apply here.

Let’s dive in 🤿

What I’ve been thinking about:

Developing for A/B Tests - Quick, Ugly, Out the Door

I recently had another developer ask about how I code features for A/B tests like it was some kind of forbidden knowledge. 

It’s not.

Let’s get into it.

First (and most obvious): if you can build it in the tool, build it there. Usually these tests involve hiding elements, swapping out text and images, changing CSS.

But what if you need to make bigger changes to functionality?

Let’s say our client wants to increase subscriptions, so they want to test splitting the CTA for subscriptions and one-time purchases into two separate buttons:

That’s probably going to take some Liquid, some CSS, and - depending on the theme - some changes to core JS functionality (adding to cart). We could probably figure this out entirely in our client’s A/B testing tool of choice, but it’s going to be a lot quicker and cleaner to build if we make changes to the code base instead.

Because most A/B tests fail, we should approach the actual work a little differently:

Focus on the MVP

If the test loses, any extra time you spent writing production-quality code, hooking up theme settings, getting those sleek animations perfect? Wasted. 

Code A/B tests like a maniac — hardcode that text, inline that CSS, add your script tag right there in the Liquid. It’s ugly, it’s messy, it’d be unsustainable if this was production code. We all know better, but this is an A/B test. It’s not about perfection. It’s about figuring out if this thing is even worth exploring before sinking in the time to go from “good enough” to “perfect”.

Separate your Code

If the test loses, we’re going to remove it; if the test wins, we’re going to refactor it. Make it easy to find. I like to keep any code for an A/B test in separate snippets prefixed with a key unique to that test.

Hide it by Default

When the page loads, the code for your feature should be on there, hidden. Then, when you build the actual test variation in your client’s A/B testing tool, simply show the feature using CSS.

That’s it! Hopefully this is helpful to you as you build out those A/B tests!

3 links you can’t miss:

A few Shopify A/B testing tools that my clients love:

https://visually.io/

📌 Visually.io

📌 Intelligems

& if you’d like to learn more about A/B testing: Check this out:

📌 https://www.nngroup.com/articles/ab-testing/

1 app I like:

Shoplift is great for smaller clients that have the traffic and want to start testing ASAP, though because it’s testing one theme against another, it can be challenging to manage if you’re actively making updates to the site.

One learning as a freelancer:

Always come back to what your client is actually trying to solve for.

For example, sometimes the folks I’m working with want redesigns launched as A/B tests - think new templates like a product page, collection, home page or new features like an inline cart or nav menu.

With these types of large-scale changes, “get it out the door” may not be the right approach. It’s best to have a conversation with the client to understand where (if anywhere) to cut time.

Are they going to launch this feature whether or not it wins the test? How do they feel about you spending more time to polish before running the test — or spending some extra time to polish the feature after (if it wins)? And why are they running this as a test - are they unsure about the design? Do they want to prove its value to someone higher up who wants numbers?

All relevant to the decisions you’re making at the top of the project - and none of them answers that come out unless you ask the right questions.

-Elisa