4 years ago from Bappy Golder, UX Developer & Tech Lead
We know we can easily do A/B testing once designs are coded and they are live as a website or an app.
I'm wondering if there is a tool that allows designers to do A/B testing before a design is live. Something that potentially allows also to test non web designs (i.e: a/b test letter heads or pamphlets etc.)
Create two pictures and put them into a google form with two radio buttons. Send this link to your test candidates and done.
But what are you actually testing? Opinions on design or usability?
That depends on what I am testing. Sometime it is a matter of testing opinion if you are comparing say 20 different design of the same logo.
Other times you might want comments on usibility of 2 different version of a same app.
That'll test which ones people like, not which ones are effective.
Hey thanks Ole, I tried doing this with Typeform. But doing it with form presents some problems like image zooming.
Comments on images etc.
Depending on what feedback you're looking for, A/B testing might not be the best tool.
A vital concept of A/B testing that often gets overlooked is that you're going for statistical significance. Without statistical significance results are often little better than guessing. The catch is that it often requires a larger sample size depending on the variables involved, which is why they are commonly conducted with a live audience. There's a great overview here.
But sometimes it's incredibly helpful to just get a quick read on preferences, potential trouble spots, what stands out to visitors, etc. A couple of tools I've found really useful for that in the past are:
Hey Sam thanks for sharing the links of Zurb & Usibility Hub. They both look like great resources. I think what I'm after is more like the Preference Test option provided by Usibility Hub.
Yes, there is. I’ve been using a tool called Chalkmark that lets you upload a bunch of static images to test “first clicks”. It’s been such a dream using it.
It’s kind of like a usability test for mockups. You can give users a series of screens accompanied by a task, and then they have to click on wherever they feel helps them achieve that task.
For example, if you want to test if users know where to click to checkout on an online shop, can have a task like: “You just added a pair of headphones to your cart and want to checkout. Where would you go to do so?”. You can set your cart icon to a “correct area”, and Chalkmark will record if users clicked on it or not, and give you a % score.
To achieve an A/B test on a static design, simply upload the A version and B version(s) to Chalkmark as separate tests. Chalkmark will give you an option to recruit users from their panel (very cheap, compared to usertesting.com), or you can simply use the provided link and share it with your own contacts.
To automate the process further, you can use Ethnio to intercept users on your site and send them randomly to different tests. You should also be able to build some kind of workflow or pipeline around the data that Chalkmark gives you since it’s a simple .xlsx file.
Hey Andric, thanks for the talking about Chalkmark. I'm keen to try Chalkmark out. It seems kind of pricy for one designer to get started though.
But it definitely looks like something I'm looking for.
I have long wanted to build a much simpler tool for this. I feel like the examples already posted are either too feature crazy, or with the usability hub, too simple. Another feature I would like for this, is allowing the user to view multiple versions of the same design, so they can see how it evolved over design iterations.
Hey Billy, I totally agree with you. In fact this is one of the reasons I posted this question in the first place.
I would like to build something for my team's internal use. Something that is not too feature heavy and possibly free for designers.
Would you be interested to collaborate with me? Even if you do some early beta testing that might be helpful.
I would absolutely love to! I have been talking with my design team about creating wires and designs for this tool. Hit me up at billy [at] threadless dot com! It would be so fun if we could work something out.
I run a team of 5, and for the amount of traffic we get and the type of product we have, we've gone with VWO.com.
As was stated previously, the most important part of any A/B testing of variations is the data. Make sure you are eliminating as many variables against the most robust data available. It's just the scientific method:
Thanks heaps Paul VWO looks like a great resource.
I use UsabilityHub and get good feedback from each user tested.
usabilityhub.com
Upload two designs and ask community which they prefer
If you want to try A/B testing on the flow - you can try http://CanvasFlip.com . Records user videos, heatmap & even shows how conversions will look like..
Hope it helps.
[Disclaimer: I am associated with CanvasFlip.]
Hey Bappy, what you're looking for is not really an A/B Test but a survey – check out Google Consumer Surveys, they are great for that: https://www.google.com/analytics/surveys/
Thanks Johannes
Login to Comment
You'll need to log in before you can leave a comment.
LoginRegister Today
New accounts can leave comments immediately, and gain full permissions after one week.
Register now