Some quick notes after reading A field guide to usability testing and re-reading the Smashing Book #1:

1. A/B testing

– always test both versions simultaneously
– wait for it 🙂 (use a calculator to determine when to end it, and don’t give up earlier)
– keep the A/B tests for new visitors only (don’t surprise the regulars)
– but make sure that a new visitor gets the same version on consecutive visits
– be consistent: make sure that the variation appears on all pages (ex – if you have a promotional price on version A, make sure that the user will always see the promotional price on all the pages)
– the results might be un-intuitive
– naturally, the higher the number of users, the more reliable the result
More

Who would be involved in an A/B test:
– the UI/UX team – to propose the 2 versions and analyze the metric results
– the dev team – to implement the metric, manage sessions and make changes consistent across all the interfaces
– the network team – to handle various types of redirects (ex – run the A/B test only for users in a given geographical area, or only users on mobile)

2. MVT (multi variate testing)

– it needs a lot of traffic and time
– keep the number of combinations to 25 or less and make sure you preview them all
– global vs local optimum (A/B vs MVT)
– if you don’t have the traffic and cannot use full factorial testing, you can still use partial factorial testing.

3. Some usability rules/principles

(more…)

Two years ago I was posting 2 Ted talks having as central idea doing the things you believe in. I recently reviewed them and reflected more about the second – called ‘The puzzle of motivation‘ where Dan Pink makes a case on how the current types of activities we’re involved in require different types of incentives.
Dan Pink sees a mismatch between what science knows and what business does: even though there is scientific proof that creative jobs do not work well with rewards, the current business model works just like that.
There are a few exceptions, like the Google initiative to allow his employees to spend one day per week working on an independent project, Atlassian’s FedEx days or – most powerful example – how Wikipedia succeeded and Microsoft Encarta failed:

Encarta’s closing is widely attributed to competition from the much larger online encyclopedia, Wikipedia
– Wikipedia page about Encarta (!)

Dan Pink identifies 3 building blocks of an entirely new operating system for our businesses:
Autonomy – the urge to direct our own lives
Mastery – the desire to get better and better at something that matters
Purpose – the yearning to do what we do in the service of something larger than ourselves.

If we bring our motivation, notions of motivation into the 21st century, if we get past this lazy, dangerous, ideology of carrots and sticks, we can strengthen our businesses, we can solve a lot of those candle problems, and maybe, maybe — we can change the world.

More:
* What motivates us
* The Motivation Trifecta: Autonomy, Mastery, and Purpose

A/B vs Multivariate Testing

A/B testing: two versions (A and B) are compared, which are identical except for one variation that might affect a user’s behavior. Total number of variations: 2. More
Multivariate testing: multiple variables are modified for testing a hypothesis. The goal of multivariate testing is to determine which combination of variations performs the best out of all of the possible combinations. [Total # of Variations] = [# of Variations on Element A] X [# of Variations on Element B] ... More

Permutations, Arrangements, Combinations

Given a set of n elements (ex – for n=3, the set is A, B, C)
Permutations: each ordered set of n elements P(n) = n!
In our example with n=3, P(3)=3!=6: АВС, АСВ, ВАС, ВСА, САВ, СВА
Arrangements: each ordered set of k elements A(n,k) = n! / (n-k)!
In our example with n=3, ordered pairs of 2, A(3,2)=3!/(3-2)!=6: AB, BA, AC, CA, BC, CB
Combinations: each unordered set of k elements C(n,k) = n! / k! (n-k)!
In our example with n=3, un-ordered pairs of 2, C(3,2)=3!/2!*1!=3: AB, AC, BC
And the relationship between P, A, C: C=A/P
Remember that for the permutations you don’t need a k! More

About learning

Learning isn’t done to you, it’s something you do. You need to take responsibility of your education. There will always be a new technology to learn, but this is not that important. Is the constant learning that counts.
Andy Hunt – Pragmatic Thinking and Learning

We all tend to learn best by doing and teaching. Active learning is a much more effective way to learn than any other way.
It seems a bit strange, but it should really be no surprise that play is a powerful mechanism for learning. […] This simple process that comes natural to us all, but somehow gets “taught” out of us, is the simplest and purest way to learn.
John Sonmez – Soft Skills

Random links:

(more…)