Some quick notes after reading A field guide to usability testing and re-reading the Smashing Book #1:

1. A/B testing

– always test both versions simultaneously
– wait for it 🙂 (use a calculator to determine when to end it, and don’t give up earlier)
– keep the A/B tests for new visitors only (don’t surprise the regulars)
– but make sure that a new visitor gets the same version on consecutive visits
– be consistent: make sure that the variation appears on all pages (ex – if you have a promotional price on version A, make sure that the user will always see the promotional price on all the pages)
– the results might be un-intuitive
– naturally, the higher the number of users, the more reliable the result
More

Who would be involved in an A/B test:
– the UI/UX team – to propose the 2 versions and analyze the metric results
– the dev team – to implement the metric, manage sessions and make changes consistent across all the interfaces
– the network team – to handle various types of redirects (ex – run the A/B test only for users in a given geographical area, or only users on mobile)

2. MVT (multi variate testing)

– it needs a lot of traffic and time
– keep the number of combinations to 25 or less and make sure you preview them all
– global vs local optimum (A/B vs MVT)
– if you don’t have the traffic and cannot use full factorial testing, you can still use partial factorial testing.

3. Some usability rules/principles

– ideally everything should be within 3 clicks, unless the navigation is really easy
– Pareto: 80% of effects comes from 20% of the causes
– Fitts law: it’s all about distances and human movement
– inverted pyramid – started from journalism (conclusion first, details later)
– users satisfice (settle with the solution that is good enough)
– users don’t read, they scan
– users ignore banners
– human’s short memory can keep between 5 and 9 items
– humans are wired for instant gratification
– humans suffer from the baby-duck syndrome (stick to the first design they learn – blue floppy disk for saving) and are biased by baby-faces
– humans can’t stand uncertainty
– stay above the fold and aim for the foveal area
– avoid minesweeping: keep your links blue and avoid changing their :visited state color
– don’t decide for the user (ex: don’t open links in new windows, don’t resize window)
– form labels work best above the field
– use well known conventions (top right search) and metaphors (shopping cart)
– keep the forms as short as possible
– if you must use multi-level dropdowns, look how Amazon did it
– users started scrolling, so don’t be afraid of long pages as long as they are easy to follow
– keep the text readable (don’t make the font size too small)
– don’t allow the users to ruin their experience (Undo function, Are you sure you want to leave? – when filling a big form)
– let the users try the product before you ask for personal /financial details
– ABC (Always Be Closing) – provide next steps
– eliminate choice paralysis: keep the number of options small and make the differences very clear
– be aware of the human brain bugs – like anchoring or framing and use them for the good reasons (ex – anchoring to show a default pricing option to avoid the choice paralysis)
– Ultimately, remember that quality of design is an indicator of credibility.

See all Smashing Magazine’s UX books