A/B, MVT testing and usability

Written on 30 March 2015, 10:31pm

Tagged with: , ,

Some quick notes after reading A field guide to usability testing and re-reading the Smashing Book #1:

1. A/B testing

– always test both versions simultaneously
– wait for it 🙂 (use a calculator to determine when to end it, and don’t give up earlier)
– keep the A/B tests for new visitors only (don’t surprise the regulars)
– but make sure that a new visitor gets the same version on consecutive visits
– be consistent: make sure that the variation appears on all pages (ex – if you have a promotional price on version A, make sure that the user will always see the promotional price on all the pages)
– the results might be un-intuitive
– naturally, the higher the number of users, the more reliable the result

Who would be involved in an A/B test:
the UI/UX team – to propose the 2 versions and analyze the metric results
the dev team – to implement the metric, manage sessions and make changes consistent across all the interfaces
the network team – to handle various types of redirects (ex – run the A/B test only for users in a given geographical area, or only users on mobile)

2. MVT (multi variate testing)

– it needs a lot of traffic and time
– keep the number of combinations to 25 or less and make sure you preview them all
– global vs local optimum (A/B vs MVT)
– if you don’t have the traffic and cannot use full factorial testing, you can still use partial factorial testing.

3. Some usability rules/principles


1. Improve ColdFusion performance by 1000x?

I recently had to deal with some ColdFusion page optimization. Although the result was not 1000 better like in this case, the approach was more or less similar. Instead of SQL indexes + rewriting queries inside loops + caching queries, I only had to do a combination of:
– reducing the number of DB queries by moving them outside loops
– using cfqueryparam. In addition to making things more secure, turns out it also has a significant impact on performance tuning.

Update: Some more improvements:
– ColdFusion whitespace management
getting rid of the old cfform
gzipping the content transferred to the browser

The end result was a 30x-50x performance improvement.

2. About helping others

Apparently helping others is embedded in every human’s DNA. Here’s an excerpt (no spoilers) from Andy Meir’s ‘The Martian’ – one of the best SF novels I ever read:

Every human being has a basic instinct to help each other out. It might not seem that way sometimes, but it’s true. […] This is so fundamentally human that it’s found in every culture without exception. Yes, there are assholes who just don’t care, buy they’re massively outnumbered by the people who do.
The Martian (coming in a theater near you in 2015)

The downside of this ‘feature’ of the human behavior is that it can be exploited. For example, in computer security, individuals who are attempting to social engineer some piece of information strongly rely on this aspect of the human nature:

Don’t rely on network safeguards and firewalls to protect your information. Look to your most vulnerable spot. You’ll usually find that vulnerability lies in your people. […]
Most people generally want to help somebody who is requesting help.
CompTIA Security

3. The Rosie Project

The Rosie project is an amazingly enjoyable book about a socially-challenged scientist who is working his way through the process of finding a life partner. Also recommended by Bill Gates. If anyone makes a movie out of it, I hope they pick Jim Parsons for the main role 🙂

Cryptography basics

Written on 3 December 2014, 11:07pm

Tagged with: , ,

1. Steganography

The science of hiding the existence of a message, as opposed to cryptography. A type of security through obscurity.
Ex. message written on the head of a messenger and sent only when it’s covered by the messenger growing hair; microdots; physical templates applied to a long text to highlight only some of the words.

2. Cryptography

The practice of secured communication. The science of encrypting a message, or concealing the meaning of a message.

  • Transposition ciphers – letters do not change, but move position
  • Substitution ciphers – letters change, but keep position
    1. Caesar shift: all the letters of the alphabet shift a number of positions (from 1 to 26)
    2. Simple monoalphabetic substitution: substituting a different letter for every letter. The cipher alphabet is fixed throughout the encryption. Both methods fail to basic frequency analysis
    3. Monoalphabetic with Homophones: a plaintext letter can be enciphered in many ways (typically numbers or symbols) – making the encryption resistant to a basic frequency analysis
    4. Polyalphabetic substitution – alphabet matrix + password repeated until it has the same length as the plain text message (Vigenère cypher). The cipher alphabet changes during the encryption; the change is defined by a key. The longer the key, the more secure; but less practical for everyday use.
  • A mix between transposition and substitution: ADFGVX (used to send Morse code messages)
  • One time pad – the only form of encryption that is unbreakable, relying on a random key that is the same length as the message. Each key can be used only once. Impractical for extended use.

3. Cryptanalysis

The science of deducting the plain text from a cyphertext, without knowledge of the key.
One of the most used methods at the beginning: frequency analysis

substitution cipher

basic cryptanalisis