Morrie the Toupee Salesman

A nice piece from Jeremy Liew this morning. I have long felt that the kind of testing described there (which I’m going to condense into the catchphrase “A/B Testing”) is a great alternative to the vast assortment of mechanisms favored by orthodox marketers. Focus groups, polling, and the rest of the panoply of how marketers determine what we want, are well known to be deeply flawed, but the conventional wisdom (and significant vested interests) keeps those methodologies hanging around. One of the most touted benefits of the internet was (and still is, at least at some companies) the ability to not measure behavior by what people say they do, but by what they actually do. And websites, especially with the whole web 2.0/Permanent Beta phenomenon, are the perfect products to use split testing, due to their infinite malleability, their plasticity.

But A/B Testing, to be done right, requires that your engineering is done in a way that allows the presentation of different content to small segments (small in either time or number of visitors or both) and that can cause problems internally. Engineers will push back at having to maintain multiple copies of stuff, and marketing will dislike having to beg for hundreds of little changes to be made (and unmade) to your website. Standard logging packages don’t really offer good support for testing small changes in functionality. I think it has to be a philosophy that is followed right from the start, and built into your technology from day one.

Analytic Hierarchy Process is something I find interesting, partly because I had to learn and implement it once upon a time. It’s a way of making complex decisions entirely based on pairwise comparisons, something that is related to split testing. Stick the two of them together and perhaps you end up with something like a methodology for determining corporate strategy in a satisfyingly “push-button” way.

Jakob Neilson outlines the strengths (and weaknesses) of split testing here.