Think of the eternal vastness of configuration spaces (even in domains as restricted as, say, static HTML/CSS pages) and then you know why AB testing is not a solution to everything. (Multivariate testing, too. It’s just slightly more elegant in some respects, but suffers from the same basic deficits and additionally has its own issues.)
Time is limited and traffic is limited. You simply cannot test everything, in fact, the amount of variants you can actually test during, say, a year may be shockingly low compared to the amount of variants possible (at least if you are not Google – but most aren’t).
Without a strong sense of direction and discerning eyes and hands that provide plausible variants to test, AB testing can be quite useless. AB testing is a great tool to verify hypotheses (and those may well be competing hypotheses), but I really question its usefulness as a creative tool.
If you can plausibly only do a couple of AB tests a year (the possibility of running an AB test being a function of traffic and time) you better not waste that opportunity on button colors or some similar nonsense.
Sure, evolution through natural selection produced some awesome designs and is basically a completely blind process based purely on testing of different variants with only slight and random differences with no involvement of any designer at all (sound familiar?) – but it also took billions of years. Most designers just don’t have that kind of time …
To more directly answer your question, AB tests can certainly be used to defend against bad hypotheses. If there is a culture of testing and new designs have to be battle tested then that’s less of an opportunity for someone in a position of power to screw things up by fiat. However, that’s also a tremendous waste of time and efficiency if bad designs that could be sieved out quickly by simple heuristics have to also first be tested, which is just a slow process and a wasted opportunity to test something better.
Maybe think of AB testing as a last line of defense, but be careful, AB tests can be paralyzing. (Or rather, first AB tests paralyze everything, then the boss is annoyed by how slow everything is going and changes everything by fiat and immediately without any testing at all. From one extreme to the other …)
Time is limited and traffic is limited. You simply cannot test everything, in fact, the amount of variants you can actually test during, say, a year may be shockingly low compared to the amount of variants possible (at least if you are not Google – but most aren’t).
Without a strong sense of direction and discerning eyes and hands that provide plausible variants to test, AB testing can be quite useless. AB testing is a great tool to verify hypotheses (and those may well be competing hypotheses), but I really question its usefulness as a creative tool.
If you can plausibly only do a couple of AB tests a year (the possibility of running an AB test being a function of traffic and time) you better not waste that opportunity on button colors or some similar nonsense.
Sure, evolution through natural selection produced some awesome designs and is basically a completely blind process based purely on testing of different variants with only slight and random differences with no involvement of any designer at all (sound familiar?) – but it also took billions of years. Most designers just don’t have that kind of time …
To more directly answer your question, AB tests can certainly be used to defend against bad hypotheses. If there is a culture of testing and new designs have to be battle tested then that’s less of an opportunity for someone in a position of power to screw things up by fiat. However, that’s also a tremendous waste of time and efficiency if bad designs that could be sieved out quickly by simple heuristics have to also first be tested, which is just a slow process and a wasted opportunity to test something better.
Maybe think of AB testing as a last line of defense, but be careful, AB tests can be paralyzing. (Or rather, first AB tests paralyze everything, then the boss is annoyed by how slow everything is going and changes everything by fiat and immediately without any testing at all. From one extreme to the other …)