A/B Testing for Adsense is Often Pointless

Google recently posted a little blurb about The ABCs of A/B Testing. I hate to be a wet blanket, but for most of us lower volume publishers, that's a pointless exercise with little to gain. Unless you have very high volume, you simply don't have enough data to draw any useful conclusions from this kind of test. Don't get completely discouraged: sometimes a small change can produce visible results. For example, when I added ads that mixed in with the text (like the one just above or below this paragraph), my revenues increased enough to notice. However, it wasn't from those ads: it was the column ads that picked up activity. So, yes, I do track my Adsense. Mostly that's just idle curiousity: I just like to know where the money is coming from. For example, you see Google ads in three different places on this page, and I know how much money each position makes per day. That's how I know that adding the middle ad increased the performance of the side column ads. But what's interesting (and unfortunate with reference to A/B testing) is that while the monthly overall income is relatively steady, the distribution of that money varies pretty widely. Today the top position ads are the best performing, tomorrow it might be the ones that mix in with the text, and next week Wednesday the ones in the columns take the prize. Why does this happen? Who knows? Daily stats swing too: A new advertiser might pop into the mix and change everything - nothing to do with anything you did. For whatever reason an old ad that attracted nothing in the past suddenly becomes interesting to the visiting audience.. that kind of thing. That means that tracking changes can be confusing and noisy. Unless you run a product like