comScore - The Great Debate

Or again, perhaps people aren’t worried about what tools or pages drove repeat visits or cross-session sales behaviors. But it seems important to me. For a portal site like the one tested by comScore, the bellwether analysis is content impact on engagement. I’d like to say that this analysis may still be possible with this level of error, but I’m not sure that it would be.

Not concerned about these little things? Well how about the fact that every one of your campaigns might be significantly mis-counted in terms of conversion? That seems pretty consequential.

In short, just because comScore isn’t focused on analytics but only on traffic reporting doesn’t mean the impact of their findings is limited to traffic. And the impact for analysis of this level of error in 1st Party cookies would be very bad indeed. That’s not an admission you’d probably expect to hear from a web analytics consultancy and I’m sorry if this constitutes breaking ranks, but our first commitment ought to be to the truth of the numbers.

On the other hand, I too have some caveats about the general applicability of these results to every site. First, it’s important to understand the difference between each statistic. The 31% 1st Party cookie rate is, in my view, probably the most damning statistic in the study. Why? Because unless the behavior is somehow related to the specific site in question, you’d expect it to hold up for every site. Or then again, maybe you wouldn’t. If we accept that this statistic represents manual deletion of cookies, then there are two alternatives. The first is that people simply erase all cookies. That’s certainly possible – maybe even likely since it’s so easy to do. But a user may also scan the cookies on the target system and since cookies are often quite identifiable it’s possible that some cookies are much more likely to be deleted than others. It may be that an Amazon or Charles Schwab cookie is much less likely to be deleted than an AOL one. That’s one of the reasons it would be very interesting to understand the qualitative part of the comScore study – namely, what deleters (especially "serial" deleters) were actually doing. In addition, I’d like to understand if this problem is especially severe for certain browser populations (Firefox). And are users simply setting their browser to delete all cookies every time they close? This might explain the behavior of deleters – they aren’t doing any manual work at all. It would also imply that for this segment even same day uniques are overstated.

As comScore positions the study, however, the most damning finding is probably the one about the potential traffic impact of serial cookie "deleters." And for their particular chosen site, the impact would indeed be considerable. But for many of our sites, this group would probably have much less of an impact. Why? Portal sites are unusual in attracting a very high volume of repeat visits. comScore reported an average of nearly 13 1st Party cookies during a month for the 7% of visitors with 4 or more cookies detected. For most of our client sites (that aren’t portals), the overwhelming majority of visitors will come much less often than this. And if only a very small percentage of visitors are multiple repeaters and only a small percentage (7%) of those are multiple deleters, then the impact is likely to be considerably less (in terms of percentage of total visitors) than for a portal site. It’s likely true that no set of sites has a higher percentage of frequent return visitors than a true portal.

I’ll be honest – I find the study worrisome. It’s not that I haven’t realized that cookie deletion is a significant issue or that the potential impact of frequent return visitors on visitor traffic eluded me. But every analyst knows there is a line between data that is fuzzy but useful and data that is too messy to analyze. And I’ll be honest as well in saying that we see many sites where the number of visitors flagged as "New" would be better explained by the comScore view of the world than by the one prevalent (including by us) in the web analytics world. Depending on how the comScore study shakes out, web analysts may be facing a significant rethinking of where we are relative to that "fuzzy" line for almost every kind of interesting analysis. For some types of sites, at least, this might drive our short-term tool kit back to the pathetic same session analysis we were stuck with in tools a couple of years back.

comScore has their own agenda, here, obviously. But to me, the issue is ultimately about much more than unique visitor counts. Perhaps Anil is right to discount the importance of that statistic. But in doing so, he’s surely missing the larger point. In my view, the basic analytic toolkit of visitor segmentation and cross-session tracking is at risk in light of these findings. What’s the point of visitor segmentation when your visitors are lost every couple of days? This is an issue the measurement community just can’t ignore. I remember when 3rd Party cookie rejection rates began to climb and the ostrich like mentality that insisted this wasn’t an issue. Till suddenly the vendors made it possible to use tagged 1st Party cookies and everyone began implementing first party cookies. Sadly, we may not have solved our problem.

Are there short-term solutions? Possibly. And they begin with getting a handle on the scope of the problem for your site and understanding how to protect your analysis from these problems. You can begin to get a handle on the issue for your site by examining the trends in "New" visitors – especially if you can measure from significant events like a measurement start, a new cookie, or major traffic spikes. You might also want to look at the percentage of New/Returning visitors by Browser type. If you have strong behavioral cues on your site (like Login), you can measure the degree to which these behaviors are used by "NEW" visitors. This percentage compared to total visitor usage is an excellent way to get a reasonable read on how much or little of a problem you have. Handling bias is trickier, but if you want to analyze, for example, whether types of tools or content drive repeat visits, then you’ll be wise to start with a universe of visitors that you can prove has remained cookie constant. How might you do this? By insuring that you segment for behavior in both the 1st and last month of your study. This will narrow your analysis, but insure that you’ve ruled out cookie issues.

Found an error or have a suggestion? Let us know and we'll review it.