We Can't Judge Relevance

Once we get to a site, design and other “irrelevant” factors affect our perception of a site, making it difficult to isolate ‘relevance’ alone as a cause for someone to hit the ‘back’ button. And who’s to say that hitting the ‘back’ button means a site is irrelevant, anyway? How many times have you gotten the information you needed and were done with a site?

Objective measures of relevance, on the other hand, are made in a vacuum. They are far outside the real world and our realm of experience. In an objective measure of relevance, the tester types in a query, which they probably didn’t choose. [Apple], perhaps.

And then tester judged how relevant the results are. But “relevance” here isn’t determined by what the searcher really wanted when they typed in the query: it’s what the research team decided was the “right” answer when you ask a search engine “[Apple]?” If their definition doesn’t include Braeburns, suddenly the search engine is wrong.

One of my college professors called this problem with research “The Utterly Boring World.” In this world, The man bit the sandwich is a perfectly fine construct, while *The sandwich bit the man is ungrammatical because it is nonsensical. But there is a place for nonsense in the real world—and a place for Braeburns on a SERP, even if that wasn’t what you were looking for. It might be exactly what someone else wanted.