Comparing Search Engine Results - My Experiment

A couple months back I was covering the launch of a new shopping search engine. As part of my event coverage I was allowed to be part of the beta group, prior to the launch only a handful of people [mostly employees and family members of employees] were able to take part in the Become beta test. Read coverage of spreadsheet comparing the search results of the top search engines for a generic shopping related term "furniture" but when I saw what the data looked like I decided to dig a bit deeper and limited the scope of the engines I compared but broadened the scope of the terms I used in my comparison. Before I tell you what terms I used and how I picked them I'd like to first let you in on a little secret. I had a hypothesis. Yes, that's right, the guy that never brought is high school lab notebook to class actually had a hypothesis before he conducted an experiment. Dr. Scott McCord would be proud. My Hypothesis On Shopping Search Result Comparison I believe there is no significant difference in the search results of Google, Yahoo and Microsoft. This statement applies to their search results in general but, for sake of limiting the scope of this experiment, I will be focusing soley on shopping related queries. The Method Step 1: Getting A List of Objective Shopping Related Search Terms First, I selected keyword phrases from Froogle's year end Zeitgest list for 2004. This consisted of 6 different categories of queries with 10 keyword phrases per category. So that gave me a total of 60 terms and 600 results I could use in my experiment. Second, I used the top 100 searches on MySimon for the week ending February 25, 2005. This list contained 100 queries that were completly unique. Step 2 Data Aggregation and Comparison of Search Results In order to determine the similarity of search results it's important to know how unique their results are. To accomplish this, I performed 3 different sets of comparisons [because there were 3 unique sources of data used as noted in step 1 above]. Then, within each report I compared the top 10 search results of each keyword phrase per search engine. If a url was found within the search results of a single search engine [engine A] but the url didn't exist in the top 10 results of either one of the other engines [engines B, C & D] then that url was considered unique for the search engine in which it was found [engine A]. This comparison could be drilled down to even further if people request that. For example, I could compare the number urls in the search results of Search Engine A to the urls of Search Engine B to determine how 2 specific search engines fare against one another. For my purposes, I only wanted to compare the results each engine tested against all the engines so I'd have an unbiased report. Let me give an example to clarify that. For the term furniture, the url "http://www.nhfa.org/consumer.asp" was found within the results of Become but not in the results of Google, Yahoo or MSN so this would count as a "unique result" for become but not for any of the other search engines. The graphs below show the uniqueness of each search engine per keyword set. The keyword sets [explained in step 1] are labeled Froogle, Shopping.com and MySimon for clarification. Some terms overlapped between each of those 3 sources of keywords but I felt it was important to not merge the lists and run a single report for objectivity purposes. I wanted to see how each search engine fared when compared on 3 completely different sets of shopping related keyword searches without trying to create one master list.