Bulk Inclusion with Yahoo and Google Sitemaps

One of the most common problems many large web sites have with rankings is that portions of their web sites are not included in the search engines. The cause can be any of a variety of reasons that range from complicated URL strings to slow server response or lack of a proper internal linking structure. If Google can't find your pages, they won't get into the search results. Search engines benefit from having as many good quality pages in their databases as possible because it improves user experience. Better user experience equals more traffic and that means more advertising revenue. As a result, several efforts have been made to make it easier for webmasters to have their web sites indexed more effectively. This article discusses efforts Google and Yahoo have made to augment current methods of discovering as many URLs of a web site as possible. First there was Google Sitemaps Earlier this summer Google announced their Google Sitemaps tool that allows web site owners to create an XML or plain text file listing all or some of the URLs of a web site for inclusion in the Google index. Up to 50,000 are allowed per document and if you're over that, you can create another site map file. Google offers this tool as a compliment to what Googlebot, Google's search engine spider, can find on it's own by crawling the web. In cases where a site might have some difficulty getting indexed, providing a sitemap to Google may assist in getting missing pages into Google's search results. Google does not charge for this beta service and it's pretty easy to do in three easy steps. However, as a beta, there are no guarantees. More information is available on Google Sitemaps at: http://searchmarketing.yahoo.com/srchsb/ ) Search results display intermingled with URLs that are discovered naturally by Slurp, Yahoo's web crawler. There is a cost per URL for inclusion and also a cost per click based on the category of your business. Unlike sponsored listings, Yahoo claims there is no effect on ranking to use the Search Submit paid inclusion program, only the guarantee that your web site will be included in the Yahoo search engine index and that all URLs will be revisited every 48 hours. You can also pay Yahoo $299 per year for your domain name to be included in the Yahoo Directory. With this option, there is no guarantee; even in you pay, that you will be accepted or that your suggested title, description and category will be used. Submissions that are accepted into the Yahoo Directory are also automatically submitted to the organic index. There is plenty of speculation in the SEO/SEM community about the value of a Yahoo Directory link, both pro and con. And then there was Yahoo Sitemaps A free, bulk URL submission option for Yahoo was first presented at the recent Search Engine Strategies conference in San Jose from Yahoo's Tim Mayer during a session on indexing. Previously, web sites were discovered for the organic search results by the Slurp web crawler or by submitting one URL at a time. Now Yahoo offers the option of submitting many URLs at once using Yahoo's free submission service at http://www.domainname.com/urllist.txt . If you are providing a plain text list of URLs to Google Sitemaps, then you can use that same file for Yahoo. Place the file in the main directory of your web site and submit the URL to the Yahoo free submission form at: TopRank Online Marketing, specializing in organic SEO, blog marketing and online public relations. He's been cited as a search marketing expert by publications including U.S. News & World Report and The Economist and has implemented successful search marketing programs with top BtoB companies of all sizes. Odden shares his marketing expertise at