First, God created Yahoo, (In reality, it was David Filo and Jerry Yang.) In the mid-1990's that site was a directory of web links compiled by hand. By 1997, I had moved to Alta Vista. Relatively soon after that, I stopped typing searches there and went to Google not because of its clean interface or 'coolness', I moved because it was simply better at providing a world of useful links. Google was able to comb through and index more of the Internet than any competitor. When we're looking for something, access to as much as possible is what matters most.
Unfortunately, when one's the king of the hill there's no place to go but downhill. Google's supremacy spawned an entire industry which tried to 'optimize' a site to rank well there. Google has had to respond by creating algorithms that weed out spam and underhanded methods to manipulate their results. Dealing with these concerns has eliminated their once, undisputed lead in searches. Now the company is simply too nervous and secretive to let people know they are throwing out a lot of babies with the bath water. This is too bad.
Here's some slightly-modified, recent correspondence:
ORIGINAL MESSAGE
To: Google Support
From: Dennis
Subject: pages in domain search but not indexed.
Date: Wed, 20 Apr 2005 14:24:09 -0000
I know you are busy with important details. I have tried to look around the net to find a reason for the following but have been unsuccessful.
I have a blog that presently contains around 90 weekly pages of information. At one time googlebots used to stop by daily and all pages were indexed. My blog section has a PR of 5.
Now, I find only about 10% of my site's pages can be found with snippets in a search of my domain. The majority appear there without title or text. Do you know what factors can lead to this?
Google Search: site:www.dennissylvesterhurd.com
Language: en
WebUserLocale: en
IssueType: other_webmaster
GOOGLE'S RESPONSE #1
Thank you for your note. The Google index contains two types of pages: fully indexed and partially indexed pages. Some of your pages are currently partially indexed. Because our robots were unable to completely review its content during our last crawl, your site appears without a cached copy or detailed title. Instead, it's listed by its URL.
We understand the frustration this situation may cause you. We're always working to increase the number of fully indexed pages in our index. You may be able to improve this page's visibility in our search results by ensuring that a number of high-quality sites link to it. While we can't guarantee that pages in our search results will always be fully indexed, "crawler-friendly" pages have a greater chance of being fully indexed. Guidelines for creating a "crawler-friendly" site are available at http://www.google.com/webmasters/guidelines.html
Regards,
The Google Team
MY NEXT MESSAGE
What a wonderfully quick response! I'm happy to hear back so quickly and completely. Further questions:
Your Paragraph 1 - (a) Am I correct in that the 'partially indexed' pages are not actually scanned and therefore information on them will not be returned on Google searches?
Your Paragraph 2 - Most traffic on my site goes to my blog, therefore most links go to www.DennisSylvesterHurd.com/blog/ . That index page has always been listed. Although with a PR of 5 it now only receives visits a few times per month. The 'partially indexed' pages are the archives. (b) Can 'partially indexed' pages stay that way forever?
(c) Could it be googlebots don't want to hit them because all the pages start with initial text and they're trying to avoid duplicates?
I'd love to hear your ideas on questions (a) (b) and (c) at your leisure.
==Dennis
GOOGLE'S RESPONSE #2
Hi Dennis,
Thank you for your reply. As you may know, we don't personally review individual sites, nor do we comment on webmaster techniques or the details of our search technology beyond what appears on our site.
We've dedicated an entire section of our site to answering the most common questions from those who maintain and/or promote websites. You'll find all of our publicly available information posted at http://www.google.com/webmasters/index.html
Besides this section of our site, we've created a newsgroup discussion forum for passionate Google users. At http://groups.google.com in the http://groups.google.com/groups?q=google.public.support.general group, many webmasters and Google users share their questions and expertise.
We recommend performing an advanced search on this group if you feel your question is particularly challenging and you've been unable to find an answer on our site. To do so, go to http://www.google.com/advanced_group_search?hl=en and enter your search terms in one of the "Find messages" fields at the top of the page. Type "google.public.support.general" in the "Newsgroup" search field, and click "Google Search." If you don't find an answer to your question, you can always post your question to the group to see if other newsgroup users have helpful advice. Please be aware that this content isn't posted by Google, and we cannot verify its accuracy.
Regards,
The Google Team
THE END
Thank you for copying this bit of standardized text from a template.
I guess what I was trying to politely say is something is broken at Google. Your "mightier than thou' attitude may not last forever. I see your company fast trying to use cash to expand into areas outside of search.
This is a good for two reason, I can immediately think of:
1) Yahoo and MSN completely index my site without forcing me to resort to any additional strategies. Eventually the average web user is going to figure out you're not the only game in town.
2) I remember when Alta Vista was king of the hill. We will love you dearly until the next best thing comes along.
Have a nice day.
Dennis