Google Weblog

October 30, 2002: The End of Frequent-Crawling? [UPDATED]

UPDATE: Google spokesman Nate Tyler responds:

We are in the process of an index switchover and have had to focus some of our technical resources on deploying it. I expect that we'll have things back on track shortly. Don't worry, the fresh crawl isn't going away. Google's ongoing goal is to enable our users to find more of the web's freshest content and this includes bloggers. Apologies for any inconvenience this may have caused.

Previous entry: Not too long ago, Google introduced a system where the best frequently-updated would be crawled about every other day. This made Google much more useful when searching for rapidly updated news.

It appears this has been stopped. The cached pages for sites that previously got this treatment are all from October 25th.

I'm not sure if this is because of a bug, part of the new ranking system, or because they feel Google News replaces it. But it is disappointing. The frequent-crawling gave us access to many independent sites that aren't part of Google news, and the cached pages were very useful during major events. If anyone has any official information about this problem, I'd appreciate it.

Posted by Aaron Swartz on October 30, 2002 05:16 AM