Port80 ThreadMonitor

The Secret To MASSIVE Googlebot Crawls

Posted on March 8th, 2008 by Bompa

One webmaster, Sajal Kayan, faced a serious decline of activity from Googlebot.

In a nutshell, Sajal runs a news site. Things were going well, in fact, he was seeing an increase in traffic due to taking on a news agency partner. To accomodate the increased traffic, Sajal moved his site to a new server. The next thing he knew, Googlebot stopped crawling his site.

Here are some excerpts from his explanation:

“I am running the website thaindian.com which on September of 2007 finally(after 3 attempts) got approved as a Google news search source. Since then we have seen a steady increase in traffic at the portal.

“[due to adding a partner news agency] Sometime in November, we suddenly increased the rate of putting new content (from approx 30 articles/week to about 200 articles/day) this got a HUGE surge in traffic.

“On about November 26th or 27th, our news stories stopped being picked up by Google News. Even the main pages which were re-cached often were now outdated in Google’s cache.

“I had assumed that Google is penalizing me for running a bad server

“Perhaps G was penalizing me for adding too many(200/day) stories without getting more inbound links.

“[blank]…and immediately Googlebot started showing the love it once used to, my 2 week long ordeal was over. It was something like I had just turned on the magic switch. The crawl rate went from 1 request every 3 or 4 minutes to 4 to 6 requests a minute. The same days news articles was immediately shown on Google News.

Flipping the magic switch? Sounds a prelude to an ebook pitch.