Free Sitemap Generator

free sitemap generator.com

Googlebot bandwidth usage

Posted at September 10th, 2006
We host a couple of sites that got Googlebot's attention for some reason. Yesterday it fetched 22,388 pages and today 42,933. That's a good thing I guess: to be found, you must be spidered first. But this is taken a lot of bandwidth. I browsed around and found a page (on google.com) saying I should e-mail google about this. Well, see the next post on this page: why not compress HTML output to reduce bandwidth usage? If you have the right modules loaded in your Apache webserver, just a single line in the .htaccess file will do:
AddOutputFilterByType DEFLATE text/html text/plain
This reduced bandwidth use (by Googlebot and others) with 60%!

Back to the new archive | Back to the homepage.