[TriLUG] MSN bot is pounding my website...

[Rick DeNatale]

I did a little googling this morning.

MS claims that msnbot respects the "pseudo standard*" Crawl-delay
directive in robots.txt.

So you might try this in robots.txt

User-agent: msnbot
Crawl-delay: 20

This should keep msn bot from hitting you more than once every 20
seconds but still get your site indexed if you wish. I just did this
and I plan to keep an eye on my logs.

* This apparently isn't something which MS dreamt up themselves,
several other bots such as Yahoo already supported this.