Kamis, 19 April 2012

Is this Robots.txt file correct?

I have been getting a lot of CPU spikes recently on my server and somehow I believe it's not the real traffic or some part of it isn't real. So I want to only allow Google bots, MSN and Yahoo for now. Please guide me if the following robots.txt file is correct for my requirement. User-agent: Googlebot User-agent: Slurp User-agent: msnbot User-agent: Mediapartners-Google* User-agent:

0 comments:

Posting Komentar