I have been getting a lot of CPU spikes recently on my server and somehow I believe it's not the real traffic or some part of it isn't real. So I want to only allow Google bots, MSN and Yahoo for now. Please guide me if the following robots.txt file is correct for my requirement. User-agent: Googlebot User-agent: Slurp User-agent: msnbot User-agent: Mediapartners-Google* User-agent:
Kamis, 19 April 2012
Langganan:
Posting Komentar (Atom)
0 comments:
Posting Komentar