documented

Controlling crawling and indexing now documented on code.google.com

Webmaster level: All Do you know how Google's crawler, Googlebot, handles conflicting directives in your robots.txt file? Do you know…

14 years ago