Robots.txt1994 July 3rd

Martijn Koster presented the robots.txt standard (Robots exclusion standard or Robots exclusion protocol) as part of the W3C www-talk mailing list. The rules defined in the robots.txt file are used to prevent or restrict indexing robots from accessing a website.

Robots.txt  proposal in 1994
Robots.txt proposal in 1994

Send us your suggestions for interesting websites

Carbon Ads