Important Notice: Our web hosting provider recently started charging us for additional visits, which was unexpected. In response, we're seeking donations. Depending on the situation, we may explore different monetization options for our Community and Expert Contributors. It's crucial to provide more returns for their expertise and offer more Expert Validated Answers or AI Validated Answers. Learn more about our hosting issue here.

Does Ask observe the Robot Exclusion Standard?

April 26, 2017exclusion observe Robot standard

0

Posted

Does Ask observe the Robot Exclusion Standard?

1 Answer

0

Posted

Yes, we obey the 1994 Robots Exclusion Standard (RES), which is part of the Robot Exclusion Protocol. The Robots Exclusion Protocol is a method that allows Web site administrators to indicate to robots which parts of their site should not be visited by the robot. For more information on the RES, and the Robot Exclusion Protocol, please visit http://www.robotstxt.org/wc/exclusion.html. Q: Can I prevent the Ask crawler from indexing all or part of my site/URL? A: Yes. The Ask crawler will respect and obey commands that direct it not to index all or part of a given URL.