General questions about CMSimple
3 posts • Page 1 of 1
There is no generally valid robots.txt.
In the robots.txt file you can specify which behavior of search robots you want on your site.
Note: It is not possible with this technique to protect web pages from access by robots or persons. You can only control the appearance in the search results.
Note: There is no guarantee that search engines will adhere to the prohibitions in robots.txt. The vast majority of robots in modern search engines take into account the presence of a robots.txt, read it and follow the instructions. Robots that search the web with bad intentions are unlikely to follow the instructions.