robots.txt

General questions about CMSimple
Post Reply
ustalo
Posts: 164
Joined: Mon Aug 16, 2010 7:42 am
Location: Russia
Contact:

robots.txt

Post by ustalo » Sun Nov 08, 2020 9:26 am

Masters
could I get right robots.txt for CMSimple XH
Exuse me, stupid one. I wish to have correct one.
Aleksei

frase
Posts: 5085
Joined: Thu Apr 21, 2016 6:32 am
Location: Saxony
Contact:

Re: robots.txt

Post by frase » Sun Nov 08, 2020 11:11 am

ustalo wrote:
Sun Nov 08, 2020 9:26 am
Masters
could I get right robots.txt for CMSimple XH
Exuse me, stupid one. I wish to have correct one.
There is no generally valid robots.txt.
In the robots.txt file you can specify which behavior of search robots you want on your site.

Note: It is not possible with this technique to protect web pages from access by robots or persons. You can only control the appearance in the search results.

Note: There is no guarantee that search engines will adhere to the prohibitions in robots.txt. The vast majority of robots in modern search engines take into account the presence of a robots.txt, read it and follow the instructions. Robots that search the web with bad intentions are unlikely to follow the instructions.

ustalo
Posts: 164
Joined: Mon Aug 16, 2010 7:42 am
Location: Russia
Contact:

Re: robots.txt

Post by ustalo » Tue Nov 10, 2020 10:13 am

Thanks for answer. I did upon this information.
Aleksei

Post Reply