Bot rules
Set access for bots that obey robots.txt.
REKAWEB WEBTOOLS
100% Free Robots.txt Generator
Build a robots.txt file that tells search bots which pages they may crawl.
Crawl delay
Ask bots to wait between requests.
Sitemap
Point to the sitemap from the same file.
Paths
Keep folders such as /cgi-bin/ out of the crawl.
About Robots.txt Generator
Bina fail robots.txt untuk memberikan arahan perayapan kepada bot enjin carian yang mematuhinya.
Tetapkan direktori yang dibenarkan atau disekat dan semak kandungan fail sebelum meletakkannya di akar domain. Robots.txt bukan kaedah melindungi fail sulit atau mengawal akses pengguna.
