First, I want to note that the generated file for the site is an important aspect of search engine optimization. Why do I need robots.txt? For example, in SEO, robots.txt is needed to exclude pages that do not contain useful content from indexing and much more. You can learn how to use it correctly in the article what should be in my robots.txt file
https://www.psd2html.com/blog/robots-txt-what-when-and-why.html . And remember that the robots.txt file is a text file in the .txt format that restricts search engine crawlers from accessing content on the http server.