Is it necessary to use Robots.txt file?

[jonathan brown]

Hi,

No, it's not necessary to add Robot.txt file but it can’t be ignored because it helps to navigate search engine robots like Google robots to crawl and index websites’ pages.

Benefits of Robot.txt file are as follows:
Index only those areas of the website you want to index.
Helps in indexing particular files on a site (images, content, videos, PDFs)
Avoids duplicate site content pages from showcasing in SERPs