Best Robots.txt of a WordPress

1. Best Robots.txt of a WordPress Website hello@flippercode.com Flipper Code Pvt. Ltd. www.flippercode.com

2. Robots.txt is a text file that is put on the website to tell the crawlers which pages you would like them to visit and the ones to avoid. Robot.txt is not a firewall or password protection for search engines. It doesn’t prevent the search engine from crawling on the website. If there is any data that does not need to be shown in search results then don’t rely on robots.txt to prevent it from being shown in search results. Robots.txt WordPressfile should be in the main directory as only then the search engines can find them. If the search engines are unable to find them in the main directory, then they assume that the robots.txt file does not exist on the website. If robots.txt is not placed in the main directory then the search engine will display everything they find. Syntax of a Robots.txt File The syntax of the Robots.txt WordPress file is the following: - User-agent: * Disallow: / Search engine crawlers will be written in user-agent and the files which should be displayed are written in front of Disallow. Important Things for Best Robots.txt of a WordPress Website The CGI-bin and the standard WP directories like wp-admin, wp-content, and wp-includes should be disallowed. Only the subdirectory ‘uploads’ should be allowed in the robot.txt. If using categories then block the tag archives from search engines and vice-versa. PHP files, JS files, INC files, and CSS files have to be blocked. Other things which are not to be blocked are your choice. www.flippercode.com