How to Avoid Indexing your Website Development Version

If you are working on your new website in an online development environment, or if you have a copy of your live site for staging / development purposes, it is imperative that you don't inadvertently let Google (or any Search Engine, for that matter) crawl your development site.

Allowing search engines to crawl both the live and development version of your site will harm your SEO, sometimes significantly.  This happens because of a number of factors, but not least due to splitting your SEO score across two sites and falling foul of duplicate content issues.

Block All Agents in Your Robots.txt File

The easiest way to impede Search Engines from indexing your development site is to add a robots.txt file to the primary domain directory (i.e. the parent web directory for your website).  This robots file should contain the single directive, given below:

User-agent: *
Disallow: /

Blocking Search Engines on WordPress

On a WordPress site, you don't need to trouble yourself with manually editing and uploading the robots.txt file.  You can simply visit the settings->reading management page and tick the box that says: discourage search engines from indexing this site.

Don't Forget to Remove the Directive when Going Live

If, ultimately, your development site will be deployed as the live website, then don't forget to edit the robots.txt file accordingly and / or deselect the setting on WordPress.