DISCLOSURE: This article may contain affiliate links. Which means we make a commission, at no cost to you, on products and services we recommend that you decide to buy. Read our full disclosure here.
Sometimes it makes sense to block search engines from indexing your website.
For example, you might buy a brand new domain name and build your WordPress website on a live server to save the hassle of moving it once it’s finished (make sense for beginners), or you might want to redesign an existing site on a temporary subdomain.
In some cases, you might want to build a private website that you don’t want to be found through any search engine. You could be using it for a specific project, an intranet or for a private family blog.
If any of these situations apply to you, and you don’t want people to find your site through organic search results, there’s a setting inside WordPress you can use to instruct search engines like Google not to crawl or index your pages.
Sadly, this directive isn’t followed 100% of the time. So there’s still a chance search engines may index some piece of content or other from your site. It’s a small chance, but a chance nonetheless.
With that said, unless you password protect your site, you have little choice but to use the method I’m about to describe.
How to stop search engines indexing your WordPress website or blog
Log into the admin area of your WordPress website and navigate to Settings > Reading and check the box next to Discourage search engines from indexing this site.
Checking this box does two things:
It adds this command to the source code of every page on your site.
<meta name="robots" content="noindex,follow">
These two commands – noindex and follow – tell search bots to not index the page but to follow the links within the page to discover more content. Because this code exists on every page, Google won’t show your pages in the search results.
And it adds this to the robots.txt file
User-agent: * Disallow: /
This tells search engine bots to disallow all content. When a bot lands on your domain it checks for a robots.txt file. If it finds one containing the protocols above, it won’t crawl and index your site.
At least, that’s the theory.
The problem with each of these methods is that you can’t guarantee your pages and other content won’t get indexed.
That’s all you have to do to block search engines indexing your content.
Working on your site while blocking search engines
If you want to work on your site and block people from viewing the pages while it’s in development, you need to install a maintenance mode plugin.
There are tons to choose from.
A maintenance mode plugins installs a splash page across your site so the only people can’t see any other content besides the homepage. Once you’re logged in, you can view the site as if it is fully available on the internet and work on getting your content right.
You MUST remember to change the settings when your site is ready!
When your site is ready for the world to see, you really must go back to the settings page and untick/uncheck the Search Visibility button. If you don’t do that, the search engines won’t index your website.
Many people have fallen foul of this over the years. And only found out weeks or months after the site went live.