Noindex — that’s the subject of this lesson. What is noindex? How and when do you use it? Is noindex a hint or a directive? What exactly does noindex do? Does the crawl rate of a noindexed page fall away over time? When is the nofollow tag the better choice instead? These and plenty more are what we’ll be covering.
As you’ll know, in our SEO Academy we cover every topic connected to ranking on search engines. If you’re new here, have a look at the earlier lessons — we’ve covered a number of things that will help you follow this article.
If this lesson interests you, it’s probably because you need to keep content out of the index or remove it from the index, and you want to do it properly, avoiding mistakes that would hit your visibility and traffic.
So let’s not waste any more time — here we go!
Want to become an SEO guru?
Thinking about starting a career in SEO?
Got a site you want to optimize for search yourself?
Get in touch and ask about our SEO courses!
I want more info
What is noindex?
A definition of noindex you’ll find all over the web goes like this: noindex is the instruction given to search engines not to include a particular page or website in their databases (called indexes).
More technically it’s an HTML meta tag — a piece of code hidden from the user but visible to the search engine bot, through which it is given instructions and/or extra information (meta-information).
The noindex directive isn’t always obeyed by search engines, though, and they interpret it in slightly different ways from one another. Over time, however, the crawl rate of a noindexed page declines as long as the directive stays in place. It can take two or three months before the crawler comes back to check whether the page may be indexed after all.
So we can say noindex is a directive that Google and the other search engines don’t apply instantly, but the page’s visibility drops all the same, edging towards invisibility as time passes.
Noindex & the robots meta tag
The noindex tag is part of the robots meta tag, whose job is to tell search engine spiders specifically which pages are accessible and which pages or sections of them must not be indexed, as well as to give them information about the contents of a web page.
The meta tag’s structure is simple and involves only two attributes:
<meta name=”property” content=”value“/>
where:
- the name attribute states the type of information the tag carries
- the content attribute states the information itself
Syntax of the noindex meta tag
The noindex directive goes in the “head” section of a web page’s HTML source. The syntax is:
<head>
<meta name=”robots” content=”noindex”/>
< /head>
Beyond the noindex directive, the content attribute can also carry the following:
- index (a request to include a page’s content in a search engine’s archive);
- follow (a request to follow every hyperlink to other connected pages);
- nofollow (an instruction not to follow the links on the page or pass link equity through them – more here);
- nosnippet (an instruction not to show a snippet in the search results – more here);
- noarchive (an instruction not to keep a copy of the page in the search engine’s cache);
- unavailable_after (an instruction to stop indexing a page from a given date). Earlier indexing isn’t deleted, and there’s no meta tag that sets a date from which the page stops appearing in the engine’s results and/or its cache.
The index and noindex directives can then be combined with the others as needed, giving syntax such as:
…content=”index, follow”/>
Want an SEO analysis of your site?
Want to know how many keywords you rank for on Google?
Want to know your website’s organic traffic?
Want to know who your competitors are and how they’re performing?
Want to know the number of backlinks pointing to your site?
No problem, we’ll tell you. FREE!
When and why do you use noindex?
The noindex directive is mostly used in these situations:
- pages and sites still under development;
- pages you want kept private (such as the parts of a site meant only for employees or members of an organisation);
- temporary pages;
- thank you pages: here the instruction stops those pages being indexed, and so stops a visitor landing on one before filling in the opt-in on your landing page;
- pages with duplicate content, irrelevant content or thin content;
- a tag page that outranks the content you actually want to place in the SERP;
- pages you’d rather not index because they’d make for a poor user experience (UX).
How do you use the noindex tag?
The noindex directive can be implemented by hand, editing the page’s HTML source, or through plugins for the various CMSs (WordPress, Joomla, Drupal and others).
Implementing it by hand
Here’s how to implement the noindex instruction manually.
Noindex on a whole page:
<head>
<meta name=”robots” content=”noindex”
<title> don’t index this page </title>
</head>
#2 Noindex directive for specific bots
The noindex directive can be restricted to certain bots by giving a different “name” value in the meta tag. To block Google’s bot (Googlebot), for instance, the syntax is:
<meta name=”googlebot” content=”noindex”>
#3 the robots.txt file
The robots.txt file is a UTF-8 text file containing commands and directives that can block crawling. They also let you specify which content, folders or files the search engine may access or index.
The file is part of the REP (Robots Exclusion Protocol), which applies controls both globally (through robots.txt itself) and at page level, and it always belongs in the root directory of your domain.
The “Disallow” command used by the robots file to block content is another instruction a search engine may disregard. Google Search Console helps here: you can test the robots file, see which instructions Google is ignoring, and decide what to do accordingly.
Take care with the robots file: if you don’t want a piece of content indexed but it links to many internal pages, rather than blocking it with disallow you’re better off using the robots meta tag set to “noindex, follow”.
#4 Noindex on part of a page
Keeping part of a web page out of the index can be attempted with various techniques, not all of them recognised by search engines:
Noindex tag
<p>
don’t index this text.
<noindex>don’t index this text</noindex>
<!—noindex–>don’t index this text<!- -noindex–>
</p>
Microformats
Microformats (more here) are a piece of markup in a web page that, given the right specifications, can serve the same purpose.
In this case the REP checks for the value class=”robots-noindex” in the HTML tag.
The syntax is:
<p>Do index this text.</p>
<div class=”robots-noindex”>Don’t index this text.</div>
<span class=”robots-noindex”>Don’t index this text.</span>
<p class=”robots-noindex”>Don’tindexthistext.</p>
As above, values can be combined here too:
<div class=”robots-noindexrobots-follow”>Text.</div>
Yoast SEO plugin
Implementing it with a plugin
As mentioned, several plugins can implement the noindex directive; among the most widely used are Yoast SEO, All in One SEO Pack, Screaming Frog and Visual SEO.
Chances are that you, like almost everyone running a website these days, already use the Yoast SEO tool for SEO. So here’s how to do it in a few simple steps:
- go to the front end of the page or article you don’t want indexed, then in the Yoast meta box select the “advanced” tab, marked with a cog;
- clicking the icon immediately brings up a set of questions, the first of which is: do you want to allow search engines to show this post in search results?
- by default the plugin is set to show your content; choose “no” and the content won’t appear in the SERPs. With existing content it will take some time before it stops showing up in search results.
Noindexed pages & links
Does applying the noindex tag to a page strip the value from the links on it? In other words, do those links still build PageRank and pass link equity to other pages?
With the plugin route, Yoast automatically uses the setting “noindex, follow”. Search engines will therefore keep following the links on those pages.
If you’d rather they didn’t, answer “no” to the plugin’s question “should search engines follow links on this post?” when you set it up.
Conclusion
In this lesson of your favourite SEO Academy we’ve covered the noindex tag. You now have what you need to understand exactly what it is, when to use the directive and how to set it up yourself — but that alone isn’t enough.
SEO is a field in constant motion, and the innovations that keep arriving call for new skills. We’ll keep covering the recent developments, and where they’re heading, in the lessons ahead.
Stay tuned and keep following us — in the next lessons of the SEO Academy we’ll get into plenty more interesting territory…
Need help using or implementing the noindex tag? Get in touch! We’ll help you keep your page or your whole site out of the index.
Want to improve your site’s SEO ranking? Want to optimize your site, get found by your customers on Google and beat the competition? Get in touch! We’ll work out the best SEO strategy for your site.
Want to sharpen your SEO knowledge? Get in touch! We’ll find the right SEO course for you.
Want to ask us something or just talk to us? Get in touch! That’s what we’re here for.