If you happen to be a fan of The Hobbit, then the most pragmatic quote quipped by the literature genius that was J.R.R Tolkein was this; “It does not do to leave a live dragon out of your calculations, if you live near him.”

Powerful stuff. And with these words, comes a very important message. That we’ve got to plan ahead.

It’s no different in the world of SEO. And the first step to any effective SEO audit is indexing. But why?

It’s simple really. If your site is yet to be indexed, then there’s no way that search engines such as Google or Bing can read it. So if the search engine can’t read it, then no amount of top dollar SEO will improve how highly your web page(s) will rank.

As a rule of thumb, for your site to be ranked, it first needs to be indexed.

How can you tell if your site is being indexed?

Thankfully, there are many tools out there that can give you a heads up as to whether your site is being indexed.

Indexing, at it’s primary level, is a page-level procedure. In layman terms, search engines read these pages and treat them as individual entities.

A quick and sure way to tell whether one of your pages is being (or has been) indexed by Google is to take advantage of the site:operator via a Google search. By entering the domain, Google will avail to you all the pages in that domain that have been indexed. You can even key in a specific page URL to confirm whether the single page has been indexed or not.

Instances when a page has failed to be indexed

You’ve just found out that one of your pages has not been indexed. What could be the main cause of this? The biggest culprit in such a situation is the Meta robots tag that is currently present on the said page. Not only that, it could also be the inappropriate utilization of disallow in the robots.txt file.

One thing you’ve got to realize is that the Meta tag that is present on the page level, as well as the robots.txt file, provides detailed instructions to search engine bots on how they should interpret the current content of your page or website.

The main difference between the two is that while the robots Meta tag is present on each individual page, the robots.txt relays instructions about the site as one single entity. However, when it comes to the robots.txt file, you can isolate directories and pages and instruct how the bots should handle these pages during the indexing process. Here’s how you can examine using robots.txt.

Robots.txt

In the event that you’re not certain whether your site utilizes a robots.txt file, there’s a pretty simple way to affirm this. All you need to do is to key in your domain in your browser, then followed by /robots.txt.

Google search control further comes with a convenient robots.txt tester tool, which can help you fish out any errors in your robots file. Furthermore, you can also test one of the pages on your website via the bottom bar to determine whether your robots file, in its current layout, could be hindering the Google bot.

Thus, if a directory or page is disallowed, then it will be shown after Disallow while in the robots file.

Robots Meta tag

At the header of the page is where a robots Meta tag is placed. Normally, there is no point to utilize both the robots.txt and the robots Meta tag at the same time to disallow indexing of a page.

Apart from disallow, the robots Meta tag can be used to instruct search engines on which links in the entire page should not be followed. Especially when search engine optimization purposes are taking place. In fact, such an instruction can come in handy in situation like having press release pages.

The most common directives used with this tag in relation to SEO are:

Index follow: This is implied by using default. Hence, search engine indexing robots are instructed to index the information that is currently present in a particular page. Furthermore, search engine indexing robots are instructed to follow the links of the given page.

Noindexfollow- Search engine indexing robots are instructed NOT to index the information that is currently present in a particular page. Furthermore, they are instructed NOT to follow the links of the given page.

That being said, there are plenty of uses of the meta-tag that I will discuss comprehensively in an upcoming blog.

All in all, you’ve got to keep in mind that your site has to be indexed first in order for it to be ranked. Therefore, make sure you always check your site’s index ability first when executing an SEO audit.

That being said, if you’d like to get started with page and site indexing, contact 4Spot Marketing today. You can email us at Info@4SpotMarketing.com or call us at 702-721-9763. We offer a free consultation to determine what approach will serve your company best.