Solution: how to avoid Google's crawling and indexing non-existing pages

Many webmasters are affected from the weird issue: Google is indexing (at least crawling) non-existing URLs. The issue isn't depending of whether one uses Wordpress or other CMS. This question about why Google is crawling and / or indexing non-existing URLs appears in all webmaster forums, Google Groups and so on, but without a clear solution.

The fact, that Googlebot creates and crawls a bunch of non-existing URLs, lets arise some questions:

• Where non existing URLs are coming from?
• Why is it not optimal, if non-existing URLs are crawled respectively indexed?
• How to minimize risks related to non-existing URLs?

SEO query string universal solution

URLs with query strings can be a real poison for a SEO. The main and mostly harmful damage untreated URLs with string query do, is a not calculable rise of amount of existing URLs with same content, HTTP answer code 200 and untreated indexing management, also called duplicated content. Another issue caused by query strings in URLs is overspending of crawl budget to URLs with query strings, which must be better excluded from crawling and indexing.

On this way a site with untreated URLs with query strings gets on the one side such URLs into index, which don't belong here, on the other side the crawling budget for good URLs could be missed, cause overspend.

There are some passive techniques to deal with query strings in URLs. Actually i planned to publish existing techniques for dealing with query strings in URLs and my solution for SEO problems caused by query strings in URL into my ultimate htaccess SEO tutorial, but then this topic got some more details, so i decided to create an extra article about query strings in URL and SEO.

Preceding kinds of SEO dealing with query strings in URLs

• while Google means, it could deal with query strings in URLs, it recommends to adjust the bot's settings in Webmaster Tools for each of existing query strings.
• URLs with query strings could be disallowed in the robots.txt with a rule like
  

  Disallow: /?*
  Disallow: /*?
  

• If header information of HTML or PHP files, available with URLs with query strings, can be edited, so it is possible to add rules for indexing management and URL canonicalisation, like
  

  <meta name="robots" content="noindex, nofollow">
  <link href="Current URL, but without query string" rel="canonical">
  

These methods are mainly manual, require unpredictable workload and solute problems partly. But the good news is: i have an universal solution, working for all URLs with query strings and getting rid of all SEO troubles caused by query strings in URLs.

H for htaccess: part 5 of the HASCH the OnPage SEO framework

.htaccess (hypertext access) is a text file, placed mostly in the root folder of the given site and invisible cause of the point at the begin. .htaccess contains directives for server, server software, robots and browser about handling of files, folders and paths / URLs.

Generally there are 2 topics, where .htaccess can be used for SEO purposes:

• Mod_alias and mod_rewrite directives (URL redirects and rewrites)
• load time optimization

Site security has in my opinion only indirectly to do with SEO, so i decided not to make it to a topic of this article.

The last, fifth part of my HASCH OnPage SEO framework is about the SEO mission of .htaccess. I aim to create a kind of multipurpose explained and examples-illustrated checklist about .htaccess usage for mod_rewrite and robots manipulation and load time optimization as advanced SEO objectives. This ".htaccess for SEO" tutorial will be helpful (for me and you) on performing site audits and building new strictly SEO-minded sites. Read the tutorial →

How to redirect HTTPS to HTTP without certificate

Today we all got to know from Google officially, that HTTPS secured connection till now is a ranking signal for Google. Some of us run to buy one. Some other will try to get one for free, e.g. by StartSSL. While we don't know at the moment, how much weighs the secured connection as the ranking signal for Google, another issue exists for all website owners:  what if somebody tries to reach your site with HTTPS, but you haven't any certificate installed? Any browser will rise an error, something like ERR_SSL_PROTOCOL_ERROR. and the visitor will not see your site. What to do? If you will search for an answer, you will mostly get an answer, any redirection isn't possible, cause the SSL handshake happens before the whole connection, so any redirection will not work. But there is a little mighty thing named htaccess, which will allow us to make our sites visible for any visitor independently of with which URL the visitor tries to reach your sites. The trick is

30 cheat sheats for successful SEO

What a SEO needs to know?

The web is flooded with infografics and cheat sheets. Somebody meant once, such giveaways are good for SEO as linkbuilding assets and now everybody makes some. At least as copy and share. I will not speculate about whether or how many of them bring real value, imo most of them are redundant, but my personal biggest problem with them was - THE cheat sheet was NEVER present, if it was really needed (at least for me). Indeed, the sense and the convenience of cheat sheets is if they are there just in time, at the moment, whem one needs them. So i decided to create a collection of all cheat sheets i ever used on my SEO activities and share it. This cheat sheets suite is an evergreen knowledge, hints and tricks, which will be always helpful. Surely this knowledge isn't enough to call oneself an expert, but for somebody who does SEO, specially technical SEO and Onpage SEO, these cheat sheets will render a great service. And for somebody who learns SEO at the moment, they will give a great summary of things which must be learned. These cheat sheets cover already all essential knowledge segments a SEO brings daily into action. Befor publishing i reviewed all cheat sheets to find eventuallya fresher version - for some of them i finded one indeed.

How to minify javascript / css?

We are agreed, that assets like html, css and javascript must be better minified. If not, YSlow, Google Page Speed test and similar tools will give your site less scoring points and advice you to minify. It is enough said about importance of load optimizing for SEO, so lets look, what we can do and what we use to achive best possible result.
I tested 17 free tools to minify JavaScript online: after minifying with the best tool there are ca 43% of code remained,  the "worst" tool (not really worst) remained ca 51% of code. It worth also to minify. To minify CSS gives ca 60% of less code - It worth also to minify too. Read further - i list all tested online minifying tools with stats and a few server-side traffic-saving hints :