Now that we have seen a little about gaining trust from readers in our venture of secrets of successful blogging, let’s see how to tame those crawlers to obey our directives, after all they are our traffic officers.  Find below a set of action that I performed on this blog to be bot friendly though this may not be a complete list of what to do on gaining trust of those bots as algorithms change often; however, I’m confident that it throws at least some light for a beginner or somebody who is in trouble with supplemental indexing and less traffic. 

• Now that the whole world of Copybloggers, Probloggers and Daily Blog Tippers are blank and grudging themselves on PageRank downgrade, I would rather say “If you think Google hates your site, get over it. Algorithms don’t hold grudges.” A set of pages that appear in Google’s index today will vanish tomorrow and will surface again day after tomorrow. It’s a hide-and-seek game by those algorithm writers; else, they’re going to be out of business because of discontented searchers. Even if you write some minty fresh article that’s alien to the Milky Way or at least to the internet right now, you’re going to see it under Google’s minty fresh supplemental duff the very next minute, don’t know how they classify it as less important; Google being big brother on the internet, nobody can raise a question on its dictatorship (Democracy cannot be advocated here, Google being an Inc., lol)
• Similarly don’t worry about Alexa ranking. Given the fact that Alexa ranking for a site is calculated on the basis of how many visitors with the Alexa toolbar installed have visited that particular site, the results of traffic are definitely going to be inaccurate. Not every single visitor will have the Alexa toolbar installed on his browser. So there are millions of websites that, even if they have a lot of traffic, are not ranked high enough by Alexa.
• I started with a little knowledge about optimizing for those search engines which I shared in the post “Search engine optimization tips for bloggers,” you may refer that for throwing some light in that aspect but that is not the whole list of things.
• Age of a domain is the most important thing to bring in trust for search engines. As I said in the previous post, many blogs fade out or vanish in a few months from launch and crawlers too take into this account. Trust builds up only with growing age.
• I was liberal earlier to all those bots with few restrictions in robots.txt, but soon noticed that Yahoo was sending more traffic than Google which shouldn’t be the case. With further investigation, I could find all of my pages in Google’s supplemental index. I was just beginning with a few posts but the site command on Google was showing thousands of pages indexed by Google (with static and dynamic URLs) and all of them in the supplemental index. Then started doing trial and error in placing curbs to Google alone. It was just like training a sniffer dog to “Do this, don’t do that, don’t pee here, don’t shit there, sniff that corner” etc., and am still pulling my pages from the uncomfortable supplemental wedgie.
• I was displaying whole posts everywhere, i.e., in the home page, its continuation, categories, archives, etc., but while doing the revamp I accidentally came across the article “Use post excerpts everywhere” on Daily Blog Tips, thanks Daniel Scocco, I too switched to that idea. Except for single post page, I started using excerpts everywhere. You too make sure that a single content is inaccessible to the spiders with different URLs (even on your own domain). Else, even if your content is original, search engines will stamp it as duplicate and dump it under the supplemental rump. For eg., this blog’s home page can be accessed with at least eight URLs viz., mtherald.com, mtherald.com/index.php, mtherald.com/blog, mtherald.com/blog/index.php and the other four with a prefix www with these URLs. All of these URLs display the same content and a search engine bot (or at least the Google bot) will stamp the home page as a massive duplicate one. So with the proper use of robots.txt, restrict the robots to visit only any one URL for a single content. At the other URLs where the same contents can be found as in archives or categories, display only excerpts.
• Decide whether or not to use www in your URLs and be consistent about it. Be sure to set your preference in Google Webmaster Console.
• Prefer static URLs over dynamic URLs as search engines don’t like dynamic URLs. A dynamic URL is the one with variable characters like ?, &, =, etc.
  Example of a dynamic URL: http://www.mtherald.com/blog/thread.php?threadid=AX325&sort=date
  Example of a static URL: http://www.mtherald.com/blog/tame-crawlers-to-obey-your-directives
  Static URLs are typically ranked better in search engine results pages, and they are indexed more quickly than dynamic URLs. Static URLs are also easier for the end-user to view and understand what the page is about. If a user sees a URL in a search engine query that matches the title and description, they are more likely to click on that URL than one that doesn’t make sense to them.
• Prefer dashes over underscores in URLs. If you have a choice about how to structure your URLs, use dashes to indicate word breaks. If you have used underscores already, don’t bother making the switch but be sure to use only one of the two.
• Place this PHP code below the title tag in your header template:

  <?php if(is_single() || is_page() || is_home()) && ($paged <= “1″)) { ?>
  <meta name=”robots” content=”index,follow” />
  <?php } else { ?>
  <meta name=”robots” content=”noindex,follow” />
  <?php }?>

  Meaning of this code: You are directing the bots to only index the home page, single posts and pages. For the rest of the areas like home page continuation, archives, category etc., the bots should only follow the links but not index them. Great relief with pagination and all other nonsense links that used to get indexed in Google!

• Remove those admin/useless files indexed by Google with the aid of removal tool at webmaster console area of your Google account after duly blocking robot access to those files either with robots.txt or meta tag or 404/410 error page.
• With increase in the number of sites that point your pages, your authority too increases. Hence to gain such links voluntarily, submit articles to blog carnival.
• Similarly submit your blog to directories that offer free, one-way links. There are many out there on the internet. This way you can gain links without having to shelve a reciprocal link.
• Write at least two to three posts a week (I prefer two, the reason for which I described in the previous post) to keep search engine spiders visit your blog frequently; search engine spiders like indexing new contents. 

Let’s see some other tidbits on improving user/bot experience in our next post.