How to view a webpage like a Search engine crawler

I have come across many situations where we SEO professionals need to test a website (mostly when we are launching new pages) and would require to access the webpages in the same way a search engine crawler will do. Many a time’s software engineers feel that crawlers will see the same webpage as you see in a browser window. And my answer to that is ‘Not always’.
Nowadays the new web applications are complex and web developers can introduce some logic which can block crawlers (Most of the time when you add some script to check user agents – to deal with cross browser compatibility). It’s also seen that load balancers and servers can block search engine crawlers accidently (Most of the time when you want to block certain user agents – Scraping sites or certain search engines which you don’t want to get index – like Baidu ).
The best way to test your website for Search engine accessibility is to test with the search engine user agent in Mozilla Firefox. Here is the step by step procedure to start your testing:
1. Get the user agent switcher for Firefox.
I prefer using the add-on “User Agent Switcher” which can be downloaded from here .
If you access that link from Firefox itself, its a one click install to get it added to your browser.


2. Change the user agent in your browser
After you install the add-on, go to Firefox option and you can find the add-on user agent switcher added to the list of other add-ons. This add-on comes with a preset Google bot user agent that you can select.If you don’t want to select the existing user agent (sometimes they can get outdated when Google changes their user agents), I recommend you edit and add your own user agents.

For example, the latest user agents for major search engines are :
Google :
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Bing :
msnbot-webmaster/1.0 (+http://search.msn.com/msnbot.htm)
Yahoo:
Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)
Change the description and user agent entry as shown in the image below.

3. Start visiting WebPages without closing the browser

While being in this setting, any pages that you visit will be same as a crawler seeing when analyzing your page content.You can make sure that your pages are loaded completely and all the important content related section of the pages are looking good.

Possibly Related Posts:


Tags: , , , ,

3 comments

  1. […] How to view a webpage like a Search engine crawler […]

  2. Excellent post thank you so much this will come in very handy.

  3. this is very nice
    i used this and found it very usefull
    thanks

Leave a Reply