Changing your Firefox User Agent to Emulate Googlebot

The User-Agent field is used by various websites to identify when web crawlers such as Google's Googlebot and Yahoo!'s Slurp, visits their website. Interestingly, many webmasters naively use this to open up protected pages on their website (otherwise not visible without e.g. logging into the website) to Google, so it may mine and list those pages in its web search. Unfortunately, as the User-Agent can be easily modified, this form of authenticating provides almost no protection against spoofing attacks! In this video we will see how one can trivially change the User-agent string in Firefox and emulate Googlebot or any other web crawler's User-Agent string. The websites used in the below video are listed in the Links section below.

Enable Javascript or Download Flash Playe if you see this!

For commenting on this video Click Here!

Links:

1. Googlebot User-Agent string

2. HTTP Header dumper

Vivek Ramachandran is a security evangelist and has been working in computer security related fields for the past 7 years. In 2007, Vivek spoke at world renowned conferences Defcon (WEP Cloaking Exposed) and Toorcon (The Caffe Latte Attack). The discovery of the Caffe Latte Attack was covered by CBS5 news, BBC online, Network World etc news agencies.In 2006, Vivek was announced as one of winners of the Microsoft Security Shootout contest held in India among 65,000 participants. He has also been a recipient of a Team Achievement at Cisco Systems for his work on 802.1x and Port Security modules on the Catalyst 6500 switches. Currently he spends all of his time maintaining Security- Freak.Net , SecurityTube.Net and is the founder of an online startup (currently in stealth mode). Vivek, is a Bachelor in Electronics and Communications Engineering from the prestigious Indian Institute of Technology, Guwahati.You can contact him at vivek[at]securitytube.net