Here’s a tidbit of useful SEO information for everyone. It appears that Googlebot is attempting to become human by doing such wonderfully frustrating things like leaving feedback and reporting problems via forms. It’s almost as monumental as Data (from Star Trek: The Next Generation) receiving his emotion chip. While we are happy for Googlebot we see ourselves a lot like Picard going in with Data to fight the Borg . . . “perhaps you should shut off your emotion chip, Data” or in other words “perhaps you should stop trying to be so damn human Googlebot!”
For all you techies out there here’s an excerpt of an email from one of Tribune’s kick ass programmers, Tony Coconate:
http://theguide.orlandosentinel.com/lists/staff
[The above] URL has a few user-generated lists. If you click into those lists, then you can see a link that says “Report a problem”. For each list, people can report problems to our producers via a list of “flagged” content in the CMS. When our application sees that a piece of content, like a user list, has been flagged more than [x] times, we pull it automatically from the site. The problem is that Google has decided to follow those links, which are suppose to be POST requests …, but The Big G’s Bot is turning them into GET requests and managing to flag a lot of our content when it spiders the site. This problem is happening all over our site, not just for anchors (“a” tags) that are using JavaScript, but also div elements of the site that contain JavaScript links and actions (Ex. AJAX effects).
Sample code:
<a href=”#” onclick=”new Ajax.Request(‘/user_lists/flag/16′, {asynchronous:true, evalScripts:true}); return false;”>Report a problem</a>
…
—
We’re going to institute a ‘nofollow’. We think that should shut off Googlebot’s overly ambitious desire to be more human for now.
I’ve also seen Googlebot request URLs which it “found” in copy that looked kinda like a URL, so it’s not just javascript. Watch out especially for copy which includes forward slashes, preceded with quotes or apostrophes.