[clug] googlebot doing funny things in logs

Hal Ashburner hal at ashburner.info
Tue Jun 14 23:43:43 MDT 2011

On 15/Jun/11 2:07 PM, Angus Gratton wrote:
> On Wed, 2011-06-15 at 13:08 +1000, Hal Ashburner wrote:
>>> Second possibility I can think of is you (or someone who accesses your
>>> Mythweb) uses the Google Toolbar with the Google History feature[1], and
>>> sent those URLs to Google as they were visited.
>> I really don't /think/ so. I do use chrome and have 2 android devices.
> Oops, missed this line the first time around. Do you use any Chrome
> Synchronisation features? How about the Google Safe Browsing Service
> (which is on by default I think) together with anonymous Usage
> Statistics? On second look, there are quite a few clauses in the Chrome
> privacy policy where it may sometimes send out a URL and/or 'certain
> other data'.
> As for Android, a quick Google didn't find any reference to the Android
> browsers' privacy policies or any "phone home" features in them...

Well there's certainly plenty of pages there that google attempted to 
scrape and seemingly failed to that I've never been to.
It seems more likely that either:
a) somehow it was left unpassworded on setup for a brief time and with 
no robots.txt googlebog snarfed the lot (but didn't cache any of it).
b) googlebot is doing some "research" (similar to the kind google did 
with wireless when photographing streetview that caused all the hoo-har?)

I don't recall ever linking mythweb to my front page. Can't swear to it, 

My usual mythweb setup over the past 18 odd months has not used the 
/mythweb path but instead something like

As far as google watching my browser is concerned, If I've agreed to 
share my private browsing habits with google by using chrome and/or 
android I surely was not aware of this. It would be a total deal-breaker 
and I'd consider it morally unconscionable if Google slipped a submarine 
clause in the fine print. You'd have to ask a lawyer if it would also be 
legally unconscionable conduct. Either way my gut feeling is that the 
Goog would not be odiously and stupidly contemptuous of its customers. 
It just doesn't add up. I would like to know that for sure but wouldn't 
know how to get a straight answer from the impenetrable monolith that is 
Goog from someone who knows and is saying rather than a PR who doesn't 
but will use rather a large volume of words to not say so.

More information about the linux mailing list