[clug] googlebot doing funny things in logs

[Hal Ashburner]

On 15/Jun/11 1:58 PM, Angus Gratton wrote:
> On Wed, 2011-06-15 at 13:08 +1000, Hal Ashburner wrote:
>>> http://www.google.com.au/search?q=site:ashburner.info&hl=en&prmd=ivns&ei=QRz4TdDXAuTZiAK85an9DA&start=0&sa=N&filter=0&biw=1400&bih=698
>>>
>> Most of these pages I didn't actually know for certain (and some at all)
>> exited and I'm pretty sure I've never visited them.
>> I've learned something about mythweb features just by looking through
>> the 4 pages of links google lists there.
> Well, that's definitely weird. Did you ever run mythweb without a
> password set (even briefly)?
Can't swear to it. Might have happened during setup and migration of 
config from the previous box. Or original setup 2 years or so ago. I 
don't even remember exactly when I did it.
> If GoogleBot got one good look at one of the pages it could easily have
> scraped all the other links just from there once upon a time, but the
> question does still remain about how it found it...
Maybe this is most likely despite the nagging doubt in my mind.
Did I link it from the front page 2 years ago for half an hour or so 
when originally setting it up? Possible. Sounds daft but I can't claim 
I've never done anything daft. ;-)
>> Nobody else has access.
>> Someone who isn't me? I don't think anyone else has used it...
> Can I ask what browser/OS combos you use to access it?
Android 2.2 default browser, android opera
fedora 13 firefox, chrome, epiphany
fedora 12 firefox, chrome, epiphany
OSX firefox, chrome
N900 (I've forgotten the browser, the phone got pinched and android & 
win7 look to have destroyed mee-go in the minds of nokia management, who 
hired a microsoftie to trash everything they've done and do microsoft - 
but will it work?)

> Did you ever (even briefly) have ashburner.info serving up / as a
> directory, without an index page?
Not that I recall, definitely not on this machine.
> How far back do you keep your apache logs? Would you be able to find the
> first ever time GoogleBot accessed /mythweb ?\
not far. The old machine is unplugged in the corner but was in use for a 
good few years before the addition of myth, I didn't bother making an 
effort to back up logs.

Dunno if this is going to get anywhere.
Thanks for all your thoughts.
Hal