[clug] googlebot doing funny things in logs

[Angus Gratton]

On Wed, 2011-06-15 at 13:08 +1000, Hal Ashburner wrote:
> >
> > http://www.google.com.au/search?q=site:ashburner.info&hl=en&prmd=ivns&ei=QRz4TdDXAuTZiAK85an9DA&start=0&sa=N&filter=0&biw=1400&bih=698
> >
> Most of these pages I didn't actually know for certain (and some at all) 
> exited and I'm pretty sure I've never visited them.
> I've learned something about mythweb features just by looking through 
> the 4 pages of links google lists there.

Well, that's definitely weird. Did you ever run mythweb without a
password set (even briefly)?

If GoogleBot got one good look at one of the pages it could easily have
scraped all the other links just from there once upon a time, but the
question does still remain about how it found it...

> Nobody else has access.
>Someone who isn't me? I don't think anyone else has used it...

Can I ask what browser/OS combos you use to access it?

Did you ever (even briefly) have ashburner.info serving up / as a
directory, without an index page?

How far back do you keep your apache logs? Would you be able to find the
first ever time GoogleBot accessed /mythweb ?


- Angus