[clug] googlebot doing funny things in logs
gus at projectgus.com
Tue Jun 14 21:58:18 MDT 2011
On Wed, 2011-06-15 at 13:08 +1000, Hal Ashburner wrote:
> > http://www.google.com.au/search?q=site:ashburner.info&hl=en&prmd=ivns&ei=QRz4TdDXAuTZiAK85an9DA&start=0&sa=N&filter=0&biw=1400&bih=698
> Most of these pages I didn't actually know for certain (and some at all)
> exited and I'm pretty sure I've never visited them.
> I've learned something about mythweb features just by looking through
> the 4 pages of links google lists there.
Well, that's definitely weird. Did you ever run mythweb without a
password set (even briefly)?
If GoogleBot got one good look at one of the pages it could easily have
scraped all the other links just from there once upon a time, but the
question does still remain about how it found it...
> Nobody else has access.
>Someone who isn't me? I don't think anyone else has used it...
Can I ask what browser/OS combos you use to access it?
Did you ever (even briefly) have ashburner.info serving up / as a
directory, without an index page?
How far back do you keep your apache logs? Would you be able to find the
first ever time GoogleBot accessed /mythweb ?
More information about the linux