[clug] googlebot doing funny things in logs

Hal Ashburner hal at ashburner.info
Tue Jun 14 20:59:56 MDT 2011


On 15/Jun/11 12:44 PM, Alex Satrapa wrote:
> On 15/06/2011, at 11:55 , Hal Ashburner wrote:
>> I changed my server on the weekend and after restoring the mythtv database forgot to put the usual
>> What web server are you using? Is there any indication from the server string that it's a MythTV installation?
Nope, it's a fairly vanilla fedora install.
> Telnet to your web service and issue the following:
>> GET /robots.txt HTTP/1.1
>>
>> (make sure you have two end-of-lines there)
>>
>> What does it return?
There is the appropriate robots.txt there now, however bad requests look 
like this:

HTTP/1.1 400 Bad Request
Date: Wed, 15 Jun 2011 02:48:23 GMT
Server: Apache/2.2.15 (Fedora)
Content-Length: 317
Connection: close
Content-Type: text/html; charset=iso-8859-1

etc.


More information about the linux mailing list