Topic: Error notifications with crawlers


Let me introduce myself, I'm a web developper working with Rails for 6 months in a two guys company, we just love RoR smile

We published a couple of website, and on both of them we got a weird problem in production.

We installed the exception notification plugin, which sends us an e-mail every time an error occurs.

We receive almost every day errors, that always comes from crawlers with UltraSpider3000. The URL called is the homepage which works fine in a browser :

You'll find bellow the notification we reveive, do you have any idea what happens ?

Thanks a lot for your help,


edit: i removed the logs

Last edited by iMatt (2011-05-05 06:14:32)

Re: Error notifications with crawlers

Hi again,

I think I found the error, the crawler's header mentioned HTTP_ACCEPT: text/* (insteed of text/html).

By adding respond_to in my controllers I prevent RoR to search for a view.txt.erb

Hope it's clear smile