Topic: All mongrels crash after intensive data gathering request begins
I have a 512Mb slice on Slicehost. I've got 3 Mongrels running on nginx. If I start an intensive data gathering process (specifically, gathering data from Wikipedia via their API), all other requests to the box are not serviced. Eventually, nginx times out and I have to restart the Mongrels to get requests to be processed again. It seems to me like all 3 Mongrels are being used by this intensive process, but can that be right?
I've tried making the intensive task less intensive by putting in 3 second sleep periods between each API request, but to no avail.
Has anyone any ideas?
Thanks very much.