Topic: Page response time erratic
We have a Rails app that we host for several clients. Part of it uses web services calls. We use mongrels and apache load balancing.
We have a monitoring feature that looks into the production log to see how long each response took. These times look great.
We also have a requirement to monitor the response time outside of the log, so I have written a small script that does a wget on one of the web services urls. This is going great too. Most of the time.
The problem is that a few percent of the wget calls results in a long time before it recieves the response. Sometimes it is a couple of minutes. I trace the long calls through the apache log and the production.log, and the calls seem to get through at the same moment that the script makes them, and seem to get a response time in the production.log that is quite fast.
Where is the extra delay coming from? If the wget takes 120 seconds, and the production log says the mongrel took only 0.1 second to respond, where do I look for the problem? It happens even if I run the wget from the same server that is running the rails app.