500 error using fastcgi + rails

I see a possible issue with rails and fastcgi, as posted in textdrive: http://forum.textdrive.com/viewtopic.php?id=1467

I am getting a similar situation where my site runs fine other than seemingly random 500 errors. Here's an example of my log:

[Mon Aug 15 14:51:21 2005] [error] [client xxxx] FastCGI: incomplete headers (0 bytes) received from server "/home/nerdbuc/gladiator/public/dispatch.fcgi"

Since things run fine most of the time, I can only assume this issue is linked to the issue posted in text drive where fastcgi thinks it's run out of file handles and just spews a 500 error. Any ideas how to get around this? I doubt I'll be able to run my rails app in normal cgi mode without being asked to switch servers (assuming it ends up being popular), which is not an option...

I think the key to the textpattern thread is this post:

"Jason kindly checked out the error.log and unfortunately found no errors relating to fcgi or apache. The only entries in there are 'not found' links relating to my old blog which ran textpattern e.g the url http://www.mpet45.co.uk/article/559/whatever which generates an error in RoR given that there is no such url anymore."

I was getting a bunch of 500 errors due to not having a robots.txt file. Any time a request comes in that you have no route for, and the url doesn't fit the default route, you'll get 500 errors.

I also got them from my 500 page itself as I had a stylesheet linked to it using a relative url. When the 500 page was generated from other than the top level of the site, the stylesheet could not be found and that generated another 500 error. (Changing it to /stylesheets/my.css fixed it -- that is, adding the leading slash.)

In the end, many, many of the 500 errors are tracked down to some little thing in the application. However, I also found a new fcgi_handler yesterday that I've been using the last 24 hours and it seems to eliminate those last few. At least I haven't seen once since installing it. It really seems stable and fast:

http://convergentarts.com/articles/2005/08/16/new-fcgi_handler-rb-cuts-500-errors-or-so

The processes seem to get the shutdown notice and are able to gracefully exit when they are not in the middle of a process (which I think is what would happen in the prior version).

I think with the old handler, once a process got broken in some way it didn't go away until it was in the middle of another request. I think this handler sets them to expire before handling the request so a good process can take over.

hth,
Tom


-Tom Wilcoxen
convergent arts

No, my application will get 500 errors on pages that work when you refresh them. I do make my share of stupid errors, but this one was definitely not something I was doing... or at least not something obvious. I could hit a page without problems, then refresh and get an error, then refresh and see the error go away.

However, the link you posted isn't all that helpful... the poster mentions a new fcgi_handler, but links to the dispatch.fcgi, which is exactly the same as the one I already have. I don't know the rails code structure well enough to know how to find the correct file. Could you point me to the real one?

Thanks!

DOH! The poster was me. Here's the correct url:

http://dev.rubyonrails.org/svn/rails/trunk/railties/lib/fcgi_handler.rb

I corrected it on my blog. Sorry about that!

-Tom

-Tom Wilcoxen
convergent arts

Heh, didn't realize that was you. I'm not the most observant person, that's for sure. Thanks for the update.

So I just put that into my app's lib directory and it'll pick it up, I take it?

EDIT: Nevermind, read your comments on your post and realized that's exactly what I am supposed to do. Thanks for all your help.Edited by nerdmaster on 08/19/05 01:58 PM (server time).

:)

Yes, if you want to expedite the change you can go to the shell and:

killall -9 dispatch.fcgi

That'll reload them right away.

-Tom Wilcoxen
convergent arts

This exact thing is happening to me, and it's as annoying as all-get-out. In fact, I'm on the verge of losing some programming work because it's just about impossible to get anything done.

I'll get 500 errors at random. I'll hit refresh, and for several minutes, I'll keep getting errors. Then, the pages will suddenly start working again (just refreshing a list action, mind you -- no form submissions or anything like that). That'll last a few minutes, and then I'm back to the 500 errors.

I've tried going through dreamhost's support, and all I get back is "the URLs resolve for me". But clearly, looking through this forum and elsewhere, there is something going on.

I'm at my wit's end about this, and am just about to go back to php (as much as I love everything that I've done and seen with RoR) just to keep from losing my customers. Either that or find another host, stat.

I too have had random 500 errors and lots of fgci crash log entries...

I've been having the same annoying, random 500 errors from my rails app. It's funny though: There are two rails apps running under the same domain, one has issues, one does not.

So, I put the new fcgi_handler in my lib directory, but don't want to wait around for it to reload (when does that happen anyways?)

But, i've had some bad experiences with the killall command -- last time i used it it killed my dispatch.fcgi and it never restarted. If I have sub1.domain.com and sub2.domain.com, is there any way to kill the processes for sub2.comain.com while leaving the other intact?

You can use:

killall -USR1 dispatch.fcgi

That will tell all the processes to go away next time they become idle. In my experience though, those processes never go out of your process list. They won't get dealt any more requests though, but just hang around.

I don't know of any way to stop only one app's processes. You could probably rename the dispatch files to be dispatch_1.rb, dispatch_2.rb to differentiate them (I'm guessing).

Finally, you might try running the app that has more errors from the command line:

ruby ./dispatch.fcgi

See if it loads ok or gives any errors. And check the logs to wipe out any application errors you have. That will settle it down the most.

The random errors are caused because one or more dispatch.fcgi processes gets corrupted and is later dealt a request. The way the processes get corrupted is from application errors, so squashing those is the long term solution. (Or making fastCGI processes more robust somehow.)

-Tom Wilcoxen
convergent arts

running from cmd line i get

[rosecrans]$ ruby ./dispatch.fcgi
Content-Type: text/html
Set-Cookie: _session_id=b89b0d890509d0a18462a68923058d71; path=/
Status: 200 OK
Cache-Control: no-cache

<html><body><h1>Application error (Rails)</h1></body></html>[rosecrans]$

Additionally, my production.log is full of only (200 OK)'s, except for an error generated by running from the command line. The error looks like this:

TypeError (can't convert nil into String):
/usr/lib/ruby/gems/1.8/gems/actionpack-1.9.1/lib/action_controller/base.rb:807:in `+'
/usr/lib/ruby/gems/1.8/gems/actionpack-1.9.1/lib/action_controller/base.rb:807:in `complete_request_uri'
/usr/lib/ruby/gems/1.8/gems/actionpack-1.9.1/lib/action_controller/benchmarking.rb:45:in `perform_action_without_rescue'
/usr/lib/ruby/gems/1.8/gems/actionpack-1.9.1/lib/action_controller/rescue.rb:80:in `perform_action'
/usr/lib/ruby/gems/1.8/gems/actionpack-1.9.1/lib/action_controller/base.rb:356:in `send'
/usr/lib/ruby/gems/1.8/gems/actionpack-1.9.1/lib/action_controller/base.rb:356:in `process'
/usr/lib/ruby/gems/1.8/gems/rails-0.13.1/lib/dispatcher.rb:32:in `dispatch'
/lib/fcgi_handler.rb:150:in `process_request'
/lib/fcgi_handler.rb:64:in `process!'
/lib/fcgi_handler.rb:55:in `each_cgi'
/lib/fcgi_handler.rb:55:in `process!'
/lib/fcgi_handler.rb:22:in `process!'
./dispatch.fcgi:24

This doesn't mean much to me, does it man anything to you?

Well, it's probably nothing -- just the result of calling dispatch.fcgi without actually being a client with a well-formed request.

You can find the line where the error is:

  /usr/lib/ruby/gems/1.8/gems/actionpack-1.9.1/lib/action_controller/base.rb:807:in `+'

Which leads to this code in action_controller/base.rb:

  def complete_request_uri
    request.protocol + request.host + request.request_uri
  end

It is trying to apply the + operator to request.host which is nil.

So anyhow, I'd just keep an eye on your production log and fast cgi crash log. If you see an entry in the crash log, look at the production log for that time and see if there's anything to explain it.


-Tom Wilcoxen
convergent arts

Quick Question:

What does it mean if you get the 500 error, there is an entry in the log/fcgi.crash.log, but no error entry in the log/production.log?