fcgi problem -- site down

Hi folks. First day of my site going live on dreamhost and it's been down twice already.

The site is ruby on rails using fcgi.

Seems to be a result of my killing the fcgi processes to pick up changes to controllers, etc.

Here's the restart script:
killall -USR1 dispatch.fcgi
echo 'done'


Here's the error in the error.log:
[Tue Jan 02 12:05:45 2007] [error] [client 205.175.225.22] FastCGI: comm with (dynamic) server "/home/wmnf_admin/wmnf.org/wmnf/public/dispatch.fcgi" aborted: (first read) idle timeout (60 sec), referer: http://www.google.com/search?hl=en&q=wmnf radio&btnG=Google Search
[Tue Jan 02 12:05:45 2007] [error] [client 205.175.225.22] FastCGI: incomplete headers (0 bytes) received from server "/home/wmnf_admin/wmnf.org/wmnf/public/dispatch.fcgi"

Before the site went live I used that restart script many times with no problems.

I've seen mentioned elsewhere that clearing the session files might help; tried that, no change.

I've run a console session and queried one of my models, and that returned results as expected, so it's not a coding issue or a database connection issue.

[edited to add crash log]
Here's the last few entries from the fcgi.crash.log. If someone can at least help me get fcgi started again, I would be very grateful.

[02/Jan/2007:06:42:16 :: 25734] starting
[02/Jan/2007:06:42:43 :: 16346] asked to terminate immediately
[02/Jan/2007:06:42:43 :: 16346] terminated by explicit exit
[02/Jan/2007:07:55:17 :: 23641] asked to terminate ASAP
[02/Jan/2007:07:55:17 :: 4262] asked to terminate ASAP
[02/Jan/2007:07:55:19 :: 4262] terminated gracefully
[02/Jan/2007:07:58:35 :: 23641] asked to terminate ASAP
[02/Jan/2007:08:09:35 :: 23641] asked to terminate ASAP

Any suggestions would be greatly appreciated.

thanks


Edited by matthewc on 01/02/07 01:36 PM (server time).

Hi Matthew,

I've had the same problem... I moved two of my sites http://windandtides.com and http://gearandboats.com to dreamhost over the weekend and they run fine for a while but after 12 hours or so I visit one of the sites to find out it's down with the "FastCGI: comm with (dynamic) server". Killing all of the running dispatch.fcgi processes and waiting a bit seems to bring it back to life.

I've been running one of the sites for over a year on my own machine under FastCGI with no problems so I know it's not something in my code, must be setup related. I have a request in with dreamhost support... if you figure your issue our let me know and I'll do the same if I resolve mine.

-Todd

My fcgi troubles seem to have settled down. It was up/down/up/down for a couple of days. The response I got from dreamhost support was, essentially, shared servers aren't that great for rails and you're on your own.

I posted in another rails forum, and scoured every wiki, forum, and blog post I could find. Made several changes, which may help someone else:

.htaccess --
Made some changes here to allow access to the stats page; here's the whole file:
# General Apache options
AddHandler fastcgi-script .fcgi
AddHandler cgi-script .cgi
Options +FollowSymLinks +ExecCGI

... comments removed ....


RewriteEngine On
# enable analog stats pages
RewriteCond %{REQUEST_URI} ^/stats/(.*)$ [OR]
RewriteCond %{REQUEST_URI} ^/failed_auth.html$
RewriteRule ^.*$ - [L]

# If your Rails application is accessed via an Alias directive,
# then you MUST also set the RewriteBase in this htaccess file.
#
# Example:
# Alias /myrailsapp /path/to/myrailsapp/public
# RewriteBase /myrailsapp

RewriteRule ^$ index.html [QSA]
RewriteRule ^([^.]+)$ $1.html [QSA]
RewriteCond %{REQUEST_URI} ^/stats/(.*)$ [OR]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ dispatch.fcgi [QSA,L]

# In case Rails experiences terminal errors
# Instead of displaying this message you can supply a file here which will be re
ndered instead
#
# Example:
# ErrorDocument 500 /500.html

ErrorDocument 500 "<h2>Application error</h2>Rails application failed to start p
roperly"



dispatch.fcgi: two changes here. One, limit the number of processes to 6. Two, use the frao_handler to restart gracefully. Entire file:

#!/usr/bin/ruby1.8
... comments removed ...
#
# # Custom log path, normal GC behavior.
# RailsFCGIHandler.process! '/var/log/myapp_fcgi_crash.log'
#
require File.dirname(__FILE__) + "/../config/environment"
require 'fcgi_handler'
### add this if we're getting 500 errors
class RailsFCGIHandler
private
def frao_handler(signal)
dispatcher_log :info, "asked to terminate immediately"
dispatcher_log :info, "frao handler working its magic!"
restart_handler(signal)
end
alias_method :exit_now_handler, :frao_handler
end

RailsFCGIHandler.process! nil, 6

Ok, I take it back. My site has been down 4 times in 3 days, without any code uploads/restarts which seem to have triggered it in the past.

Fcgi just dies; when I restart by killing all the fcgi and ruby processes, it takes some unpredictable amount of time for them to restart.

They should start immediately upon a request coming in.

Any help with this would be most appreciated.

Hi Matt!

I'm having very similar problems. Did you figure anything else out?

Please let me know.

Thanks!

Terry

Hi Terry,

No, not really. DH support replied that my fcgi processes were taking too much memory and so were being killed. That's part of the problem; the bigger problem is that they don't start up again until sometimes 10-15 minutes has passed.

Unfortunately I'm moving to a different host. DH has great bang for the buck and I'm sure they're fine for other applications, but I've found that rails isn't really working that well in the shared environment there. They weren't able to offer any suggestions on getting things stable, and I've tried all the suggestions I've found on forums etc.

Matt,

I think I've solved the problem. I haven't received any application errors at all in the past 3 or 4 days since making some changes. I'm going to document them and post them on my rails blog at http://www.rubynoob.com

Hopefully I'll get it posted today or tommorow.

Terry

Thanks for the news. When you do get it up, could you post one last followup here?

Wholly

I mean this in the most constructive of possible ways, and I'm very interested in what you found, but I just tried to go to your blog and got an application error page..