subdomain shut down due to phpbb

I set up a subdomain (mysubdomain.dreamhosters.com) and did the one-click install of phpBB for use by me and 3 other friends. After about a week of using the message board, I get this email from support@dreamhost.com that says:

"Unfortunately I've had to disable your site, <mysubdomain>.dreamhosters.com, this afternoon because it's been causing severe instability issues on the server. Your <mysubdomain> user's php.cgi scripts have been taking down the machine on a daily basis. I'd recommend taking a look at your access.log file to see where all the php requests are coming from (and what php script they're requesting) and then see if you can figure out the problem."

I have no idea what to look for in the access.log file. All I did was the one-click install of phpBB and then did a basic setup. There are a total of 4 users on the message board and we spend maybe an hour or two a day each reading and posting. I replied to the tech support email asking more questions and all I got back was:

"All I can tell you is that the daemon we run to monitor runaway procs killed your php.cgi's over 7000 times just this morning."

Like that helps me (roll eyes). I'm a novice at this and don't even know what php.cgi stands for.

Any help on what to do or ask next?

Thanks.

Well, you could probably ask myself or the other gurus to look into this for you.

Download the access logs using FTP. They should be in the "logs/domain/http.nnnn" directory, like so:

access.log
access.log.0
access.log.2005-10-12
access.log.2005-10-11.gz

Do not download access.log.0 - it is simply a link to access.log

Rename the ones without the ".gz" on the end to ".txt":
access.log.txt
access.log.2005-10-12.txt
access.log.2005-10-11.gz[/pre]

And then e-mail them to me at dh-forum@openvein.org

Then I could find out what script is being requested and from where. Hopefully you just need to patch phpbb.

cool Perl / MySQL / HTML+CSS

Sent.

Thanks.

Well I don't see any attempts to exploit a vulnerability.

The only curious thing is someone who is hiding their browser name and version is hitting several pages (faq.php, search.php, etc) at the same time. I believe this to be possible be more than one person - they appear to sometimes use Firefox 1.0.7; however most of the time whatever browser they are using (not Firefox) is following links wrong:

/forum/viewtopic.php?t=53&amp;view=next&amp;sid=hexadecimal

The HTML entity should not appear in Apache log. This is most likely a bot or other automated process, like a bad proxy.

Why this would cause a high server load eludes me. There are not 7,000 requests for PHP scripts in the 5 days logs that you sent me, much less a single day. On 2005-10-11, there were 3,363 log entries and only 1,605 had to do with PHP scripts.

Did you install any phpbb mods after you completed the one-click install?

cool Perl / MySQL / HTML CSSEdited by Atropos7 on 10/14/05 01:09 AM (server time).

No mods at all. I did the one-click install and took the defaults during the installation. I restored the database from a previous installation then added a few avatars to the avatar library. That's all. I started this message board in a folder in my personal domain but had moved it to a subdomain of dreamhosters.com. To do so, I had to install phpBB in the subdomain, back up the database in the original installation, then restore that into the new location. I believe that only brings over the user accounts and threads/posts. There were no mods on the original installation either.

I don't know if this could be related but we have noticed that the board seems to hang on occasion. Like it will just freeze and timeout on us. When that happens, we just come back maybe an hour later and things seem fine. I even started a thread on it before. http://discussion.dreamhost.com/showflat.pl?Cat=&Board=3rdparty&Number=30260&page=5&view=collapsed&sb=5&o=186&part=

Would it help if I made the message board private? Meaning only the 4 of us can view it and we would have to be logged in? Use a different message board software? I'd hate to lose the threads we have now.

What happens now? Do I email DH Tech Support and tell them to reactivate my subdomain so I can run the message board again as a private board? Install a different message board software?

Thanks for checking the logs, and replying to my post.

In reply to:
Would it help if I made the message board private? Meaning only the 4 of us can view it and we would have to be logged in? Use a different message board software? I'd hate to lose the threads we have now.

I doubt making it private would help - there were very few unique IPs visiting, so its not like you were getting too much traffic, much less DOS/unwanted traffic. I'm leaning toward a bug in phpbb - perhaps an infinite loop or a poorly thought out mysql query, considering the observation that it hangs on occasion.

In reply to:
What happens now? Do I email DH Tech Support and tell them to reactivate my subdomain so I can run the message board again as a private board? Install a different message board software?

I would try letting support know you don't appear to be the target of DOS or vulnerability exploits and that it might be a bug in phpbb. And ask them if you can install it yourself from the phpbb site instead of using the one-click install; perhaps if it is a bug it has been fixed. Delete all the old phpbb files before re-installing.

Were you using PHP-Apache or PHP-CGI, and which version, 4 or 5? Do you know if the machine was running Debian Sarge?

cool Perl / MySQL / HTML+CSS

> Do not download access.log.0 - it is simply a link to access.log

Actually, unless your server is different, access.log is the most recent (i.e. current day) log, and access.log.0 is linked to yesterday's complete day log, so they are different. Here's from my ls:

access.log.0 -> access.log.2005-10-14

> Rename the ones without the ".gz" on the end to ".txt"

Is that a w1nd0s/Luckout problem?

Cheers,

BobS

> I'd recommend taking a look at your access.log file to see where all the php requests are coming from (and what php script they're requesting)

You might also look at

<mysubdomain>.dreamhosters.com/stats/resources/

and the files, your-user.sa.analyzed.x, to see if those confirm the story, or narrow down the problem.

Good luck,

BobS

One last thing from me :-)

You could always tell Support: Thanks, you looked at the logs, and confirmed it was
caused by the comment spam problem mentioned in DH's blog, so turn the subdomain back on now...

Good luck,

BobS

Tech Support responded to my email and said they will activate it again. They said they haven't had problems with other users who did the one-click install of phpBB. *shrug*

So, I deleted everything in the subdomain and did the one-click install again. I've told my 3 users that I will start from a fresh empty message board again. They're ok with it. We'll see how it goes.

>Were you using PHP-Apache or PHP-CGI, and which version, 4 or 5? Do you know if the machine was running Debian Sarge?

That went whoosh! over my head.

In reply to:
Actually, unless your server is different, access.log is the most recent (i.e. current day) log, and access.log.0 is linked to yesterday's complete day log, so they are different.

Oops, you're right.

In reply to:
Is that a w1nd0s/Luckout problem?

Yeah, I don't use Outlook myself so not sure how well it would handle a file attachment whose filename did not indicate that it was text/plain. I do have .txt associated with a text editor though.

cool Perl / MySQL / HTML CSS

In reply to:
>Were you using PHP-Apache or PHP-CGI, and which version, 4 or 5? Do you know if the machine was running Debian Sarge?

That went whoosh! over my head.

On the web panel, go to Domains -> Manage and click on the "Edit" link next to the domain.

Fill out the checkboxes on this page as you seem the in the webpanel:
http://atropos.openvein.org/web/dreamhost/panel/domain.html
And when you click on the button on my page, the Results area will tell summarize the settings for you if you have JavaScript enabled.

cool Perl / MySQL / HTML+CSS

I did the checkboxes as you showed in your example.

Thanks!

Actually, I meant for you to use it to tell me what settings you have setup, in case that might have caused the problem with phpbb.

cool Perl / MySQL / HTML+CSS

Oh. I misunderstood. When I set the checkboxes to how I had it before, the results are:

Your domain is configured to run PHP version 4 as CGI. The Apache module mod_security has been enabled. The Apache module mod_fastcgi has been disabled.

I just set it back to the way it was. Should I run it as PHP Version 5 or Version 4?

You were right. DH Tech Support emailed me and said when they moved someone else off the server I was on, the problems went away. The server where they moved this other customer is now crashing all the time. My stuff wasn't the culprit after all.

Thanks again.

subdomain shut down due to phpbb