How do I LIMIT traffic to my site?

I'm not exactly sure where to search for help on this, so thought I'd make a post in the forum.

I set up with Dreamhost back in April and overall I've been very happy with the service. I run a blog that has slowly been gaining in traffic week on week.

Occassionally one of our contributors will post a story that gets picked up by other blogs and we see a temporary spike in traffic. I usually dies back down after a day or two, but we're still grateful for the traffic it brings, because a few of those people hang around and become regulars.

Today, however, Dreamhost took my site offline after one of our stories got picked up by quite a few other blogs and the traffic started to approach 5k visitors an hour. The reason I was given is that we had too many simultaneous users, and we were taking up 35% of the server.

While I understand that we're on a shared server, and realize that I can only take up so many connections at a time, how do I know what the threshold is, and how do I plan for that? Ideally, I'd need some way of turning off a switch myself before things get out of hand. because now I'm sitting here with no website, and I'm kind of at the mercy of Dreamhost as to when/if it's turned back on again. The only thing I can think of is, not to write any stories that will be too popular, which is clearly an absurd solution. Moving to some kind of dedicated server is also not really an option because, after the spike goes down, I know our traffic will be back down to it's low regular size (which is probably no more than 7-10,000 visitors a day. A spike sees that shoot up to maybe 35-40k visits a day). This is just an amateur site - we make a couple of bucks a day in ad revenue from Adsense, so no way can I afford $100 a month on a dedicated server. I'm so under our bandwidth quota I don't see us EVER using all that up.

Anybody have any ideas on what we do now? Anyone here had a similar problem?

Thanks

Steve

This is my guess, and I coudl be wrong. Support should be very wielling to help you with this, so here are my sigguestiosn...

I think that your best bet would be to work on way to stream-line your website, so that when people view your webiste it takes fewer server resources to send out the page.

For instance, Wordpress because of it's dynamic content system, pulls information from the datbase, and layout files to assemble the page the user sees, for every page load. This obviously wold take up a lot more resources than just feedign out a singe HTML page, and maybe a seperate CSS page.

Also, you might consider designing your site, (possibly using php includes) so that only the content of the page reloads and not the parts of the page like navigation and other stationary objects. I allways used to use I frames for this, but I'm told this by my wife that this is a simply horrible concept that is SO outdated.

One last thing, I'm concerned that you said that "Today, however, Dreamhost took my site offline..." It was, as I understood, dreamhost's policy to move persons such as your self to 'testing' servers so you site would re-main functional while you worked out the issues, and then either be retured to a regular server, or upgrade to dedicated. Please expand a little as to what all went on, if you don't mind.

Hope this is helpful.



-Matttail

I will have to wait until my site gets back up before I can assess the filesize of that particular page. My site is generated by MT, and the pages are served dynamically from MYSQL.

The chain of events was as follows;

1 - at about 5.00pm EST I noticed my site was not responding, so I emailed customer support.
2 - at about 5.20, support responded saying it looked like heavy server load, and they were trying to figure it out
3 - at 7.56, they emailed again saying that it was one particular post on my site that users were requesting. They then said they had temporarily disabled my site, and would not reinstate until I deleted that file
4 - at 8.19 I emailed them to say I had disabled the offending post, and asking for my site to be switched back on.

That was nearly 4 hours ago - still waiting for them to put my site live again. They did not offer me the option of moving to a testing server - they just disabled the site with no warning. Very disarming.

I agree, that is very disappointing to hear of support. I hope your site is turned back on soon.

Pershaps you should press support for a little more help and sigguestions as to what to do about your CPU usage - if they're enforcing rules about this, they need to be helpful about getting things worked out and not just leave it up to the end user to figure out on their own.

good luck...



-Matttail

My site is generated by MT, and the pages are served dynamically from MYSQL.

I'm not very familiar with Movable Type, but I believe it generates static HTML pages by default. Requests for static HTML pages cost nearly nothing in terms of server resources, unlike dynamic PHP/MySQL-generated pages. Switching back to static pages just may fix the problem.

Your site is getting traffic, so we're going to take down your site until you stop getting traffic...

lame!

Note that we will only completely take down websites when there is no other option. If a site is taking down a server hosting hundreds of other users we opt to take down one site and restore service to the others.

If it is possible we move the site to another server and communicate with the user to figure out what might be going on. That is not always possible, unfortunately.

- Dallas
- DreamHost Head Honcho/Founder

Static mode is indeed the default for MT installs (and until 3.1, the only option). if you want to confirm that's how you're running it, check out http://www.sixapart.com/movabletype/docs/3.2/06_publishing/ .

> Note that we will only completely take down websites when there
> is no other option. If a site is taking down a server ...

What about the option of DH doiing something to protect the server from being taken down by one site?

What we now have is a situtation where DH have imposed a CPU limit but provided no way for the site owner to cap his site within that, leaving every site at risk of being completely shut down by DH because of a peak in load. Very poor.

In a shared hosting environment some websites will use more of the available resources than others. There's no reason for us to limit every website to exactly the same amount of resources as that doesn't best meet the needs of the majority of our users.

- Dallas
- DreamHost Head Honcho/Founder

Well fine, but it is no consolation to the site owner to know he's in the majority whose needs you are meeting... at times when he's not in the minority whose site's you've taken down.

But regardless, how can we protect our sites against being taken down by DH?

Very few sites will ever be in danger of being taken down for server resource use. Even in the case of busy sites we are almost always able to notify the site owner when we start to see a problem so things can be worked out without the site ever being taken down. It is only in extreme situations that we are forced to disable a website and when we do we also immediately notify the site owner about the situation. If the site owner gets back to us right away the site will be restored very soon after and we will work with them to isolate the problem and resolve it.

Sometimes spikes in resource use for a website can be caused by an overzealous search engine spider so having an appropriate robots.txt can help prevent those. Spikes are also sometimes caused by blog comment spammers. Having an appropriate anti-spam setup in your blog will help prevent that.



- Dallas
- DreamHost Head Honcho/Founder

Aha. Does that mean there's no way to protect our sites against being taken down by DH?

This is slightly off-topic, but related. I just had my site taken down because of extreme use of cpu-time on teh MySQL server. I was given 48 hours to reduce from 1600+ cpu-minutes to 10. Since I was on holiday the site had been taken offline by the time I read the message. This is similar because there was a limit I knew nothing about, and when I crossed it my site was taken offline.

Now I don't object to their being a limit, but it would have been nice to know a while back, when the problem was not so severe, that there was a problem. This would have meant that I could have dealt with code / database changes well in advance, without disrupting my users, rather than having a site down for 5 days and having to spend three solid days intensively changing code.

Really I found the whole way the site was taken down to be very abrupt and unplesant. It would be nice to have the feeling that the support teams could see problems coming and help work with site owners to avoid them, rather than doing desperate damage control when its all hit the fan.

I've passed these comments along to our technical team. We do have tools that monitor database usage and tell us about potential problems so those might be able to send out some automated notifications as well.

- Dallas
- DreamHost Head Honcho/Founder

Static mode is good, but the big MT resource drainer is usually mt-comments.cgi which gets run any time a comment is added, but also every time a comment is viewed.

DreamHost does have a system in place to warn people of high resource usage on the web servers side. Every monday warnings are sent out to people with overly high resource usage.

In regards to helping out the "end user", DreamHost really can't hold the hands of people who install software or drive tons of traffic which ends up taking down a machine with many other users on it. In that situation they have no other choice but to disable the offending site.

Actually there is! You have the option to turn on CPU usage monitoring for your websites.

All you need to do is go to your users menu in your webpanel, select the user you wish to monitor, and check it on. Thereafter, you will see a resources directory in your logs directory, and there you will see <username>.analyzed.0, <username>.analyzed.1 and so on. They are generated each day at 8 a.m. and will help you make sure that your resources are staying where they should be.

> You have the option to turn on CPU usage monitoring for your websites.

Eh, and how does that provide protection? As far as I can see there's no throttling for CPU usage, as there is for transfer.

In response to a few of the comments above, and to send a message to dreamhost who seem to offer longer answers on this board than they do when I write to tech support:

1) Yes, I am running MT with dynamic pages, rather than static. We often change the template of our pages, and there are over 2000 pages on the site. Constant rebuilding of the pages was taking forever, so we switched to dynamic building, which really enhanced the user experience, allowed us to show trackbacks, etc. So while it may be a bad decision CPU wise, it was a conscious one. If anyone out there is actually willing to help me with more information on the pros and cons of each, I'd be incredibly grateful. Is it really too much to expect an mysql server to not buckle under the wight of a few thousand visitors?

2) The spike we saw in traffic was high, but it seemed to me not gargantuan, but then, I have nothing to compare to. Anyone interested can check some stats here: http://www.sitemeter.com/default.asp?action=stats&site=sm3Scaramouch&report=0 and maybe give me a reality check.

Perhaps I'm expecting too much from a shared environment? Everything I read when I set up with Dreamhost talked only about bandwidth and disk usage, and we're still well under our limits there. If Dreamhost actually have a limit of traffic & connections we can take as well, it would be good for them to give guidance on that too. But there's no meter of that that I can see on the control panel, so I don't know how much is "too much" apart from when the site stops working.

3) The last thing I'd want to do is upset someone else's site with our own. Our blog is just a hobby, not a business, so downtime to us is just annoying, not income threatening. But for the same reason, jumping from $8 a month to $100 a month on a dedicated server (which was the only solution offered to me by tech support) seems excessive. Is there some middle ground where we can increase capacity without having to go to the whole dedicated server solution? It took forever to get the blog working like it is now - the thought of having to set it all up again on another server would probably force me to just give up the whole thing.

The most annoying thing is that I think it's just happened again. I wrote a post today that looks like it got picked up by a couple of big US blogs, and the traffic was high all day. Then I noticed a few visitors coming in from Japan tonight, and suddenly the site slowed to a crawl, so I suspect the japanese readers are hammering the site, and I run the risk of being shut down again. And there's not a damn thing I can do about it.

Dreamhost - if you are reading this, can you please help with some advice? I don't want to find another hosting provider - so far, my experience with you guys has been great. I have banners on my site driving traffic to you. But just as we start to get some success, it feels like you're shrugging your shoulders and saying "not our fault, just don't get as much traffic". I'm not expecting the world for $8 a month, but I guess I am expecting you to treat me like I could be a long term customer who continues to upgrade and bring you revenue.

It looks like my site is running smoothly again, so maybe I was getting alarmed too quickly, but getting taken down a couple of weeks ago has made me very nervous. The fact remains that I'd love our traffic to continue to increase so some guidance on how and when is the right time to upgrade would be appreciated.

> jumping from $8 a month to $100 a month on a dedicated server
> (which was the only solution offered to me by tech support)
> seems excessive. Is there some middle ground where we can
> increase capacity without having to go to the whole dedicated
> server solution?

How about splitting the load between two or more $8 plans?

> How about splitting the load between two or more $8 plans?

But if you are using Movable Type, it only has one database, there is no way of spreading it across servers.

And dreamhost already splits the http server and the mysql server.

I think if you want high traffic you need to go with the inconvenience of static pages.
You shouldn't need to change your templates too often. Use a small independent MT blog to test template changes quickly, then apply them on your real site.