Funny how you describe "normal"

Nearly 2 hours ago, everything went down across all my sites. I saw on status.dreamhost.com that they were having problems and the message wasn't updated until a few minutes ago (nearly 2 hours later) with this message:

"Network Problems Resolved
The network problems described earlier have subsided. A switch error appears to have been the cause. We're still investigating, but again, things are normal and we're keeping a close eye!"

Email is getting through somewhat BUT EVERY SINGLE WEBSITE IN MY DOMAIN IS STILL UNREACHABLE. That is not NORMAL to me by any stretch. Yeah, I just submitted a support ticket since panel.dreamhost.com seems to be back up...I wonder if I'll find a new host before I actually get an answer. ;-)

Yeah, seems dreamhost.com itself is back up, but none of my domains are either.

Not only are all my websites offline, but all the ones by everyone I know who has a DreamHost website... it's a little embarrassing after having convinced them to go with DreamHost in the first place, but I've never used a web host that didn't have outages from time to time.

I've stopped suggesting DH to friends...I continue to use it myself, but my patience is wearing thinner by the outage.

Yes, same here - except that I can reach the webmail interface, but not my sites and I can't get FTP either. That's not my definition of "normal", and it's very frustrating.

Over the last 10 minutes or so, all of my sites have "started" to come back to life. Some still can't connect to their respective MySQL database, but at least they are resolving. I suspect within the next 15-30 minutes or so, everything will be back.. sigh..
rlparker

Whatever was fixed must be propagating throughout the servers. My sites were up but the ftp was not functioning. Now ftp and http are both working.

now "that" describe a "normal" re-birthing process...loading...stuff will come back on line incrementally and over a bit of time...for other's reading, depending upon your sites' construction, make sure you clear you caches (possible even DNS, depending on how you are set up) in order to see "actual" status (especiall using Internet Exploded or *any* form of AOL)

rlparker

I'm back up too. Still not happy and DH owes us a better explanation. A two-hour network-wide outtage is not something to take likely, especially when it happens during business hours. I can almost forgive a power outtage that was out of their control, but a "switch error". Come on!!!

I hear you...and you have a point...as for me and my clients, I'm hanging in there (though I have been "hanging *out* there" a bit more than I would like lately) in hope DH gets a handle on this stuff. They *have been* exceptional in the past (I've been with them since 1997!), and I am hopeful/confident? they will get back to that level soon (please, pretty please, with sugar on top?). They have had "bumbs" before, and always managed to pull it tigether, so I am hopeful. Lately, though, it has been discouraging...especially email.

rlparker

well I don't know if this is related to the outages this morning, but two of my sites got Brazillian Boy hacked about an hour ago. I hope that DH has some recent backups to restore from...

backups are made on an hourly basis. Just log into FTP and then move into the hidden directory called .snapshot You'll find Backups!

Mysql also have backups regurarly, but you'll have to contact support if you need access to that.



-Matttail

They should, as I have never found their backups to be lacking. Depending on whether or not your sites are database driven you may need to reload you MySQL databases.

Just a friendly note:

I strongly suggest that, in spite of DH good record with back-ups, you develop and implement *your own* backup plan, because "sh*t *does* happen". I would *never* rely *exclusively* on a web hosting providers back-up for *any* critical site. That is *our* responsiblity as webmasters.

I would also be curious to know what was hacked, and how. Are you using PHPbb, AwStats, etc., and is all your code current/patched?

Regards,
rlparker

I'm really getting so sick of all the problems. And my site is still not back up! I've tried FTP and www, both still down. I have to say not only have I stopped recommending dreamhost to friends, I'm seriously considering moving myself. I came here because my friends raved about dreamhost - I can't say I agree with my experience so far.

*EDIT - site finally came back up just now.

Edited by LLusive on 11/18/05 10:47 AM (server time).

Your post was probably more useful to the poster than mine; you told when how to find the back-ups which I just "brain farted" and forgot to include...thanks!

I still think, though, that none of us should *rely* on those backups...if there is prolonged lack of connectivity, we won't be able to get to the backups, and if we have to get the site back up ASAP on a different host, we are "out of luck" if the only backups we have are on DH, which can't be reached.

--rlparker

That is an understandable position to take, especially if you are "new" to Dreamhost, as things have been worse of late and your experience would be heavily negative in the "short term". For me, while recent events have been problematic, to say the least, over the 5 years I have hosted numerous sites with DH the total downtime has been insignificant.

Now, it is *never* "insignificant" if you are "down" when you need to be "up", but these *are* machines, and they burp, hiccup, belch, fart and break....no matter *who's* machines they are...

I think it also behooves us all to remember something that we all too often forget (primarily, I believe, because DH works *so well* most of the time):

No *mission critical" site should ever be hosted on "shared servers".

In a very real sense, we all "get what we pay for" and while I sympathize with anyone whose site is broken, if 100% uptime is critical to you (or 99.999% or 97% or whatever your "pain threshold allows) the proper business decision is to suck it up and spend some *real money* to arrange your own connectivity, redundancy, storage, mail, etc...

For me, all this equates to DH still being a reasonable compromise between expense and functionality. The low cost allows some sites to operate profitably that would be completely in the red if they had to provide their own infrastructure.

Now, having said all that, I think it *is* important that DH service does seem to be deteriorated of late, and if it continues, at some point, the cost/functionality ratio *will* change. It is my exsperience that DH, in the past, has managed to pull it together before I, or my clients, have felt it was time to change.

As usual, this is just my opinion, and YMMV...

-rlparker

Thanks for the pointer to the backups - I'd no idea they were there. I do have local backups, not recent but I hadn't made any changes recently so they probably would have sufficed.

They didn't get to the database, thankfully, but they did delete a slew of files in my web root directory as well as some image folders, etc. Seemed like random vandalism. I'm almost fully restored now. I've no idea how they got in, though. I'm running the Nucleus CMS but besides that, nothing that wasn't installed by DH in the basic package.

Bastards.

Thanks for the update, and I'm glad you are gonna recover! I just happened to be on the Nucleous website last night, and noticed a series of security updates recently added. I don't know how recently you have checked over there, but it might be worth a visit to see that you are running the latest code "patched" for the most recently discovered exploits.

Cheers,
rlparker

"No *mission critical" site should ever be hosted on "shared servers"."

I agree with this, however ... that isn't the problem lately (or hardly ever, in our experience over the last 2 years).

Their network keeps failing. Big time. Wouldn't matter if you were paying them $100 a month for their dedicated server option - you'd still be hosed. This is twice in a two month period that they've had a serious outage related to the network. Luckily this one didn't last an entire day like the last one. They also had another one several months ago (That's 3 in a one year period).

And nevermind the ongoing email problems which never seem to get fixed (for example, just now I didn't get an email that my business partner did - and it was sent to an email alias that goes to the both of us)

That's the reason we're looking at other options. I love the features you get at DH, their support people seem very good and usually reply quick to trouble tickets, and of course the price is great ... but with a flakey network, none of that really matters :(

Everyone is going to have problems ... and once a year would be acceptable (IMHO), but when things keep breaking (especially with increasing frequency), you start to lose faith.

- Brian Roach

Edited by BRoach on 11/18/05 12:20 PM (server time).

Broach,

All your points are valid, but I think I was not clear in my statement that you quoted.

"No *mission critical" site should ever be hosted on "shared servers".

was only stated as a general comment, and is true, as far as it goes. The real "meat" of what I was trying to say was:


"the proper business decision is to suck it up and spend some *real money* to arrange your own connectivity, redundancy, storage, mail, etc..."

I just assumed (always a dumb thing to do!) that my meaning was taking responsibility for your own infrastructure - not just 'upping" the ante on a commercial hosting service.

The *real money* I was talking about is several orders of magnitude above a $100/month "dedicated server", and I see now that I wasn't clear.

What I was referring to was leasing your own pipes, arranging for necessary connectivity and peering arrangement and back-up, fault tolerant archetecture for you redundant web, mail, and DNS servers, ad nauseum...in short, the whole magilla.

Big, Big, bucks....

While a dedicated server at DH or another hosting service may be more robust than a shared server, it alone is not acceptable for a mission critical site precisely because it is subject, as you pointed out, to many of the same frailties as a "hosted" shared server: it all depends on the *host*.

Hosting yourself (if you can afford it, have the technical expertise to design it and maintain it, and are willing to accept the full resposibility when you missed something or when something breaks) is how those kind of sites should be run. Everything else involves some kind of cost/performance tradeoff, and only *you* can decide what is a resonable compromise between cost and fuctionality/redundancy.

--rlparker

I have been experiencing *lots* of email problems over the last three months. It's really too bad, since service has been fine until recently.

Specifically, inbound mail is getting dropped (I know this because other people to whom messages are addressed send me notes saying things like "why haven't you replied?" -- highly embarrassing), and outbound SMTP has gone up and down.

I would get email somewhere else like fastmail.fm, and just change the MX record so I could keep my sites intact on DH, but I'm concerned that the DNS servers are having problems as well.

If you are moving, I'd be curious to hear what other deals you are considering.

rlprarker -

Completely agree with your points.

I don't think anyone here has the size of business that could afford that type of infastructure though. I know we don't. The problem being that real bandwidth (not ADSL/cable modem) is cost prohibative. If I could justify shelling out the $500-$600 a month for a T1, and have our existing DSL line as a backup, that'd be great - but for that I could probably get managed hosting with a SLA somewhere that would be an equally or more robust solution.

As you mentioned in your post, it is a cost:performance compromise with any hosting company you choose. My point was that DH is starting to have too many problems compared to what we expect from this level of hosting solution.

We're still here because they do have a lot of things in the "plus" column, and moving the site would NOT be fun, and it's something I really don't want to do with the Holliday shopping season being here. But we are looking into other hosting companies. The problem being, of course, that you don't want to end up right where you started in terms of problems. Basically, I'm looking into the hosting companies our competitors use - I have never seen their sites down or unavailable like ours, and they're using the same types of hosting solution (shared or dedicated servers).

- Roach

Roach,

We are thinking the same way, and I agree completely with you that the fact that we are hosted here most likely is indicative of the fact that we cannot afford, or do not have the expertise, to manage everything ourselves. I know that I don't have either the money *or* the expertise :-).

I also agree that, once DH can no longer meet our needs, managed hosting is probably the only viable option for most of us.

I think what is so dissappopinting to many of us, especially those of us that have been here a long time, is that DH *used* to be relatively free of the kinds of troubles we have been having of late, and we miss the "good old days".

What is a little scary for me, as I host numerous small business websites for clients on DH, is that one of these days, one of my "little" sites is likely to "hit" ("please, God, let me be slashdotted", followed immediately by, "Oh My God! I've been slashdotted! Help!") and when it does, I'm fearful DH will not be up to the task and I will be forced to "scramble" for a more robust infrastructure at the very time the site needs the disruption the least. It's kind of a "chicken or the egg" kind of thing; no one wants to invest the money till needed, but when it's needed it's likely to be too late to do it smoothly and without significant disruption.

My thoughts were mostly aimed at the personal/hobby/social networking , not really "business", sites that make up a large part of the DH poulation in hopes of getting some of these operators to realize that, for what they receive for what they pay, it may well be worth showing "patience" while DH grows and deals with these issues.

If DH can fix these issues, and they *have* fixed such issues in the past, these operators can continue to enjoy impressive capability for very little money.

It just stikes my as as little ignorant to hear these users' outrage at every DH shortcoming, and makes me suspect they really do not understand all that is involved in producing waht they receive for a few dollars a month.

Not to say all is well at DH right now! Email unreliablility is becoming a major issue for me and my clients, and it may well be time to move that fuction away from DH. To me, the danger in this is that, given any opportunity to do so, I am quite certain that DH would just as soon get completely out of the email business! I know I would, if I were a hosting provider, and *if* the market would let me!

Thankfully, for those us us with extremely limited resources, the market is not yet ready to allow DH to dump email as a basic component of a hosting plan, or I suspect we all might be dealing with a web site host *and* a separate email provider, with a *separate cost* whether we like it or not. To me, that extra aggravation, complication, and cost is something I would just as soon avoid so, as long as I can, I'm trying to be patient and have faith, that since DH has responded to problems well in the "not so distant" past, they can well do so again.

Regards,
-rlparker