Earthquake contingency plans

Gentlemen, I have made a place, http://wiki.dreamhost.com/Redundancy,
for Dreamhost to list their earthquake contingency plans.

Good one. I did not know these.

I believe the article will help people to be confident to host their websites in DH.

Uh, I don't think those are the serious contingency plans.

Use promo code 50DISK50 for 50% extra disk and $50 off
More Dreamhost coupons

I have no experience on maintaining a data center. Actually as customers, we can't do anything about. What we can do is just to sit down and listen what they say.

No, I mean they are a joke!

Use promo code 50DISK50 for 50% extra disk and $50 off
More Dreamhost coupons

At least there's a spot for the real info.

However! It's not DreamHost's responsibility to back up *your* data. One needs their own disaster plan. I'd bet that DreamHost has their own. Should their data center come crumbling down, I bet they have a plan. It'd take a while to recover by setting up shop elsewhere, but they most likely won't be restoring your data; just your accounts.


-Scott

I guess the issue was too sensitive. If not,
I hope when they have some spare time
they could put an article back into their wiki
addressing
http://www.livescience.com/environment/060621_san_andreas.html
in terms of how they will reduce possible downtime. Thank you.

In reply to:

---

I guess the issue was too sensitive. If not,
I hope when they have some spare time
they could put an article back into their wiki

---

First, the wiki is self-maintained. I don't think Dreamhost staff do most of the editing.

Second, natural disasters typically fall under the category of "Acts of God" and it isn't the responsibility of the web hosting provider or data center to have backups or contingency plans for your website. It is common business practice in the industry (I believe), for each business itself to take responsibility for having a disaster recovery plan and a plan for contingency operations.

Do you have information to the contrary?

Use promo code 50DISK50 for 50% extra disk and $50 off
More Dreamhost coupons

I asked support about this a few months ago. As far as hosting goes, we'd just be screwed... but my concern was domain registrations and how we would go about getting our domains back if LA fell off into the ocean.

They said that firstly, if something happened bad enough to completely destroy the area that it would have such a significant impact on the overall web that we'd have more serious problems than that, but that the domains would still be listed with ICANN. I asked if private regs would make this a more complicated issue and never received a clear answer.

As far as hosting in and of itself goes, you should never under ANY circumstances depend on your hosting provider for backups or disaster plans. Always keep your files and databases on YOUR local machine and always have your own secondary host picked out so that you could quickly sign up for services and change your DNS. It would be illogical to expect survivors of a major disaster to concern themselves with your petty websites!

"First, the wiki is self-maintained. I don't think Dreamhost staff do most of the editing."

Ralph deleted it. Ralph is a Bureaucrat, Sysop. Ralph is Dreamhost staff.

In reply to:

---

Ralph is Dreamhost staff.

---

Ah, I didn't notice that. Thanks for pointing it out.

I still think the idea of having a wiki page dedicated to earthquake contingency plans doesn't make sense.

Use promo code 50DISK50 for 50% extra disk and $50 off
More Dreamhost coupons

I agree with Lensman - the article was quite useless in regards to providing any valid information as to what Dreamhost will do in the case of an Earthquake. As mentioned, there's nothing they can do to help keep your website online or data backed-up in such an instance. A much more expensive, dedicated hosting facility might offer remote backup features (ie. to a site far outside their main facility), but Dreamhost doesn't charge nearly enough to maintain such a backup contingency feature nor of course do they offer it. Which isn't to say you couldn't make a suggestion for them to add such a feature in the future...

However - my suggestion would be to perhaps create a page dedicated to "Natural Disasters" in general, so as to explain to other Dreamhosters that Dreamhost itself cannot be held accountable for maintaining your website and data in case of such an event. There are alternative contingency plans users themselves could put into affect that would work just as well to prevent any prolonged downtime in case of some major downtime on DreamHost's end and the mention of such ideas would, I imagine, be most welcome for many users who aren't familiar with setting up redundant systems.

This official statement sounds very good, at first. "On every DreamHost plan we keep extensive back-ups of your data." You have to read four sentences in to the middle before you see less solid language, "We recommend you always keep your OWN copy of your entire web site at a remote location as well, but we'll do our best to make sure that's never needed."

A more complete wiki article covering DreamHost's recommendations and plans for Redundancy and Disaster recovery would be a good thing. That they casually deleted the opportunity and called it a joke is disappointing. Given the history of "disasters" from more minor "disasters".

http://blog.dreamhost.com/2005/09/12/power-outage-update/
http://blog.dreamhost.com/2006/09/19/anatomy-of-a-disaster-part-2/
http://blog.dreamhost.com/2006/08/01/anatomy-of-an-ongoing-disaster/
http://www.dreamhoststatus.com/2007/06/11/web-hosting-break-ins-security-update/
http://blog.dreamhost.com/2007/08/21/it-all-falls-down/

They work miracles for low prices. I wouldn't assume they can't find a way to have redundancy at separate locations. Google does. ;-)

Google also makes infinitely more money than Dreamhost

Like I said - redundant systems are expensive. I know because I've set them up before. If DreamHost wants to offer such an addition to its plans then I'll bet they just haven't had enough people interested in it to go about implementing the system. Which is, as I said, best to go about finding out by submitting a suggestion. Also, just FYI, redundancy is *not* covered in the current backup clause (re: "hosting-features") and normally isn't covered unless specifically implied in the TOS or other such official documentation.

Not surprisingly, I agree with Mousee.

Redundancy done right costs more than twice as much, if you count the replication that has to go on between sites.

If you need two-site redundancy, you should either find a vendor that provides this or you should do what everyone else does, which is go with another company altogether. The latter is better anyway because it provides you with backup for the contingency of any business interruption by your first hosting company.

My assessment is that the people who know they need business contingency plans are either not using $5.95-$10.95 a month shared hosting or are using two separate hosting companies. In my mind, your contingency plans are not satisfied by having only one hosting company no matter what that company's contingency plans are.

Use promo code 50DISK50 for 50% extra disk and $50 off
More Dreamhost coupons

In reply to:

---

Like I said - redundant systems are expensive. I know because I've set them up before.

---

That is cool!

I've learned only the concept of redundant system in my collage. I believe it should be expensive because of the hardwares. Hope I will have a chance to be involved in the set up of the system some day.

In reply to:

---

Hope I will have a chance to be involved in the set up of the system some day.

---

Don't do it. It's the most boring and thankless job ever.

Use promo code 50DISK50 for 50% extra disk and $50 off
More Dreamhost coupons

hahaha - most agreed! Better to be the systems admin instructing others to do it

In reply to:

---

Don't do it. It's the most boring and thankless job ever.

---

ha ha, i can roughly tell how it is now

In reply to:

---

Better to be the systems admin instructing others to do it

---

I think you should be over that level by now arleady

If it's "boring and thankless" it should be cheap. ;-) At the scale Dreamhost buys hardware and "pipes", I expect decreasing prices and good "economies of scale" so it doesn't cost more than double.

I don't know what "redundancy" *you* have in mind. I suggest there are many options, including "Our filers are completely RAIDed and hot-swappable so if any disk fails no data is lost, and we can replace the disk without any interruption of service!" which is limited, disk redundancy. Something for data like dreamhoststatus.com or the 3rd nameserver (I think), which provide limited remote functionality, not complete "redundancy," is possible.

In the wiki DreamHost could describe what *they* have in mind in more detail than "pure" marketing descriptions. Planning is tough, I know, and there are more than earthquakes to consider. Like fires and leases. And parties. :-)

http://blog.dreamhost.com/wp-content/uploads/2007/12/buildingonfire.jpg

http://blog.dreamhost.com/2007/12/21/were-so-high-right-now-you-dont-even-know/

I have a feeling this is rather fruitless, but I'll address your points...

The fact that the job is boring and thankless has absolutely no relevance to the cost of such a system, much less what the position pays, which is what we were talking about.
Now Dreamhost is welcome to buy all the hardware they want but redundancy is NOT, under the situation(s) we've been talking about (re: Natural Disasters), accounted for by having a lot of hardware in a single location. The redundancy that DreamHost would need in case of any Natural Disaster is created through the distribution of hardware through several geographic points. You typically want no less than two different geographic points (ie. a source and a slave) in a redundant system of this nature. If DreamHost wanted to offer it I'm sure they could - to that extent you are correct, considering the new DreamHost-East location - but the expense would still be a great deal more than what we pay them now to actually maintain it, keeping in mind that the OC192 lines alone would cost several hundred thousand dollars per month.

Take into consideration that you'd need a pipe between DH-West and DH-East fast enough to convey customer data between both locations. We're talking several OC192 lines here - not cheap even at half the price. Then you'd need hardware equivalents on both ends (ie. Person 1 in DH-West has an account on a server in DH-East and Person 2 in DH-East has an account on a server in DH-West.), plus the software to link both accounts and transfer data as well as make sure the links between both points are active (some of you might recognize this is also referred to as "High-Availability").

Now all of that said, going by your idea of "redundancy", DreamHost may as well be offering us all free Dedicated Servers with milk and cookies on the side. I mean seriously, what do you think those "redundant" systems are going to do when an Earthquake hits?
Jeff's Last News Letter: "Sorry guys! All of those amazing redundant systems we just purchased last month are swiftly coasting into the Pacific along with their cousins. Hopefully you all made back-ups of your data. Peace!"


I'll throw in some random links here also.

http://en.wikipedia.org/wiki/High-availability_cluster
High Availability and Disaster Recovery
http://en.wikipedia.org/wiki/Homer_Simpson

> fruitless

In reply to:

---

my suggestion would be to perhaps create a page dedicated to "Natural Disasters" in general, so as to explain to other Dreamhosters that Dreamhost itself cannot be held accountable for maintaining your website and data in case of such an event.

---

On the OP suggestion topic, I agree an article on disaster recovery - what to expect and do - would be good.

On how to be prepared, it doesn't cost less for all of us to periodically download copies of all our data through our straws, versus DreamHost having a more efficient system of some kind. If it "costs less" it's only because we're not counting or paying cash for our time and storage.

> going by your idea of "redundancy",

The silliness starts. ;-)

I've only quoted DreamHost's related statements and suggested there is a range of possibilities, like between local raid and your fully redundant, geographically separated, super-centers. The point of the OP, as I think we agree, is there would be some value in a good article on *DreamHost's* position and recommendations.

I believe dreamhoststatus.com is hosted in Northern CA, which is probably far enough away from LA for most disasters. This is part of their "plan" for communicating during fubars or disasters. I thought one of the name servers might be located there too. If not, my mistake. Even a few miles can be enough to be independent from local problems like building fires and building power outages.

This is more along the lines of *my* ideas on the topic:
http://en.wikipedia.org/wiki/Backup_site

> OC192 lines alone would cost several hundred thousand dollars per month.

I'm not the expert, ;-) but I bet for a lot less we could have daily or weekly minivan transport of an entire set of backup disks from LA to a rental storage unit out in the desert east of LA. Then customers would get an email, more likely from Josh than from Jeff, ;-) saying "LA is smoking rubble, but destro and I are getting things going again. For $199 we'll send you a date and time you can download your data. For $499 we'll get you and your site back online in a week. Sorry it took so long to get in touch, but gmail accused us of spamming."

In reply to:

---

The silliness starts. ;-)

I've only quoted DreamHost's related statements and suggested there is a range of possibilities, like between local raid and your fully redundant, geographically separated, super-centers.

---

Yup, and as I've already explained, your idea is never going to happen unless they dramatically increase their prices. Redundant systems of that nature are not cheap - period.
Local redundant systems Dreamhost already has and has had for quite some time, so more of those clearly aren't going to help out much.

In reply to:

---

I'm not the expert, ;-) but I bet for a lot less we could have daily or weekly minivan transport of an entire set of backup disks from LA to a rental storage unit out in the desert east of LA. blah blah, etc

---

Okay Ardco - in the mean time, I'll go back to worrying about my own data and making sure it's backed up.

Another thing to consider (in this fruitless thread) is are we talking redundancy or recoverability?

Aside from when Los Angeles sails past San Francisco and out to sea twenty million years from now (it's a popular myth that California west of the San Andreas is going to sink into the Pacific, but that's not going to happen), let's talk about preparedness for a major earthquake. The idea is that earthquakes are survivable and that though there may be infrastructure and individual structures that may need to be repaired, that on the whole the fabric of Southern California will survive.

Depending on the epicenter of the quake, the damage to the Dreamhost datacenters may be non-existent, light, medium, or heavy. It is unlikely that that the datacenters will be destroyed because, well, that would mean that the earthquake was large enough to destroy an earthquake reinforced building. The question would be when services would be restored to the building.

What many people don't seem to understand is that the buildings and infrastructure in California are designed or retrofitted to survive a major earthquake. Sure, there is the possibility of a quake larger than predicted, but the possibility of such a quake in non-earthquake prone zones around the world is suprisingly large. Buildings in these other places are neither designed nor retrofitted for earthquakes.

At any rate, everyone's contingency plans should include business interruption preparedness and it seems to me that such a plan should include preparing for rehosting your site with another web host - this means keeping your own backups. If you are really serious about this, you will have another hosting account prepared with another web host as your "failover" host. I don't want to pay one penny for the kind of redundancy that you're talking about because it is superceded by my own contingency plans.

And yes, I know very well that companies pay well for contingency planning jobs, but they're just a part of the overall job for IT staff at most companies. We have two redundant data centers and manage the backup and failover plans ourselves on an application-by-application basis based on how critical the application is.

Use promo code 50DISK50 for 50% extra disk and $50 off
More Dreamhost coupons

"and as I've already explained, your idea is never going to happen unless they dramatically increase their prices. Redundant systems of that nature are not cheap - period."

As I've already explained, I'm not espousing any particular "idea." Disaster plans have wide range of alternatives with wide range of costs; some are relatively cheap, and even more boring.

"Local redundant systems Dreamhost already has and has had for quite some time, "

Other than raid disks, they don't seem to have much.

"I'll go back to worrying about my own data and making sure it's backed up."

In summary, a Disaster Plan wiki page would be appreciated by some people, but some people are satisfied with the status quo, and with boring and thankless jobs.

Thanks for the thoughtful post. I think the OP was more on Recovery, but it seems obvious that some Redundancy can help with Recovery.

> manage the backup and failover plans ourselves on an application-by-application basis based on how critical the application is.

Sounds very sensible. ;-)

In reply to:

---

If you are really serious about this, you will have another hosting account prepared with another web host as your "failover" host.

---

Where do we find your promo codes for the 2nd best hosting?

In reply to:

---

Where do we find your promo codes for the 2nd best hosting?

---

Thankfully, not littering the DreamHost forums.

--rlparker

I never even thought about the earthquake scenario. I think if that happened, I'd be so concerned for all my friends and family in CA that my web sites wouldn't even cross my mind..

Use promo code "DREAMHOSTFIFTY" for $50 OFF
Use promo code "STINGYREFERER" for $1 OFF

No wonder they aren't worried about earthquakes.
All those cheery Support .signatures are probably coming from India :-)

In reply to:

---

All those cheery Support .signatures are probably coming from India :-)

---

Heh...well, that is *one* way to indicate that you don't know where those "cheery Support .signatures" come from.

The most recent DH blog post actually has pictures of the Downtown L.A. tech support "bullpen", on the 50th floor of the AON building (which should be a *hoot* during an earthquake).

There is also tech support staff working in Brea, CA., and (according to the information on the DH "Jobs" page, you may even get the occasional support response from a support staffer's home (from where they are allowed to work some days).

DH insists that tech support employees be available in L.A. (see that blog post and the comments from DH staffers), so I don't think it's likely, for now at least, that even a "from home" support ticket would be sent from India.

--rlparker