Panel/Webmail Maintenance Today, Noon-1pm

Hey, we're doing some database maintenance to speed up the Web Panel and related central services (like Webmail/KBase/etc).

This will take things down between Noon and 1pm (PDT). We expect the actual downtime to be about 30 minutes.

This really should help things a lot, and we really want to get things up and zippy before the weekend so the problem isn't dragging on any longer!



nate.

If this makes a difference in the web panel's performance, then all I can say is Halleluyah! But to take the panel down smack in the middle of the day? Most hosts would schedule this kind of a "take the whole thing down" maintenance at 11 pm, not noon/3pm (my time).

What happens if something goes wrong right in the middle of everyone's workday? I'm not as concerned about the glacial web panel, but I have a lot of clients that rely on webmail and if it gets any slower than it is I'm in serious you-know-what. I'm getting a lot of complaints.

I have to say that this is a REALLY bad time to take down webmail.

It's Friday afternoon before a three-day holiday and people that I'd like to communicate with before Tuesday are getting antsy and thinking about jetting early for the day.

I guess the downtime is happening now because IT staff at Dreamhost are getting antsy and thinking about jetting early for the weekend.

It ain't so bad. Time-critical businesses have contigency plans for outages, and most businesses operate on schedules with more than one hour margin anyway. Besides, email is not affected. Better to get this kind of a thing out of the way ASAP.

The thinking is that before a long weekend, when lots of people (like us) want to do stuff like go out of town, you get it done and have the rest of Friday to make sure it's working.

I know it's really not optimal, and in the past when we've done stuff like this it has all been late at night on a Saturday (and I've been deep in the datacenter for most of them!).

But these database problems are slowing things down enough that we really need it to be fixed ASAP, and have a window to watch over things before the 3 day weekend!


nate.

I'd suggest you guys make deadlines for stuff like this significantly different than just before major holidays and long weekends. It's a repeating pattern. Do it significantly before holidays, with time to recover when Murphy's Law kicks in, or just put it off until after the holiday. Maybe you got lucky (or were skillful, as the case may be), this time, and it went well, but it does not always go well...

Oh yeah, and send the announcements a bit earlier. :-)

Funny, sort of, I just got the (email) Announcement of the outage/work many hours afterwards, and it wasn't supposed to effect email. Also, even Spamassassin knows it was an Excuse. :-) I swear I'm not making this up.

Received: from ...dreamhost.com (...dreamhost.com [66...])
by ...dreamhost.com (Postfix) with ESMTP id ...
for <...>; Sat, 28 May 2005 00:17:19 -0700 (PDT)
Received: from ...dreamhost.com (...dreamhost.com [66...])
by ...dreamhost.com (Postfix) with ESMTP
id ...; Sat, 28 May 2005 00:17:17 -0700 (PDT)
Received: by ...dreamhost.com (Postfix, from userid 0)
id ...; Sat, 28 May 2005 00:17:19 -0700 (PDT)
To: ...
...
From: DreamHost Announcement Team <support@dreamhost.com>
Subject: [...] [Announcement] Panel/Webmail Maintenance 05-27-2005
Message-Id: <...@....dreamhost.com>
Date: Sat, 28 May 2005 00:17:19 -0700 (PDT)
X-Spam-Checker-Version: SpamAssassin ... (...) on ...dreamhost.com
X-Spam-Status: No, score=-1.2 required=... tests=AWL,BAYES_00,EXCUSE_1
autolearn=ham version=...
X-Spam-Level:

Hello Dreamhosters,

We're doing more maintenance on some of our central databases this afternoon which should
help alleviate Web Panel slowness and keep our automated background jobs running faster as
well.

The Web Panel/Webmail/KBase and other centralized web services will be down between Noon
and 1pm (PDT) today, May 27th 2005. We expect the actual downtime to only be about 30
minutes.

THIS WILL NOT AFFECT CUSTOMER SITES OR EMAIL!


Happy DreamHost Fix-y Team!

Hi!

I think it's great, that DH is working on speeding up the CP - but that the email about the maintenance reaches me many hours AFTER the maintenance is really strange.

The annoucement emails take a really really long time to go out. It's dependent on each individual's contect preferences whether they receive them via email, and there are many accounts with multiple contact points. The announcement I sent on Friday was sent to something like 75,000 people, which can take awhile.

That said, it can definitely be made faster than it is now!

And as often happens, we weren't expecting to do the maintenance that took place on Friday, but the Panel and backend jobs were getting so borked up that we felt it was important to do it pretty much ASAP.


nate.