Progress on current error

Hi,

This post is a mess and I apologize in advance. And for the record, I am a satisfied Dreamhost customer who is glad Dreamhost is growing, and I don't mind some growing pains. I like to think I'm a realist about technology.

I cannot get DNS to resolve for my site. Invariably I get "host not found" from ping and other utilities. In addition, I cannot get through to the Dreamhoststatus.com blog. My questions are:

1. Is the cause of the problem known?

2. Is there an estimate of when it will be fixed?

3. Is there anything I should be doing to help (notifying DH personnel that "This specific server is down")?

Thank you to anyone who can answer. I brag on Dreamhost all the time to friends, colleagues and clients, so I'm hoping they don't notice this downtime (which appears to have been going on from Thursday afternoon through Friday morning here).

Technical writing blog

Does your domain point to ns1.dreamhost.com (and ns2 and ns3) ?

Does your domain is on the panel ?

When did you check it, it can take some times.

Try to check using
http://www.dnsstuff.com/tools/lookup.ch?name=foo.com&type=A

They don't use cache, it's very useful.

There is a general outage at the moment. DreamHost engineers are working to fix the problem, but there is no specific time frame for getting it done. Monitor the support page of the Control Panel until the status blog is restored.

-- si-blog --

Domain has been on the panel for years.

In reply to:

ERROR: One or more of the nameservers listed at the parent servers are not listed as NS records at your nameservers. The problem NS records are:
ns.newdream.net.
ns2.newdream.net.

Thanks for the link to that DNSstuff.com utility. This was its diagnosis.

The domain has been working for several years, and encountered problems yesterday afternoon that are ongoing today.

I'd like to get it back up before inbound links assume it's a dead domain.

Technical writing blog

In reply to:

Monitor the support page of the Control Panel until the status blog is restored.

Thank you. I'll do that when I get home. I couldn't get to http://jessey.net/ either, so I'm guessing this problem is widespread.

Technical writing blog

The problem seems to be effecting about half of my domains, willy-nilly. Also E-mail is not working for me. According to the support pages of the control panel:

"Sorry about the downtime experienced today. We have network and DNS issues today. If you are writing in because your site is down please hold off if you can. We are currently running an update that will fix all of your domains, but unfortunately the update takes a while to run. If by tomorrow your site is not up please write directly to dhstatus@dreamhost.com rather than posting to normal support. We have a specialized support team ready to assist you.

You can keep updated at dreamhoststatus.com! (posted 10 hours 26 mins ago) "

--Matttail
art.googlies.net - personal website

Update from control panel:

! Critical Announcement! Please Read!
We are still experiencing DNS issues for a large number of our domains. After the network problems yesterday, a lot of dns records were somehow dropped from our ns servers, the cause of which we're looking into.

We've been regenerating all dns since last night, but it's a very slow process, slower than we'd like, and there are still a lot of domains not up.

On top of it all, it seems as though our remote dreamhoststatus.com server is under a DDOS attack, taking it down.

Please rest assured we're doing everything we can to fix this situation. If you'd still like to contact us about it, please use the form below as usual.

You can keep updated at dreamhoststatus.com.. assuming we stop the DDOS soon. Otherwise, we'll just be updating the message on this page. (posted 18 mins 55 secs ago)

Despite all one's legitimate gripes about DH service, one begins to wonder if there has not been a hostile hack here. If the status page is under attack, maybe the network was exploited, too.

> DDOS attack, taking it down.

Is that what you call 10,000 customers trying to find out what's wrong?

.......

I should think DH can tell the difference between a DDOS attack and heavy load.

--Matttail
art.googlies.net - personal website

> I should think DH can tell the difference between a DDOS attack and heavy load.

I probably should think so too, but I can't quite do it. What *is* the difference?

.......

All DDoS attacks result in heavy loads, but not all heavy loads can be attributed to DDoS attacks. DDoS attacks are almost always the result of an automated attack from zombies, whereas heavy loads are almost always the result of putting too much dirty laundry in the washing machine.

-- si-blog --

In reply to:
I probably should think so too, but I can't quite do it. What *is* the difference?

Depending on the kind of DDoS, one can easily figure out the difference. For example, a smurf attack using ICMP ping messages with spoofed return addresses is easily distinguished from legitimate http requests.

Apologies if I missed the joke.

The really bad thing is that Dreamhoststatus.com is also dead. They said they had set that up externally so we could find out what was going on. Apparently that is not the case which is really piss poor.

In reply to:
The really bad thing is that Dreamhoststatus.com is also dead. They said they had set that up externally so we could find out what was going on. Apparently that is not the case which is really piss poor.

That is not entirely accurate. http://dreamhoststatus.com/ is at a different physical location so that it is not susceptible to the same power outages as the bulk of the DreamHost network. In addition, the site is apparently undergoing a DDoS attack.

-- si-blog --

In reply to:

We've been regenerating all dns since last night, but it's a very slow process, slower than we'd like, and there are still a lot of domains not up.

On top of it all, it seems as though our remote dreamhoststatus.com server is under a DDOS attack, taking it down.

Thank you for the clarification. I am sorry to hear it's such a mess, and on a Friday, too.

I understand that many spammers are very angry at Dreamhost for refusing to host their domains, but a DDOS attack is going overboard. They should know better. No matters ;)

In the meantime, my domain has come back online and I am very thankful for that. I know this must mean people working behind the scenes to clean up what sounds like a very sticky situation, and I appreciate their efforts.

Technical writing blog

The status page might be at a different site but it appears it is dependant on the mysql server to provide it's information, so that's not really an independent source.

> Depending on the kind of DDoS, one can easily figure out the difference. For example, a smurf attack using ICMP ping messages

Wouldn't you best take down a web server by pretending to be super google or super surfer?

DNS Outage Identified
Posted 44 minutes ago (August 17th, 2007 at 12:52 pm PST) by Dallas

We have identified a runaway process that has been erroneously eating domain DNS records and that has been killed. The missing DNS data is being restored as quickly as possible now so any domains still down should be restored in the next couple of hours. We have already pushed out new software with safeguards against this situation happening again. The network intermittence yesterday caused database connection problems which triggered a race condition in our code.

We will be checking all domains for missing DNS records and restoring all of them so there is no need to contact us to let us know your domain is down. We will update this site when we believe all domains should be working again.

Severity: High Resolved: No

This entry was posted 42 minutes ago on Friday, August 17th, 2007 at 12:52 pm and is filed under Major Outage.

.......

In reply to:
Wouldn't you best take down a web server by pretending to be super google or super surfer?

I'm not a DoS expert, but have a couple of thoughts on this:
1. The fact is that there are tons of DoS methods that don't involve doing a DDoS using http requests. You have to protect against and/or resolve these.
2. I'd suspect that http DDoS is hard to do because you have to zombie a large number of machines in a relatively deep way to get them to issue http requests. Additionally since the responses are large, there's probably a high probability of taking down the zombies or saturating their connection unless you somehow forge a return address - and I dunno if you can do that in an undetectable way with the http protocol.
3. Finally - you'd take down one web address with this - and even then with load-balanced redirection you'd really need some horsepower (read a very large botnet) to crawl properly.

At any rate, I think DNS DDoS is the current fashion - again, not that I'm the expert on this in any way, shape, or form.

Dallas didn't stick with the DDOS story in the "all clear" at dreamhoststatus.com, so maybe it was self-DDOS anyway, and external DDOS was a false alarm.

.......