How to use a large mail box in real life?

I see that with 99% of hosting companies you can create your gibibyte email account (or even bigger). I have, but at one point when you have hundreds of emails, your email client gets really really slow if you use IMAP. Also, the search feature doesn't work very well, you search something that you know should return results but you get no results... If you use POP, you don't have these problems, but you can't access your mails from any computer, as you download them onto your local computer (and your mailbox is empty 99% of time).

In the end, Gmail, which is free, works better than a paid mailbox from a web hosting company because your inbox pops up instantly, when you search something, you get the search results in half a second. And you can use several gigs easily, it makes no difference.

How can I use a gibibyte mailbox with thousands of mails, lots of attachments, with a fast/efficient search feature, and have it working as fast as Gmail? Is IMAP flawed? What do you think?
Thanks.

In reply to:
at one point when you have hundreds of emails, your email client gets really really slow if you use IMAP.

Yes. Yes it *does*. Especially if you leave lots of mail in your inbox. Moving mail from your inbox to other folders on a regular basis so that your actual inbox is "lean and mean" goes a long way toward addressing that problem.

Frankly, I use POP at Dreamhost in conjunction with a Gmail Account (where I forward stuff I might want to keep available "from anywhere").

--rlparker

Yes, but I guess this "Archive" directory would get messy the same way and I can't imagine having to access it... My email client would go crazy and freeze...

In reply to:
Yes, but I guess this "Archive" directory would get messy the same way...

Well, it certainly could if you used a single directory. Using a series of directories (by month/year, project, or other "sub-sort" criteria) would help to a point but, at the end of the day, if you just leave it all on the server there will be a trade off between convenience of searching and speed.

--rlparker

Just for the record what are you using as mail client?

I use Opera's email client.

I've personally come to the same conclusion and switched to Google Apps for email, and web stuff here at Dreamhost. I doesn't seem that (most?) shared hosting setups are directed at supporting moderate to large email stores over IMAP.

Maybe this will get better if/when database-driven email systems become more widely used.

Yes. The problem is that no e-mail client (except webmails) are compatible with database-driven email (they only work with POP & IMAP).

I was thinking more along the lines of a database backend, something along the lines of DBMail - http://www.dbmail.org/index.php?page=overview

I've never used DBMail, or seen it used, but I imagine it would scale much better than most of what is currently used. There are other, similar servers out there which do even more for the index and searching aspect of mail storage. But I'm sure it would be a huge task to make such a switch so I doubt it will happen any time in the near future if ever.

Thanks for the link. It looks nice but it stores attachments into the database (it should use the file system). Also, I don't think I'll be able to install it in DH as it's a C++ application, from what I read :(

I find it incredible that web hosting companies accomodate with situation. Maybe they prefer it this way because people kind of "have to" download emails to their own computers, and their email accounts are virtually always empty, using less HDD space on the web host end.

In reply to:
Maybe they prefer it this way because people kind of "have to" download emails to their own computers, and their email accounts are virtually always empty, using less HDD space on the web host end.

Tat could be, I suppose, but I suspect the HDD space is less of an issue than the processing power required to handle manipulating really large collections of received, sent, stored, etc. email.

It's interesting to me that web hosting companies are in the "email business" at all. They certainly do not *have* to be (well, except at present for competitive reasons). Email and Web hosting are two different functions, and I wouldn't be at all surprised if one day we find web hosting companies just saying "We don't do email" or "it's an add-on service, and here is what it costs". wink

--rlparker

Are you serious?!! Eh eh :P
I mean, a company that says this is like saying "We don't offer PHP, only as an addon". Everybody uses PHP, everybody uses email :)
Not everybody wants to use Google Apps (although it's great).

In reply to:
Are you serious?!! Eh eh :P

Absolutely! he he he

I didn't say I would *like* it, and I *did* point out that competitive forces wouldn't allow it to happen at present. I said I wouldn't be "surprised" if it happened; that doesn't mean that I think it *will happen*! wink

Int he past, there *were* companies that said one thing or another that we now expect to be "included" was an "add-on" (for example, DH *used* to have MySQL only as an add-on).

"Email" seems to be particularly suited to such treatment because of the varying levels of service needed for different users, the difficulty of keeping "shared" email server off of blacklists, and the SPAM issues altogether.

Dedicated email services could possibly focus on different things, and provide a wider range of services, than a "hosting" company that is primarily focused on serving websites.

--rlparker

Companies such as DH have made the unfortunate distinction of calling themselves "Webhosts." Unfortunate in that people think that Web==Internet. A more accurate title would be "Domainhosts." And typically, a domain consists of email, FTP, web, and the back end to both of these services, including spam filtering and databases. I think shell access is super-cool and goes way beyond simple domain hosting. It makes sense that all services related to mydomain.com are hosted by the same company.

So I don't think email should be an add-on, and should be part of basic service. DH has already taken the steps to separate the services hardware-, and probably network-wise. In a way, they are offering dedicated email service by isolating the email part of your domain. It would make them even more competitive to build up their email feature offerings. Maybe they'll even grow to create a PS for email.

-Scott

In reply to:
So I don't think email should be an add-on, and should be part of basic service.

I agree with you completely on that, though I worry that it may not always be that way. Fortunately, it looks to me like it *will* remain part of "basic service" for the near future, though moving to "enhanced" dedicated email services is already almost a requirement for many businesses. wink

In reply to:
It would make them even more competitive to build up their email feature offerings. Maybe they'll even grow to create a PS for email.

You make a really good point about the segregation that is already in place, and an email PS would be *very nice*! smile

--rlparker

Alright. Anyway, I added a suggestion in the Dreamhost web panel: "Use DBMail as mail server backend". It doesn't cost to ask :)

From glancing at DBMail's website it looks like DBMail would be incompatible with Maildir email storage. I get the impression that DBMail uses a database as the mail store, not merely a means of indexing the mail for quick searches.

If DBMail were used merely for searching email server-side (essentially handling only the IMAP search command(s)) it would be nice. I could, theoretically use some program to get quick server-side searches. Perhaps a program better suited for searching large volumes of text (like the text indexers Beagle and Tracker use) would be a better choice than PostgreSQL, MySQL, or SQLite (DBMail's backends). Such an arrangement would theoretically make Thunderbird's search feature (which, for IMAP accounts, hands off the real searching work to the server) work without so many timeouts on large mail folders.

But I want the easy/powerful manipulation server-side filtering/sorting Maildir affords me via procmail. If I had any improvements to suggest to Dreamhost it would be that all of my email addresses (current and future) should be exposed to me as Maildir folders on my account so that I can use procmail server-side to manage the email for me before I connect with my IMAP client.

I don't want to have to connect with an IMAP client to do email filtering and sorting. And no MUA's filtering I've seen can compete with what procmail feature-wise.

In reply to:
So I don't think email should be an add-on, and should be part of basic service.

To me, the question is whether the service should be disaggregated from the web hosting. In my own personal case, I'd like it to be because I don't use it - well, I do have a few addresses that I have forwarded to my "primary" non-Dreamhost account.

Given that I don't use the service, do I want to pay for it? No, if it is a significant part of the price I pay. But do I think it is? No, I don't, so I'm satisfied with the bundling.

Then again, I see a possible future where web hosts become mere aggregators of primary service providers. For instance, you'd have email provided by gmail, blogs provided by wordpress.com, bulletin boards provided by some specialized phpbb hosting provider, etc. In this world, there'd still be a slice of the pie for honest-to-god apache/html/php/RoR custom sites, but a lot of the services that can be asp'ed away would be.

What are 50DISK50, 3DOM50, and 1IP1DOM50?
More Dreamhost coupons

One thing that drives "professional" email hosting apart from "webhost" email hosting is the simple fact that proper antivirus and spam-filtering takes humongous amounts of CPU power and memory; on a reasonably busy domain with just a few users, it can easily be more than the "average" dynamic webpage. That particular gap will only widen ... It's simple to provide email service per se, it's not so simple to provide well-maintained spam-filters, virus-scanners, keep the host(s) out of blacklists, and be reasonably nice to the internet-community at large (which includes the option to not accept emails in the SMTP session due to spamfiltering or antivirus-filtering instead of trying to send a bounce to an unrelated address later on -- and this makes it necessary to do filtering in real time).
Even a somewhat optimized setup of spamd (from SpamAssassin), clamd (from ClamAV), procmail (for user-defined filtering -- and note there is no daemonized form of this, so every time you load that huge ruleset of yours, almost all the regexes need to be compiled and applied), a variety of realtime blacklists (dnsbls as well as Razor and such) can easily grind a server to a screeching halt with a small flood of mails (which may even be legitimate) -- and this doesn't even take into consideration accessing the whole thing through IMAP and the like (you'd like to have sieve-support, server-side fulltext searches, etc.).

Brave new world, that is.

You need mail service connected to your own domain if you want to use addresses in your own domain instead of tying your e-mail address to something outside your domain like an ISP or something, which is less portable and also (in my opinion) looks less professional. (If your company site is at www.yourcompany.com, which looks more professional for a contact email, info@yourcompany.com or yourcompany@aol.com [gag, vomit, retch]?)

I suppose this could be done by an external e-mail service with your domain MX record pointed at it, but it's easier to do through your Web hosting provider so it's all in one place.

-- Dan

In reply to:
I suppose this could be done by an external e-mail service with your domain MX record pointed at it, but it's easier to do through your Web hosting provider so it's all in one place.

Yep, that's what I'm suggesting. I do admit that it's a little harder right now in that you have to manually set the MX record for the domains where you want to use an external email provider.

As eike says, firms specializing in email hosting are becoming *very* good at what they do. There are economies of scale and specialized email expertise that they develop which enables them to have an easier time in addressing the problems that web hosts only have limited time and resources to devote to. If we look particularly at the problem of spam, they've got solutions that are superior to anything that web hosting providers typically provide.

Even corporations are going ASP for email.

BTW, I'd argue that external email done right actually is easier than learning how to configure your email and spam filtering properly at each web host you have.

What are 50DISK50, 3DOM50, and 1IP1DOM50?
More Dreamhost coupons

Well, that's the age-old argument of "all from one provider" or "mix & match". Same could be said for domain registrations (and I PREFER registering with somebody other than my webhost, since that gives easier portability should problems arise), and an argument could even be made that having your access provider being the same as your hosting provider can be advantageous.

External mail service via MX-record-juggling is common enough. You can get it from most of the big email providers (IIRC gmail has an option for this as well), and it really helps if your mailbox is flooded with spam every which way :)

How to use a large mail box in real life?