Warning: Dreamhost may delete your custom procmail

In this new upgrade process it looks like you will not be able to have a custom procmail install. Another user informed me that:

"The pre-upgrade checklist on the Web panel advises me to make sure that 'Nobody on your account is using .procmail or .forward files'"

But it also appears that even those of us with Grandfathered in accounts are going to lose them as well. This is from another user:

"They said they wouldn't allow new accounts to have email access from the shell, and I thought I'd been grandfathered in, just like you. But without warning, I found my custom spamassassin install broken since I'd been moved to the new servers.

They said they'd be moving folks over the course of the next year, so maybe I was just really unlucky to be one of the first accounts moved. But as a heads up, you've got at the most one more year before you're moved, too... I hope they move everyone soon, so someone smarter than me can figure out how to get custom spamassassin installs working again. :) "

And all of this happened with no warning. This user lost his custom procmail setup in a manner of minutes with no warning or explanation from dreamhost.

So those of you out there with custom Spam Assassin installations, or other custom procmail setups be prepared to lose them any day, you could be "upgraded" tomorrow.

Also, Dreamhost promised to upgrade the mail-filter tab in the panel almost 6 months ago now. And yet we still do not have a filter by custom header option.

This is going to get really stressful and annoying if Dreamhost just starts switching people without warning. The smart thing to do would be to at least tell me the name of the new server that I am moving to and give me a time that it will happen at. This way I can at least attempt to prepare for it.

In reply to:

---

I hope they move everyone soon, so someone smarter than me can figure out how to get custom spamassassin installs working again.

---

lmao mate that cracked me right up. Good stuff.

I'm in the same situation and discussing the issue with tech support.

The advice given so far has been to setup e-mail filters on the Panel so that mail is forwarded to your shell account. The idea is that it will create a .procmailrc file on your mail account, which then forwards to your hosting account. It doesn't work.

I found more fallout from this change:
1) Pine no longer works on your shell account. Try it. Maybe yours works but mine continually tries to log in but fails every time.
2) My Maildir contents are gone - moved to my new mail server, no doubt. The only contents are the e-mail text from a weekly cron job that used to clean up my Spam folder! Note to DH: cron e-mails go to the shell account mail and not my new mail server account? It doesn't make sense.

Assuming my shell account could process mail, I still need a way to make this line in my .procmailrc file point to something meaningful: MAILDIR=$HOME/Maildir. I need a way to mount my mail server's Maildir directory, thereby giving Procmail (and SpamAssassin) access, but that needs to be done at the administrator level.

I can pretty much delete my good ol' SA installation guide unless a few tweaks can get things working again.

-Matt

Hey Matt!

Hmm, as for pine, I think you may have to set it to connect via imap to mail.yourdomain.com as opposed to locally. pine can definitely do that, in fact that's what I do exactly for email myself!

Also, for the .procmailrc problems from the panel, it looks like the issue was that we had copied over your old .procmailrc file and so our panel stuff won't work! Doh. I've removed your old .procmailrc so now what you set up should work (the forwarding of everything to your shell account).

For the cronjob emails, if you set it up from our panel, then it would work, but if you've set it up yourself, you'll need to manually specify the email address you'd like the generated emails to be sent to with a line like:
MAILTO="blah@blah.com"

About the custom SpamAssassin installation, I would like to ask.. have you given our spam filtering a try recently? We've actually put a lot of work into it lately and I'm pretty curious to know what you like about your set up that ours doesn't provide. Because, really having all these custom spamassassins running all over the place on web servers is something of an administrative foo, and it's just better overall if we can centralize it in an acceptable way. You can now have your spam go to an IMAP folder for example, and we'll be adding bayesian filtering soon, and we're keeping our spamassassin version up to date, etc... any feedback on that would be really appreciated!

Thanks,
josh!

I haven't tried the DH installation of SA. The Bayesian filtering is why I became a diehard fan of SA. It's not unusual to go seven days with only 5 spam messages slipping through the filters.

I've finally figured out a strategy to use my custom SA installation all the Bayesian data I've accumulated. The problem is now updating the Bayes db. To be continued...

-MattEdited by matthew on 10/15/08 09:20 PM (server time).

My body is hardwired to spamassassin:
http://jidanni.org/comp/spam/spamdealer.html
http://www.google.com/search?q=spamassassin+jidanni+site:gmane.org
https://issues.apache.org/SpamAssassin/buglist.cgi?query_format=specific&order=relevance+desc&bug_status=__all__&product=Spamassassin&content=jidanni

Certainly Dreamhost will continue to provide a method to feed an
email to a program.

Certainly Dreamhost will continue to provide IMAP to serve messages I
place in front of it.

If someone can figure out how we can do this in Dreamhost's new setup
please post here. If it involves forwarding back and forth so be it.

Josh> and we're keeping our spamassassin version up to date, etc.

jidanni@spyro:~$ for i in '' 'env -'; do $i which spamassassin; $i spamassassin -V; done
/home/jidanni/bin/spamassassin
SpamAssassin version 3.2.5
running on Perl version 5.8.4
/usr/bin/spamassassin
SpamAssassin version 3.0.3
running on Perl version 5.8.4

Up to date for your version of Debian
http://packages.debian.org/search?keywords=spamassassin, but I cannot
participate in the SpamAssassin mailing list or report bugs using that
version. Sorry.

Josh said:

> About the custom SpamAssassin installation, I would like to ask.. have
> you given our spam filtering a try recently? We've actually put a lot of
> work into it lately and I'm pretty curious to know what you like about
> your set up that ours doesn't provide. Because, really having all these
> custom spamassassins running all over the place on web servers is
> something of an administrative foo, and it's just better overall if we
> can centralize it in an acceptable way. You can now have your spam
> go to an IMAP folder for example, and we'll be adding bayesian
> filtering soon, and we're keeping our spamassassin version up to date,
> etc... any feedback on that would be really appreciated!

I was one of those people who run customized spamassassin (on a PS server though) and my reasoning was that DH provided spamassassin wasn't powerful enough. It seems to be missing SPF/DomainKeys checking and it doesn't seem to use a lot of more complex rules, which were really helping me to get almost zero spam messages to my INBOX.

But, I switched to DH Spamassassin because I am brave and I want to be moved early to the new servers. So I said goodbye to procmail and spamassassin, took a last look to my spam-free INBOX and jumped.

Yes, the current DH spamassassin still doesn't recognize SPF/DomainKeys and sometimes for example classifies some newsletters as spam when they are having SPF included (my customized spamassassin was happy to let me read those commercial newsletters because they were legit). Setting up IMAP folder for spam was a bit weird because there isn't any documentation available (it disables the tagging option and doesn't let you tweak those in mailboxes.example.com but it let's you do that in panel, which then will ruin the whole thing).

I got it working. And I am quite okay with that - I wish there would be bayesian already now and I wish DH would include some of the more advanced rules (SPF/DomainKeys etc). I get more spam into my INBOX now but maybe that will get changed (and I get few true emails to spam folder when they shouldn't go there).

With DH spamassassin solution, you apparently need to setup a lot of whitelists when with customized solution you wouldn't need to.

I've almost got a scheme figured out. The only remaining snag is training false negatives - type II errors. That's critical for good Bayesian scoring.

It's easy to filter for spam and forward the good e-mails to the mail server. Unfortunately, when it's time to move false negatives to a 'junk' or 'spam' folder then we have a problem. Those messages are only moving on the mail server - the trick is to get them back to the server where your SA installation resides and scan them every week.

It will take more research on my part, however, the isync utility looks very promising. Using mail forwarding, Maildir isync-ing, some cron updates, and testing this can all work again.

-Matt

(Bayes? Not for me. "use_bayes 0" is what I say.)

So they're going to have one machine where all you can do is read by IMAP,
and another machine where all you CAN'T do is read by IMAP? Instead of
me idly guess, maybe this is a safety in numbers motif, where a lot of
people will start scream... OK, no more idly guessing. Do post what
you reverse engineer. Thanks.

What we would need is DreamHost PS Mail; following the PS concept (PS, PS MySQL) it would provide a virtual server allowing more customization and also would dedicate CPU/memory for procmail and spamassassin (and whatever the user would like to run for their email).

Of course customized webmail would be running there and all the email addresses would optionally have filesystem/shell access to their email Maildirs.

I would pay for that kind of service.

I've just come across IMAP Spam Begone. It's a Python script that feeds IMAP mail to Spamassassin. It's from 2003 but still seems to work, and it's pretty short, so it shouldn't be hard to adapt.

There will always be ways to roll your own spam filtering. When Dreamhost removes all direct access to mail files as they announced, it will just have to be over IMAP: same effect, bigger overhead. Or they will actually fix their Spamassassin install and add per-user Bayes--never give up hope...

Well, I finally have a complete solution. I haven't had time to post to my website but here's the gist:

1. Created a new email-only account (call it the "validation account")
2. Updated all my email forwarding addresses to go to the new validation account
3. Created an e-mail filter on the panel to forward any mail from the forwarding account to my hosting (spamassassin) account
4. Scan incoming mail on my hosting account with SpamAssassin - spam goes to a local .Spam folder and the rest gets forwarded to my actual user email account
5. Setup a cron job to synchronize my .Spam folders on the hosting account and mail server every hour. This is so I can check for false positives. I use a Python script called "offlineimap" to do the synchronizing.
6. Every Sunday, like previously, I run spamassassin and update the Bayes database. Junk messages are purged on both servers.

If I break this into digestible chunks, it will probably double the number of steps in my existing tutorial on SpamAssassin. Still, I'm very pleased a solution is possible. I was losing hope and almost switched to Gmail permanently. I'm sure that decision would be welcomed by Dreamhost. Then Gmail started having outages and my decision was simple: stay the course.

I'll post again once my tutorial is complete.

Josh is talking about the version of spamassassin on our junkmail filter machines, not the version on your hosting machine.

- Dallas
- DreamHost Head Honcho/Founder

I maintain my spamassassin filters via scripts, not by loggin in to the panel by hand every time I want to adjust something.

(Can you imaginine one needing to fire up a browser just to add a name to some white/black list.

I do this:
$ scp user_prefs server:.spamassassin/
to update my user_prefs after my script on my home computer runs it through grammar checks, as seen with spamassassin --help. I would like to apologize today, I am at the public library hence cannot spell check this post, etc.)

Today I received the DH 72 hour notice that I am to use the panel.

It says
* If you set up a custom .procmailrc (filtering) or .forward (forwarding)
file, you NEED to go to our panel's "Mail" section and from there mimic
whatever functionality you need. You can set up forwarding, filtering,
whitelists, blacklists, and even sending emails to a script for further
processing, all from our interface!

OK, on https://panel.dreamhost.com/index.cgi?tab=mail&subtab=filters¤t_step=Mailbox&next_step=Add
I found an entry:
Email Filters
Adding new filter for: my@account...
When an email's Subject From To CC Headers Reply-To Body contains does not contain ,

Do this: Move it to folder:

(For sub-folders, do like this: Friends.Josh)
Add this to the subject:

Forward it to email address:

Forward it to a shell account:
jidanni (spyro)

OK, maybe that last choice means that I can keep on using my present setup.

And maybe if I don't fill in any sbject it will match all subjects, even mails with no subject lines.

But who knows.

I am just guessing that "and even sending emails to a script for further processing" means one should click "Forward it to a shell account: jidanni (spyro)". OK, and then how to get it them off of SPYRO using IMAP?
Or do we then sent it back to the server with a X-Loop header or something?

Wait, on https://panel.dreamhost.com/index.cgi?tree=mail.addresses& we see Create New Email Address, OK,
I should create FILTERED@mydomain.com, and set Forwarding-Only Addresses MYoriginalAdreess@mydomain.com to forward to MYshellAccount@dreamhost.com, and then the messages that pass my filter I should then have my scripts send to FILTERED@mydomain.com, which will have IMAP attached.

I suppose, maybe.

Edited by jidanni on 10/24/08 11:43 PM (server time).

Posted on my site if you're interested: http://www.unsaturated.com/projects/spamassassin-changes-for-dreamhost/

>>>>> "D" == DreamHost Customer Support Team <support@dreamhost.com> writes:

D> I'd be happy to clarify things for you. What this was trying to say that
D> you can set up all those filters using our interface, which is the web
D> panel itself. For example, you could set your email addresses to forward
D> mail to a shell user where they can be processed by the script that's in
D> the shell user's home directory.

D> So go to https://panel.dreamhost.com/index.cgi?tree=mail.filters&, edit
D> the email address you have listed there, then click the Add Filter
D> button, and you'll see all the options listed for filtering. Among them,
D> there is the "Forward it to a shell account:". That's what that note
D> refers to, that you can do all this through our web panel. Sorry about
D> the confusion.

I assume not touching "When an email's ... contains ..." will match
all mail no matter what headers they have or lack.
Is this open source so I can be sure? Maybe it is in one of the
$ dpkg -l ndn* packages.

D> you need to use that option to send the message to your shell user
D> for the processing you set up there. Also, you can either use Pine
D> to check mail directly under your shell user, or you can use the
D> X-Loop to send it back to your original address.

D> It appears that another customer posted a reply too, you can check his
D> explanation at
D> http://www.unsaturated.com/projects/spamassassin-changes-for-dreamhost/

That is wildly more complicated than older user me can handle, And
indeed, as I do not use bayes, I need not take that route. So all it
seems I would need is add the X-Loop as above.

However I could then not also read the messages on the shell account
(with mutt). I could only to use IMAP or webmail. I could no longer
use all three depending on what is best from what computer I'm
connecting from that day (home:IMAP; public library:webmail as no
putty allowed; friends house:putty ssh then mutt).

Ah, I will in my (hosting account's .procmailrc) forward back
with X-Loop, and also save a local copy, so I can read the messages by
all three methods (IMAP, webmail, mutt).

And I will rig up a script to do
$ ssh hosting_server expire_messages
after I am happy I read them all via IMAP or webmail so I don't need a
copy still.

Anyway, perhaps the .procmailrc making script on the new
closed mail server is open source, so I can see what will happen with
no Subject, and if X-loop will really work.

Also we see "This feature will not work if you have a .procmailrc file
already." on the panel. But perhaps that refers to on one machine that
now becomes two, etc. etc.

Anyway, please don't start counting the 72 hours until I can
understand and get this all in place. I estimate it will need a few
weeks, especially as I cannot see the code of how you intend to handle
X-loop so fear disaster.

Another solution I will examine is
$ ssh hosting_server 'start up a personal IMAP process which I then connect to to download my mail'

That would eliminate the need to "forward backward", but still means I
couldn't use webmail, unless there is a webmail listener alive on the
hosting server.

Of course all this would take a few weeks for me to figure out.

Another option is to please put me deeper into the waiting list of
users to be affected, and hope other users complaints will cause
Dreamhost to back off on requiring this tangled mess and then when the
time rolls around for me to be affected, I needn't do anything.

This reply posted to discussion board and emailed to support.

Sent to Dreamhost staff:
It would make 100 times more sense and less CPU to just let me have an
account on the mail server so I could maintain my own spamassassin
user_prefs file and not need to install a current copy of SA on my
shell machine or shuttle mail back and forth. Why risk me or you
causing a loop?

I've used procmail for its entire ~20 year /usr/share/doc/procmail/HISTORY.gz,
Please don't make me have to go thru a web application to edit it:
http://www.unsaturated.com/projects/spamassassin-for-dreamhost/#comment-30
Bait and switch:
http://www.unsaturated.com/projects/spamassassin-for-dreamhost/#comment-28

You say your goal is to get mail off of the web servers, so no sense
moving it back and forth to there just for processing.

I will explore using
$ mutt imap://imapserver:port/INBOX
on my shell account, (after doing forward to shell account, filter, forward "ham" back to mailserver + X-loop header)
OK, thanks everybody. Bye.

Instead of guessing about X-Loop, the following seems to work:
me@example.org -> my_shell_account_where_my_filters_live -> final@example.org
And in the .muttrc on the shell account:
set spoolfile=imaps://final@example.org:passwd@mail.example.org
my_hdr From: me@example.org

Hi!

What do you think of "Procmail consolidation"?

I mean suppose you've had several shell mailboxes all filtering email via custom SA.

Now, when all's broken and you have to start from scratch, would it be feasible to forward mail from ALL your mailboxes to a SINGLE shell account, then forwarding back to respective addresses?

Multiple INBOXes? not me. All I know is I filter for my single address.
(Also never dared to use X-loop, using a different final destination mailbox instead.)

Anyway, with a fancy enough .procmailrc, you could certainly do
many->one->many. Note that some mail is "undisclosed recipients", so
you need to bifurcate on "Received ...for" headers.

Anyways everything is hunky dory, except if I am at a terminal at a
public library that blocks port 22 so I can't use putty to read via
mutt on my shell account.

Whereupon all I can use is http(s) to webmail, which is like "driving
with the steering wheel locked"...Edited by jidanni on 11/15/08 02:22 PM (server time).

See, I keep mailboxes for quite a few people and many of them are complaining about spam.

So if I were to help them fight spam, what would be easier -

create an additional shell user per each mail user (tens of people),

OR

setup a centralized one-shell-user system and bulk-edit all mailboxes just adding a forward to them
?

Let's see: the only way to not need to teach those many users new tricks is to forward filtered mails back to their mailboxes with X-loop added (better test first to see if it works, as it is all "closed source" (configuration is on closed access mail server).

This would be your
"setup a centralized one-shell-user system and bulk-edit all mailboxes just adding a forward to them"

So in total you would still have your e.g.34 mailboxes, plus one additional shell account. Better than 68 mailboxes though. P.S.: untested.

Note however http://wiki.dreamhost.com/SMTP_quota, which by the way sometimes breaks... I got a "overquota bounce" when I filtered the reply notice that I got from you replying to this very message, even though I've never been anywhere near any limits. So all bets are off.

Is there other interest in this sort of service?

It sounds interesting but you would end up losing out on a lot of redundancy and fault tolerance in exchange for the added flexibility. That's a trade-off you're willing to make, I guess?



- Dallas
- DreamHost Head Honcho/Founder