SpamAssassin

Has anyone installed SpamAssassin on a dh server? How is it working? Worth the install?

I've never used it on a Dreamhost server but I use it on our internal office mail gateway and I gotta say that I love it, catches 90% of my spam and labels it as spam and filters it to a spam folder. Only thing missing from it is catching all these Klez email viruses.

- wil

It's installed (but not "officially supported") on all of our mail machines. The example documentation and man pages may or may not be installed on your user machine, but can be installed easily.

Since it's somewhat processor intensive, we haven't made an official announcement about this. Since there aren't that many people on this forum, I don't mind mentioning it.

I can forward directions to people who are interested; they do assume that you are comfortable messing around with this stuff, and that you know how to edit text files etc.

Is it setup on all Dreamhost mail accounts with some default minimal options or something? As I don't seem to be recieving that many number of spam on my dreamhost account compared to a few others ...

- wil

> Is it setup on all Dreamhost mail accounts with some default minimal
> options or something?

Nope. By default we perform no filtering on customer mail. We'd rather err on the side of caution where customer mail is concerned.

We'd like to be able to do some global tagging (but not blocking) of mail for spam, but we'd have to perform a lot of optimizations before we'd be able to process all of our incoming mail through spamassassin (remember that our largest group of email servers has about 22k users on it at the moment).

At some point, I'd like to integrate the per-user UCE patch for Postfix, which would allow customers to "opt-in" to different blocklists on a per-user basis. This would require some backend adjustments, but would be a good compromise between customers who want to agressively reject spam and those who don't want to risk losing important business related mail.

I'm glad to hear you're receiving less spam to your DH account than to others... my only guess would be that you've been more careful with addresses at domains that are hosted with us.

Hm. The per user UCE control patch for Postfix looks good, Will. I'm sure a number of people would appreciate something like that.

I can understand and appreciate the reluctance to generally filter any incoming mail. That's a good thing. However, I guess my only concern with the Postfix patch would be to educate users. It looks like quite a powerful tool and if someone's quite not sure what they're doing it could result in some lost emails.

A global tagging of spam email -- now this idea I like. I know this would mean a big impact on the mail servers, and unfrotunately on the mail servers is somewhere where you try to avoid any unneccessary overheads -- but if there was such a patch or a hack for postfix that would just add an extra header to spam email flagging it as spam so the user would have the choice of easily fileting out to a special mailbox on their side.

Hmmm.. Sorry, I'm just wondering out aloud to myself here. :-)

Me, myself, I don't see spam as much of a problem (thankfully) as I do have SpamAssasin installed on our local gateway here, but I don't know how much of a problem spam email is for others.


- wil

We've had some discussions on global content filtering recently.

There was an interesting thead on the postfix-users mailing list that had some good suggestions on how to optimize performance with Spamassassin...

These threads (as well as some threads that Jeff forwarded from the SA-Talk list) were interesting (follow the 'next in thread' links to view the next message).

http://marc.theaimsgroup.com/?l=postfix-users&m=102407222401993&w=2
http://marc.theaimsgroup.com/?l=postfix-users&m=102616301303570&w=2

This comment (from the SA-Talk list; too lazy to google for a link right now) was also interesting:
> To use SpamAssassin in a production environment, we do a number of
> optimisations. We patched Net::DNS to remove the use of $& , which
> speeds up all SA regexp matches by orders of magnitude.

So most likely it would have to be run as a Postfix content-filter, make no changes to the body (spam report in the headers), and we'd have to skip checks that are more epensive.

Other (good) suggestions were to run spamd on a different machine from the mail machines. However the content-filter itself needs to be very robust, and failure tolerant. I am leaning towards Amavis at this point (well one of the many Amavises that are around right now), even though I have very little experience with any of the Amavis variants (we use RAV for virus scanning here in the office).

Anyway an interesting idea, and we're certainly weighing our options. Jeff can attest to the fact that I was *very* hesitant to even consider something like this. I still have some reservations about Spamassassin for such a large, production level project, even though I find it very effective for keeping spam out of my inbox.

Ok - back to sleep - kinda tied one on last night, and I keep trying to enter vi commands (hitting escape and what not) in this browser text box (not sure if the two are related)..... Sick, I tell you.

This all sounds very interesting, Will, and would certainly be yet-another-added-value service for Dreamhost customers.

I'll take a ganda over some of those links laters.

> Ok - back to sleep - kinda tied one on last night, and I keep
> trying to enter vi commands (hitting escape and what not)
> in this browser text box (not sure if the two are related).....
> Sick, I tell you.

So now you're so desperate to keep up with my message count you're staying up all night making sure you grab any new messages first?! Sheeeeeeeesh. :-)

- wil

I'm having major issues getting spamassassin to work -- can you forward me some hints to getting it working?

Thanks,

Jonathan

Sure... send me a message on here with your email address, or email support.

We're working on getting official support for SA going, as well as possibly allowing customers to opt-in to individual blocklists (on a per-user basis)...

I'm also starting a DNS based blocklist for internal use, which will block some of the major offenders....

jsalkoff(at)yahoo.com

tx

hey, i'd love some tips, too...pretty new around here...perhaps SA is way too much overkill...

thanks

jack greenwood

Send me your email address privately on here (or submit a support request) and I'll get you DH-specific directions. There are also some hints at a link on one of the kbase articles...
http://donkin.org/bin/view/Main/SpamAssassin
also, the general procmail article at:
https://kbase.newdream.net/index.cgi?area=2626
may be of help.

Note that SA is broken on the web machines, but it's not necessary for it to work properly there. It is installed (and works) on the mail machines.

What does broken on the Web machines mean? I am using it on frigga and it seams to be working fine. I would think that we really can't use it on the mail machines anyway since we can't login and create the necessary procmail settings.

Jeff

The version installed on the web machines doesn't work:
frigga% spamassassin
Can't locate Pod/Usage.pm in @INC (@INC contains: /usr/lib/perl5/5.005/i386-linux /usr/lib/perl5/5.005 /usr/local/lib/site_perl/i386-linux /usr/local/lib/site_perl /usr/lib/perl5 .) at /usr/bin/spamassassin line 8.
BEGIN failed--compilation aborted at /usr/bin/spamassassin line 8.

Since it's *run* on the mail machine and not on the web machine, it works anyway.

>I would think that we really can't use it on the mail machines anyway since
> we can't login and create the necessary procmail settings.

Wrong... the mail machines mount the same home directory as the web machine - so even though you can't login, you can make the necessary changes.

If you were actually using it on frigga, it wouldn't process any of the mail coming into your domain - just mail coming directly to frigga.

Ok, I think I understand now. The shell accounts as well as the mail accounts are mounted on both the mail and web server. Since the mail server is actually doing the mail handling thats why it works.

Jeff

Right - we use a central filer for data storage, so your home directory is the same on both machines.

I've been using and loving SpamAssassin for at least a couple months now. One thing that doesn't seem to work for me, though, is the whitelist. I have a few thoughts about this:

1) Here's what my /.spamassassin/user_prefs file looks like (the whitelist portion):

# Whitelist and blacklist addresses are now file-glob-style patterns, so
# "friend@somewhere.com", "*@isp.com", or "*.domain.net" will all work.
# whitelist_from someone@somewhere.com
whitelist_from ADDRESS_THAT_ALWAYS_ENDS_UP_IN_SPAM_FOLDER

2) How does auto-whitelisting work?
# use spamassassin -Pa if you want to use auto-whitelisting

3) I have my own spam block list in a recipe file that is called BEFORE spamassassin is called, so I'm guessing certain addresses from particular friends are being caught by this for some reason, although I have no idea why. The messages are NOT being labeled with the same text that SpamAssassin uses for messages, so that's the only thing I can figure.

4) Here's the end of my .procmailrc file:

#INCLUDE FILES
#=============
#INCLUDERC=$PMDIR/rc.inbox
INCLUDERC=$PMDIR/rc.autoresponders
INCLUDERC=$PMDIR/rc.lists
INCLUDERC=$PMDIR/rc.spam
INCLUDERC=$PMDIR/rc.toblocks
INCLUDERC=$PMDIR/rc.spamassassin
~
# Messages that fall through all your procmail recipes are delivered
# to your default INBOX (to find out yours, see step 2 above)
:0
$HOME/Maildir/


I'm not sure why that tilda (~) is there after the last include. Should it be? It may have inadvertently ended up in that file.

Thanks,
Gabe

1) I don't use the whitelist at all, but looking at some other employees' user_prefs files, that looks like the right syntax.

2) Auto whitelisting works by remembering the score of previous messages from an address; thus if someone sends you lots of un-spammy messages and then one spammy looking one, the high score will be mitigated by the whitelist adjustment; similarly, if someone sends you lots of spammy email, they'll be scored up higher. That's my understanding of it, at least.

I honestly rarely have to whitelist people, but I filter 96% of my mail before it even gets to Spamassassin. Only unmoderated lists and stuff that's going to my inbox goes through SA.

3) *coughLOGGINGcough*.

In other words, turn on procmail logging, and see why messages from those people are being filtered. If they're not being tagged by SA, this is almost definitely your problem. I hope that you're filtering them to a separate folder and not to /dev/null? Be very careful when sending stuff to the great bit-bucket in the sky. I try to use very specific recipes when I killfile people to avoid accidents.

4) The tilde shouldn't be there (perhaps you cut and paste from a vi window)?

Hi Will-

Thanks for the reply and suggestions. I've removed the tilda and have turned on autowhitelist to see if it helps. The only messages I send to /dev/null are from specific spammers when their messages leak into my inbox - once is enough! I don't fliter to /dev/null on subject - only FROM addresses.

I do have logging enabled and that doesn't give me any clues as to why a particular friend's messages are consistently dumped into my spam box:

From USER@usa.net Mon Oct 14 19:36:42 2002
Subject: Re: [RE: finally]
Folder: .spam/new/1034649402.31858_2.ludo

Here's my one clue, though: If I "fake" an email from this friend using her address and send to myself, it does NOT go into spam box. So I'm wondering if my own spam recipe is somehow picking up hidden header info from usa.net? I've double-checked that this address, nor any of the words in subject line, are filtered on. Could it be the [ ] that show up in reply? I don't think so, but you never know...

Thanks,
Gabe

My guess would be that there's some sort of non-obvious error in your procmail recipes; perhaps you've setup a regular expression that does something slightly different than you think it does.

You could try setting:
VERBOSE=yes
in your .procmailrc
and have your friend send you another message (or use Pine's 'bounce' feature to bounce the message back to yourself). This should give you an idea of which rule is being matched.

** Important ** - turn verbose logging back off when you're done, or your logfile will get very large.

I tried posting this the other day, but apparently I forgot to confirm or it didn't show up...

In any case, thanks, will, for the suggestion about turning on verbose logging. I did so and realized that one of my subject-based filters was catching non-spam as spam (even though the words in the filter were not in the message - go figure). in any case, i removed all subject-based filters of my own and so far, so good.

On another note, last week I was out of town and didn't check my email for several days. When I finally did on Sunday, there were a TON of spam messages in my inbox, so I immediately knew that something was wrong with either procmail or spamassassin. Sure enough, it seems that the auto-whitelist function stopped working (it HAD been working fine):

From tirzahyofbiz@especially.optingnow.com Sat Oct 19 10:10:49 2002
Subject: Work from home, earn THOUSANDS! jwlzuq
Folder: /home/USERNAME/Maildir/new/1035047449.29309_2.ludo 3470
Cannot open auto_whitelist_path /home/USERNAME/.spamassassin/auto-whitelist: No locks available
procmail: Program failure (70) of "/usr/bin/spamassassin"
procmail: Rescue of unfiltered data succeeded

I've since deactivated the auto-whitelist and no problems have occurred since. Any thoughts?

Thanks,
Gabe

Hrmm - looks like a locking problem - occasionally NFS has some weird locking problems.

If you re-enable it, do you still have the problem? There's an 'autowhitelist.lock' file in your .spamassassin directory, so try removing that.

The WIKI says to run spamassassin directly, which I'm told is slow and a burden on the system. spamc/spamd is supposed to be the solution, but when I switch to spamc I get no x-Spam headers in my emails at all, as if spamassasin is not working at all this way.
Anybody else got this to work?

The WIKI says to run spamassassin directly, which I'm told is slow and a burden on the system. spamc/spamd is supposed to be the solution, but when I switch to spamc I get no x-Spam headers in my emails at all, as if spamassasin is not working at all this way.
Anybody else got this to work?