Spamassassin has stopped working

I have had Spamassassin 3.1.7 working quite happily for a couple of months now but it has suddenly stopped.

I've switched on logging and the issue appears to be with Spamasassin trying to find certain perl modules, i.e.

==================== LOG EXTRACT ===============
From Lorraine@SoftoSoft.com Mon Feb 26 14:34:29 2007
Subject: Credit Pensions
Folder: /mnt/smasher/vol/boot/randy/saffy/macrae/Maildir/new/1172529 21249
procmail: Assigning "MAILDIR=/mnt/smasher/vol/boot/randy/saffy/macrae/Maildir"
procmail: Assigning "INCLUDERC=/mnt/smasher/vol/boot/randy/saffy/macrae/procmail/spam.rc"
procmail: Executing "/mnt/smasher/vol/boot/randy/saffy/macrae/sausr/bin/spamassassin"
Can't locate Mail/SpamAssassin/Util/Progress.pm in @INC (@INC contains: /home/macrae/sausr/share/perl/5.8.4 /etc/perl /usr/local/lib/perl/5.8.4
/usr/local/share/perl/5.8.4 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl) at /mnt/smasher/vo
l/boot/randy/saffy/macrae/sausr/bin/spamassassin line 84.
BEGIN failed--compilation aborted at /mnt/smasher/vol/boot/randy/saffy/macrae/sausr/bin/spamassassin line 84.
procmail: Program failure (2) of "/mnt/smasher/vol/boot/randy/saffy/macrae/sausr/bin/spamassassin"
procmail: Rescue of unfiltered data succeeded
procmail: No match on "^X-Spam-Level: ********"
procmail: No match on "^X-Spam-Status: Yes"
procmail: Assigning "LASTFOLDER=/mnt/smasher/vol/boot/randy/saffy/macrae/Maildir/new/1172529814.30632_0.randymail-mx2"
procmail: Notified comsat: "macrae@0:/mnt/smasher/vol/boot/randy/saffy/macrae/Maildir/new/1172529814.30632_0.randymail-mx2"
==================== LOG EXTRACT ===============


Any ideas?

Dave

If you haven't changed anything, probly is fallout from power outage and router and filer communication problems. Junk filter also has problems:

http://www.dreamhoststatus.com/2007/02/26/mail-delay-on-junk-filtered-domains-2/


tor.eff.org
Promos: MaxTOR: Save Maximum TOR: Save Maximum and give $1 to TOR

I think I've fixed the problem.

I've changed the references to $HOME to be explicitly /home/macrae.

Dave

I got the same error, but only on some messages. (I think it was related to the DH networking problems.) It seems to have gone away now, so I think it was a coincidence that your fix solved the problem.

What I plan to do from now on is send any mail that does not get an X-Spam-Status header to a Rescan folder and re-run procmail on those messages (eventually through a cron job).

I've just noticed that my spamassassin install (updating to 3.1.8 is when i noticed it the other day) is getting errors on occassion like the following:

procmail: Program failure (-11) of "/home/XXXX/bin/spamassassin"

it's not on every email, by any means, but I don't see any other reason for it, and can't find what the return codes are (about to resort to digging into the source). This appeared to start around 2/26/07. has anyone else experienced this?

My installation of 3.1.7 started crashing on March 5 with core dumps and the same error code -11 to procmail. man spamassassin has no such return code. It crashes about half the time, but not in a regular pattern and not correlated with density of incoming local mail. It had been running regularly without this error for months and the inception of the problem was not correlated with any changes such as local upgrades to spamassassin or perl.

Dreamhost support gave this not very encouraging response:

In reply to:

---

This is an issue we're aware of, and looking into, though as we don't normally support custom software installations such as yours I don't know if we'll be able to figure it out.

---

If dreamhost's system-wide installation of its own custom spamassassin were not so far out of date this wouldn't even come up because local installations would not be necessary.

Yeah, I'm getting the same thing; Sometimes SA runs, sometimes it doesn't and the procmail log shows Failure (-11) of spamassassin.

Its pretty weird, especially considering from command line if I:
$ procmail < ~/Maildir/cur/offendingmessage
it will parse the message through SA every time. Likewise, if I:
$ spamassassin < ~/Maildir/cur/offendingmessage
It returns the (correct) SA headers to stdout.

It appears to me that somewhere SA is hanging on procmail's invocation of it, but for the life of me I can't figure out why or where. I'm having poor luck trying to google up some info on what error -11 is.

I got the exact same crappy support response to the same problem. It is the same as the previously reported problem at http://discussion.dreamhost.com/showflat.pl?Cat=&Board=forum_programming&Number=59994&page=0&view=&sb=&o=&part=1&vc=1 which they fixed by re-installing perl. If you look carefully at your messages, if a message is processed by spaceymail-mx2, spamassassin runs fine. If a message is processed by spaceymail-mx1, spamassassin segfaults (signal -11 is segfault).

So, the issue is clearly not with any of our individual spamassassin installs, but with inconsistency between mx1 & mx2 mail servers. Since we can't log in to mx1 or see any logs to narrow the problem down, only dreamhost can truly fix the problem. It may be possible to do your own non-broken perl install, but I think that crosses the line of what I expect from a hosting provider.

PLEASE, for everyone's sake, if you are experiencing this problem, PLEASE submit a support request, and keep nagging them til they fix it. It is THEIR problem, and as ewv mentioned, if their default spamassassin didn't suck so bad because they crippled it, they wouldn't have to deal with people's own individual spamassassin problems. This is ridiculous.

Wow, what insight. What you say (and the other post says) is exactly what's happening - Spaceymail-mx1 is fouling up and mx2 has all the right headers.

It appears that the same problem last November was fixed by reinstalling PERL on mx1, so hopefully they will do this (and soon).

I just wrote support and I agree with skiingac that the more ppl that send in this support request, the more attention it will get (and faster it will get fixed.)

in november, it took me about 1 week to convince them it was THEIR problem, and another week for them to fix it. So, don't hold your breath. Right now I am working on installing my own perl to see if that makes any difference, but it is a real pain in the neck getting the paths right and getting all the modules installed.

Well, I hope they can fix it sooner this time. If you make any progress installing your local version of PERL, let us know.

That explains why I haven't seen a problem; my mail is handled by the "randy" cluster.

As I mentioned earlier, you can detect whether SpamAssassin ran or not, and if something broke, filter the mail into another folder. Then you can re-run procmail (eg from a cron job), and it would be running on your "home server" rather than the mail server.

Another thing that you can do is add something like the following in a procmail file:

# diagnostics
:0fw
* ^Subject: procmail-diagnostics
| $HOME/bin/procmail-diagnostics

Then when you send a message with the subject procmail-diagnostics, you can run a program to collect information from the point of view of the mail servers. This could run SpamAssassin with debug flags, which might reveal exactly what's wrong.


Larry

and then when their perl install on my "home" server is broken (which apparently is MY fault), what am I supposed to do?

their response was:

In reply to:

---

We are working on this issue, and have tried re-installing perl on
spaceymail-mx1, to no avail. The amount of time our admin and
development team can give to custom software is small due to the large
amount of custom things customers have installed, and of course because
they have plenty to do anyways.

---

So, apparently perl is "custom software", because the problem is CLEARLY with perl. Perl segfaults on spaceymail-mx1, but not on spaceymail-mx2 or gatorade. Just because the perl script in question is spamassassin and not some 5 line perl script that I could probably concoct so that perl crashes in the same way if I really wanted to piddle my time away, does not mean I am using "custom software". Note that my complaint is not "help me setup spamassassin" or "spamassassin gives error X", it is that perl, which is listed as a feature of their hosting, is seg faulting. That's ridiculous.

So... I finished installing my own perl (5.8.8 with all the defaults except the paths obviously), and what do you know spamassassin no longer crashes on spaceymail-mx1! I told them this, but apparently they are really busy so who knows when/if they'll fix it. I wonder whether they even lifted a finger so far on this issue, because it worked on the first try for me when I installed perl.

If you want to get it working yourself, first complain to DH again for good measure, then download perl 5.8.8, install it with all the defaults but tell it different paths as described at http://forum.dreamhosters.com/3rdparty/28969-thread-enabled-perl-breaks-Interchange.htm and then set your PERL5LIB in your ~/.bashrc to /home/<username>/software/lib/site_perl/5.8.8/:/home/<username>/software/lib/5.8.8/ and re-install spamassassin & its pre-reqs as described at http://wiki.dreamhost.com/index.php/SpamAssassin using CPAN & your new perl install.

In reply to:

---

We are working on this issue, and have tried e-installing perl on spaceymail-mx1, to no avail. The amount of time our admin and development team can give to custom software is small due to the large amount of custom things customers have installed, and of course because they have plenty to do anyways.

---

What a ridiculous, evasive response. The problem is not spamassassin, which is not 'custom software' to begin with; the problem is one of the dreamhost mail servers. No one is asking that dreamhost support 'custom software'; they are supposed to be supporting the ability to run normal applications without crashing them. And if the specific problem is perl, it is the dreamhost system installation of perl, not 'custom software'.

Local spamassassin is working on mx1 again. Dreamhost support reports:

In reply to:

---

We just moved spaceymail-mx1 onto new hardware to address it's mysterious troubles with spamassassin and perl, and everything seems to be working now. I've tested out your spamassassin and it's not segfaulting now. If there are still any problems please let me know.

---

I have verified that my X-Spam headers are working on mx1.

Great news, got a new response:

In reply to:

---

We just decided to move spaceymail-mx1 to new hardware, and now that
that's done spamassassin seems to be working on it again. If you're still
having any problems with your email please let me know.

---

I'm afraid to mess up my brand new spamassassin & perl install to test this. Anyone out there with just a custom spamassassin install (and not your own perl) that is getting mail successfully filtered by spaceymail-mx1? Please let me know so I can thank the person who fixed it!

i just got the same email you did.

just turned procmail logging back on.
i will post if i continue to have problems.

no problems yet.

no problem so far but spamassassin isn't working as well as i like and is requiring to much oversight.

i am going to give gmail a try and see how that works out.

For spamassassin to work well, make sure bayesian filtering and auto-learning are on. You should also lower your spam threshold to around 3.5 or so. After 2 weeks or so, enough mail will have been processed so the bayes database works well (I think it takes ~300 emails, you can speed this up using sa-learn on any existing email), and auto white listing will help eliminate any false positives.

You should also set up a folder where you can drop missed spam to have spamassassin learn from it, that has helped me tremendously and is easy to use since you just drop the messages in the folder.

I can post my config files and the cron entry to learn from missed spam if you want.