Topic: ASSP Anti-SPAM HOWTO

[Gene Cooper]

Hi all,

I've just written my first HOWTO based on various sources of info...including this list.  Thanks for your help!

Though I see much room for improvement, I hope this HOWTO will help others.

Please e-mail me with any comments, fixes, updates or rewrites.

PDF here: http://www.sonoracomm.com/sonoracomm/pdf/ASSP_HOWTO.pdf
or
RTF here: http://www.sonoracomm.com/sonoracomm/pdf/ASSP_HOWTO.rtf

G

[Michiel]

Maybe you could also publish your howto on http://contribs.org. That way we keep all great SME contributions in a central place.

Michiel

[Cyrus Bharda]

Gene,

Congratz on your first howto, looks great! You really should move it over to contribs.org, just so that more people can gain access to it

Your howto is for 5.6, is there any reason it wont work on 5.5 ?

Cyrus Bharda

[Gene Cooper]

Not that I know of...

G

[Gene Cooper]

A couple of ommisions weere pointed out...and fixed.  New version posted.

G

[Tom H.]

In your howto you say...
You will want to change:
Base (/usr/local/assp)
Spam Error (edit for a valid mail address)

I cannot find either of those options.  The only base is the Directory Base, but that will not accept /usr/local/assp

thx

[Tom H.]

after applying changes etc and rebooting, I cannot log onto that page anymore?

admin connection from 192.168.1.66 rejected by allowAdminConnectionsFrom

crap...anyone can help with this?

[gage]

I untared the package, and I get this error wile untaring  -Im starting to think that something was wrong with my tarball. Anyone have any ideas?


asspsmpl/notspam/2671

gzip: stdin: unexpected end of file
asspsmpl/notspam/2672
asspsmpl/notspam/2673
asspsmpl/notspam/2674
asspsmpl/notspam/2675
asspsmpl/notspam/2676
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
[root@pompano root]#

[Steve]

Sounds like you have accidently barred yourself from the admin webserver ?

I think you need to edit the cfg file or can you replace it with the working backup, failing that delete it and it will be recreated (you will of course loose your settings)

[gage]

everything seems to be running correctly except -
when I go and try to build the SPAM database and update the DNS blackhole list
I get this error , damn, can someone pls. help?

Thanks for this great HOWTO!!!!

Length: unspecified [text/html]

    0K .....                                                  @  44.94 KB/s

11:18:29 (44.94 KB/s) - dns.bz2' saved [5568]

mv dns dns.sav
mv: cannot stat dns': No such file or directory
bunzip2 dns.bz2
bunzip2: dns.bz2 is not a bzip2 file.
perl dnsbl.pl dns
db=./dnsbl
Can't open dns: No such file or directory at dnsbl.pl line 11.
 records written


[root@pompano assp]# perl assp.pl
loading config -- base='.'
Jun-22-03 11:23:10 ASSP version 0.2.1 initializing
Jun-22-03 11:23:10 Warning: bayesian spam database is small or empty: './spamdb'
Jun-22-03 11:23:10 Warning: DNS blacklist database is small or empty: './dnsbl'
Jun-22-03 11:23:10 Warning: Whitelist is small or empty: './whitelist' (ignore if this is a new install)
Jun-22-03 11:23:10 Listening for mail connections at 25 and admin connections at 55555
Jun-22-03 11:23:10 Starting

[Steve]

Yeah me too! today anyway. I assume the DNSBL file is not available, corrupt or the format has changed. Try another time.

The DNSBL only add a weighting and is not relied solely upon for determining spam, so dont worry.

The other warning about whitelist & spamdb will go away as soon as the files contain enough information

[gage]

thanks steve. Im going to just observe in test mode for a couple of days.

[Barry S]

Hi all,

It seems that the filename has changed for the DNS BlackList.

Update your updatednsbl.sh file to wget this file:

wget http://spfilter.openrbl.org/data/output/DEFAULT.octets.bz2 -O dns.bz2

It was originally looking for SPFILTER.octets.bz2

This should solve your problem.

-Barry

[Gage]

Thanks Berry, that did helped alot. But Im still not able to update the updatednsbl.sh

What wiered is that I have seen the log say that the whitelist has been updated.



[root@pompano assp]# ./nightly.sh
updatednsbl.sh: dns.bz2: command not found

01:27:53 (145.60 KB/s) - dns.bz2' saved [634382/634382]

[root@pompano assp]# loading config -- base='.'
Jun-29-03 01:28:21 ASSP version 0.2.1 initializing
Jun-29-03 01:28:21 Warning: bayesian spam database is small or empty: './spamdb'
Jun-29-03 01:28:21 Warning: Whitelist is small or empty: './whitelist' (ignore if this is a new install)
Jun-29-03 01:28:21 Listening for mail connections at 25 and admin connections at 55555
Jun-29-03 01:28:21 Starting
updatednsbl.sh: dns.bz2: command not found
bash: updatednsbl.sh:: command not found
[root@pompano assp]#

[Barry S]

Hey Gage:

I think that this is an issue related to the script (nightly.sh) that you are running. Most probably its not executing in the home directory of the updatednsbl.sh script (or there are path problems).

Two things:

(1)
As a test, can you 'cd' to the home directory of ASSP (ie. /usr/local/assp), and execute the following:

sh updatednsbl.sh

If that leads to the same error, try this:

(2)
It sounds that there is an error with the shell script updatednsbl.sh. Perhaps,  dns.bz2 is on a line by itself?

It should be on the same line as "wget".

See if this helps,
-Barry

Gage wrote:
>
> Thanks Berry, that did helped alot. But Im still not able to
> update the updatednsbl.sh
>
> What wiered is that I have seen the log say that the
> whitelist has been updated.
>
>
>
> [root@pompano assp]# ./nightly.sh
> updatednsbl.sh: dns.bz2: command not found
>
> 01:27:53 (145.60 KB/s) - dns.bz2' saved [634382/634382]
>
> [root@pompano assp]# loading config -- base='.'
> Jun-29-03 01:28:21 ASSP version 0.2.1 initializing
> Jun-29-03 01:28:21 Warning: bayesian spam database is small
> or empty: './spamdb'
> Jun-29-03 01:28:21 Warning: Whitelist is small or empty:
> './whitelist' (ignore if this is a new install)
> Jun-29-03 01:28:21 Listening for mail connections at 25 and
> admin connections at 55555
> Jun-29-03 01:28:21 Starting
> updatednsbl.sh: dns.bz2: command not found
> bash: updatednsbl.sh:: command not found
> [root@pompano assp]#

[Gage]

Thats exactly what the problem was. Some how dns.bz2 was on the next line.
I fixed it and ran the updatednsbl.sh and it worked w/o errors.
But,  When I run the nightly.sh I still get this error about the white list!!!
I know that Im getting errors like couldnt creat server socket on port 25 cause I already have it running. But the whitelist is always emptly.



[root@pompano assp]# sh nightly.sh
--23:20:03--  http://spfilter.openrbl.org/data/output/DEFAULT.octets.bz2
           => dns.bz2'
Connecting to spfilter.openrbl.org:80... connected!
HTTP request sent, awaiting response... 200 OK
Length: 642,410 [application/octet-stream]

    0K .......... .......... .......... .......... ..........  7% @  81.57 KB/s
   50K .......... .......... .......... .......... .......... 15% @ 157.73 KB/s
  100K .......... .......... .......... .......... .......... 23% @ 152.44 KB/s
  150K .......... .......... .......... .......... .......... 31% @ 157.23 KB/s
  200K .......... .......... .......... .......... .......... 39% @ 157.73 KB/s
  250K .......... .......... .......... .......... .......... 47% @ 152.91 KB/s
  300K .......... .......... .......... .......... .......... 55% @ 157.73 KB/s
  350K .......... .......... .......... .......... .......... 63% @ 153.37 KB/s
  400K .......... .......... .......... .......... .......... 71% @ 156.25 KB/s
  450K .......... .......... .......... .......... .......... 79% @ 157.73 KB/s
  500K .......... .......... .......... .......... .......... 87% @ 152.91 KB/s
  550K .......... .......... .......... .......... .......... 95% @ 157.73 KB/s
  600K .......... .......... .......                         100% @ 158.11 KB/s

23:20:08 (145.32 KB/s) - dns.bz2' saved [642410/642410]

[root@pompano assp]# loading config -- base='.'
Jul-6-03 23:20:46 ASSP version 0.2.1 initializing
Jul-6-03 23:20:46 Warning: Whitelist is small or empty: './whitelist' (ignore if this is a new install)
Jul-6-03 23:20:46 Couldn't create server socket on port '25' -- maybe another service is running or I'm not root (uid=0)?
Jul-6-03 23:20:46 Couldn't create server socket on port '55555' -- maybe another service is running or I'm not root (uid=0)?
Jul-6-03 23:20:46 Listening for mail connections at 25 and admin connections at 55555
Jul-6-03 23:20:46 Starting
[root@pompano assp]# loading config -- base='.'
bash: [root@pompano: command not found
[root@pompano assp]# Jul-6-03 23:20:46 ASSP version 0.2.1 initializing
bash: Jul-6-03: command not found
[root@pompano assp]# Jul-6-03 23:20:46 Warning: Whitelist is small or empty: './whitelist' (ignore if this is a new install)
bash: syntax error near unexpected token (i'
[root@pompano assp]#

[Gene Cooper]

Hi All,

I finally got back to this and fixed/updated the howto.  ASSP is at version 0.3.2 and heading for 1.0, I think.  ASSP is aggressive, configurable and uses several techniques to attack the spam problem.

Please point out any errors and I'll fix the HOWTO.

See: http://assp.sourceforge.net

and my updated howto at:

http://www.sonoracomm.com/sonoracomm/pdf/ASSP_HOWTO.pdf.

You might also want to look at Darrell May's DSBL contrib at http://mirror.contribs.org/smeserver/contribs/dmay/mitel/contrib/dsbl/.  This is a far simpler tool that's super-easy to install, zero maintenance and cuts out a good chunk of your spam.

See: http://dsbl.org

G

[Stewart Midwinter]

Gene, is there any harm in running DSBL as well as assp? or will they step on each other?  I started on installation of assp, wasn't getting any filtering, then installed DSBL.  Now I see from the .pdf how-to on assp that I have to untar the sample spam list and update my smtp port number, so I could finish the installation of assp if it is a better tool.

thanks
Stewart in Calgary
running SME Server 5.5

[Ron]

Darrell has just posted an ASSP contrib for SME.
http://mirror.contribs.org/smeserver/contribs/dmay/mitel/contrib/assp/

I am running DSBL as well as assp at the moment in test mode.  If you decide to try it let everyone know how you made out.

[Neal Collins]

IRon wrote:
>
> Darrell has just posted an ASSP contrib for SME.
> http://mirror.contribs.org/smeserver/contribs/dmay/mitel/contrib/assp/
>
> I am running DSBL as well as assp at the moment in test
> mode.  If you decide to try it let everyone know how you made
> out.

I'm running it and it's all working fine. Just make sure you run it in test mode initially as it tends to label just about everything SPAM first off. And make sure you set the Local Domains setting first up, otherwise it refuses all incoming email (I lost a couple of messages due to that)!

--
Neal

[Steve]

The DNSBL feature of assp is depreaciated in favour of the greylist feature.

[Ed Form]

Ron wrote:
>
> Darrell has just posted an ASSP contrib for SME.
> http://mirror.contribs.org/smeserver/contribs/dmay/mitel/contrib/assp/
>
> I am running DSBL as well as assp at the moment in test
> mode.  If you decide to try it let everyone know how you made
> out.

Is it actually possible to use ASSP if your smtp server is at your ISP? I loaded it onto my test machine and the configuration page boggled my mind, and the instructions on the Sourceforge page where slightly less use than a chocolate fireguard.

Ed Form

[Alexander Ziemann]

Hi folks,

we already had ASSP 1.0.6 working before on our SME 6.03 b Server.

But Installation following the Sonora.com Howto, using Darrels "old" beta rpm from June and additional the new ASSP 1.0.6 was ugly for me.

But we got it working at last - after a testing-period of 4 weeks, we are german  - it lasted quite a time to rebuild the spam and esp. notspam databases for our needs.

I have now reinstalled ASSP with the excellent noarch.rpm from Darrel and it works even better, because the starting skripts are in the proper place and there even is a hourly status ? script.

And that is needed, becaus assp quite often stops without any known reason.

ASSP works better than SpamAssassin, which i never really got to work on 6.03b

Alex

[Alejandro Lengua]

I installed Darrell's rpm's, however it seems it is marking all my incoming email as ::SPAM:: Why is this?

I have Testmode enabled.

[Neal Collins]

Yes, same thing happened to me. You have to train the software to recognise stuff that isn't spam. You do this in 3 ways:

1. If you send an email to someone, their address is added to the "white list", which means email from that address will not be marked as spam.

2. If you receive a message and ASSP has marked it as spam you can forward the email to assp-white@yourdomain.xxx and ASSP will add the sender to the "white list" as above.

3. If you receive a message and ASSP has marked it as spam you can forward the email to assp-notspam@yoursomain.xxx and ASSP will alter its rules to (hopefully) no longer consider that kind of mail as spam.

Conversely if a spam email manages to get through undetected, you can forward it to assp-spam@yourdomain.xxx to again update the rules.

Once I did a bit of training ASSP has been working really well. I especially like the blocking of executable attachments.

Oh, do run the assp-status.sh script, as mentioned by someone else, ASSP does sometimes just die and has to be restarted (which is what that script does). I run it from cron every 15 minutes.

[Jim Danvers]

Guys...

I've been following this thread and have but one simple question:  what about hardware requirements for either of the solutions ( the assp one or Darrell's ).  I run my little 6.0b3 box on an old pii 233 w/192 meg 'o ram.  It does have some disk space though - 30gig.  I know from previous experience playing with another product, (on another project) IPCop v1.3, where I wanted to implement the Dansguardian content filtering piece using similar hardware.  Come to find out that while the IPCop itself requires very little in hardware, the DG piece actually recommends at least a pii 400...  it showed too as performance on that box got kind of flaky.  In a nutshell - the filtering requires a lot of additional overhead on the cpu...  I can only imaging that the same would apply for the mail filters discussed in this thread.

Just curious - what are ya'll running for hardware, and have you noticed any difference in performance as a result of using either (or both) solutions?  I too get a lot of spam and fortunately I've got my mozilla client fairly well trained now to catch it all, but web mail is a different story.  ( I have to rely on the web interface when @work as I haven't convinced my net admin guy to open the imap port that I would need to hit my server from there....   ;(  )   I can upgrade hardware (somewhat) if required - it'd be nice to catch it 'at the door' so to speak.  

Thanks...

-=- jd -=-

[Alexander Ziemann]

Hi,
just follow the instructions

In the first 3 weeks or so (depending on your language and your "normal" mail-traffic) the system will just train an build up proper databases. This is a self-learning system with very rigid blocking attitude at the start.

Do not mind and do not forget to forward "wrong spam" to assp-notspam@yourdomain.com .

It will stop to bring up false positives soon, esp. if you got lots of outgoing mail.

az

[Alexander Ziemann]

Hardware Question:

I was running a self-made installation of 1.0.6 and the beta RPM from Darrel with 1.0.3 on my home-server PI /233 with 164 MB RAM and 9 GB disk it did not work out: Stopping all the time and  loooong database buildups at night, sometimes dying.

I took it off. I reinstalled Darrels new contrib.rpm with assp 1.0.7 now on the same server. It works fine! But: I do not make use of additional dnsbl, because that even stopped my office server.

You will see, that database rebuilds need about 7 minutes compared to my office server that is long (XP 1700, 512 MB->70 sec). But database rebuild is once at night, so i dont care about that.

The major improvement is darrels status-script, which checks the "oftendying" assp daemon and rewakes him as mentioned before.

az

[Neal Collins]

Is it just me or is anyone else using ASSP seeing a problem where the host stops accepting connections on port 25, but the ASSP daemon is still running (so the assp-staus.sh script thinks everything is ok)?

Restarting ASSP fixes the problem.

Also the status script has a bug - the status variable is called ASSPSTATUS, but later is referenced as BDSTATUS!

Anyway I might put see if I can change the status script to actually check that port 25 is functioning.

--
Neal

[Greg Zartman]

Darrell May has a descussion going on over on his boards about ASSP.  Seems the ASSP deamon dies from time to time, thereby killing SMTP.   I believe they've solved the problem.

Have a look at his boards and contrib area:  myezserver.com

Greg Zartman

[Neal Collins]

To follow up my own post, it seems that when ASSP is restarted after the nightly update, it fails to start up correctly because the ports (25 & 55555) are still in use.

I modified the /etc/init.d/assp script to put a 'sleep 5' between the stop and start commands to give it time to finish shutting down and it now seems to ok.

Otherwise ASSP has worked a treat!

--
Neal

[Alexander Ziemann]

Hi Neal,

how interesting! This may be it, but i am linux dummy  ; is this correct:

<        fi
        ;;
'sleep 5'
'stop')
        $stop
        RETVAL=$?
<
ASSP was dying just for that reason. I will test.

--
az

[Steve]

There is no requirement to shut down ASSP to rebuild the spam dbase - if thats what you mean by nightly update

[Neal Collins]

Steve wrote:
>
> There is no requirement to shut down ASSP to rebuild the spam
> dbase - if thats what you mean by nightly update

Interesting - this is the assp-update.sh script which is run daily at 4am. The last thing it does is a /sbin/service assp restart.

Anyway I changed line 66 of the /etc/init.d/assp script from:

$stop && $start

to

$stop && sleep 5 && $start

No more problems (so far anyway). BTW the 5 second delay was chosen at random - but it seems to work.

--
Neal

[Alejandro Lengua]

I have installed the last version of the DMay ASSP Contrib. And after 5 days
of being working no problems have been reported. BTW I haven´t changed anything.

I have a question: If I use the email delegation of SME will this allow me to
protect from SPAM another email server (eg: an Exchange Server or another SME BOX).

[Ross E]

I forward all emails (incoming and outgoing) to a Lotus Notes server (through a 5.6 box). If I installed ASSP would it work for me as SME is not really processing or storing the messages?

Would the install be any different?

[Alexander Ziemann]

Hi Neal,

i checked it over again at my 2 servers: No need for a 5 seconds delay.

ASSP worked fine, after i stopped using dnsbl , which i was doing additionally in the past.

Now i am using the standard installation provided by Darrel Mays contrib and it is working perfectly for some days now. Darrels scripts and no "dnsbl" entry in the config-file.

So this

> $stop && sleep 5 && $start

is not needed it seems.

az

[Neal Collins]

Alexander Ziemann wrote:
>
> Hi Neal,
>
> i checked it over again at my 2 servers: No need for a 5
> seconds delay.
>
> ASSP worked fine, after i stopped using dnsbl , which i was
> doing additionally in the past.
>
> Now i am using the standard installation provided by Darrel
> Mays contrib and it is working perfectly for some days now.
> Darrels scripts and no "dnsbl" entry in the config-file.
>
> So this
>
> > $stop && sleep 5 && $start
>
> is not needed it seems.
>

Odd. There must be something strange in my configuration - it needs the sleep here (and I don't have dnsbl enabled either). Oh well, it's working well now, so I'm not too fussed about it

- Neal

[Ruben]

Hi Neal,

I am also experiencing assp shuting down. My environment is: ASSP 1.0.9 (Darrels latest contrib) in SME 5.5 with P166MX laptop. In my case I isolated the problem easily since assp always stoped at the same time in the morning (looking at the maillog). No random hangs of assp service have happened so far (probably because the server has a low load).

What my server is experiencing is that whenever I run the assp-update.sh process assp is not restarted properly. It shuts down after the spam-db-rebuild (which always completes).

Curiously when I commented the spam-db-rebuild line in assp-update.sh this script restarts assp properly. In addition, when I manualy do a spam-db-rebuild, assp main process does nos hang. The problem (faulty restart) happens when the restart is done within the assp-update.sh after a spam-db-rebuild has been asked in the script. With or without low ram checked in the assp config.

Now, I am trying to alter the assp-update.sh script to avoid it happen (it always does), maybe my laptop needs to sleep a bit after the spam DB has been rebuild and the assp service is restarted... I'll write any progress...

regards,

Ruben

[Ruben]

BTW, I forgot to say that hints are welcomed ;-)

[Anonymous]

There is NO requirement to shut down assp to perform the rebuild of the database.

Why all these posts about restarting it and failure to start it ? Start it on server boot and leave it running.

[Ruben]

There is no requirement to shutdown assp in order to build the spam database, however the update script restarts assp after the spam db has been rebuild. My question is... is this necessary? does assp need to be restarted once the spam db has been rebuilt?

regards,

Ruben

[Anonymous]

It doesnt need to be stopped therefore as its running why would you need to restart it.

[Ruben]

I don't know how assp service interacts with the spam database. Maybe to restart the assp service is necessary after the database has been changed. I am quite new into this, but why would the assp-update.sh which runs daily do a spam db rebuild and afterwards force an assp service restart?
If the assp service works properly without any restart with a recently updated database taking into consideration all db changes, certainly it would be absolutely useless to restart the service as the assp-update.sh script does.


cheers,

Ruben