Topic: Fighting spam with SME dedicated DNSBL

[Knuddi]

I have been building my own DNS blacklist over the last years for my own company (ScanMailX) and can see the positive effect of this in the constant battle against spam and other bad stuff which is email borne.

I was wondering whether there would be an interest for you to provide data to this list and in return get access to the total list for your SME server?

The philosophy would be that all users (SME Servers) would have to provide data in order to be allowed to use this DNSBL. This data will be collected automatically via a plugin to qpsmtpd and an associated script that runs on your server.

The plugin will collect information from all (or some) hard rejected emails and send data to the central server which then analyzed and add to the DNSBL. This data that would be sent is sending IP address, sender email address, qpsmtpd module that rejected and the reject message.

Interested?

Regards,
Jesper

[Stefano]

yes, I am

[brianr]

.. and me, although I am only a minor player in sme installations these days (I've got 5).

And my latest mailstats will log the use of the DNSBL and we can see if it genuinely is worth it!!

oh - and I'd be happy to help test it as well!

[Knuddi]

For a starter it will be entirely IP based but I also have an URI based blacklist that I can add to the mix.

@Stefano & Brianr,
When I have more ready I will let you know. Please btw. send me an email on jesper@swerts-knudsen.dk so that I can send you install information and other details.

/Jesper

[Fumetto]

I've one server with wan ip on 192.168.x.x (after a router I can't manage) and 4 server with wan public IP; the 1st is SME 8.x all other 9.1.
I can give my willingness to act as a tester, but the servers are in production, so testers for something that works ...

[Knuddi]

Its not a problem that the one server sits behind a NAT router - so does on of mine. The system will add a configuration file to SpamAssassin for the DNSBL and a qpsmtpd plugin that collects rejected data.

I am quite sure it works, but I need to have the installation process pipecleaned hopefully by Stefano and/or Brianr over the next days. If you connect as described above, then I will send you installation guide when ready.

The DNSBL currently has 1.3 mio validated spam IPs and around 350.000 URIs so we do not start from scratch.

/Knuddi

[calisun]

I don't know about you guys, but built is spam filer in SME Server 9 works great for me. I hardly ever get any spam in my in box.
When I check the junkmail folder, it has a lot of mail in there daily.

[Knuddi]

I have created a Howto for this at https://wiki.contribs.org/SMEOptimizer

The solution has now successfully been though several alpha testers and I think that its ready for additional users.

[brianr]

Jesper

I followed your howto, but at the end there was nothing in the /var/log/smeoptimizer.log (indeed it did not even exisit!).

[Knuddi]

Hi Brian,

I cannot see that any registration was attempted in the backend.

Did you remember the:

./SMEOptimizer.pl –initialize

Can you check whether these files are present:
/usr/local/smeoptimizer/smeoptimizer.cron
/etc/e-smith/templates/etc/crontab/smeoptimizer.cron.template


/Jesper

[brianr]

Neither of those file exist, when I run

./SMEOptimizer.pl –initialize

I get:

Code: [Select]

[root@bjsserver smeoptimizer]# ./SMEOptimizer.pl –initialize

         SMEOptimizer - Optimize your SME server
by SMEOptimizer.com - Copyright (c) 2016, all rights reserved.
 Servers hosted and operated by ScanMailX - www.scanmailx.com

Command line options:
-help:      Shows this help
-initialize: Register and retrieve the configuration and enable the cronjob services.
            When the registration has been confirmed, then all services will be activated automatically.

[root@bjsserver smeoptimizer]#


[brianr]

Found it - you need

./SMEOptimizer.pl --initialize

(note the --).

[Stefano]

you, just like me, just did a copy and paste

[brianr]

you, just like me, just did a copy and paste

the only way to get it right, except in this case it didn't!

[Knuddi]

Interesting as it works with one "-" on my 8.2 server. I all cases I have corrected the Howto to reflect two "-"

[brianr]

Interesting as it works with one "-" on my 8.2 server. I all cases I have corrected the Howto to reflect two "-"

9.1 here.

PS where can i find your spamassassin modifications? I've looked at your perl and the qpsmtpd plugin, but would like to eyeball the spamassassin bits as well.

[Knuddi]

The SpamAssassin additions will only appear when the server has contributed to the system and thereby the common DNS Blacklist. This means at least 50 spam samples right now.

When that happens you will get a smeoptimizer.cf file with DNS setting which will placed in the /etc/mail/spamassassin directory.

/Jesper

[brianr]

The SpamAssassin additions will only appear when the server has contributed to the system and thereby the common DNS Blacklist. This means at least 50 spam samples right now.

When that happens you will get a smeoptimizer.cf file with DNS setting which will placed in the /etc/mail/spamassassin directory.

/Jesper

Very clever - do i get a prize as well? 

Looking at the MySQL DB I'd say I was half way to that already!!

[holck]

I just installed the contrib. The log-file show messages like these:

Code: [Select]

7-10-2016, 11:18:12 - Cannot deliver spam report (500 read timeout)
7-10-2016, 11:18:12 - Providing 0 spam reports
7-10-2016, 12:18:02 - Providing spam report - Last: 2016-10-07 10:04:45
7-10-2016, 12:18:12 - Cannot deliver spam report (500 read timeout)
7-10-2016, 12:18:12 - Providing 0 spam reports
7-10-2016, 13:18:02 - Providing spam report - Last: 2016-10-07 10:04:45
7-10-2016, 13:18:12 - Cannot deliver spam report (500 read timeout)
7-10-2016, 13:18:12 - Providing 0 spam reports
There is also an error message in /var/log/qpsmtpd/current:

Code: [Select]

@4000000057f780a70c73bc8c 31088 smeoptimizer plugin (deny): Cannot insert into log - Duplicate entry '2016-10-07 13:01:49' for key 'PRIMARY'

Suggestions?

[brianr]

I've also got one of those:

Code: [Select]

7-10-2016, 08:34:25 - Executing: sv t qpsmtpd
7-10-2016, 08:56:01 - Providing spam report - Last: 0
7-10-2016, 08:56:02 - Providing 2 spam reports
7-10-2016, 09:56:02 - Providing spam report - Last: 2016-10-07 08:47:54
7-10-2016, 09:56:02 - Providing 2 spam reports
7-10-2016, 10:56:01 - Providing spam report - Last: 2016-10-07 09:41:13
7-10-2016, 10:56:02 - Providing 6 spam reports
7-10-2016, 11:56:01 - Providing spam report - Last: 2016-10-07 10:51:22
7-10-2016, 11:56:02 - Providing 5 spam reports
7-10-2016, 12:56:01 - Providing spam report - Last: 2016-10-07 11:40:19
7-10-2016, 12:56:03 - Providing 6 spam reports
7-10-2016, 13:56:02 - Providing spam report - Last: 2016-10-07 12:49:18
7-10-2016, 13:56:12 - Cannot deliver spam report (500 read timeout)
7-10-2016, 13:56:12 - Providing 0 spam reports
[r

[Knuddi]

@Holck & Brianr,

I have used the timestamp as key in the table and it seems that its not unique enough (I guess I should have known that...). Should not cause you any problems though and is not the cause for the "500 Read Timeout" issue. This is related to the Raid Battery in the server that picks up the reports is dead and the server now wants all writes to be comitted to disc before ack. This means a VERY slow server. Again, should not cause you any problems but just some replacement work on my side...

/Jesper

[brianr]

@Holck & Brianr,

I have used the timestamp as key in the table and it seems that its not unique enough (I guess I should have known that...). Should not cause you any problems though and is not the cause for the "500 Read Timeout" issue. This is related to the Raid Battery in the server that picks up the reports is dead and the server now wants all writes to be comitted to disc before ack. This means a VERY slow server. Again, should not cause you any problems but just some replacement work on my side...

/Jesper

Autoincrement is the only safe way to go....

[Knuddi]

I agree - and also now in the new codebase

[brianr]

I agree - and also now in the new codebase

Have you got a mechanism to update our DBs, or will we need to start again?

[Knuddi]

I might - Lets see whether it works. The script will at the next run modify the table and use id with auto increment.

[brianr]

I might - Lets see whether it works. The script will at the next run modify the table and use id with auto increment.

Seems to have worked - I can see autoincrement field "id" now...although I would question whether 6 sig figures is enough - remember the increment will not drop back after the table is emptied.  I'd go for 11...for extra safety.

[Knuddi]

All the log records are deleted as they have been delivered - all which is important is that they are unique.

[brianr]

All the log records are deleted as they have been delivered - all which is important is that they are unique.

Aha - i've never thought about it - i suppose the autoincrement will wrap round when it gets to 99,999?

[holck]

During the last, few days, one of my users has received a large number of spam messages. They promote all kinds of things - dating sites, company services etc.  They come from different IP addresses, and with different "From:" addresses. Interestingly, the "From:" addresses often show ordinary, Danish names, all different. Apparently the blacklists at Spamhaus and other places have not been able to register them yet.

At my work, we use Microsoft's Office 365, and have also seen a large amount of spam recently.

Jesper, Denmark

[Knuddi]

I have also seen these in high volume across user on many danish domain. Did a lot of Bayes training and had to esemi-manually build dedicated rules for these.

[Knuddi]

SMEOptimizer has now been enhanced with remote monitoring of the registered SME server. This means that it (its public IP) will be checked daily for registration in about 80 different public DNS blacklists. It will also on a regular basis (every 30 minute) be checked on the SMTP channel to see whether its online.

If the server has issues, then the admin will receive an alert via email.

See the updated Howto on how to configure these settings.

https://wiki.contribs.org/SMEOptimizer

Enjoy,
Jesper

[brianr]

Got this:

[root@bjsserver smeoptimizer]# ./SMEOptimizer.pl -status

         SMEOptimizer - Optimize your SME server
by SMEOptimizer.com - Copyright (c) 2016, all rights reserved.
 Servers hosted and operated by ScanMailX - www.scanmailx.com

Use of uninitialized value in printf at ./SMEOptimizer.pl line 236.
Contact Email       :   
Alerts              :   Yes
Spam Reports        :   111
Registered          :   2016-10-07 09:34:24
Last SpamReports    :   2016-10-09 20:44:01

Looks like no email set, so I set one (used -connect), and the uninit var went away!!

How do I know if I've been allowed to use DNS for spamassassin?

Also MY server is on a dynamic IP address, so your alert system will not really work?  My others ones which I'll add once I've got some more confidence are however on fixed IPs.

[Knuddi]

Yes, the first few servers registered did not set the default value for contact email - you were one of them

If you try the "--status" again, you will see a new line indicating whether you have DNS Blacklist access.

Alerts              :   Yes
Spam Reports        :   88
Registered          :   2016-10-07 16:03:52
DNS Blacklist       :   Active via SpamAssassin
Last SpamReports    :   2016-10-09 22:23:01

If you are on a dynamic IP, then you are right that SMTP checks are quite unstable and the check for blacklist listning most certainly. You should therefore consider to disable Alerts for that server.

[brianr]

So I see you are using spamassassin score of 5 for the situation in which one of your rules fire. This should at least push the email into the "move to the junkmail folder" category.

I am interested to know why you have chosen to use spamassasin rather than treating it as a "genuine" DNSBL or writing a custom plugin for qpsmtpd (which quite clearly you understand enough to do).

My hidden agenda is that I'd like to be able to identify the emails which are found by your package and identify them on the mailstats report. Unfortunatly the latest qpsmtpd (0.96) has removed the logging of the Spamassassin rules that have been applied so I've had to take out the spamassassin league table.

[Knuddi]

The reason for choosing SA is that I didn't want to make a hard trigger from start. If I created a qpsmtpd plugin that hard triggered a reject it could cause annoyance for now. Having said that, then it is the plan to do just that when the confidence level has been build up.

I can see that SA adds these to the mail header (X-Spam-Status) , so I will be able to pull these out and count hits on rules.

[brianr]

The reason for choosing SA is that I didn't want to make a hard trigger from start. If I created a qpsmtpd plugin that hard triggered a reject it could cause annoyance for now. Having said that, then it is the plan to do just that when the confidence level has been build up.

I can see that SA adds these to the mail header (X-Spam-Status) , so I will be able to pull these out and count hits on rules.

Good news that you intend to write a qpsmtpd plugin - I'll then be able to count the uses of it.  Althyough spamassassin adds a x-spam-status header to the email, it no longer drops it into the qpsmtpd log file, so the mailstats package cannot access it just now. I'll just have to look out for more emails ending up ion the junkmail folder...

[bosco555]

Hi All,

installed on a couple of servers this morning, but now I get this:

Contact Email       :   admin@mail.com (Email has been altered, of course)
Alerts              :   Yes
Spam Reports        :   418
Registered          :   2016-10-17 07:44:59
DNS Blacklist       :   Awaiting enough spam reports to be activated
Last SpamReports    :   2016-10-17 08:13:26

How many Spam reports are needed for the DNS Blacklist to be activated?

Thanks

[Knuddi]

That is odd - the trigger is 50 spam reports. Let me check why it hasn't been upgraded.

Got it - upgrade script has been fixed and you should have received mail confirmation of upgrade.

[brianr]

This seems a bit odd?

SMEOptimizer status:
   Alerts              :   No
   Spam Reports        :   20093
   Registered          :   2016-10-07 09:34:24
   DNS Blacklist       :   Active via SpamAssassin
   Last SpamReports    :   2016-10-19 00:26:05

20,000 is more than I'd expect in a year! Are you sure you are not keeping track through my IP which changes...

[Knuddi]

The counter is not related to the IP but the servers unique SME ID. So I guess that you receive more shi... than you think. Please notice that also directory attacks are counted and they often come in big waves.

[brianr]

The counter is not related to the IP but the servers unique SME ID. So I guess that you receive more shi... than you think. Please notice that also directory attacks are counted and they often come in big waves.

Actually you are right!!  I had 19,000 in 3 hours yesterday afternoon! All rejected as non-conformant.

[bosco555]

That is odd - the trigger is 50 spam reports. Let me check why it hasn't been upgraded.

Got it - upgrade script has been fixed and you should have received mail confirmation of upgrade.

Thanks Jesper..all good now...

[Knuddi]

Now some weeks later there are 39 servers registered that have provided 2598279 spam reports resulting in additional 5767 harvested IP addresses in the community DNS BL.

I can also notice that several of the registered servers are listed in various other international blacklists (e.g Spamhaus) so that I assume causes some admin thoughts as well

[Knuddi]

Just a small update on the status of the DNSBL. As of today (April 24, 2017), 23.987.906 spam reports have been registered from the 54 contributing servers in the setup. This has resulted in a DNSBL that serves 22448 active and bad IPs back to the contributing server.