Topic: Hash/SHA256 based attachment filter - catched WannaCry

[Knuddi]

Dear all,

I have now enhanced the SMEOptimizer (https://wiki.contribs.org/SMEOptimizer) with a new attachment filter functionality that showed its value during the recent WannaCry attack. As with the DNS Blacklist for IPs and URLs, this is also SME community enhanced (you contribute). All attachments will based on their SHA1 hash signature be checked against a central DNS based database - if matched they will be rejected. The actual attachment will never leave your server, only the signature.

If not matched directly as malicious, then the SHA1 and SHA256 signature is stored locally and then afterwards checked up against VirusTotal. If there is a match here, then the signatures (as well as size and filename) will be sent to the shared DNS database to all users benefit and can get this matched next time.

Old users should get the modules automatically and only need to configure the VirusTotal API. As of right now 25,827,548 spam reports have been submitted for the DNS Blacklist!!

Question or suggestions are as always welcome.

Enjoy,
Jesper

[jameswilson]

I'd like to impliment this but it seems a script install and not something that could be updated via yum?
I may be wrong of course but that's the only reason i haven't.

[Knuddi]

To make it available via yum, I would need to build an RPM and I just haven't come around to that (I do not have a build environment at the moment). But technically there is no problems with doing that. I would also like to make the server-manager UX but again - time hasn't permitted.

[Knuddi]

Pleased to see many user contributed malware hashed in the system already. Just shows that the SME server with all best practices enabled (as I am sure all do) still allow bad stuff through. When use base of this system grows a little we can catch these thing inline and before they reach the users inboxes...

[Stefano]

first of all thank you for your work

I installed your "plugin" on all servers of mine (mail servers, of course), 'cause I'm very interested in virus filtering
 

[Knuddi]

@Stefano,
Make sure to also register for the VirusTotal API key to get full effect.

[Stefano]

done, I'm waiting to see if I have some interesting feedback in the next few days

[ReetP]

I can probably flesh out a basic rpm with templates but would need a hand with some stuff.

Anyone willing to help?

[Stefano]

I could be interested.. Next week I'd have plenty of time

[ReetP]

OK,

I have vaguely fleshed out some files here:

https://github.com/reetp/smeserver-SMEOptimizer

Some notes/ comments:

I have physically added the qpsmptd/plugin to the rpm. The script can check for updates to this
I can't see the /etc/mail/spamassassin/smeoptimizer.cf on my testbox - I would have added it otherwise
DB needs to be created manually. Ideally it should be created from the RPM and a random password used

These keys could be in a DB entry (and later in a server-panel):

-alerts=[Yes|No]:        Enable checks and alerts for SME server downtime or registration in DNS Blacklist - default Yes.
-contact=[Email]:        Set the contact email address where alerts are sent to - default admin@<your domain>.
-VTAPI=[API Key]:        This is the VirusTotal public API key used to check attachments (will remain local).
-DNSBL=[qpsmtpd|sa]:     This configures whether the DNS blacklist lookup rejects directly (qpsmtpd) or scores (sa=SpamAssassin).

A smeoptimizer enabled/disabled key could also be added

e.g.

config set smeoptimizer service alerts Yes contact admin@somewhere.com VTAPI 12345wxyz DBSBL qpsmtpd

[root@test plugins]# config set smeoptimizer service alerts Yes contact admin@somewhere.com VTAPI 12345wxyz DBSBL qpsmtpd
[root@test plugins]# config show smeoptimizer
smeoptimizer=service
    DBSBL=qpsmtpd
    VTAPI=12345wxyz
    alerts=Yes
    contact=admin@somewhere.com

Those would need some rewriting in the main script, and with some files being templated in the RPM some of the setup/init stuff would not be required, but I do not want to touch the script right now !

You can either fork the repo or I can add you as a contributor if you are interested in helping.

To test:

Copying over the files manually

Add the DB

Run the file

/usr/local/smeoptimizer/SMEOptimizer.pl --initialize

See what breaks......

B. Rgds
John

[Knuddi]

@John,
Awsome good start. There is actually no need for more files that the SMEOptimizer.pl to start with, the rest is being pulled when using the "-initialize" command. What would be really interesting would be to automatically create the MySQL DB and store a random username and password in the smeoptimizer config DB. I can make the script pick up these rather than using pre-configured default username/password.

The .spec should also have a requirement for perl-Email-MIME (for the attachment filter).

I would obviously be ready to contribute to get this packaged well.

[ReetP]

Hi Jesper,

Thought I'd get the ball rolling

Get all that. I thought it might be easier to have the initial files in the RPM and then updates can be pulled. I'm never keen on having to drag stuff in on install personally. Not a biggy though.

Added the require to the spec thanks.

If I add some keys as described will you use them ?

eg:

smeoptimizer=service
    status=enabled
    DBSBL=qpsmtpd
    VTAPI=12345wxyz
    alerts=Yes
    contact=admin@somewhere.com (Have to think about this one)

DB - that's a good one and I need to have a look at that as I am not sure of the answer. I'll have a look at it for sure.

As far as contributing I can either add you to the git repo or you can fork it, or take it over yourself - I just did it for convenience really.

I can also give you a login to my build box if you want - it has git and a whole RPM mock build system on it. Let me know and I can give you an account.

B. Rgds
John

[Knuddi]

Its important that the configuration changes are made through the script and not directly in the DB. This because f.ex the contact address must be communicated to the central server or it will not be able to send alerts when server is blacklisted or having SMTP issues.

So for a starter, only the "smeoptimizer" DB should be created. The rest will be created during "-initialize" and when being configured.

It would be awesome to have access to a build environment and then jointly get this ready for release. Again, it should be super simple

Also remember that this contribs is and will not be static as the advantages (DNSBL and Attachment filter) will only be available and enabled if you contribute by sending spam reports and SHAs that VirusTotal detected. Therefore the backend (smeoptimizer.com) will start and stop these services on the fly. Its a community system - share to benefit.

[ReetP]

OK - well if you are happy to have a hack I'll get you an account on my box PDQ

I do think a 'enabled/disabled' key would be good though, unless you are going to build a web panel too !

[SchulzStefan]

I Installed the contrib, thank you for your work.

I registered at VirusTotal and got an email from alert@smeoptimizer.com:

"Thank you for registering your SME Server (9.2) with SME Optimizer. Your server is registered with its unique key and its public IP 84.120.123.53. With SME Optimizer you and your server contributes to the DNS blacklist that you also get access to. When your server has provided enough spam samples, then the SpamAssassin based DNS Blacklist rules will be downloaded automatically and enabled.

SME Optimizer will also on a daily basis check your servers public IP for listings in more than 80 international DNS blacklist and inform you via email in case you get listed. Additionally, the system will check your server from remote for downtime or other email issues. This is tested by trying to connect to the SMTP port 25.

Enjoy!

Best regards,
SME Optimizer"

As I have no static IP does this affect SMEOptimizer?

Thank's for answering.

Regards,
stefan

[SchulzStefan]

@Jesper

How can this be solved?

# ./SMEOptimizer.pl --initialize

         SMEOptimizer - Optimize your SME server
by SMEOptimizer.com - Copyright (c) 2016-17, all rights reserved.
 Servers hosted and operated by ScanMailX - www.scanmailx.com

Trying to register with SMEOptimizer...
ERROR: Couldn't register right now - try again later!Online registration for this specific SME server (Version 9.2), requires you to e-mail below unique indentifier to register@smeoptimizer.com.

UUID: 12345

You will receive a regitration confirmation back within 24 hours and the services will automatically be activated.


Sent an email as requested. Here's the answer:

MAILER-DAEMON@swerts-knudsen.dk
failure notice

"Hi. This is the qmail-send program at swerts-knudsen.dk.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.

<jkn@swerts.dk>:
Connected to 176.222.239.100 but sender was rejected.
Remote host said: 550 Sorry but domain not registered with ScanMailX (xyz.de)!"

Domain is registered...

[Knuddi]

@SchulzStefan

I assume that either the SME Optimizer server was offline or the box you try to install on not on the internet?

The UUID which is printed during the attempt to register is odd (do you obfuscate it?). What does your server indicate for SystemID when issuing (12345?):

#config show sysconfig

/Jesper

[SchulzStefan]

Jesper,

thank you for answering.

Server has access to the internet. Server is behind a firewall. Primary (local) domain is different to the subscribed domain. That means the local domain is xyz.local, the domain I registered is xyz.de. Output of config show sysconfig is:

# config show sysconfig
sysconfig=configuration
    InstallEpoch=1291393193
    KeyboardType=pc
    Keytable=de-latin1-nodeadkeys
    Language=de_DE.UTF-8
    PreviousSystemMode=serveronly
    Registration=none
    ReleaseVersion=9.2
    SystemID=12345

The UUID which is printed during the attempt to register is odd (do you obfuscate it?).

Yes, but it's the same as SMEOptimizer wants as unique identifier.

Regards,
stefan

[SchulzStefan]

Another server, which seemed to be successfully installed, brings up the following message:

Your SME server on IP address 84.131.659.13 seems to have SMTP problems.

SMTP Error log
Error message    Cannot open SMTP connection to 84.131.659.13 on port 25!
Last check    2017-06-18 12:03:18


Best regards,
SME Optimizer

Server is behind a firewall and is sending and receiving all email nearly (see Bug wbl/qpsmtpd bounces...) properly.

How to investigate on this?

Regards,
stefan

[Stefano]

If you're using fetchmail your server is not receiving emails, just fetching them

[Knuddi]

SME Optimizer tries to connect to your server (84.131.659.13) on port 25. If that is not possible, then it will report what you listed. So if you are on a DynDNS IP and it changes often then you can have a problem.

[SchulzStefan]

@Jesper,

I tried again registering, no luck.

# ./SMEOptimizer.pl --initialize

         SMEOptimizer - Optimize your SME server
by SMEOptimizer.com - Copyright (c) 2016-17, all rights reserved.
 Servers hosted and operated by ScanMailX - www.scanmailx.com

Trying to register with SMEOptimizer...
ERROR: Couldn't register right now - try again later!Online registration for this specific SME server (Version 9.2), requires you to e-mail below unique indentifier to register@smeoptimizer.com.

UUID: 12345

You will receive a regitration confirmation back within 24 hours and the services will automatically be activated.

I sent again an email and your server says you don't like.

<jkn@swerts.dk>:
Connected to 176.222.239.100 but sender was rejected.
Remote host said: 550 Sorry but domain not registered with ScanMailX (xyz.de)!

As I reported the local domain is different from the email domain. Helo is *.local, email From (Return-Path) is the registered domain. Server has no static IP.

What is checked? The domain from the HELO or the domain in the email-adress? If it's the HELO it's clear that there's nothing registered. It is a LOCAL domain.

If so, then the domain *must* be renamed to a LIVE/REGISTERED domain. Otherwise it'll not work. Am I right?

Regards,
stefan

[Stefano]

https://wiki.contribs.org/Email#Set_Helo_hostname
https://wiki.contribs.org/DB_Variables_Configuration#smtpd

[ReetP]

At a guess I think the issue is that the server uses a dynamic IP which I am not sure the script allows for.

I think it works on the basis that mail servers normally run on a static IP with properly configured dns records.

Whether it could be modified to handle dynamic dns I do not know.

[Stefano]

there's no problem in using a dynamic IP for a mail server (for receiving emails.. sending is another topic)

I have some servers on dynamic IP.. I use dyndns to setup an host for them and point my MX record on that host.. never had any issue, at all

[Knuddi]

I cannot see an attempt to register in the backend (from the -initialize command), so I suspect that it cannot resolve smeoptimizer.com or cannot connect with HTTPS.

Forget the email failure for now - its simply just because this email hasn't been enabled on the server.

[SchulzStefan]

or cannot connect with HTTPS.

Here's the reason - firewall was not opened for this port.

# ./SMEOptimizer.pl  --initialize

         SMEOptimizer - Optimize your SME server
by SMEOptimizer.com - Copyright (c) 2016-17, all rights reserved.
 Servers hosted and operated by ScanMailX - www.scanmailx.com

Trying to register with SMEOptimizer...
Registration successfull!

Will report further if I am on track now.

Regards,
stefan

[SchulzStefan]

@Stefano and ReetP,

thank you for following.

Regards,
stefan

[SchulzStefan]

there's no problem in using a dynamic IP for a mail server (for receiving emails.. sending is another topic)

I have some servers on dynamic IP.. I use dyndns to setup an host for them and point my MX record on that host.. never had any issue, at all

All the years I used my ISP as email relay (SME was/is configured to use fetchmail and SMTP to/from the catchall from my ISP).

From my point of view with SPF, DKIM, DMARC and at least the efforts to fight spam and viruses with SMEOptimizer it's time for me to think about a fix IP. It seems to me a lot easier to configure/control all these settings with a fix IP. If I'm wrong, please point me in the right direction. For now I ordered a fix IP from my provider, at the 26th this month we'll have it.

Maybe there are coming up new/other questions, for now, thank's to all who followed and helped.

Regards,
stefan

[Knuddi]

A static IP is scoring much better (lower spam score) in the receiving end so I would always choose that. Secondly, you are in better control with your SPF and it can be "tighter". Lastly your ISPs SMTP is guaranteed in poor spam conditions... Static IP and no ISP relay - Yes!

[warren]

@Knuddi
will this also catch the Erebus Linux Ransomware http://blog.trendmicro.com/trendlabs-security-intelligence/erebus-resurfaces-as-linux-ransomware/

Seems Virust Total Has some hashes for it :
SHA256 detected as RANSOM_ELFEREBUS.A:

    0b7996bca486575be15e68dba7cbd802b1e5f90436ba23f802da66292c8a055f
    d889734783273b7158deeae6cf804a6be99c3a5353d94225a4dbe92caf3a3d48

[Knuddi]

No these SHA256 are not currently registered in the DB which means no one has seen them yet. As of right now the community part has found and added 238 different signatures that will be rejected.

I will look into a model to add "preventive" SHAs to the DB.

[guest22]

Heads-up, it seems there is another attack under way (June 27, 2017).


@Knuddi, any prove of this in your monitoring systems please?

[Knuddi]

I cannot see any specific patterns that are unusual today or yesterday. The community has provided 9 attachments with bad stuff (trojans, virus, etc.) and most of these are zip attachments with a few Java variants (jar).

A good example of the many caught:
https://virustotal.com/en/file/79d5ae8f94e5320458f3ba5f7556590b7d3366ebd9eda21a77289b07687deba1/analysis/

[SchulzStefan]

SMEOptimizer reports from a wrong public IP. How can this be fixed? IP was changed in a static one, it seems that it's still the old dynamic IP.

Regards,
stefan

[Knuddi]

@Stefan,

SMEOptimizer just uses a stamdard HTTPS connection from your SME server towards the smeoptimizer.com server. So whatever IP your server uses it will use. Why do you think it uses a wrong IP and what is the problem with that?

[SchulzStefan]

@Knuddi,

Blacklist warning for your SME Server von SMEOptimizer Alert:

Your SME server with public IP address 84.130.159.73 has been listed in international blacklist databases.

This is not the public server IP.

Regards,
stefan

[SchulzStefan]

@Knuddi,

today I'm receiving again this email:

Your SME server with public IP address 84.130.159.73 has been listed in international blacklist databases. It has been observed registered now 21 day(s) in this database(s). This blacklist registration very often means that the emails sent from will not successfully reach their intended destination.

We assume that you are not a spammer and suggest that you right away initiate actions to identify the reasons for the listing. This could require security updates of client PCs in your organization including checks for virus and botnets and potentially, the reconfiguration of your mail server.


Blacklist    Reason    Return code
pbl.spamhaus.org    https://www.spamhaus.org/query/ip/84.130.159.73
   127.0.0.10


Best regards,
SME Optimizer

My public IP is a different one. I assume this IP was formerly the dynamic public IP and it's obviously still in the database of SME Optimizer. Do I have to re-register with my new static IP or how is this going to be updated?

Regards,
stefan

[SchulzStefan]

Opened a bug https://bugs.contribs.org/show_bug.cgi?id=10385

regards,
stefan

[Knuddi]

@Stefan,

Thanks for opening a bug, this makes it much easier to track and follow for me. Do not get to the forums that often

The problem that you have reported has now been resolved.

Enjoy,
Jesper