Topic: qpsmtpd - default 'helo' configuration seems to allow dictionary attack

[mmccarn]

After upgrading to SME 9.2 I was getting about 1000 "Failed Authentication" connections to qpsmtpd daily according to holck's mailstats.pl script - emails denied by qpsmtpd's "auth_cvm" plugin.

After a bit less than a month I installed Fail2ban , which reduced the count from 1000 per day to a bit over 100 per day.

Oddly, Fail2ban was showing lots of repeated 'Found' entries within seconds of each other in blocks of 8, 9 or more - despite having "MaxRetry" set at 3. The qpsmtpd logs showed up to 11 auth_cvm failures per second from these hosts.

Digging into this, I found that the offending IPs had all been flagged as 'naughty' by the helo plugin -- meaning that the connection was destined to fail, but would be allowed to attempt authentication first, in case it was a valid user connecting from an offending network.

In my case (2 users on my home SME), I felt that I didn't need to let failed helo hosts try to login.

I have done this:

1) Create a custom template fragment for the helo plugin that uses 'HeloReject' from the qpsmtpd settings instead of the literal 'naughty' **

Code: [Select]

mkdir -p /etc/e-smith/templates-custom/var/service/qpsmtpd/config/peers/0
cd /etc/e-smith/templates-custom/var/service/qpsmtpd/config/peers/0
echo "helo policy { \$qpsmtpd{HeloPolicy} || 'lenient' } reject { \$qpsmtpd{HeloReject} || 'naughty' }" > 15helo


2) set 'HeloReject' to '1' to reject offending hosts during connection and 'HeloPolicy' to 'rfc' to catch offending hosts that would not be caught by 'lenient' ("helo User" was quite common, which passes the plugin's 'lenient' checks but fails the 'rfc' checks)

Code: [Select]

config setprop qpsmtpd HeloReject '1'
config setprop qpsmtpd HeloPolicy 'rfc'

3) activate the changes

Code: [Select]

signal-event email-update

4) prepared myself to deal with any difficulties my wife users encounters

Since making this change (~ 24 hours ago) I have not had any new auth_cvm failures in the qpsmtpd logs.


To remove:

Code: [Select]

rm -f /etc/e-smith/templates-custom/var/service/qpsmtpd/config/peers/0/15helo
config delprop qpsmtpd HeloPolicy
config delprop qpsmtpd HeloReject
signal-event email-update



** here's what /etc/e-smith/templates-custom/var/service/qpsmtpd/config/peers/0/15helo should look like when you're done:

Code: [Select]

helo policy { $qpsmtpd{HeloPolicy} || 'lenient' } reject { $qpsmtpd{HeloReject} || 'naughty' }


[Jean-Philippe Pialasse]

Nice work.
Sounds like an interesting NFR.

[ReetP]

4) prepared myself to deal with any difficulties my wife users encounters

ROFLMAO....

The new version of qpsmtpd has a lot of new options and I think we are all still learning about it.

Can you add this as a NFR bug in the tracker please ?

[Daniel B.]

I fail to see the adventage of doing this. The new qpsmtpd doesn't allow more dictionary attack than before. This setting might reduce log noise, but it doesn't add security. Offending hosts could just as well brut force the IMAP service for example.

[ReetP]

I fail to see the adventage of doing this. The new qpsmtpd doesn't allow more dictionary attack than before. This setting might reduce log noise, but it doesn't add security. Offending hosts could just as well brut force the IMAP service for example.

He didn't say it did add security, and not every setting we have does...... It just cuts down the amount of noise you get which can be a distraction when you are looking for other issues.

In my case (2 users on my home SME), I felt that I didn't need to let failed helo hosts try to login.

Perfectly justifiable reason IMHO.

[Daniel B.]

Ok, I do not have a problem with that. It's just that the topic title is misleading. The default settings do not allow more dictionary attacks than before, and with this customization, it won't reduce the dictionary attack risks, it'll just reduce log noise (in some situations)

[mmccarn]

the topic title is misleading

Lack of imagination.

If you suggest a better subject, I'll change it.

Offending hosts could just as well brut force the IMAP service for example.

I hadn't thought of that... but then, I don't have any unexplained failed logins appearing in my dovecot logs so I don't have anything to work on.

[whine=self justification ]

I've become a bit paranoid about dictionary attacks in the last couple months after discovering a slow-motion attack at my office -- remote attackers attempting login to active-directory-authenticated services at a rate of 1 attempt every 2 minutes -- slow enough to avoid tripping the usual Active Directory account lockout (3 bad attempts in 5 minutes), but still letting an attacker try 700+ username/password attempts per day.

Combined with the various breaches of major online service username/password databases (onelogin, google, twitter, yahoo, etc...), I worry that a dictionary attack using the information from those breaches could find a user who is using the same password on one of my servers that he/she once used on a service that was breached.

I have also heard from folks who look like they know what they're talking about that the threat landscape has changed in recent years - shifting from lone attackers executing opportunistic attacks to coordinated groups leveraging advanced tools, botnets, etc to harvest and exploit vulnerability information in a "big data" way --  today's attack from the Netherlands may feed the same database as yesterday's attack from Iowa, or last week's attack from China.

[/whine]

[Stefano]

I've seen similar attacks on some servers of mine (but to wordpress)..

I suggest you to create a new rule for the needed services.. 5 failed login from the same remote IP during 12 hours.. enough for me to block the attacker.. if you receive some ranting from your locked out users, then you'll know that it's not an attacker but a stupid monkey

[mmccarn]

Update:

Since changing the settings of qpsmtpd's 'helo' plugin, that plugin has blocked 60% of the delivery attempts to my server:

Code: [Select]

# awk -F"[\t]" ' /logterse/ { svc=$6; count[svc]++; count["Total"]++; }  END { for (j in count) print count[j] "\t" j "\t" expr count[j]/count["Total"]*100"%" ; }' $(find /var/log/qpsmtpd /var/log/sqpsmtpd -ctime -14 -type f)|sort -n
12 auth::auth_cvm_unix_local 0.40282%
14 resolvable_fromhost 0.469956%
20 rhsbl 0.671366%
21 tls 0.704935%
26 loadcheck 0.872776%
72 check_goodrcptto 2.41692%
112 greylisting 3.75965%
119 naughty 3.99463%
323 queued 10.8426%
470 earlytalker 15.7771%
1790 helo 60.0873%
2979 Total 100%

Most of the blocked IPs  are listed in multiple RBL services. 

One IP (74.127.31.17) is responsible for 30% (554 of 1790) of the denied connection attempts, but doesn't appear in any RBL list according to mxtoolbox (as of this writing)

I can't find any evidence that any valid / desired emails are being blocked (but this server only has two users...)

NFR bug created:
NFR: template the qpsmtpd helo plugin 'reject' treatment

[devtay]

I tried this config and I can also confirm that the changes reduce the amount of SPAM I get and the amount of auth_cvm messages has gone to basically 0. However, I did get a bunch of valid emails blocked because of the HELO plugin. Two examples:

116.212.208.170   remote.airflite.com.au            helo   903   HELO hostname does not exist   msg denied before queued
207.35.254.194   mx.innotech-execaire.com            helo   903   HELO hostname does not exist   msg denied before queued

I'll grant that the reason the messages are being denied is poor configuration on the admin's part for these domains. However, I do need to get email from these domains.

In addition, I don't recall ever seeing this messages like this in my logs before:

45.59.119.139   Unknown            helo   903   no rDNS for 45.59.119.139   msg denied before

I know this is spam and should be blocked. It was something out of the 'norm' that I'm used to seeing in my log files. Just an observation.

The removal procedures work as advertised and I'm back to emailing with these non-conforming domains. For me, I choose to live with the log noise and allow the other plugins to do their thing. As a side note, I love the email filtering in 9.2. Major upgrade from prior versions. Thanks so much for all the hard work.

[mmccarn]

Thanks for the report.

I have no problem doing DNS lookups of remote.airflite.com.au and mx.innotech-execaire.com; I wonder if there's a related DNS component at work here...

If you wanted to keep working on it you could try setting 'HeloReject' set to '1', but leave HeloPolicy unset or set it to 'lenient' to restore the default policy:

Code: [Select]

config setprop qpsmtpd HeloReject '1'
config setprop qpsmtpd HeloPolicy 'lenient'

The given reason for defaulting HeloReject to 'naughty' is to allow mobile users to relay email... but I've never had any trouble sending or receiving email remotely from my phone or Thunderbird with HeloReject set to '1'; it's not clear to me what situation is improved by having HeloReject set to naughty...

[Daniel B.]

The given reason for defaulting HeloReject to 'naughty' is to allow mobile users to relay email... but I've never had any trouble sending or receiving email remotely from my phone or Thunderbird with HeloReject set to '1'; it's not clear to me what situation is improved by having HeloReject set to naughty...

The answer is in the helo plugin doc

Code: [Select]

If you have Windows users that send mail via your server, do not choose
I<policy rfc> without setting I<reject> to 0 or naughty.
Windows PCs often send unqualified HELO names and will have trouble
sending mail. The B<naughty> plugin defers the rejection, giving the user
the opportunity to authenticate and bypass the rejection

Indeed, with the default lenient policy, using naughty doesn't bring a lot, but it's recommended when using rfc or strict, or you most likely have valid users being rejected.

[devtay]

I'll try your suggestion for a few days and see how it goes. Thanks for following up. Here's a sample of the full log entries for two of the offending domains. I agree about DNS being the issue.

Code: [Select]

2017-08-21 07:21:36.829982500 11567 Accepted connection 1/40 from 116.212.208.170 / remote.airflite.com.au
2017-08-21 07:21:36.830097500 11567 Connection from remote.airflite.com.au [116.212.208.170]
2017-08-21 07:21:38.223728500 11567 (connect) earlytalker: pass, not spontaneous
2017-08-21 07:21:38.224534500 11567 (connect) relay: skip, no match
2017-08-21 07:21:38.224696500 11567 (connect) check_badcountries: GeoIP Country: AU
2017-08-21 07:21:38.224961500 11567 (connect) check_badcountries: Country AU RemoteIP 116.212.208.170
2017-08-21 07:21:38.473388500 11567 (connect) dnsbl: pass
2017-08-21 07:21:38.473761500 11567 220 ftmain2.ftav.com ESMTP
2017-08-21 07:21:38.726128500 11567 dispatching EHLO EXCHANGE.Airflite.local
2017-08-21 07:21:38.728282500 11567 (ehlo) helo: karma -1 (-1)
2017-08-21 07:21:38.728337500 11567 (ehlo) helo: fail, no such host
2017-08-21 07:21:38.728568500 11567 (deny) logging::logterse: ` 116.212.208.170 remote.airflite.com.au helo 903 HELO hostname does not exist msg denied before queued
2017-08-21 07:21:38.728659500 11567 550 HELO hostname does not exist
2017-08-21 07:21:38.728710500 11567 click, disconnecting
2017-08-21 07:21:39.248374500 2368 cleaning up after 11567

Code: [Select]

2017-08-21 07:39:35.295405500 14066 Accepted connection 1/40 from 207.35.254.194 / mx.innotech-execaire.com
2017-08-21 07:39:35.295534500 14066 Connection from mx.innotech-execaire.com [207.35.254.194]
2017-08-21 07:39:36.692931500 14066 (connect) earlytalker: pass, not spontaneous
2017-08-21 07:39:36.693768500 14066 (connect) relay: skip, no match
2017-08-21 07:39:36.693930500 14066 (connect) check_badcountries: GeoIP Country: CA
2017-08-21 07:39:36.694191500 14066 (connect) check_badcountries: Country CA RemoteIP 207.35.254.194
2017-08-21 07:39:37.091645500 14066 (connect) dnsbl: pass
2017-08-21 07:39:37.091945500 14066 220 ftmain2.ftav.com ESMTP
2017-08-21 07:39:37.143980500 14066 dispatching EHLO IAL-Iron.innotech-execaire.com
2017-08-21 07:39:37.184381500 14066 (ehlo) helo: karma -1 (-1)
2017-08-21 07:39:37.184439500 14066 (ehlo) helo: fail, no such host
2017-08-21 07:39:37.184656500 14066 (deny) logging::logterse: ` 207.35.254.194 mx.innotech-execaire.com helo 903 HELO hostname does not exist msg denied before queued
2017-08-21 07:39:37.184748500 14066 550 HELO hostname does not exist
2017-08-21 07:39:37.184797500 14066 click, disconnecting
2017-08-21 07:39:37.206636500 2368 cleaning up after 14066

Thanks for the report.

I have no problem doing DNS lookups of remote.airflite.com.au and mx.innotech-execaire.com; I wonder if there's a related DNS component at work here...

If you wanted to keep working on it you could try setting 'HeloReject' set to '1', but leave HeloPolicy unset or set it to 'lenient' to restore the default policy:

Code: [Select]

config setprop qpsmtpd HeloReject '1'
config setprop qpsmtpd HeloPolicy 'lenient'

The given reason for defaulting HeloReject to 'naughty' is to allow mobile users to relay email... but I've never had any trouble sending or receiving email remotely from my phone or Thunderbird with HeloReject set to '1'; it's not clear to me what situation is improved by having HeloReject set to naughty...

[mmccarn]

The EHLO hostnames  actually do fail DNS lookup:

...
2017-08-21 07:21:38.726128500 11567 dispatching EHLO EXCHANGE.Airflite.local
...
2017-08-21 07:39:37.143980500 14066 dispatching EHLO IAL-Iron.innotech-execaire.com

HeloPolicy 'lenient' instead of 'rfc' should ignore that -- although the admins of those servers probably have lots of trouble sending email to google, microsoft, yahoo, aol...

[devtay]

Agreed. The messages for bad HELO have dropped off and I'm able to process mail from the offending sites. I'm going to watch it for a little bit before moving forward with this setting. Thanks again for the assistance.