Topic: Too many connections: 10 >= 10. Waiting one second.

[bunkobugsy]

Great! Would this also work for port 465/sqsmtpd?

db configuration setprop ssmtpd DenyHosts xxx.xxx.xxx.xxx
signal-event remoteaccess-update

Thanks.

[ReetP]

Great! Would this also work for port 465/sqsmtpd?

db configuration setprop ssmtpd DenyHosts xxx.xxx.xxx.xxx
signal-event remoteaccess-update

In short yes it will.

Code: [Select]

config setprop ssmtpd DenyHosts 1.2.3.4

Code: [Select]

signal-event remoteaccess-update

Code: [Select]

grep "1.2.3.4" /etc/rc.d/init.d/masq

# ssmtpd: TCPPorts: 465, AllowHosts: 0.0.0.0/0, DenyHosts: 1.2.3.4 --destination $OUTERNET --src 1.2.3.4 --jump denylog

However, it means you have to keep adding it manually.

In the event of an attack your mail server will be locked up before the IP is banned.....

Fail2ban is a more effective method of stopping this.

[bunkobugsy]

Indeed.
I also upped Instances to 40 for ssmtpd too since all our laptops are set up to 993/465.

Curious, ssmtpd doesn't show an InstancesPerIP in db config as smtpd does. Coincidence?

[ReetP]

Curious, ssmtpd doesn't show an InstancesPerIP in db config as smtpd does. Coincidence?

Check the templates. Always check the templates

Look here:

/etc/e-smith/templates/var/service/sqpsmtpd/runenv

cat INSTANCES
{
    return "INSTANCES=" . ($ssmtpd{Instances} || $smtpd{Instances} || "10");
}


cat INSTANCES_PER_IP
{
    return "INSTANCES_PER_IP=" .
   ($ssmtpd{InstancesPerIP} || $smtpd{InstancesPerIP} || "5");
}

So it defaults to 10/5 if there is no config entry set (a lot of template set defaults in the absence of properties)

The issue (which is what the bug is about) is that qpsmtpd seems to ignore the PER_IP setting.

[kruhm]

Clarifying and consolidating for anyone who follows. The issue is concerning the sqpsmtpd.

qpsmtpd is for receiving mail (port 25).

sqpsmtpd is for sending mail (port 465).

The following command shows the configuration for sending:

Code: [Select]

# config show ssmtpd
ssmtpd=service
    Authentication=enabled
    Instances=10
    TCPPort=465
    access=public
    status=enabled

Instances=10 meaning 10 people can send email at anyone given millisecond. Once the connection is finished, it is available for the next.

InstancesPerIP is absent on most configs and will use the default of 5. InstancesPerIP can be set if needed but typically most servers will not have 5 people send email at the exact same millisecond. Even if this scenario does occur, the connections would free up in a second or two as the mail is sent. The connection is available for the next sending message.

InstancesPerIP is not the issue here.

The issue is that incoming connections are connecting and not being terminated. A timeout should occur at 120 but this doesn't happen.

Code: [Select]

# cat /var/service/sqpsmtpd/config/timeout
120

You can see your connections by:

Code: [Select]

ps fax |grep forkserver

Which will show something like:

Code: [Select]

30312 ?        S      0:00  |   \_ /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 465 -c 10 -m 5
32479 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [142.93.23.42 : bitcoin-redeem.com : 09:31:34 2018-10-25]
  988 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [142.93.23.42 : bitcoin-redeem.com : 09:57:40 2018-10-25]
 1554 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [142.93.23.42 : bitcoin-redeem.com : 10:23:48 2018-10-25]
 1559 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [176.126.245.65 : Unknown : 10:24:06 2018-10-25]
 1994 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [142.93.23.42 : bitcoin-redeem.com : 10:49:53 2018-10-25]
 2136 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [176.126.245.65 : Unknown : 10:51:58 2018-10-25]
 2659 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [151.80.47.101 : ns3009097.ip-151-80-47.eu : 11:01:53 2018-10-25]
 2842 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [45.55.26.204 : Unknown : 11:08:08 2018-10-25]


[ReetP]

InstancesPerIP is not the issue here.

If you read the bug you would understand that this is EXACTLY the issue.

https://bugs.contribs.org/show_bug.cgi?id=10639#c3

It appears that qpsmtpd IGNORES the instances per IP instead of honouring it.

You can set it to what you like and it will not work, which it should.

So a single IP can open multiple connections and lock out your mail server.

Please, read the bug, and follow it (but don't comment unless you actually have something new to add to it)

[Daniel B.]

Note that there might be another issue with timeout not being applied. But it would be a separated bug

[bunkobugsy]

Note that there might be another issue with timeout not being applied. But it would be a separated bug

My thoughts exactly, thats why I attached our entire log from yesterday's DOS to bug 10639:

2018-10-24 23:40:57.050624500 11860 Accepted connection 9/10 from 162.243.25.159 / Unknown
2018-10-24 23:40:57.050701500 11860 Connection from Unknown [162.243.25.159]

2018-10-25 00:20:05.477450500 11860 (connect) tls: fail, unable to establish SSL
2018-10-25 00:20:05.477587500 11860 (deny) logging::logterse: ` 162.243.25.159   Unknown            tls   903   Cannot establish SSL session   msg denied before queued
2018-10-25 00:20:05.477650500 11860 550 Cannot establish SSL session
2018-10-25 00:20:05.477660500 11860 click, disconnecting

2018-10-25 00:20:06.414817500 5105 cleaning up after 11860

That's almost 40 minutes !!! On the other hand, is TLS configured correctly? Isn't there a mismatch in cipher versions?

[ReetP]

The point is if the limit per IP was working as expected the timeout would be far less of an issue.

Instances per IP should block any further connections from the IP.

If that was working as expected, a longer timeout almost works against the attacker and would keep their IP locked for longer preventing them making further connections.

Resolving any potential issues with timeouts won't fix the underlying issue with Instances per IP which is the bit that really needs fixing.

E.g. so the timeout is say 30 seconds. All that will happen is the connection gets dropped and they immediately make another. You are in the same position.

The fail2ban filter previously mentioned seems to have kept things under control for me so far.

[bunkobugsy]

You're right again    You should add fail2ban regex to wiki.

But in my case I don't think it was an attack, only an unfortunate DOS from a single IP spammer.
And its throttle kept making new connections before old ones were closed, thus hitting 10/10.

Qpsmtpd / TLS plugin doesn't care about timeout.

[ReetP]

You're right again    You should add fail2ban regex to wiki.

ROFLMAO I'm no genius..... just try and see the logic. The timeout issue is really a bit of a 'red herring' here and really only serves to confuse things.

OK, until we can resolve qpsmtpd.... how can we stop some of this?

fail2ban gets a little confusing as the qpsmtpd.conf filter looks at both qpsmtpd and sqpsmtpd logs.

Couple of points - the first line in the fail2ban qpsmtpd.conf regex will not work (in my experience see https://bugs.contribs.org/show_bug.cgi?id=8955 which I have reopened) The second one will.

Here's the regex:

Code: [Select]

failregex = ^\s*\d+\s*logging::logterse plugin \(deny\): ` <HOST>\s*.*90\d.*msg denied before queued$
                ^\s*\d+\s*\(deny\) logging::logterse: ` <HOST>\s*.*90\d.*msg denied before queued$
Here's the sort of lines we are looking for (or similar) from both qpsmptd and sqpsmtpd:
               

(deny) logging::logterse: ` 71.6.199.23   ubuntu1619923.aspadmin.com   openssl.client.net         tls   901   TLS Negotiation Failed   msg denied before queued
(deny) logging::logterse: ` 191.53.200.26   191-53-200-26.dvl-wr.mastercabo.com.br            tls   903   Cannot establish SSL session   msg denied before queued

The first regex line used to match the old logs, so we need the second line.

There is also an issue with my old regex from https://bugs.contribs.org/show_bug.cgi?id=8952

Here's the old regex:

Code: [Select]

^\s*\d+\s*count_unrecognized_commands plugin \(unrecognized_command\): Unrecognized command 'auth' '<HOST>'$
This will not work with the newer qpsmtpd as the log line has changed, and it also won't see things like this:

2018-10-25 16:29:29.485465500.s:49001:@400000005bd14c80027ed8ec 31545 Accepted connection 1/40 from 71.6.199.23 / ubuntu1619923.aspadmin.com
2018-10-25 16:29:29.485465500.s:49002:@400000005bd14c8002804434 31545 Connection from ubuntu1619923.aspadmin.com [71.6.199.23]
2018-10-25 16:29:29.485465500.s:49005:@400000005bd14c81116bc144 31545 (connect) earlytalker: pass, not spontaneous
2018-10-25 16:29:29.485465500.s:49006:@400000005bd14c811182df3c 31545 (connect) relay: skip, no match
2018-10-25 16:29:29.485465500.s:49007:@400000005bd14c81118a0f14 31545 (connect) ident::geoip: US
2018-10-25 16:29:29.485465500.s:49008:@400000005bd14c8111cb0014 31545 (connect) dnsbl: karma -1 (-1)
2018-10-25 16:29:29.485465500.s:49009:@400000005bd14c8111cb4e34 31545 (connect) dnsbl: fail, NAUGHTY, zen.spamhaus.org
2018-10-25 16:29:29.485465500.s:49010:@400000005bd14c8111ceebfc 31545 220 esmith.myserver.com ESMTP
2018-10-25 16:29:29.485465500.s:49011:@400000005bd14c811a2876ec 31545 dispatching EHLO openssl.client.net
2018-10-25 16:29:29.485465500.s:49012:@400000005bd14c811abfbd7c 31545 (ehlo) helo: pass
2018-10-25 16:29:29.485465500.s:49013:@400000005bd14c811acc409c 31545 250-myserver.com Hi ubuntu1619923.aspadmin.com [71.6.199.23]
2018-10-25 16:29:29.485465500.s:49014:@400000005bd14c811acd4a3c 31545 250-PIPELINING
2018-10-25 16:29:29.485465500.s:49015:@400000005bd14c811acd8ca4 31545 250-8BITMIME
2018-10-25 16:29:29.485465500.s:49016:@400000005bd14c811acdea64 31545 250-SIZE 20000000
2018-10-25 16:29:29.485465500.s:49017:@400000005bd14c811ace4ff4 31545 250 STARTTLS
2018-10-25 16:29:29.485465500.s:49018:@400000005bd14c812326bdbc 31545 dispatching STARTTLS
2018-10-25 16:29:29.485465500.s:49019:@400000005bd14c81232857e4 31545 220 Go ahead with TLS
2018-10-25 16:29:29.485465500.s:49022:@400000005bd14c812b88284c 31545 (deny) logging::logterse: ` 71.6.199.23   ubuntu1619923.aspadmin.com   openssl.client.net         tls   901   TLS Negotiation Failed   msg denied before queued
2018-10-25 16:29:29.485465500.s:49023:@400000005bd14c812b8a08c4 31545 500 TLS Negotiation Failed
2018-10-25 16:29:29.485465500.s:49025:@400000005bd14c9e030d78a4 31545 dispatching ��
��1=H<��_,��b�+
2018-10-25 16:29:29.485465500.s:49026:@400000005bd14c9e035873f4 31545 (unrecognized_command) count_unrecognized_commands: '��
��1=h<��_,��b�+', (1)
2018-10-25 16:29:29.485465500.s:49027:@400000005bd14c9e03587bc4 31545 500 Unrecognized command

It seem to throw this in the fail2ban logs:

fail2ban.filter         [26870]: WARNING Error decoding line from '/var/log/qpsmtpd/current' with 'UTF-8'. Consider setting logencoding=utf-8 (or another appropriate encoding) for this jail. Continuing to process line ignoring invalid characters: '@400000005bd1d6182eba1ad4 24134 dispatching \x05\x00\x80\x03\x00\x80\x01\x00\x80\x07\x00\xc0\x86\x8eD\xae%[\x9e\xae\x14K\x7fe\xfe\x06\x11\xc9\n'

I have made a note on the bug and will look at it again next week.

But in my case I don't think it was an attack, only an unfortunate DOS from a single IP spammer.
And its throttle kept making new connections before old ones were closed, thus hitting 10/10.

Yes, can be a single spammy IP but it keeps banging away which creates the DoS. It should be stopped before that happens which is what Instances per IP should do.

Qpsmtpd / TLS plugin doesn't care about timeout.

Possibly it should, but that is another issue.....

[mmccarn]

Bug 10639 updated with proposed fix to /var/service/sqpsmtpd/run

[ReetP]

Bug 10639 updated with proposed fix to /var/service/sqpsmtpd/run

Nice

Wonder why....

[Mophilly]

Is the fix in bug 10639 something I can test? I have a production server experiencing this problem.

[ReetP]

It's not a definitive 'fix' but a possible workaround.

By all means test, but you should really try on a test machine... yes you can try on your production box... at your own risk & YMMV.....