Topic: Too many connections: 10 >= 10. Waiting one second.

[ReetP]

Last few days I have been getting some form of DoS attack on my server.

These lock the mail server and prevent us sending mail.

Any thoughts on how to cut this down would be appreciated !



/var/log/sqpsmtpd


2018-10-17 15:04:15.415534500 14589 Accepted connection 9/10 from 138.197.162.236 / Unknown
2018-10-17 15:04:15.416239500 14589 Connection from Unknown [138.197.162.236]
2018-10-17 15:04:15.542398500 Missing GeoIP City data! <<<< This is just log noise from GeoIP2
2018-10-17 15:04:15.542400500 Missing GeoIP ASN data! <<<< This is just log noise from GeoIP2
2018-10-17 15:04:16.408251500 2314 Too many connections: 10 >= 10.  Waiting one second.
2018-10-17 15:04:17.408505500 2314 Too many connections: 10 >= 10.  Waiting one second.
2018-10-17 15:04:18.408682500 2314 Too many connections: 10 >= 10.  Waiting one second.
................ loads more lines the same.....
2018-10-17 15:09:41.244603500 14589 (connect) tls: fail, unable to establish SSL
2018-10-17 15:09:41.246184500 14589 (deny) logging::logterse: ` 138.197.162.236   Unknown            tls   903   Cannot establish SSL session   msg denied before queued
2018-10-17 15:09:41.246186500 14589 Lost connection to client, cannot send response.
2018-10-17 15:09:41.246187500 14589 click, disconnecting
2018-10-17 15:09:41.478783500 2314 cleaning up after 14589
2018-10-17 15:09:41.478784500 2314 Too many connections: 10 >= 10.  Waiting one second.
2018-10-17 15:09:41.505464500 16818 Accepted connection 9/10 from 185.222.202.113 / Unknown
2018-10-17 15:09:41.506288500 16818 Connection from Unknown [185.222.202.113]
2018-10-17 15:09:41.659520500 Missing GeoIP City data!
2018-10-17 15:09:41.661718500 Missing GeoIP ASN data!
2018-10-17 15:09:41.852729500 16818 (connect) tls: fail, unable to establish SSL
2018-10-17 15:09:41.852869500 16818 (deny) logging::logterse: ` 185.222.202.113   Unknown            tls   903   Cannot establish SSL session   msg denied before queued
2018-10-17 15:09:41.853024500 16818 550 Cannot establish SSL session
2018-10-17 15:09:41.853092500 16818 click, disconnecting
2018-10-17 15:09:42.479020500 2314 cleaning up after 16818
2018-10-17 15:10:54.844354500 2314 Too many connections: 10 >= 10.  Waiting one second.
2018-10-17 15:10:54.852716500 17350 Accepted connection 9/10 from 93.174.93.228 / hosted-by.rainbownetworks.net
2018-10-17 15:10:54.870157500 17350 Connection from hosted-by.rainbownetworks.net [93.174.93.228]
2018-10-17 15:10:54.989002500 Missing GeoIP City data!
2018-10-17 15:10:54.989004500 Missing GeoIP ASN data!
2018-10-17 15:10:55.151380500 17350 (connect) tls: pass, connect via SMTPS
2018-10-17 15:10:55.844649500 2314 Too many connections: 10 >= 10.  Waiting one second.
2018-10-17 15:10:56.152163500 17350 (connect) earlytalker: pass, not spontaneous
2018-10-17 15:10:56.154334500 17350 (connect) relay: skip, no match
2018-10-17 15:10:56.154643500 17350 (connect) ident::geoip: NL
2018-10-17 15:10:56.164513500 17350 (connect) dnsbl: karma -1 (-1)
2018-10-17 15:10:56.171124500 17350 (connect) dnsbl: fail, NAUGHTY, zen.spamhaus.org
2018-10-17 15:10:56.171125500 17350 220 esmith.impamark.co.uk ESMTP
2018-10-17 15:10:56.177927500 17350 dispatching EHLO User
2018-10-17 15:10:56.178550500 17350 (ehlo) helo: karma -1 (-2)
2018-10-17 15:10:56.178566500 17350 (ehlo) helo: fail, NAUGHTY, not FQDN
2018-10-17 15:10:56.179043500 17350 250-impamark.co.uk Hi hosted-by.rainbownetworks.net [93.174.93.228]
2018-10-17 15:10:56.179044500 17350 250-PIPELINING
2018-10-17 15:10:56.179058500 17350 250-8BITMIME
2018-10-17 15:10:56.179068500 17350 250-SIZE 20000000
2018-10-17 15:10:56.179078500 17350 250 AUTH PLAIN LOGIN
2018-10-17 15:10:56.191533500 17350 dispatching RSET
2018-10-17 15:10:56.192219500 17350 250 OK
2018-10-17 15:10:56.204675500 17350 dispatching AUTH LOGIN
2018-10-17 15:10:56.204975500 17350 334 VXNlcm5hbWU6
2018-10-17 15:10:56.217252500 17350 334 UGFzc3dvcmQ6
2018-10-17 15:10:56.229844500 17350 (auth-login) auth::auth_cvm_unix_local: fail: authentication failure for: guest@impamark.co.uk
2018-10-17 15:10:56.230044500 17350 (deny) logging::logterse: ` 93.174.93.228   hosted-by.rainbownetworks.net   User         auth::auth_cvm_unix_local   901   auth failure (100)   msg denied before queued
2018-10-17 15:10:56.230131500 17350 535 LOGIN authentication failed for guest@impamark.co.uk - auth failure (100)
2018-10-17 15:10:56.242743500 17350 dispatching QUIT
2018-10-17 15:10:56.270632500 17350 221 impamark.co.uk closing connection. Have a wonderful day.
2018-10-17 15:10:56.270633500 17350 click, disconnecting
2018-10-17 15:10:56.844841500 2314 cleaning up after 17350
2018-10-17 15:20:50.715503500 2314 Too many connections: 10 >= 10.  Waiting one second.

[ReetP]

Copied over from other thread as these may be different issues...

There should be data in the sqpsmtpd log file above the "Too many connections:" entries showing what systems are using up your 10 connections -- or run
  netstat -an |grep :465.*EST
at the command line to see what systems have active, open connections on port 465.

Once you have the active IPs you can 'grep' the sqpsmtpd logs for more details (grep for the IP to get the connection number, then grep for the connection number to get full details of the transaction).  Here are some wiki notes on digging info out of the email log files: https://wiki.contribs.org/Mail_log_file_analysis.

"qplogtail" (attached to Bug 3418) lists qpsmtpd activity, but only looks at /var/log/qpsmtpd/current.  You could download that script and edit line 8 to use 'sqpsmtpd/current', instead of 'qpsmtpd/current' ("tail -f /var/log/sqpsmtpd/current \")

If the connections are used up by remote systems, you might want to look into Fail2ban.

There is nothing else reported and not enough to grep the logs with fail2ban which is already installed.... or at least I can't see how to trap the specific IP. GeoIP doesn't block until later in the connection process by which the connection has died (I think)

You can see all there is to see in the log extract. And I can't find the damn '10' limit either

Yes I can check the connections but I'm pretty sure they're mostly from the 'attacker' that overloads the connections, compounded by a few users trying to mail at the same time.

Note there are no 'local' users as this server is a VM up in the ether.

This may or may not be relevant:

(connect) tls: fail, unable to establish SSL

Tls   903   Cannot establish SSL session   msg denied before queued

Seems multiple SSL sessions are being attempted and dropped?

Hence Fail2ban sees a failed connection rather than a failed login.

Been scratching my head on it. Is there a  TLS timeout limit or something? I need to check.

Currently waiting for another blast to investigate further.

[warren]

Hi Reet

I Have also been hit with this.

Copied over from other thread as these may be different issues...
...
. And I can't find the damn '10' limit either

are you refering here to the the max allowed connections ( Too many connections: 10 >= 10 )

I worked this out as follows :

Code: [Select]

Change the number of incoming qpsmtpd / sqpsmtpd connections :
===============================================================

1. to change number of qpsmtpd connections ( default number connections is 40 ; default per IP is 5 )

config set smtpd Instances 50
config set smtpd InstancesPerIP 10

# expand-template /var/service/qpsmtpd/runenv

# sv t /service/qpsmtpd


2. to change sqpsmtpd connections ( default number connections is 10 ; default per IP is 10 )

config set ssmtpd Instances 30
config set ssmtpd InstancesPerIP 10

# expand-template /var/service/sqpsmtpd/runenv

# sv t /service/sqpsmtpd


Check settings are in effect :

Code: [Select]

cat  /var/service/*qpsmtpd/runenv
#------------------------------------------------------------
#              !!DO NOT MODIFY THIS FILE!!
#
# Manual changes will be lost when this file is regenerated.
#
# Please read the developer's guide, which is available
# at http://www.contribs.org/development/
#
# Copyright (C) 1999-2006 Mitel Networks Corporation
#------------------------------------------------------------
# This templated file is sourced by the qpsmtpd run
# file. Shell variables can be set here for use by the run
# script, or environment variables can be exported for use
# by qpsmtpd.
INSTANCES=50
INSTANCES_PER_IP=10

..
..
#------------------------------------------------------------
#              !!DO NOT MODIFY THIS FILE!!
#
# Manual changes will be lost when this file is regenerated.
#
# Please read the developer's guide, which is available
# at http://www.contribs.org/development/
#
# Copyright (C) 1999-2006 Mitel Networks Corporation
#------------------------------------------------------------
# This templated file is sourced by the sqpsmtpd run
# file. Shell variables can be set here for use by the run
# script, or environment variables can be exported for use
# by sqpsmtpd.
INSTANCES=30
INSTANCES_PER_IP=10


[warren]

Copied over from other thread as these may be different issues...
...

This may or may not be relevant:

Seems multiple SSL sessions are being attempted and dropped?

Hence Fail2ban sees a failed connection rather than a failed login.

Been scratching my head on it. Is there a  TLS timeout limit or something? I need to check.

Currently waiting for another blast to investigate further.

They started up on my side again.

The TLS timeout. If there is something ( haven`t found anything yet ) might help.
I watched a connection coming in  and it seems that it holds open the connection.

netstat shows :

Code: [Select]

netstat -an | grep EST | grep "...:465..."
tcp        0      0 obfusicatedmyIP:465          118.24.78.192:56454         ESTABLISHED


find the sqpsmtpd process :

Code: [Select]

[root@cpt1 ~]# ps -ef| grep 118.24.78.192
qpsmtpd  30229 27655  0 21:28 ?        00:00:00 /usr/bin/qpsmtpd-forkserver [118.24.78.192 : Unknown : 21:28:49 2018-10-20]
root     32953 26592  0 21:50 pts/2    00:00:00 grep 118.24.78.192

sqpsmtpd log ( 30229 ):

Code: [Select]

2018-10-20 21:28:49.517801500 30229 Accepted connection 0/30 from 118.24.78.192 / Unknown
2018-10-20 21:28:49.517897500 30229 Connection from Unknown [118.24.78.192]

This connection just seemed to stay open.
I sent a test mail from the admin account which i received

Code: [Select]

2018-10-20 21:28:49.517801500 30229 Accepted connection 0/30 from 118.24.78.192 / Unknown
2018-10-20 21:28:49.517897500 30229 Connection from Unknown [118.24.78.192]
2018-10-20 21:30:45.720412500 30478 Accepted connection 1/30 from 155.93.249.77 / Unknown
2018-10-20 21:30:45.720525500 30478 Connection from Unknown [155.93.249.77]
2018-10-20 21:30:45.991406500 30478 (connect) tls: pass, connect via SMTPS
2018-10-20 21:30:46.993467500 30478 (connect) earlytalker: pass, not spontaneous
2018-10-20 21:30:46.994919500 30478 (connect) relay: skip, no match
..
...
2018-10-20 21:30:47.227413500 30478 250 Queued! 1540063847 qp 30481 <996249c0-bae5-5777-4ccc-2a5275fd1b2c@.....
2018-10-20 21:30:47.234699500 30478 dispatching QUIT
2018-10-20 21:30:47.234869500 30478 221 XXXXXXX.com closing connection. Have a wonderful day.
2018-10-20 21:30:47.235003500 30478 click, disconnecting

It seems that this specific attack is somehow keeping the connection open on port 465, and then they  just continue creating more connections until eventually sqpsmtpd (  qpsmtpd ) runs out of instances.
Too many connections: 30 >= 30.  Waiting one second.

[ReetP]

Good work !!

Ok so a bit of a trawl shows this (my qpsmtpd was set 40/5 so it wasn't that)

ssmtpd=service
    Authentication=enabled
    Instances=10
    TCPPort=465
    access=public
    status=enabled

cat  /var/service/sqpsmtpd/runenv

# This templated file is sourced by the sqpsmtpd run
# file. Shell variables can be set here for use by the
run
# script, or environment variables can be exported for
use
# by sqpsmtpd.
INSTANCES=10
INSTANCES_PER_IP=5

Blah....

So is it not obeying instances per IP? Or is it there are numerous different IPs so we need to up the (total) instances but keep the number per IP low?

Agree... is there a timeout somewhere?

There is a qpsmtpd template in fail2ban but it won't pick this up as it looks for a 'denied' message.

I need to go dig some more. I'd guess it is a specific attack but not sure what can be done to mitigate it.

Might log a bug anyways but need to read a bit more first.

[tomeratch]

Hi guys I'v been hit with the same thing
I posted a week ago thinking it was an smtp problem ..
is there any way to fix this?
my server keeps those external sessions open until ssmtp dies and need to restart the service.
I also changed the instances as mentioned above but it only delayed the smtp overload.

Code: [Select]

|grep :465.*ES
               123.168.150.96:65109        ESTABLISHED
               114.222.11.237:63250        ESTABLISHED
               113.121.243.12:57729        ESTABLISHED
               121.238.64.201:53447        ESTABLISHED
               123.168.150.88:51010        ESTABLISHED
               121.236.126.64:62408        ESTABLISHED


Code: [Select]

Accepted connection 2/10 from 123.168.150.165 / Unknown
22207 Connection from Unknown [123.168.150.165]
 22207 in config(plugins)
 22207 config(plugins): hook returned (0,)
 22207 in config(plugin_dirs)
 22207 config(plugin_dirs): hook returned (0,)
 22207 Loading hosts_allow from /usr/share/qpsmtpd/plugins/hosts_allow
 22207 hosts_allow hooking pre-connection
 22207 in config(plugin_dirs)
 22207 config(plugin_dirs): hook returned (0,)
 22207 Loading peers from /usr/share/qpsmtpd/plugins/peers
 22207 in config(peers/0)
 22207 config(peers/0): hook returned (0,)
 22207 in config(plugin_dirs)
 22207 config(plugin_dirs): hook returned (0,)


[warren]

Good work !!

Ok so a bit of a trawl shows this (my qpsmtpd was set 40/5 so it wasn't that)

ssmtpd=service
    Authentication=enabled
    Instances=10
    TCPPort=465
    access=public
    status=enabled

cat  /var/service/sqpsmtpd/runenv

# This templated file is sourced by the sqpsmtpd run
# file. Shell variables can be set here for use by the
run
# script, or environment variables can be exported for
use
# by sqpsmtpd.
INSTANCES=10
INSTANCES_PER_IP=5

Blah....

So is it not obeying instances per IP? Or is it there are numerous different IPs so we need to up the (total) instances but keep the number per IP low?

Agree... is there a timeout somewhere?

There is a qpsmtpd template in fail2ban but it won't pick this up as it looks for a 'denied' message.

I need to go dig some more. I'd guess it is a specific attack but not sure what can be done to mitigate it.

Might log a bug anyways but need to read a bit more first.

Found this bug that has to do with sqpsmtpd hanging in TLS plugin?  " https://bugs.contribs.org/show_bug.cgi?id=6916 "

Patch to set client socket KEEPALIVE option.

There are a number of bugzilla entries relating to qpsmtpd and sqpsmtpd hangs but I think this may be a different issue.

I just sent a SIGALRM to a qpsmtpd-forkserver process which had been running for about 5 months. Here's the info from "ps":

27488 ?        S      0:00 /usr/bin/qpsmtpd-forkserver [41.237.204.125 : host-41.237.204.125.tedata.net : 14:43:01 2011-12-19]

From what i can tell , this patch is in the current  : /usr/bin/qpsmtpd-forkserver

[ReetP]

Tomeratch,

I can see no evidence of this in your logs:

Too many connections: 10 >= 10.  Waiting one second.

As I have said in the other thread I think there may be 2 different issues. Don't just wildly hijack threads in the hope of a solution. You just confuse things.

If your problem shows the above error then continue here. If it does not, stay in the other thread.

Thanks.

[ReetP]

PS... the error may be another number besides 10 depending on the Instances setting in:

config show ssmtpd

The key bit is:

Too many connections. X >=X Waiting one second.

[tomeratch]

Thanks Reet for your Reply
I managed to fix this for now (work around) using my firewall and Blocking the addresses that keep the ssmtp session open.
that cleared the port 465 sessions and netstat no longer shows external addresses with permanent sessions.
hope this helps others as a work around for now.
to me its clear now that its an exploit.

[Daniel B.]

If you think there's a security problem, then please open 1 bug and describe the problem and how to reproduce

[ReetP]

It would seem both this thread and the other one may have a similar issue with a remote IP opening multiple IMAPS/TLS connections quickly.

If, as indicated elsewhere, there is a 60 sec timeout on TLS connections, the number of connections allowed (in my instance 10) gets exceeded and this then locks out other connections until one of the connections gets released.

In my situation I don't seem to get to the 'denyhard'.

I'll try and go back and look at some more logs and put it altogether in a bug.

Note this does not mean the server is hacked as security has not been breached. However it is effectively DoS.

[ReetP]

OK, bug posted here:

https://bugs.contribs.org/show_bug.cgi?id=10639

A potential get around.

Add an extra regex to Fail2Ban

I have previously added to the /etc/fail2ban/filter.d/qpsmtpd.conf template as follows:

Code: [Select]

failregex = ^\s*\d+\s*logging::logterse plugin \(deny\): ` <HOST>\s*.*90\d.*msg denied before queued$
                ^\s*\d+\s*\(deny\) logging::logterse: ` <HOST>\s*.*90\d.*msg denied before queued$

That second line seems to pick up the SSL IPs - you can test like this (note the escaped ` for CLI test that is not required in the conf file itself)

Code: [Select]

fail2ban-regex --print-all-matched /var/log/sqpsmtpd/current "^\s*\d+\s*\(deny\) logging::logterse: \` <HOST>\s*.*90\d.*msg denied before queued$"
For reference/interest I also remembered I did another regex here:

https://bugs.contribs.org/show_bug.cgi?id=8952
https://bugs.contribs.org/attachment.cgi?id=5242&action=edit

Code: [Select]

^\s*\d+\s*count_unrecognized_commands plugin \(unrecognized_command\): Unrecognized command 'auth' '<HOST>'$


[ReetP]

PS - have added the line to my fail2ban and going to see what it picks up.

[ReetP]

OK,

I tested this using Daniels method from the bug.

Code: [Select]

telnet your.mail.host 465
[root@test ~]# telnet your.mail.host 465
Trying 5.6.7.8...
Connected to your.mail.host.
Escape character is '^]'.
ehlo your.mail.host
550 Cannot establish SSL session
Connection closed by foreign host.

Logs show:

(connect) tls: fail, unable to establish SSL
tls   903   Cannot establish SSL session   msg denied before queued
550 Cannot establish SSL session

Code: [Select]

grep "1.2.3.4" /var/log/fail2ban/daemon.log
2018-10-25 12:23:56,020 fail2ban.filter         [25132]: INFO    [qpsmtpd] Found 1.2.3.4
2018-10-25 12:24:06,714 fail2ban.filter         [25132]: INFO    [qpsmtpd] Found 1.2.3.4
2018-10-25 12:24:07,550 fail2ban.actions        [25132]: NOTICE  [qpsmtpd] Ban 1.2.3.4

Code: [Select]

grep "1.2.3.4" /etc/rc.d/init.d/masq
/sbin/iptables --append $NEW_Fail2Ban -s 1.2.3.4 -p tcp -m multiport --dports 25,465 -j denylog

So looks my rule is working.

Note this doesn't resolve the bug. It is just a workaround.