Koozali.org: home of the SME Server

Too many connections: 10 >= 10. Waiting one second.

Offline ReetP

  • *
  • 3,731
  • +5/-0
Too many connections: 10 >= 10. Waiting one second.
« on: October 17, 2018, 04:38:28 PM »
Last few days I have been getting some form of DoS attack on my server.

These lock the mail server and prevent us sending mail.

Any thoughts on how to cut this down would be appreciated !



/var/log/sqpsmtpd


2018-10-17 15:04:15.415534500 14589 Accepted connection 9/10 from 138.197.162.236 / Unknown
2018-10-17 15:04:15.416239500 14589 Connection from Unknown [138.197.162.236]
2018-10-17 15:04:15.542398500 Missing GeoIP City data! <<<< This is just log noise from GeoIP2
2018-10-17 15:04:15.542400500 Missing GeoIP ASN data! <<<< This is just log noise from GeoIP2
2018-10-17 15:04:16.408251500 2314 Too many connections: 10 >= 10.  Waiting one second.
2018-10-17 15:04:17.408505500 2314 Too many connections: 10 >= 10.  Waiting one second.
2018-10-17 15:04:18.408682500 2314 Too many connections: 10 >= 10.  Waiting one second.
................ loads more lines the same.....
2018-10-17 15:09:41.244603500 14589 (connect) tls: fail, unable to establish SSL
2018-10-17 15:09:41.246184500 14589 (deny) logging::logterse: ` 138.197.162.236   Unknown            tls   903   Cannot establish SSL session   msg denied before queued
2018-10-17 15:09:41.246186500 14589 Lost connection to client, cannot send response.
2018-10-17 15:09:41.246187500 14589 click, disconnecting
2018-10-17 15:09:41.478783500 2314 cleaning up after 14589
2018-10-17 15:09:41.478784500 2314 Too many connections: 10 >= 10.  Waiting one second.
2018-10-17 15:09:41.505464500 16818 Accepted connection 9/10 from 185.222.202.113 / Unknown
2018-10-17 15:09:41.506288500 16818 Connection from Unknown [185.222.202.113]
2018-10-17 15:09:41.659520500 Missing GeoIP City data!
2018-10-17 15:09:41.661718500 Missing GeoIP ASN data!
2018-10-17 15:09:41.852729500 16818 (connect) tls: fail, unable to establish SSL
2018-10-17 15:09:41.852869500 16818 (deny) logging::logterse: ` 185.222.202.113   Unknown            tls   903   Cannot establish SSL session   msg denied before queued
2018-10-17 15:09:41.853024500 16818 550 Cannot establish SSL session
2018-10-17 15:09:41.853092500 16818 click, disconnecting
2018-10-17 15:09:42.479020500 2314 cleaning up after 16818
2018-10-17 15:10:54.844354500 2314 Too many connections: 10 >= 10.  Waiting one second.
2018-10-17 15:10:54.852716500 17350 Accepted connection 9/10 from 93.174.93.228 / hosted-by.rainbownetworks.net
2018-10-17 15:10:54.870157500 17350 Connection from hosted-by.rainbownetworks.net [93.174.93.228]
2018-10-17 15:10:54.989002500 Missing GeoIP City data!
2018-10-17 15:10:54.989004500 Missing GeoIP ASN data!
2018-10-17 15:10:55.151380500 17350 (connect) tls: pass, connect via SMTPS
2018-10-17 15:10:55.844649500 2314 Too many connections: 10 >= 10.  Waiting one second.
2018-10-17 15:10:56.152163500 17350 (connect) earlytalker: pass, not spontaneous
2018-10-17 15:10:56.154334500 17350 (connect) relay: skip, no match
2018-10-17 15:10:56.154643500 17350 (connect) ident::geoip: NL
2018-10-17 15:10:56.164513500 17350 (connect) dnsbl: karma -1 (-1)
2018-10-17 15:10:56.171124500 17350 (connect) dnsbl: fail, NAUGHTY, zen.spamhaus.org
2018-10-17 15:10:56.171125500 17350 220 esmith.impamark.co.uk ESMTP
2018-10-17 15:10:56.177927500 17350 dispatching EHLO User
2018-10-17 15:10:56.178550500 17350 (ehlo) helo: karma -1 (-2)
2018-10-17 15:10:56.178566500 17350 (ehlo) helo: fail, NAUGHTY, not FQDN
2018-10-17 15:10:56.179043500 17350 250-impamark.co.uk Hi hosted-by.rainbownetworks.net [93.174.93.228]
2018-10-17 15:10:56.179044500 17350 250-PIPELINING
2018-10-17 15:10:56.179058500 17350 250-8BITMIME
2018-10-17 15:10:56.179068500 17350 250-SIZE 20000000
2018-10-17 15:10:56.179078500 17350 250 AUTH PLAIN LOGIN
2018-10-17 15:10:56.191533500 17350 dispatching RSET
2018-10-17 15:10:56.192219500 17350 250 OK
2018-10-17 15:10:56.204675500 17350 dispatching AUTH LOGIN
2018-10-17 15:10:56.204975500 17350 334 VXNlcm5hbWU6
2018-10-17 15:10:56.217252500 17350 334 UGFzc3dvcmQ6
2018-10-17 15:10:56.229844500 17350 (auth-login) auth::auth_cvm_unix_local: fail: authentication failure for: guest@impamark.co.uk
2018-10-17 15:10:56.230044500 17350 (deny) logging::logterse: ` 93.174.93.228   hosted-by.rainbownetworks.net   User         auth::auth_cvm_unix_local   901   auth failure (100)   msg denied before queued
2018-10-17 15:10:56.230131500 17350 535 LOGIN authentication failed for guest@impamark.co.uk - auth failure (100)
2018-10-17 15:10:56.242743500 17350 dispatching QUIT
2018-10-17 15:10:56.270632500 17350 221 impamark.co.uk closing connection. Have a wonderful day.
2018-10-17 15:10:56.270633500 17350 click, disconnecting
2018-10-17 15:10:56.844841500 2314 cleaning up after 17350
2018-10-17 15:20:50.715503500 2314 Too many connections: 10 >= 10.  Waiting one second.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #1 on: October 19, 2018, 01:17:14 AM »
Copied over from other thread as these may be different issues...

There should be data in the sqpsmtpd log file above the "Too many connections:" entries showing what systems are using up your 10 connections -- or run
  netstat -an |grep :465.*EST
at the command line to see what systems have active, open connections on port 465.

Once you have the active IPs you can 'grep' the sqpsmtpd logs for more details (grep for the IP to get the connection number, then grep for the connection number to get full details of the transaction).  Here are some wiki notes on digging info out of the email log files: https://wiki.contribs.org/Mail_log_file_analysis.

"qplogtail" (attached to Bug 3418) lists qpsmtpd activity, but only looks at /var/log/qpsmtpd/current.  You could download that script and edit line 8 to use 'sqpsmtpd/current', instead of 'qpsmtpd/current' ("tail -f /var/log/sqpsmtpd/current \")

If the connections are used up by remote systems, you might want to look into Fail2ban.

There is nothing else reported and not enough to grep the logs with fail2ban which is already installed.... or at least I can't see how to trap the specific IP. GeoIP doesn't block until later in the connection process by which the connection has died (I think)

You can see all there is to see in the log extract. And I can't find the damn '10' limit either :-)

Yes I can check the connections but I'm pretty sure they're mostly from the 'attacker' that overloads the connections, compounded by a few users trying to mail at the same time.

Note there are no 'local' users as this server is a VM up in the ether.

This may or may not be relevant:

Quote
(connect) tls: fail, unable to establish SSL

Tls   903   Cannot establish SSL session   msg denied before queued

Seems multiple SSL sessions are being attempted and dropped?

Hence Fail2ban sees a failed connection rather than a failed login.

Been scratching my head on it. Is there a  TLS timeout limit or something? I need to check.

Currently waiting for another blast to investigate further.

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline warren

  • *
  • 293
  • +0/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #2 on: October 19, 2018, 01:52:48 AM »
Hi Reet

I Have also been hit with this. :-x

Copied over from other thread as these may be different issues...
...
. And I can't find the damn '10' limit either :-)


are you refering here to the the max allowed connections ( Too many connections: 10 >= 10 )

I worked this out as follows :
Code: [Select]
Change the number of incoming qpsmtpd / sqpsmtpd connections :
===============================================================

1. to change number of qpsmtpd connections ( default number connections is 40 ; default per IP is 5 )

config set smtpd Instances 50
config set smtpd InstancesPerIP 10

# expand-template /var/service/qpsmtpd/runenv

# sv t /service/qpsmtpd


2. to change sqpsmtpd connections ( default number connections is 10 ; default per IP is 10 )

config set ssmtpd Instances 30
config set ssmtpd InstancesPerIP 10

# expand-template /var/service/sqpsmtpd/runenv

# sv t /service/sqpsmtpd


Check settings are in effect :
Code: [Select]
cat  /var/service/*qpsmtpd/runenv
#------------------------------------------------------------
#              !!DO NOT MODIFY THIS FILE!!
#
# Manual changes will be lost when this file is regenerated.
#
# Please read the developer's guide, which is available
# at http://www.contribs.org/development/
#
# Copyright (C) 1999-2006 Mitel Networks Corporation
#------------------------------------------------------------
# This templated file is sourced by the qpsmtpd run
# file. Shell variables can be set here for use by the run
# script, or environment variables can be exported for use
# by qpsmtpd.
INSTANCES=50
INSTANCES_PER_IP=10

..
..
#------------------------------------------------------------
#              !!DO NOT MODIFY THIS FILE!!
#
# Manual changes will be lost when this file is regenerated.
#
# Please read the developer's guide, which is available
# at http://www.contribs.org/development/
#
# Copyright (C) 1999-2006 Mitel Networks Corporation
#------------------------------------------------------------
# This templated file is sourced by the sqpsmtpd run
# file. Shell variables can be set here for use by the run
# script, or environment variables can be exported for use
# by sqpsmtpd.
INSTANCES=30
INSTANCES_PER_IP=10

« Last Edit: October 19, 2018, 02:03:31 AM by warren »

Offline warren

  • *
  • 293
  • +0/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #3 on: October 20, 2018, 10:42:06 PM »
Copied over from other thread as these may be different issues...
...

This may or may not be relevant:

Seems multiple SSL sessions are being attempted and dropped?

Hence Fail2ban sees a failed connection rather than a failed login.

Been scratching my head on it. Is there a  TLS timeout limit or something? I need to check.

Currently waiting for another blast to investigate further.

They started up on my side again.

The TLS timeout. If there is something ( haven`t found anything yet ) might help.
I watched a connection coming in  and it seems that it holds open the connection.

netstat shows :
Code: [Select]
netstat -an | grep EST | grep "...:465..."
tcp        0      0 obfusicatedmyIP:465          118.24.78.192:56454         ESTABLISHED


find the sqpsmtpd process :
Code: [Select]
[root@cpt1 ~]# ps -ef| grep 118.24.78.192
qpsmtpd  30229 27655  0 21:28 ?        00:00:00 /usr/bin/qpsmtpd-forkserver [118.24.78.192 : Unknown : 21:28:49 2018-10-20]
root     32953 26592  0 21:50 pts/2    00:00:00 grep 118.24.78.192

sqpsmtpd log ( 30229 ):
Code: [Select]
2018-10-20 21:28:49.517801500 30229 Accepted connection 0/30 from 118.24.78.192 / Unknown
2018-10-20 21:28:49.517897500 30229 Connection from Unknown [118.24.78.192]

This connection just seemed to stay open.
I sent a test mail from the admin account which i received
Code: [Select]
2018-10-20 21:28:49.517801500 30229 Accepted connection 0/30 from 118.24.78.192 / Unknown
2018-10-20 21:28:49.517897500 30229 Connection from Unknown [118.24.78.192]
2018-10-20 21:30:45.720412500 30478 Accepted connection 1/30 from 155.93.249.77 / Unknown
2018-10-20 21:30:45.720525500 30478 Connection from Unknown [155.93.249.77]
2018-10-20 21:30:45.991406500 30478 (connect) tls: pass, connect via SMTPS
2018-10-20 21:30:46.993467500 30478 (connect) earlytalker: pass, not spontaneous
2018-10-20 21:30:46.994919500 30478 (connect) relay: skip, no match
..
...
2018-10-20 21:30:47.227413500 30478 250 Queued! 1540063847 qp 30481 <996249c0-bae5-5777-4ccc-2a5275fd1b2c@.....
2018-10-20 21:30:47.234699500 30478 dispatching QUIT
2018-10-20 21:30:47.234869500 30478 221 XXXXXXX.com closing connection. Have a wonderful day.
2018-10-20 21:30:47.235003500 30478 click, disconnecting

It seems that this specific attack is somehow keeping the connection open on port 465, and then they  just continue creating more connections until eventually sqpsmtpd (  qpsmtpd ) runs out of instances.
Too many connections: 30 >= 30.  Waiting one second.

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #4 on: October 21, 2018, 01:48:32 AM »
Good work !!

Ok so a bit of a trawl shows this (my qpsmtpd was set 40/5 so it wasn't that)

ssmtpd=service
    Authentication=enabled
    Instances=10
    TCPPort=465
    access=public
    status=enabled

cat  /var/service/sqpsmtpd/runenv

# This templated file is sourced by the sqpsmtpd run
# file. Shell variables can be set here for use by the
run
# script, or environment variables can be exported for
use
# by sqpsmtpd.
INSTANCES=10
INSTANCES_PER_IP=5

Blah....

So is it not obeying instances per IP? Or is it there are numerous different IPs so we need to up the (total) instances but keep the number per IP low?

Agree... is there a timeout somewhere?

There is a qpsmtpd template in fail2ban but it won't pick this up as it looks for a 'denied' message.

I need to go dig some more. I'd guess it is a specific attack but not sure what can be done to mitigate it.

Might log a bug anyways but need to read a bit more first.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline tomeratch

  • *
  • 24
  • +0/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #5 on: October 21, 2018, 12:10:20 PM »
Hi guys I'v been hit with the same thing
I posted a week ago thinking it was an smtp problem ..
is there any way to fix this?
my server keeps those external sessions open until ssmtp dies and need to restart the service.
I also changed the instances as mentioned above but it only delayed the smtp overload.
Code: [Select]
|grep :465.*ES
               123.168.150.96:65109        ESTABLISHED
               114.222.11.237:63250        ESTABLISHED
               113.121.243.12:57729        ESTABLISHED
               121.238.64.201:53447        ESTABLISHED
               123.168.150.88:51010        ESTABLISHED
               121.236.126.64:62408        ESTABLISHED
Code: [Select]
Accepted connection 2/10 from 123.168.150.165 / Unknown
22207 Connection from Unknown [123.168.150.165]
 22207 in config(plugins)
 22207 config(plugins): hook returned (0,)
 22207 in config(plugin_dirs)
 22207 config(plugin_dirs): hook returned (0,)
 22207 Loading hosts_allow from /usr/share/qpsmtpd/plugins/hosts_allow
 22207 hosts_allow hooking pre-connection
 22207 in config(plugin_dirs)
 22207 config(plugin_dirs): hook returned (0,)
 22207 Loading peers from /usr/share/qpsmtpd/plugins/peers
 22207 in config(peers/0)
 22207 config(peers/0): hook returned (0,)
 22207 in config(plugin_dirs)
 22207 config(plugin_dirs): hook returned (0,)
« Last Edit: October 21, 2018, 12:44:24 PM by tomeratch »

Offline warren

  • *
  • 293
  • +0/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #6 on: October 21, 2018, 12:54:44 PM »
Good work !!

Ok so a bit of a trawl shows this (my qpsmtpd was set 40/5 so it wasn't that)

ssmtpd=service
    Authentication=enabled
    Instances=10
    TCPPort=465
    access=public
    status=enabled

cat  /var/service/sqpsmtpd/runenv

# This templated file is sourced by the sqpsmtpd run
# file. Shell variables can be set here for use by the
run
# script, or environment variables can be exported for
use
# by sqpsmtpd.
INSTANCES=10
INSTANCES_PER_IP=5

Blah....

So is it not obeying instances per IP? Or is it there are numerous different IPs so we need to up the (total) instances but keep the number per IP low?

Agree... is there a timeout somewhere?

There is a qpsmtpd template in fail2ban but it won't pick this up as it looks for a 'denied' message.

I need to go dig some more. I'd guess it is a specific attack but not sure what can be done to mitigate it.

Might log a bug anyways but need to read a bit more first.


Found this bug that has to do with sqpsmtpd hanging in TLS plugin?  " https://bugs.contribs.org/show_bug.cgi?id=6916 "

Quote
Patch to set client socket KEEPALIVE option.

There are a number of bugzilla entries relating to qpsmtpd and sqpsmtpd hangs but I think this may be a different issue.

I just sent a SIGALRM to a qpsmtpd-forkserver process which had been running for about 5 months. Here's the info from "ps":

27488 ?        S      0:00 /usr/bin/qpsmtpd-forkserver [41.237.204.125 : host-41.237.204.125.tedata.net : 14:43:01 2011-12-19]

From what i can tell , this patch is in the current  : /usr/bin/qpsmtpd-forkserver

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #7 on: October 21, 2018, 01:26:08 PM »
Tomeratch,

I can see no evidence of this in your logs:

Quote
Too many connections: 10 >= 10.  Waiting one second.

As I have said in the other thread I think there may be 2 different issues. Don't just wildly hijack threads in the hope of a solution. You just confuse things.

If your problem shows the above error then continue here. If it does not, stay in the other thread.

Thanks.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #8 on: October 21, 2018, 01:33:26 PM »
PS... the error may be another number besides 10 depending on the Instances setting in:

config show ssmtpd

The key bit is:

Too many connections. X >=X Waiting one second.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline tomeratch

  • *
  • 24
  • +0/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #9 on: October 21, 2018, 03:11:25 PM »
Thanks Reet for your Reply
I managed to fix this for now (work around) using my firewall and Blocking the addresses that keep the ssmtp session open.
that cleared the port 465 sessions and netstat no longer shows external addresses with permanent sessions.
hope this helps others as a work around for now.
to me its clear now that its an exploit.


Offline Daniel B.

  • *
  • 1,699
  • +0/-0
    • Firewall Services, la sรฉcuritรฉ des rรฉseaux
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #10 on: October 21, 2018, 03:19:34 PM »
If you think there's a security problem, then please open 1 bug and describe the problem and how to reproduce
C'est la fin du monde !!! :lol:

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #11 on: October 21, 2018, 04:42:06 PM »
It would seem both this thread and the other one may have a similar issue with a remote IP opening multiple IMAPS/TLS connections quickly.

If, as indicated elsewhere, there is a 60 sec timeout on TLS connections, the number of connections allowed (in my instance 10) gets exceeded and this then locks out other connections until one of the connections gets released.

In my situation I don't seem to get to the 'denyhard'.

I'll try and go back and look at some more logs and put it altogether in a bug.

Note this does not mean the server is hacked as security has not been breached. However it is effectively DoS.


...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #12 on: October 22, 2018, 03:02:54 PM »
OK, bug posted here:

https://bugs.contribs.org/show_bug.cgi?id=10639

A potential get around.

Add an extra regex to Fail2Ban

I have previously added to the /etc/fail2ban/filter.d/qpsmtpd.conf template as follows:

Code: [Select]
failregex = ^\s*\d+\s*logging::logterse plugin \(deny\): ` <HOST>\s*.*90\d.*msg denied before queued$
                ^\s*\d+\s*\(deny\) logging::logterse: ` <HOST>\s*.*90\d.*msg denied before queued$

That second line seems to pick up the SSL IPs - you can test like this (note the escaped ` for CLI test that is not required in the conf file itself)

Code: [Select]
fail2ban-regex --print-all-matched /var/log/sqpsmtpd/current "^\s*\d+\s*\(deny\) logging::logterse: \` <HOST>\s*.*90\d.*msg denied before queued$"
For reference/interest I also remembered I did another regex here:

https://bugs.contribs.org/show_bug.cgi?id=8952
https://bugs.contribs.org/attachment.cgi?id=5242&action=edit

Code: [Select]
^\s*\d+\s*count_unrecognized_commands plugin \(unrecognized_command\): Unrecognized command 'auth' '<HOST>'$


...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #13 on: October 22, 2018, 03:06:53 PM »
PS - have added the line to my fail2ban and going to see what it picks up.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #14 on: October 25, 2018, 12:32:42 PM »
OK,

I tested this using Daniels method from the bug.

Code: [Select]
telnet your.mail.host 465
[root@test ~]# telnet your.mail.host 465
Trying 5.6.7.8...
Connected to your.mail.host.
Escape character is '^]'.
ehlo your.mail.host
550 Cannot establish SSL session
Connection closed by foreign host.

Logs show:

(connect) tls: fail, unable to establish SSL
tls   903   Cannot establish SSL session   msg denied before queued
550 Cannot establish SSL session

Code: [Select]
grep "1.2.3.4" /var/log/fail2ban/daemon.log
2018-10-25 12:23:56,020 fail2ban.filter         [25132]: INFO    [qpsmtpd] Found 1.2.3.4
2018-10-25 12:24:06,714 fail2ban.filter         [25132]: INFO    [qpsmtpd] Found 1.2.3.4
2018-10-25 12:24:07,550 fail2ban.actions        [25132]: NOTICE  [qpsmtpd] Ban 1.2.3.4

Code: [Select]
grep "1.2.3.4" /etc/rc.d/init.d/masq
/sbin/iptables --append $NEW_Fail2Ban -s 1.2.3.4 -p tcp -m multiport --dports 25,465 -j denylog

So looks my rule is working.

Note this doesn't resolve the bug. It is just a workaround.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline bunkobugsy

  • *
  • 279
  • +4/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #15 on: October 25, 2018, 12:49:20 PM »
Great! Would this also work for port 465/sqsmtpd?

db configuration setprop ssmtpd DenyHosts xxx.xxx.xxx.xxx
signal-event remoteaccess-update

Thanks.

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #16 on: October 25, 2018, 01:14:27 PM »
Great! Would this also work for port 465/sqsmtpd?

db configuration setprop ssmtpd DenyHosts xxx.xxx.xxx.xxx
signal-event remoteaccess-update

In short yes it will.

Code: [Select]
config setprop ssmtpd DenyHosts 1.2.3.4
Code: [Select]
signal-event remoteaccess-update
Code: [Select]
grep "1.2.3.4" /etc/rc.d/init.d/masq
Quote
# ssmtpd: TCPPorts: 465, AllowHosts: 0.0.0.0/0, DenyHosts: 1.2.3.4 --destination $OUTERNET --src 1.2.3.4 --jump denylog


However, it means you have to keep adding it manually.

In the event of an attack your mail server will be locked up before the IP is banned.....

Fail2ban is a more effective method of stopping this.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline bunkobugsy

  • *
  • 279
  • +4/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #17 on: October 25, 2018, 01:32:14 PM »
Indeed.
I also upped Instances to 40 for ssmtpd too since all our laptops are set up to 993/465.

Curious, ssmtpd doesn't show an InstancesPerIP in db config as smtpd does. Coincidence?

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #18 on: October 25, 2018, 02:03:30 PM »
Curious, ssmtpd doesn't show an InstancesPerIP in db config as smtpd does. Coincidence?

Check the templates. Always check the templates :-)

Look here:

/etc/e-smith/templates/var/service/sqpsmtpd/runenv

cat INSTANCES
{
    return "INSTANCES=" . ($ssmtpd{Instances} || $smtpd{Instances} || "10");
}


cat INSTANCES_PER_IP
{
    return "INSTANCES_PER_IP=" .
   ($ssmtpd{InstancesPerIP} || $smtpd{InstancesPerIP} || "5");
}

So it defaults to 10/5 if there is no config entry set (a lot of template set defaults in the absence of properties)

The issue (which is what the bug is about) is that qpsmtpd seems to ignore the PER_IP setting.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline kruhm

  • *
  • 680
  • +0/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #19 on: October 25, 2018, 05:15:42 PM »
Clarifying and consolidating for anyone who follows. The issue is concerning the sqpsmtpd.

qpsmtpd is for receiving mail (port 25).

sqpsmtpd is for sending mail (port 465).

The following command shows the configuration for sending:
Code: [Select]
# config show ssmtpd
ssmtpd=service
    Authentication=enabled
    Instances=10
    TCPPort=465
    access=public
    status=enabled

Instances=10 meaning 10 people can send email at anyone given millisecond. Once the connection is finished, it is available for the next.

InstancesPerIP is absent on most configs and will use the default of 5. InstancesPerIP can be set if needed but typically most servers will not have 5 people send email at the exact same millisecond. Even if this scenario does occur, the connections would free up in a second or two as the mail is sent. The connection is available for the next sending message.

InstancesPerIP is not the issue here.

The issue is that incoming connections are connecting and not being terminated. A timeout should occur at 120 but this doesn't happen.
Code: [Select]
# cat /var/service/sqpsmtpd/config/timeout
120

You can see your connections by:
Code: [Select]
ps fax |grep forkserver

Which will show something like:
Code: [Select]
30312 ?        S      0:00  |   \_ /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 465 -c 10 -m 5
32479 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [142.93.23.42 : bitcoin-redeem.com : 09:31:34 2018-10-25]
  988 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [142.93.23.42 : bitcoin-redeem.com : 09:57:40 2018-10-25]
 1554 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [142.93.23.42 : bitcoin-redeem.com : 10:23:48 2018-10-25]
 1559 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [176.126.245.65 : Unknown : 10:24:06 2018-10-25]
 1994 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [142.93.23.42 : bitcoin-redeem.com : 10:49:53 2018-10-25]
 2136 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [176.126.245.65 : Unknown : 10:51:58 2018-10-25]
 2659 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [151.80.47.101 : ns3009097.ip-151-80-47.eu : 11:01:53 2018-10-25]
 2842 ?        S      0:00  |       \_ /usr/bin/qpsmtpd-forkserver [45.55.26.204 : Unknown : 11:08:08 2018-10-25]
« Last Edit: October 25, 2018, 05:18:11 PM by kruhm »

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #20 on: October 25, 2018, 06:07:25 PM »
Quote
InstancesPerIP is not the issue here.

If you read the bug you would understand that this is EXACTLY the issue.

https://bugs.contribs.org/show_bug.cgi?id=10639#c3

It appears that qpsmtpd IGNORES the instances per IP instead of honouring it.

You can set it to what you like and it will not work, which it should.

So a single IP can open multiple connections and lock out your mail server.

Please, read the bug, and follow it (but don't comment unless you actually have something new to add to it)

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Daniel B.

  • *
  • 1,699
  • +0/-0
    • Firewall Services, la sรฉcuritรฉ des rรฉseaux
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #21 on: October 25, 2018, 06:38:29 PM »
Note that there might be another issue with timeout not being applied. But it would be a separated bug
C'est la fin du monde !!! :lol:

Offline bunkobugsy

  • *
  • 279
  • +4/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #22 on: October 26, 2018, 10:16:01 AM »
Note that there might be another issue with timeout not being applied. But it would be a separated bug

My thoughts exactly, thats why I attached our entire log from yesterday's DOS to bug 10639:

2018-10-24 23:40:57.050624500 11860 Accepted connection 9/10 from 162.243.25.159 / Unknown
2018-10-24 23:40:57.050701500 11860 Connection from Unknown [162.243.25.159]

2018-10-25 00:20:05.477450500 11860 (connect) tls: fail, unable to establish SSL
2018-10-25 00:20:05.477587500 11860 (deny) logging::logterse: ` 162.243.25.159   Unknown            tls   903   Cannot establish SSL session   msg denied before queued
2018-10-25 00:20:05.477650500 11860 550 Cannot establish SSL session
2018-10-25 00:20:05.477660500 11860 click, disconnecting

2018-10-25 00:20:06.414817500 5105 cleaning up after 11860

That's almost 40 minutes !!! On the other hand, is TLS configured correctly? Isn't there a mismatch in cipher versions?

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #23 on: October 26, 2018, 10:30:46 AM »
The point is if the limit per IP was working as expected the timeout would be far less of an issue.

Instances per IP should block any further connections from the IP.

If that was working as expected, a longer timeout almost works against the attacker and would keep their IP locked for longer preventing them making further connections.

Resolving any potential issues with timeouts won't fix the underlying issue with Instances per IP which is the bit that really needs fixing.

E.g. so the timeout is say 30 seconds. All that will happen is the connection gets dropped and they immediately make another. You are in the same position.

The fail2ban filter previously mentioned seems to have kept things under control for me so far.

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline bunkobugsy

  • *
  • 279
  • +4/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #24 on: October 26, 2018, 10:36:32 AM »
You're right again  :-P  You should add fail2ban regex to wiki.

But in my case I don't think it was an attack, only an unfortunate DOS from a single IP spammer.
And its throttle kept making new connections before old ones were closed, thus hitting 10/10.

Qpsmtpd / TLS plugin doesn't care about timeout.
« Last Edit: October 26, 2018, 11:04:56 AM by bunkobugsy »

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #25 on: October 26, 2018, 12:58:54 PM »
You're right again  :-P  You should add fail2ban regex to wiki.

ROFLMAO :-) I'm no genius..... just try and see the logic. The timeout issue is really a bit of a 'red herring' here and really only serves to confuse things.

OK, until we can resolve qpsmtpd.... how can we stop some of this?

fail2ban gets a little confusing as the qpsmtpd.conf filter looks at both qpsmtpd and sqpsmtpd logs.

Couple of points - the first line in the fail2ban qpsmtpd.conf regex will not work (in my experience see https://bugs.contribs.org/show_bug.cgi?id=8955 which I have reopened) The second one will.

Here's the regex:

Code: [Select]
failregex = ^\s*\d+\s*logging::logterse plugin \(deny\): ` <HOST>\s*.*90\d.*msg denied before queued$
                ^\s*\d+\s*\(deny\) logging::logterse: ` <HOST>\s*.*90\d.*msg denied before queued$

Here's the sort of lines we are looking for (or similar) from both qpsmptd and sqpsmtpd:
               
Quote
(deny) logging::logterse: ` 71.6.199.23   ubuntu1619923.aspadmin.com   openssl.client.net         tls   901   TLS Negotiation Failed   msg denied before queued
(deny) logging::logterse: ` 191.53.200.26   191-53-200-26.dvl-wr.mastercabo.com.br            tls   903   Cannot establish SSL session   msg denied before queued


The first regex line used to match the old logs, so we need the second line.

There is also an issue with my old regex from https://bugs.contribs.org/show_bug.cgi?id=8952

Here's the old regex:

Code: [Select]
^\s*\d+\s*count_unrecognized_commands plugin \(unrecognized_command\): Unrecognized command 'auth' '<HOST>'$
This will not work with the newer qpsmtpd as the log line has changed, and it also won't see things like this:


Quote
2018-10-25 16:29:29.485465500.s:49001:@400000005bd14c80027ed8ec 31545 Accepted connection 1/40 from 71.6.199.23 / ubuntu1619923.aspadmin.com
2018-10-25 16:29:29.485465500.s:49002:@400000005bd14c8002804434 31545 Connection from ubuntu1619923.aspadmin.com [71.6.199.23]
2018-10-25 16:29:29.485465500.s:49005:@400000005bd14c81116bc144 31545 (connect) earlytalker: pass, not spontaneous
2018-10-25 16:29:29.485465500.s:49006:@400000005bd14c811182df3c 31545 (connect) relay: skip, no match
2018-10-25 16:29:29.485465500.s:49007:@400000005bd14c81118a0f14 31545 (connect) ident::geoip: US
2018-10-25 16:29:29.485465500.s:49008:@400000005bd14c8111cb0014 31545 (connect) dnsbl: karma -1 (-1)
2018-10-25 16:29:29.485465500.s:49009:@400000005bd14c8111cb4e34 31545 (connect) dnsbl: fail, NAUGHTY, zen.spamhaus.org
2018-10-25 16:29:29.485465500.s:49010:@400000005bd14c8111ceebfc 31545 220 esmith.myserver.com ESMTP
2018-10-25 16:29:29.485465500.s:49011:@400000005bd14c811a2876ec 31545 dispatching EHLO openssl.client.net
2018-10-25 16:29:29.485465500.s:49012:@400000005bd14c811abfbd7c 31545 (ehlo) helo: pass
2018-10-25 16:29:29.485465500.s:49013:@400000005bd14c811acc409c 31545 250-myserver.com Hi ubuntu1619923.aspadmin.com [71.6.199.23]
2018-10-25 16:29:29.485465500.s:49014:@400000005bd14c811acd4a3c 31545 250-PIPELINING
2018-10-25 16:29:29.485465500.s:49015:@400000005bd14c811acd8ca4 31545 250-8BITMIME
2018-10-25 16:29:29.485465500.s:49016:@400000005bd14c811acdea64 31545 250-SIZE 20000000
2018-10-25 16:29:29.485465500.s:49017:@400000005bd14c811ace4ff4 31545 250 STARTTLS
2018-10-25 16:29:29.485465500.s:49018:@400000005bd14c812326bdbc 31545 dispatching STARTTLS
2018-10-25 16:29:29.485465500.s:49019:@400000005bd14c81232857e4 31545 220 Go ahead with TLS
2018-10-25 16:29:29.485465500.s:49022:@400000005bd14c812b88284c 31545 (deny) logging::logterse: ` 71.6.199.23   ubuntu1619923.aspadmin.com   openssl.client.net         tls   901   TLS Negotiation Failed   msg denied before queued
2018-10-25 16:29:29.485465500.s:49023:@400000005bd14c812b8a08c4 31545 500 TLS Negotiation Failed
2018-10-25 16:29:29.485465500.s:49025:@400000005bd14c9e030d78a4 31545 dispatching ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ1=H<๏ฟฝ๏ฟฝ_,๏ฟฝ๏ฟฝb๏ฟฝ+
2018-10-25 16:29:29.485465500.s:49026:@400000005bd14c9e035873f4 31545 (unrecognized_command) count_unrecognized_commands: '๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ1=h<๏ฟฝ๏ฟฝ_,๏ฟฝ๏ฟฝb๏ฟฝ+', (1)
2018-10-25 16:29:29.485465500.s:49027:@400000005bd14c9e03587bc4 31545 500 Unrecognized command

It seem to throw this in the fail2ban logs:
Quote
fail2ban.filter         [26870]: WARNING Error decoding line from '/var/log/qpsmtpd/current' with 'UTF-8'. Consider setting logencoding=utf-8 (or another appropriate encoding) for this jail. Continuing to process line ignoring invalid characters: '@400000005bd1d6182eba1ad4 24134 dispatching \x05\x00\x80\x03\x00\x80\x01\x00\x80\x07\x00\xc0\x86\x8eD\xae%[\x9e\xae\x14K\x7fe\xfe\x06\x11\xc9\n'

I have made a note on the bug and will look at it again next week.

Quote
But in my case I don't think it was an attack, only an unfortunate DOS from a single IP spammer.
And its throttle kept making new connections before old ones were closed, thus hitting 10/10.

Yes, can be a single spammy IP but it keeps banging away which creates the DoS. It should be stopped before that happens which is what Instances per IP should do.


Quote
Qpsmtpd / TLS plugin doesn't care about timeout.

Possibly it should, but that is another issue.....
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #26 on: October 26, 2018, 01:49:48 PM »
Bug 10639 updated with proposed fix to /var/service/sqpsmtpd/run

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #27 on: October 26, 2018, 04:54:38 PM »
Bug 10639 updated with proposed fix to /var/service/sqpsmtpd/run

Nice :-)

Wonder why....
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #28 on: October 26, 2018, 09:38:41 PM »
Is the fix in bug 10639 something I can test? I have a production server experiencing this problem.
- Mark

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #29 on: October 26, 2018, 09:45:35 PM »
It's not a definitive 'fix' but a possible workaround.

By all means test, but you should really try on a test machine... yes you can try on your production box... at your own risk & YMMV.....
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #30 on: October 26, 2018, 10:25:20 PM »
I understand the risk. I can put this on a test machine, but I am not sure I can replicate the source of the condition.

I can stand by the production machine and revert the changes quickly if necessary.
- Mark

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #31 on: October 26, 2018, 11:25:20 PM »
You can replicate by following the comments in the bug.

I think Daniel explained how to test.

https://bugs.contribs.org/show_bug.cgi?id=10639#c8

There may be a commemt here in this thread too.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #32 on: October 26, 2018, 11:49:46 PM »
The patched server is behaving fine. I do see that the source of the overload is still active. However, the logs suggest the server is managing it OK.

I am tempted to ban the address and see if someone on the other end squeals.

Now, the change was only the reordering of the parameters in the service run script, as posted by mccarn. I have not changed the fail2ban regex.
- Mark

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #33 on: October 26, 2018, 11:55:50 PM »
The question is does attempt no 6 get blocked?

Open another terminal and tail the sqpsmtpd log while telnetting in the first.

Look for connection 'x of y' eg 0/10, 1/10 etc.

It should ignore connections after no 5

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #34 on: October 27, 2018, 12:18:23 AM »
Looking at forkserver it appears a connection made every 22 minutes.

The sqsmptd log shows "Accepted connection 3/15 from ... / Unknown"
- Mark

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #35 on: October 27, 2018, 12:25:49 AM »
Sorry thought you were on a test machine.

All you can do is monitor. To test you need to follow the instructions and see if it locks out the IP.

Right. Way past my bed time. Hasta maรฑana :-)
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #36 on: October 27, 2018, 12:30:53 AM »
Thanks for all the help. I really appreciate it.

I will check on this about 19:30 pacific time and post the outcome.
- Mark

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #37 on: October 27, 2018, 12:40:14 AM »
KK. NP. Ciao.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #38 on: October 27, 2018, 12:40:51 AM »
A question, for later, does come to mind: where is the limit value of 15 set?

cat  /var/service/sqpsmtpd/runenv
reveals this...
INSTANCES=10
INSTANCES_PER_IP=5
- Mark

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #39 on: October 27, 2018, 12:48:32 AM »
Mentioned earlier in thread I think :-)

Also possibly on the bug.

You can also have a dig in the templates to see for yourself.

Try looking in:

/etc/e-smith/templates/var/service/sqpsmtpd/runenv/
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #40 on: October 27, 2018, 04:40:36 AM »
I checked the server at 19:15 pacific time and the "Too many connections: 15 >= 15.  Waiting one second." entries were filling the log. forkserver reported the 15 connections.

To recap, it appears the changes to the service run script, as posted by mccarn, did not work around the bug in my case. I did not update the regex in the fail2ban, however.

I have manually banned the offending IP address for now. I will revisit this after some sleep.
- Mark

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #41 on: October 27, 2018, 06:30:27 AM »
A question, for later, does come to mind: where is the limit value of 15 set?

15 is the default value of INSTANCES in /usr/bin/qpsmtpd-forkserver itself:
Quote from: /usr/bin/qpsmtpd-forkserver
# Configuration
my $MAXCONN = 15;    # max simultaneous connections
my @PORT;            # port number(s)
my @LOCALADDR;       # ip address(es) to bind to
my $MAXCONNIP = 5;   # max simultaneous connections from one IP
my $PID_FILE  = '';
my $DETACH;          # daemonize on startup
my $NORDNS;

Look for the actual running parameters; if you're missing the "-c ##" argument, you'd end up with 15.  Of course, if you have it and it's set to 15, you'd have 15 (but you'd need to figure out why it's set to 15...)
Quote from: ps auxwww |grep qpsmtpd
root       966  0.0  0.0    108    24 ?        Ss   Oct25   0:00 runsv qpsmtpd
root       999  0.0  0.0    108    20 ?        Ss   Oct25   0:00 runsv sqpsmtpd
smelog    1012  0.0  0.0   4072   396 ?        S    Oct25   0:00 /usr/local/bin/multilog t s5000000 n30 !/usr/local/bin/qplogsumm.pl /var/log/qpsmtpd
smelog    1024  0.0  0.0   4072   392 ?        S    Oct25   0:00 /usr/local/bin/multilog t s5000000 n30 /var/log/sqpsmtpd
qpsmtpd   2468  0.0  0.6 119424 27840 ?        S    Oct25   0:14 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -m 5 -c 40 -p 25
qpsmtpd  30635  0.0  0.6 119424 28008 ?        S    Oct26   0:06 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 465 -m 5 -c 10

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #42 on: October 27, 2018, 05:36:57 PM »
Thank you. I ran the current state test.
Code: [Select]
qpsmtpd  26194  0.0  0.2  71908 16148 ?        S    Oct26   0:03 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 25 -c 40 -m 5
qpsmtpd  26326  0.0  0.2  71908 16140 ?        S    Oct26   0:02 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 465

I notice that the -m and -c parameters are absent for port 465. Perhaps that is something I should fix?

The offending IP address was released from jail at 05:21 pacific time today. The connections resumed ten minutes later and continue on the 22 minute period.

If I modify the fail2ban regex, will that suffice as a work around?

I noticed updates to bugs 10639 and 10387. Perhaps changing the crontab as suggested is a better approach. Thoughts?

Update: I created a custom template for crontab and rebooted the machine. I am waiting to see if the offending connections are held at bay.
« Last Edit: October 27, 2018, 07:52:52 PM by Mophilly »
- Mark

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #43 on: October 27, 2018, 09:20:52 PM »
I notice that the -m and -c parameters are absent for port 465. Perhaps that is something I should fix?

Yes.  My guess is that you're missing the trailing "\" at the end of the port line in /var/service/sqpsmtpd/run (colored red in the extract below):
Quote from: /var/service/sqpsmtpd/run
...
exec /usr/local/bin/softlimit -d ${SOFTLIMIT:-25000000} -s ${SOFTLIMIT:-25000000} -l ${SOFTLIMIT:-25000000} \
  /usr/bin/qpsmtpd-forkserver \
        -u qpsmtpd \
        -l 0.0.0.0 \
        -p ${PORT:-465} \
        -c ${INSTANCES:-40} \
        -m ${INSTANCES_PER_IP:-5}

...

You would get the same behavior if the backslash is there but has a space after it - "\ " instead of "\"

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #44 on: October 28, 2018, 01:00:17 AM »
Fail2ban seems to work for me.

Set these:

Code: [Select]
fail2ban=service
    BanTime=1800
    FindTime=900

I set the /etc/fail2ban/jail.conf template for qpsmtpd to max 3 retries

Copied this to a custom template:

/etc/e-smith/templates/etc
fail2ban/jail.conf/30Service20qpsmtpd

Then changed this

Code: [Select]
my $max = $maxretry*3;

To this

Code: [Select]
my $max = 3;

So we get this:

Cat /etc/fail2ban/jail.conf

Quote
Blah....
[qpsmtpd]
enabled  = true
filter   = qpsmtpd
logpath  = /var/log/*qpsmtpd/current
maxretry = 3
Blah

So more than 3 attempts get picked up (choose your own values)

Add the regex and signal-event fail2ban-conf

Obviously YMMV and it may need tweaking. It may be better to separate qpsmtpd and sqpsmtpd so you can have different settings. Currently the ban mail tells me it's qpsmtpd when it could be actually a ban from sqpsmtpd. I'll look at it more when home.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #45 on: October 28, 2018, 01:51:38 AM »
Yes.  My guess is that you're missing the trailing "\" at the end of the port line in /var/service/sqpsmtpd/run (colored red in the extract below):
You would get the same behavior if the backslash is there but has a space after it - "\ " instead of "\"

And you are correct! That's fixed.

ran signal-event post-upgrade; signal-event reboot;
then...
Code: [Select]
ps auxwww |grep qpsmtpd
smelog    1173  0.0  0.0   3940   360 ?        S    16:54   0:00 /usr/local/bin/multilog t s5000000 n10 /var/log/sqpsmtpd
smelog    1182  0.0  0.0   3940   360 ?        S    16:54   0:00 /usr/local/bin/multilog t s5000000 n10 /var/log/qpsmtpd
qpsmtpd   2735  0.0  0.2  71908 16292 ?        S    16:55   0:00 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 25 -c 40 -m 5
qpsmtpd   2765  0.0  0.2  71908 16276 ?        S    16:55   0:00 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 465
root     11987  0.0  0.0 103324   868 pts/0    S+   16:59   0:00 grep qpsmtpd

The parameters are still missing. I must have over looked something, although I double checked the run script.
« Last Edit: October 28, 2018, 02:02:27 AM by Mophilly »
- Mark

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #46 on: October 28, 2018, 02:01:44 AM »
Quote
And you are correct! That's fixed.

Lord there are some clever buggers round here :-)
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #47 on: October 28, 2018, 05:08:25 AM »
ran signal-event post-upgrade; signal-event reboot;

You shouldn't need to reboot to see if you've fixed it; stopping and starting sqpsmtpd is enough.
Quote
ps auxwww |grep qpsmtpd
root       963  0.0  0.0    108    28 ?        Ss   15:26   0:00 runsv qpsmtpd
root       992  0.0  0.0    108    28 ?        Ss   15:26   0:00 runsv sqpsmtpd
smelog    1018  0.0  0.0   3940   296 ?        S    15:26   0:00 /usr/local/bin/multilog t s5000000 n30 /var/log/sqpsmtpd
smelog    1028  0.0  0.0   3940   316 ?        S    15:26   0:00 /usr/local/bin/multilog t s5000000 n30 !/usr/local/bin/qplogsumm.pl /var/log/qpsmtpd
qpsmtpd   2464  0.0  0.7 119424 29432 ?        S    15:30   0:04 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -m 5 -c 40 -p 25
qpsmtpd   2622  0.0  0.7 119424 28484 ?        S    15:30   0:03 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 465 -c 10 -m 5

# edit /var/service/sqpsmtpd/run - move "-p ${PORT:-465}" to a different position

sv d sqpsmtpd
sv u sqpsmtpd

# ps auxwww |grep qpsmtpd
root       963  0.0  0.0    108    28 ?        Ss   Oct27   0:00 runsv qpsmtpd
root       992  0.0  0.0    108    28 ?        Ss   Oct27   0:00 runsv sqpsmtpd
smelog    1018  0.0  0.0   3940   316 ?        S    Oct27   0:00 /usr/local/bin/multilog t s5000000 n30 /var/log/sqpsmtpd
smelog    1028  0.0  0.0   3940   316 ?        S    Oct27   0:00 /usr/local/bin/multilog t s5000000 n30 !/usr/local/bin/qplogsumm.pl /var/log/qpsmtpd
qpsmtpd   2464  0.0  0.7 119424 29504 ?        S    Oct27   0:04 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -m 5 -c 40 -p 25
qpsmtpd   4683 20.3  0.4  71908 16256 ?        S    00:01   0:00 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -c 10 -m 5 -p 465

Here is my entire copy of /var/service/sqpsmtpd/run for reference:
Code: [Select]
#!/bin/sh
#----------------------------------------------------------------------
# copyright (C) 1999-2005 Mitel Networks Corporation
# Copyright (C) 2005-2006 Gordon Rowell <gordonr@gormand.com.au>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307  USA
#----------------------------------------------------------------------

exec 2>&1

[ -f ./runenv ] && . ./runenv

export QPSMTPD_CONFIG=/var/service/qpsmtpd/config

../qpsmtpd/control/1

exec /usr/local/bin/softlimit -d ${SOFTLIMIT:-25000000} -s ${SOFTLIMIT:-25000000} -l ${SOFTLIMIT:-25000000} \
  /usr/bin/qpsmtpd-forkserver \
-u qpsmtpd \
-l 0.0.0.0 \
-c ${INSTANCES:-40} \
        -m ${INSTANCES_PER_IP:-5} \
        -p ${PORT:-465}

#sub usage {
#        print <<"EOT";
#usage: qpsmtpd-forkserver [ options ]
# -l, --listen-address addr : listen on a specific address; default 0.0.0.0
# -p, --port P              : listen on a specific port; default 2525
# -c, --limit-connections N : limit concurrent connections to N; default 15
# -u, --user U              : run as a particular user (defualt 'smtpd')
# -m, --max-from-ip M       : limit connections from a single IP; default 5
#EOT
#        exit 0;
#}
#

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #48 on: October 28, 2018, 06:01:35 PM »
Thank you, again, for the detail. I revised the run script to use precisely the order of assigments in your example and restarted sqpsmtpd. Now the parameters appear as desired.

One remaining question: if this change allows sqpsmtpd to respect the limit per ip, is there still a need to apply the revision to fail2ban to trap specific log entries, suggested elsewhere?
- Mark

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #49 on: October 28, 2018, 07:57:02 PM »
One remaining question: if this change allows sqpsmtpd to respect the limit per ip, is there still a need to apply the revision to fail2ban to trap specific log entries, suggested elsewhere?

I don't know about the fail2ban trap; I'll leave that for others.

Be aware that the order of the arguments was not the solution to the CONCURRENCY_PER_IP problem - that turns out to be a problem with qpsmtpd-forkserver where it doesn't pay attention to CONCURRENCY_PER_IP until after it receives a HUP signal.  You can read more in comments 22-24 in this bug: https://bugs.contribs.org/show_bug.cgi?id=10639#c24

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #50 on: October 28, 2018, 08:14:48 PM »
Be aware that the order of the arguments was not the solution to the CONCURRENCY_PER_IP problem - that turns out to be a problem with qpsmtpd-forkserver where it doesn't pay attention to CONCURRENCY_PER_IP until after it receives a HUP signal.

Understood. I used copy and paste when I first modified the script with nano; perhaps an errant character was introduced that was removed when I edited the script this last time.
- Mark

Offline Mophilly

  • *
  • 384
  • +0/-0
    • Mophilly
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #51 on: October 28, 2018, 10:23:51 PM »
a problem with qpsmtpd-forkserver where it doesn't pay attention to CONCURRENCY_PER_IP until after it receives a HUP signal.

After the latest changes, I let the system run and noticed the log reports x/10 in connect accepted messages. After a time this appeared in the log:
Code: [Select]
2018-10-28 14:14:21.279064500 32415 Accepted connection 9/10 from 174.138.53.173 / Unknown
2018-10-28 14:14:21.279125500 32415 Connection from Unknown [174.138.53.173]
2018-10-28 14:14:22.273792500 17583 Too many connections: 10 >= 10.  Waiting one second.


At that point, the users cannot send mail. Not sure what to consider next.
- Mark

Offline bunkobugsy

  • *
  • 279
  • +4/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #52 on: October 29, 2018, 08:24:46 AM »
If you're willing to try... https://bugs.contribs.org/show_bug.cgi?id=10387#c19

uncomment  $qpsmtpd->load_plugins  at line 199 in /usr/bin/qpsmtpd-forkserver
and restart (at least sqpsmtpd and qpsmtpd)
« Last Edit: October 29, 2018, 08:27:21 AM by bunkobugsy »

Offline bunkobugsy

  • *
  • 279
  • +4/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #53 on: October 29, 2018, 10:07:22 PM »
Tried it myself, uncommenting $qpsmtpd->load_plugins at line 199 in /usr/bin/qpsmtpd-forkserver seems to fix CONCURRENCY_PER_IP problem.
The default 120 second timeout is still ignored (in the TLS plugin at least).

Don't know about any side effects of this patch, haven't tested long enough.

Offline bunkobugsy

  • *
  • 279
  • +4/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #54 on: October 30, 2018, 11:29:44 AM »
120 second timeout works regardless of this patch, but only on port 25/qpsmtpd.

Timeout not working on port 465/sqpsmtp must be a separate issue with TLS.

Offline dave simmons

  • ***
  • 125
  • +0/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #55 on: November 16, 2018, 09:52:08 AM »
I seem to also have become a victim of this.  SME 9.2 with all updates applied, no additional contribs.

Sorry for my stupidity, but could someone please explain in simple terms what I need to do?

I've read the bugs linked and a coupe of other forum threads, but I'm no further  :(

If it helps, I have a second SME 9.2 which I haven't updated (forgot!) for about 6 months.  This is showing the same problem with connections, but this machine DOES drop the connections quickly enough that I don't have the "too many connections" message.  Maybe something wrong with an update?

I've also got a SME 8 machine running (naughty!) which handles this fine - excerpt from the log file -

"2018-11-16 09:50:05.822193500 2797 hosts_allow plugin (pre-connection): Too many connections from 159.89.18.60: 6 > 5Denying connection."

ETA:  Some googling brought up this from 2015 - https://forums.contribs.org/index.php?topic=51882.0
« Last Edit: November 16, 2018, 10:41:59 AM by dave simmons »

Offline TerryF

  • grumpy old man
  • *
  • 1,826
  • +6/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #56 on: November 16, 2018, 10:33:10 AM »
I've read the bugs linked and a coupe of other forum threads, but I'm no further  :(

This:
uncomment  $qpsmtpd->load_plugins  at line 199 in /usr/bin/qpsmtpd-forkserver
and restart (at least sqpsmtpd and qpsmtpd)

Open that file in a suitable editor go to line 199  # $qpsmtpd->load_plugins
remove the # save the file and restart

This is the section the line is in, its from mine, I have removed the #:

endgrent;
$) = $groups;
POSIX::setgid($qgid) or die "unable to change gid: $!\n";
POSIX::setuid($quid) or die "unable to change uid: $!\n";
$> = $quid;

$qpsmtpd->load_plugins;

foreach my $addr (@LISTENADDR) {
    ::log(LOGINFO, "Listening on $addr->{addr}:$addr->{port}");
}
::log(LOGINFO,

 
--
qui scribit bis legit

Offline dave simmons

  • ***
  • 125
  • +0/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #57 on: November 16, 2018, 11:04:32 AM »
TerryF - thank you - I have done this and will keep an eye on it!

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #58 on: November 16, 2018, 11:15:33 AM »
Rather than cluttering the thread I have added some updated fail2ban files and instructions for use here:

https://bugs.contribs.org/show_bug.cgi?id=8955

As a workaround, that seems to have stopped overloading connections for me.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline TerryF

  • grumpy old man
  • *
  • 1,826
  • +6/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #59 on: November 16, 2018, 11:30:36 AM »
and it is very comprehensive, recommended.

Thanks John
--
qui scribit bis legit

Offline dave simmons

  • ***
  • 125
  • +0/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #60 on: November 16, 2018, 12:14:48 PM »
Rather than cluttering the thread I have added some updated fail2ban files and instructions for use here:

https://bugs.contribs.org/show_bug.cgi?id=8955

As a workaround, that seems to have stopped overloading connections for me.

Do I need to apply this in addition to the advice from TerryF?  (Sorry if it's a stupid question!)ยต

ETA:  I see in the logfile that the change has worked - the "spammer" gets rejected after 5 connections.  This is enough for us because we are only 3 users!

Thank you all for your help!  :)
« Last Edit: November 16, 2018, 12:18:49 PM by dave simmons »

Offline ReetP

  • *
  • 3,731
  • +5/-0
Re: Too many connections: 10 >= 10. Waiting one second.
« Reply #61 on: November 16, 2018, 05:44:57 PM »
It's a different way of tackling the same problem, and probably easier if you don't want to go hacking system files.

I haven't had an issue myself since I added it.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation