Topic: Too many connections: 10 >= 10. Waiting one second.

[Mophilly]

I understand the risk. I can put this on a test machine, but I am not sure I can replicate the source of the condition.

I can stand by the production machine and revert the changes quickly if necessary.

[ReetP]

You can replicate by following the comments in the bug.

I think Daniel explained how to test.

https://bugs.contribs.org/show_bug.cgi?id=10639#c8

There may be a commemt here in this thread too.

[Mophilly]

The patched server is behaving fine. I do see that the source of the overload is still active. However, the logs suggest the server is managing it OK.

I am tempted to ban the address and see if someone on the other end squeals.

Now, the change was only the reordering of the parameters in the service run script, as posted by mccarn. I have not changed the fail2ban regex.

[ReetP]

The question is does attempt no 6 get blocked?

Open another terminal and tail the sqpsmtpd log while telnetting in the first.

Look for connection 'x of y' eg 0/10, 1/10 etc.

It should ignore connections after no 5

[Mophilly]

Looking at forkserver it appears a connection made every 22 minutes.

The sqsmptd log shows "Accepted connection 3/15 from ... / Unknown"

[ReetP]

Sorry thought you were on a test machine.

All you can do is monitor. To test you need to follow the instructions and see if it locks out the IP.

Right. Way past my bed time. Hasta mañana

[Mophilly]

Thanks for all the help. I really appreciate it.

I will check on this about 19:30 pacific time and post the outcome.

[ReetP]

KK. NP. Ciao.

[Mophilly]

A question, for later, does come to mind: where is the limit value of 15 set?

cat  /var/service/sqpsmtpd/runenv
reveals this...
INSTANCES=10
INSTANCES_PER_IP=5

[ReetP]

Mentioned earlier in thread I think

Also possibly on the bug.

You can also have a dig in the templates to see for yourself.

Try looking in:

/etc/e-smith/templates/var/service/sqpsmtpd/runenv/

[Mophilly]

I checked the server at 19:15 pacific time and the "Too many connections: 15 >= 15.  Waiting one second." entries were filling the log. forkserver reported the 15 connections.

To recap, it appears the changes to the service run script, as posted by mccarn, did not work around the bug in my case. I did not update the regex in the fail2ban, however.

I have manually banned the offending IP address for now. I will revisit this after some sleep.

[mmccarn]

A question, for later, does come to mind: where is the limit value of 15 set?

15 is the default value of INSTANCES in /usr/bin/qpsmtpd-forkserver itself:

# Configuration
my $MAXCONN = 15;    # max simultaneous connections
my @PORT;            # port number(s)
my @LOCALADDR;       # ip address(es) to bind to
my $MAXCONNIP = 5;   # max simultaneous connections from one IP
my $PID_FILE  = '';
my $DETACH;          # daemonize on startup
my $NORDNS;

Look for the actual running parameters; if you're missing the "-c ##" argument, you'd end up with 15.  Of course, if you have it and it's set to 15, you'd have 15 (but you'd need to figure out why it's set to 15...)

root       966  0.0  0.0    108    24 ?        Ss   Oct25   0:00 runsv qpsmtpd
root       999  0.0  0.0    108    20 ?        Ss   Oct25   0:00 runsv sqpsmtpd
smelog    1012  0.0  0.0   4072   396 ?        S    Oct25   0:00 /usr/local/bin/multilog t s5000000 n30 !/usr/local/bin/qplogsumm.pl /var/log/qpsmtpd
smelog    1024  0.0  0.0   4072   392 ?        S    Oct25   0:00 /usr/local/bin/multilog t s5000000 n30 /var/log/sqpsmtpd
qpsmtpd   2468  0.0  0.6 119424 27840 ?        S    Oct25   0:14 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -m 5 -c 40 -p 25
qpsmtpd  30635  0.0  0.6 119424 28008 ?        S    Oct26   0:06 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 465 -m 5 -c 10

[Mophilly]

Thank you. I ran the current state test.

Code: [Select]

qpsmtpd  26194  0.0  0.2  71908 16148 ?        S    Oct26   0:03 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 25 -c 40 -m 5
qpsmtpd  26326  0.0  0.2  71908 16140 ?        S    Oct26   0:02 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 465

I notice that the -m and -c parameters are absent for port 465. Perhaps that is something I should fix?

The offending IP address was released from jail at 05:21 pacific time today. The connections resumed ten minutes later and continue on the 22 minute period.

If I modify the fail2ban regex, will that suffice as a work around?

I noticed updates to bugs 10639 and 10387. Perhaps changing the crontab as suggested is a better approach. Thoughts?

Update: I created a custom template for crontab and rebooted the machine. I am waiting to see if the offending connections are held at bay.

[mmccarn]

I notice that the -m and -c parameters are absent for port 465. Perhaps that is something I should fix?

Yes.  My guess is that you're missing the trailing "\" at the end of the port line in /var/service/sqpsmtpd/run (colored red in the extract below):

...
exec /usr/local/bin/softlimit -d ${SOFTLIMIT:-25000000} -s ${SOFTLIMIT:-25000000} -l ${SOFTLIMIT:-25000000} \
  /usr/bin/qpsmtpd-forkserver \
        -u qpsmtpd \
        -l 0.0.0.0 \
        -p ${PORT:-465} \
        -c ${INSTANCES:-40} \
        -m ${INSTANCES_PER_IP:-5}
...

You would get the same behavior if the backslash is there but has a space after it - "\ " instead of "\"

[ReetP]

Fail2ban seems to work for me.

Set these:

Code: [Select]

fail2ban=service
    BanTime=1800
    FindTime=900

I set the /etc/fail2ban/jail.conf template for qpsmtpd to max 3 retries

Copied this to a custom template:

/etc/e-smith/templates/etc
fail2ban/jail.conf/30Service20qpsmtpd

Then changed this

Code: [Select]

my $max = $maxretry*3;

To this

Code: [Select]

my $max = 3;

So we get this:

Cat /etc/fail2ban/jail.conf

Blah....
[qpsmtpd]
enabled  = true
filter   = qpsmtpd
logpath  = /var/log/*qpsmtpd/current
maxretry = 3
Blah

So more than 3 attempts get picked up (choose your own values)

Add the regex and signal-event fail2ban-conf

Obviously YMMV and it may need tweaking. It may be better to separate qpsmtpd and sqpsmtpd so you can have different settings. Currently the ban mail tells me it's qpsmtpd when it could be actually a ban from sqpsmtpd. I'll look at it more when home.