Topic: Problem: Too many connections

[Manu72]

Hi all,

We have a SME9.0 and a dedicated Antispam Box.
So the emails arrive at the Antispam Box then are being transmited to the SME9.0 for delivery to the user mailboxes.
Since more than a week we are having troubles with emails arriving hours late and by hours I mean between 3 and 24h late.
The emails are arriving in time at the Antispam Box, but then, for an important percentage of emails, it can take up 24 hours to get into the user mailbox.

Looking into logs i notice a lot of messages like:

16:06:17.180954500 2599 hosts_allow plugin (pre-connection): Too many connections from (local_IP_of_Antispam_Box): 6 > 5Denying connection.

So apparently the SME is blocking and delaying legitimate emails incoming from the antispam box.

I tried to increase the InstancesPerIP but nothing changed.

now i have the following settings:

config show smtpd
          Instances=80
          InstancesPerIP=40

config show ssmtpd
          Instances=80
          InstancesPerIP=40

But, as i said, i still get the error message and the delayed emails.

Help please

[guest22]

Hi and welcome.


please tell us a bit more about your SME Server 9.0 server. How is it configured, and what contribs did you install?

[Manu72]

Well, it is a basic install with no extra contribs, yet.
It is working in gateway mode with 2 network adapters - one with public IP address and one in the local lan.
The SMTP port is forwarded to the internal IP address of the Antispam-Box


Edit:
After modifying the InstancesPerIP parameter i noticed the following message in the logs:
45843 Accepted connection 9/40 from (local_IP_of_Antispam_Box)

then a bunch of:
2599 hosts_allow plugin (pre-connection): Too many connections from (local_IP_of_Antispam_Box)

Basically i never seen anything bigger than 9/40

[CharlieBrady]

Looking into logs i notice a lot of messages like:

16:06:17.180954500 2599 hosts_allow plugin (pre-connection): Too many connections from (local_IP_of_Antispam_Box): 6 > 5Denying connection.

If you look at the code for the host_allow plugin, you'll see some documentation:

...
The B<hosts_allow> module decides before the SMTP-Greeting if a host is
allowed to connect. It checks for too many (running) connections from one
host (see -m/--max-from-ip options in qpsmtpd-forkserver) and the config
file I<hosts_allow>.
The plugin takes no arguments.
...

If you look in /service/qpsmtpd/run you will see:

...
exec /usr/local/bin/softlimit -d ${SOFTLIMIT:-25000000} -s ${SOFTLIMIT:-25000000} -l ${SOFTLIMIT:-25000000} \
  /usr/bin/qpsmtpd-forkserver \
        -u qpsmtpd \
        -l 0.0.0.0 \
        -p ${PORT:-25} \
        -c ${INSTANCES:-40} \
        -m ${INSTANCES_PER_IP:-5}
...

which points you to "INSTANCES_PER_IP". You'll see that is set by template file, from /root/etc/e-smith/templates/var/service/qpsmtpd/runenv/INSTANCES_PER_IP:

{
    return "INSTANCES_PER_IP=" . ($smtpd{InstancesPerIP} || "5");
}

So if you do:

config setprop smtpd InstancesPerIP 40
expand-template /var/service/qpsmtpd/runenv
sv t /service/qpsmtpd

you should see a change in your system's behaviour. I suspect you only did the first of those three steps. Changing the properties in the configuration database isn't sufficient - you need to activate the changes by expanding templates and restarting services.

[Manu72]

Thank you for replying.

I made the changes as per wiki with:
config setprop smtpd InstancesPerIP  40
signal-event email-update

Then i checked the value by looking into /var/service/qpsmtpd/runenv and the value (40) was there
Then i also checked with config show smtpd and i got the values already posted in the opening of this thread.

I will try to rerun the commands mentioned and will see how it goes.

Edit: Did the commands and apparently we're getting somewhere.
In the last 10 minutes there were no more "too many connections" and i found in logs the following line:
21:16:20.594178500 1069 Accepted connection 31/80 from (local_IP_of_Antispam_Box)

Previously it was like 9/40 at most. So I assume it refers to Instances and not to InstancesPerIP as i thought.
Default it was 5/40 and to maximize the confusion  i was setting them at 40/80.

[CharlieBrady]

I made the changes as per wiki with:
config setprop smtpd InstancesPerIP  40
signal-event email-update

The email-update event only sends SIGHUP single to the running qpsmtpd, but doesn't restart it. Only restarting qpsmtpd will activate the new InstancesPerIP setting.

At a minimum the wiki should be updates, but perhaps the software could be updated so that the email-update event is sufficient.

[Manu72]

Didn't expect such a prompt answer so I edited/updated my previous post

Thanks again, will keep an eye on the server.
 

[CharlieBrady]

Previously it was like 9/40 at most. So I assume it refers to Instances and not to InstancesPerIP as i thought.

InstancesPerIP is the correct param to tweak for the hosts_allow plugin issue which is blocking too many attempts from the one IP address.

Did you also check the "AntiSpam Box" to see if it could be tweaked to be more gentle to the mail server?

[Manu72]

I know the InstancesPerIP is the culprit, but I was wondering what parameter is referred by the bolded part below.
21:16:20.594178500 1069 Accepted connection 31/80 from (local_IP_of_Antispam_Box)

It should be Instances, because this parameter is set at 80, while InstancesPerIP is set at 40. This increased the confusion as well.


Regarding the antispam_box, it does what it's supposed to do.
I assume we encountered a sudden increase in mail flow due to missed spam / ndr attack / whatever, then SME started to block the antispam and from then everything went from bad to worse since emails kept piling in the antispam because denied connections.
We removed the antispam_box from the loop yesterday and now emails arrive directly on SME, but even now it still sends the emails that piled in its queue.
Maybe I will put the antispam back in the loop after everything settle up

Regarding the culprit,  5 InstancesPerIP may be OK in a normal scenario, but seems to be quite low when all emails arrive from a single IP, which is the case with an external Mail-Filter that scan the emails before they arrive on SME Server.

[CharlieBrady]

I know the InstancesPerIP is the culprit, but I was wondering what parameter is referred by the bolded part below.
21:16:20.594178500 1069 Accepted connection 31/80 from (local_IP_of_Antispam_Box)

It should be Instances, ...

It is.

[guest22]

So what should be changed on the wiki then please?

[CharlieBrady]

So what should be changed on the wiki then please?

Correct instructions are listed here:

http://forums.contribs.org/index.php/topic,51882.msg264666.html#msg264666

Incorrect instructions (i.e. those currently mentioned in the wiki) are listed here:

http://forums.contribs.org/index.php/topic,51882.msg264670.html#msg264670