Koozali.org formerly Contribs.org

qpsmtpd-forkserver processes puts a high load on the server

Offline holck

  • *
  • 266
qpsmtpd-forkserver processes puts a high load on the server
« on: January 05, 2019, 09:53:25 AM »
Recently, I've had two incidents where qpsmtpd-forkserver puts a high load on my server. Here is a snip from the output from top:

Code: [Select]
top - 09:27:35 up 13 days,  9:19,  1 user,  load average: 0.21, 0.32, 0.32
...

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
29996 qpsmtpd   20   0  152m  40m 4252 S 20.6  1.1   0:00.62 qpsmtpd-forkser
 3139 root      20   0 1587m  21m 1908 S  0.3  0.6  62:01.88 fail2ban-server
Usually, the load averages are only a few percent.

Searching for the qpsmtpd-forkserver processes shows that new processes keep turning up, connected to the same, remote IP address:

Code: [Select]
28409 ?        S      0:15 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 465 -c 10 -m 5
29614 ?        S      0:24 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 25 -c 40 -m 5
29999 ?        S      0:00 /usr/bin/qpsmtpd-forkserver [190.64.84.98 : r190-64-84-98.su-static.adinet.com.uy : 09:27:41 2019-01-05]
30000 ?        S      0:00 /usr/bin/qpsmtpd-forkserver [103.207.37.246 : Unknown : 09:27:42 2019-01-05]
...
28409 ?        S      0:15 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 465 -c 10 -m 5
29614 ?        S      0:24 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 25 -c 40 -m 5
30014 ?        R      0:00 /usr/bin/qpsmtpd-forkserver [103.207.37.246 : Unknown : 09:28:02 2019-01-05]
...
28409 ?        S      0:15 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 465 -c 10 -m 5
29614 ?        S      0:24 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 25 -c 40 -m 5
30023 ?        R      0:00 /usr/bin/qpsmtpd-forkserver [103.207.37.246 : Unknown : 09:28:16 2019-01-05]
...
28409 ?        S      0:15 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 465 -c 10 -m 5
29614 ?        S      0:24 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 25 -c 40 -m 5
30030 ?        Z      0:00 [qpsmtpd-forkser] <defunct>
...
28409 ?        S      0:15 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 465 -c 10 -m 5
29614 ?        S      0:24 /usr/bin/perl -Tw /usr/bin/qpsmtpd-forkserver -u qpsmtpd -l 0.0.0.0 -p 25 -c 40 -m 5
30034 ?        S      0:00 /usr/bin/qpsmtpd-forkserver [103.207.37.246 : Unknown : 09:28:32 2019-01-05]

This just goes on and on...

If I ban the remote IP-address, using fail2ban, the problem is gone:
Code: [Select]
$ sudo fail2ban-client set qpsmtpd banip 103.207.37.246
Code: [Select]
top - 09:30:48 up 13 days,  9:22,  1 user,  load average: 0.12, 0.24, 0.28
...

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  481 squid     20   0  144m  80m 4388 S  0.3  2.1   1:06.91 squid 
 3139 root      20   0 1587m  21m 1908 S  0.3  0.6  62:02.43 fail2ban-server
Is this something to worry about?

Happy new year, and a warm thank you to everyone contributing to SME :-)
« Last Edit: January 05, 2019, 09:59:02 AM by holck »
......

Offline mmccarn

  • *
  • 2,350
Re: qpsmtpd-forkserver processes puts a high load on the server
« Reply #1 on: January 05, 2019, 01:51:59 PM »
That IP is located in Vietnam, and is listed on 14 different blocklists:
https://ipinfo.io/103.207.37.246
https://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a103.207.37.246&run=toolpage#

Were the connections blocked by the qpsmtpd rbl plugin?

If not, why not?

If so, why didn't fail2ban block the attacking IP automatically?

Are there any attacks on services other than smtp from the same IP?

Offline holck

  • *
  • 266
Re: qpsmtpd-forkserver processes puts a high load on the server
« Reply #2 on: January 06, 2019, 09:10:22 AM »
That IP is located in Vietnam, and is listed on 14 different blocklists:
https://ipinfo.io/103.207.37.246
https://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a103.207.37.246&run=toolpage#

Were the connections blocked by the qpsmtpd rbl plugin?
No
Quote
If not, why not?
Apparently qpsmtpd-forkserver never got so far. The IP-address wasn't checked.
Quote
Are there any attacks on services other than smtp from the same IP?
No, I haven't been able to find any other attacks

......

Offline mmccarn

  • *
  • 2,350
Re: qpsmtpd-forkserver processes puts a high load on the server
« Reply #3 on: January 06, 2019, 04:21:49 PM »
No Apparently qpsmtpd-forkserver never got so far. The IP-address wasn't checked.No, I haven't been able to find any other attacks

you might want to review your RBL configuration, and consider enabling one of the services that lists this IP. If you already have one or more of the listing services enabled, you might want to adjust other settings somewhere. 

The 'naughty' plugin changes the dnsbl disconnect behavior in a way that may be causing you headaches.

I made some customizations to the helo plugin on my SME to identify misbehaving remote systems earlier in the connection (at the risk of causing problems for remote users): https://forums.contribs.org/index.php?topic=53223.0

Offline holck

  • *
  • 266
Re: qpsmtpd-forkserver processes puts a high load on the server
« Reply #4 on: January 07, 2019, 11:18:39 AM »
Thanks for the advice

I already use Spamhaus Zen, which catches a lot bad-behaving IP-adresses, including the IP-address in question. As I see it, qpsmtpd-forkserver gets stuck early in the connection - even before the IP-address is checked against the various black-lists.

I will look into the post you mention and consider modifying the helo plugin...

Thanks,
Jesper
......

Re: qpsmtpd-forkserver processes puts a high load on the server
« Reply #5 on: January 11, 2019, 10:50:31 PM »
Recently, I've had two incidents where qpsmtpd-forkserver puts a high load on my server.

It would be useful if you could use tcpdump to collect a packet capture of what that address is sending to port 25 which is causing qpsmtpd to use a lot of CPU. Perhaps there is something that can be fixed in qpsmtpd to recognise the bad behaviour and disconnect.

Quote
Searching for the qpsmtpd-forkserver processes shows that new processes keep turning up, connected to the same, remote IP address:

...

This just goes on and on...
Quote

So maybe it is not sending anything, but just opening lots of connections...
« Last Edit: January 11, 2019, 10:53:03 PM by CharlieBrady »