Topic: CLAMD choking on emails with attachments

[gbentley]

Hi All,

Have an ongoing issue that has been increasing in frequency. Once a week or so outgoing emails with attachments get stuck in users Outlook outboxes. A while ago a quick refreshclam would fix it. However it is now becoming pretty frequent.

Against better advice I increased the size of attachments to 25MB however most of the above issues are created by emails that are less than 10MB.

Here is refreshclam output just now;

Code: [Select]

[root@mail ~]# refreshclam
Current working dir is /var/clamav
Max retries == 6
ClamAV update process started at Fri Feb 28 11:59:39 2020
Using IPv6 aware code
Querying current.cvd.clamav.net
TTL: 973
Software version from DNS: 0.102.2
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.100.2 Recommended version: 0.102.2
DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
Retrieving http://db.local.clamav.net/main.cvd
Trying to download http://db.local.clamav.net/main.cvd (IP: 104.16.219.84)
Downloading main.cvd [100%]
Loading signatures from main.cvd
Properly loaded 4564902 signatures from new main.cvd
main.cvd updated (version: 59, sigs: 4564902, f-level: 60, builder: sigmgr)
Querying main.59.93.1.0.6810DB54.ping.clamav.net
Can't query main.59.93.1.0.6810DB54.ping.clamav.net
Retrieving http://db.local.clamav.net/daily.cvd
Trying to download http://db.local.clamav.net/daily.cvd (IP: 104.16.219.84)
Downloading daily.cvd [100%]
Loading signatures from daily.cvd
Properly loaded 2199661 signatures from new daily.cvd
daily.cvd updated (version: 25735, sigs: 2199661, f-level: 63, builder: raynman)
Querying daily.25735.93.1.0.6810DB54.ping.clamav.net
Can't query daily.25735.93.1.0.6810DB54.ping.clamav.net
Retrieving http://db.local.clamav.net/bytecode.cvd
Trying to download http://db.local.clamav.net/bytecode.cvd (IP: 104.16.219.84)
Downloading bytecode.cvd [100%]
Loading signatures from bytecode.cvd
Properly loaded 94 signatures from new bytecode.cvd
bytecode.cvd updated (version: 331, sigs: 94, f-level: 63, builder: anvilleg)
Querying bytecode.331.93.1.0.6810DB54.ping.clamav.net
Can't query bytecode.331.93.1.0.6810DB54.ping.clamav.net
Database updated (6764657 signatures) from db.local.clamav.net (IP: 104.16.219.84)
WARNING: Clamd was NOT notified: Can't connect to clamd through /var/clamav/clamd.socket: No such file or directory
[root@mail ~]#

I will investigate how to reset attachment db entry to defaults. I have also noticed that qpsmtpd log when the above happens almost always includes 'virus::clamdscan 902 unable to scan for viruses msg denied before qued' and 'virus::clamdscan cannot ping clamd server could not establish connection, tried Unix domain and TCP socket at /usr/share/perl5/vendor_perl/ClamAV/Client.pm line 471'

Thanks in advance of any help

[gbentley]

I don't know if this helps or not however from the troubleshooting on Clamav

Code: [Select]


[root@mail ClamAV]# host -t txt current.cvd.clamav.net
current.cvd.clamav.net descriptive text "0.102.2:59:25736:1582892940:1:63:49191:331"
[root@mail ClamAV]# dig @ns1.clamav.net db.us.big.clamac.net

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.68.rc1.el6_10.3 <<>> @ns1.clamav.net db.us.big.clamac.net
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 54016
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;db.us.big.clamac.net.          IN      A

;; Query time: 69 msec
;; SERVER: 193.28.86.61#53(193.28.86.61)
;; WHEN: Fri Feb 28 12:48:04 2020
;; MSG SIZE  rcvd: 38



[gbentley]

I've reset to default all of the config properties listed here;

https://wiki.contribs.org/SME_Server:Documentation:FAQ:Section04#Set_max_email_size

I don't think I have ever changed the below settings but the warning sounds alarming.

Assume the defaults are fine?

These attributes could result in the rejection of a compressed attachment on a SME server:

ArchiveMaxCompressionRatio (default 300)
MaxFiles (default 1500)
MaxRecursion (default

I am now running with mail scanning off as the warnings in my original post occur as soon as its enabled.

[janet]

gbentley

Show us the output of these:

config show qpsmtpd

config show php

config show qmail

config show clamd

config show clamav

config show spamassassin

[gbentley]

[root@mail ~]# config show qpsmtpd
qpsmtpd=service
    Bcc=disabled
    BccMode=cc
    BccUser=maillog
    DNSBL=enabled
    LogLevel=6
    MaxScannerSize=25000000
    RBLList=zen.spamhaus.org,bl.spamcop.net,multi.surbl.org,black.uribl.com,rhsbl.sorbs.net
    RHSBL=enabled
    RelayRequiresAuth=disabled
    SBLList=dbl.spamhaus.org,multi.surbl.org,black.uribl.com,rhsbl.sorbs.net
    TlsBeforeAuth=1
    UBLList=multi.surbl.org:8-16-64-128,black.uribl.com,rhsbl.sorbs.net
    URIBL=enabled
    access=public
    qplogsumm=disabled
    status=enabled
[root@mail ~]# config show php
php=service
    AllowUrlFopen=Off
    UploadMaxFilesize=10M
    status=enabled
[root@mail ~]# config show qmail
qmail=service
    MaxMessageSize=15000000
    status=enabled
[root@mail ~]# config show clamd
clamd=service
    MemLimit=1400000000
    status=enabled
[root@mail ~]# config show clamav
clamav=service
    ArchiveBlockEncrypted=no
    Checks=24
    DNSDatabaseInfo=current.cvd.clamav.net
    DatabaseMirror=db.local.clamav.net
    Debug=no
    DetectBrokenExecutables=no
    FilesystemScan=weekly
    FilesystemScanExclude=/proc,/sys,/usr/share,/var
    FilesystemScanFilesystems=/home/e-smith/files
    FilesystemScanReportTo=admin
    FilesystemScanUnofficialSigs=no
    Foreground=yes
    HTTPProxyPassword=
    HTTPProxyPort=
    HTTPProxyServer=
    HTTPProxyUsername=
    HeuristicScanPrecedence=yes
    IdleTimeout=60
    LeaveTemporaryFiles=no
    LogClean=no
    LogFileUnlock=yes
    LogTime=no
    LogVerbose=yes
    MaxAttempts=6
    MaxConnectionQueueLength=30
    MaxDirectoryRecursion=20
    MaxFileSize=15M
    MaxFiles=1500
    MaxRecursion=8
    MaxThreads=20
    Quarantine=enabled
    QuarantineDirectory=/var/spool/clamav/quarantine
    ReadTimeout=300
    ScanArchive=yes
    ScanHTML=yes
    ScanMail=yes
    ScanOLE2=yes
    ScanPE=yes
    ScanRAR=no
    SelfCheck=1800
    ShowProxySettings=no
    ShowUpdateSettings=no
    SignaturesUpdated=unknown
    UpdateNonOfficeHrs=disabled
    UpdateOfficeHrs=disabled
    UpdateWeekend=disabled
    status=enabled
[root@mail ~]# config show spamassassin
spamassassin=service
    DNSAvailable=yes
    MaxMessageSize=2000000
    MessageRetentionTime=30
    OkLanguages=all
    OkLocales=all
    RejectLevel=9
    ReportSafe=0
    Sensitivity=custom
    SkipRBLChecks=0
    SortSpam=enabled
    Subject=[SPAM]
    SubjectTag=enabled
    TagLevel=5
    UseBayes=0
    status=enabled
[root@mail ~]#

[Gary Douglas]

When you run signal-event email-update or signal-event clamav-update (and maybe refreshclam) the clamd.socket can take up to 3 minutes to re-establish, during which time email transactions fail with; virus::clamdscan 902 unable to scan for viruses.

you can look for  clamd.socket  is running in /var/clamav;   ll /var/clamav
srw-rw-rw- 1 clamav clamav         0 Feb 28 08:26 clamd.socket

Not sure this helps with your issue, but just to be aware this happens, sometimes I have had the clamd.socket not restart automatically, requires /etc/init.d/clamd start

[gbentley]

You are right Gary, there is quite a delay that I was misinterpreting.

Now I am back to square one because its obviously an intermittent issue and I guess I will have to wait until one of the users reports it again and go back over the logs etc

[gbentley]

In my tests there is also quite some delay in actually scanning / processing the email messages that have attachments.

I have heard users say that email with several drawings attached i.e. 6-8 Mb can sit in outbox for ages.

[Jean-Philippe Pialasse]

Thing with clamd is it needs to load the whole db in memory on every start.

As pointed it can take 3 minutes and sometimes way more. This depends on memory available, cpu and additional definition db you add to the load.

It can even prevent clamd to start if db is bigger than available memory or limit for memory in config db.


When this occurs the symptoms is that the smtp connexion will’ refuse the email and you need to send again latter.

Check for clamd log if there are any corrupted db alert or missing memory. In that case increase max memory.
It you use unofficial clamav db reevaluate if you need them. 
Also instead of reloading clamd multiple time in the day you can change freshclam behaviour to only update once during the night so clamd is up during the day.

[gbentley]

Check for clamd log if there are any corrupted db alert or missing memory. In that case increase max memory.

Cheers JP - is this the right config param?

config setprop clamd MemLimit

In which case I'll try;

db configuration setprop clamd MemLimit 1800000000
signal-event clamav-update

[gbentley]

I've been thinking, as we have desktop Anti-Virus, it may be better to use the ISP outgoing server in the users Outlook setup.

[janet]

gbentley

I feel you are sailing too close to the wind on some of those settings.
Increase ALL these (shown below) to say 50000000 (or 50M where appropriate) & see how you go for a while, you can adjust them down to say 30000000 (or 30M) after a while if you really want to limit message size (to something lower).
All parts of your system need to support the largest expected message size plus a considerable allowance for overheads etc.
Run the required signal-event commands after making changes.

config show qpsmtpd
    MaxScannerSize=25000000

config show php
    UploadMaxFilesize=10M

config show qmail
    MaxMessageSize=15000000

config show clamav
    MaxFileSize=15M

config show spamassassin
    MaxMessageSize=2000000

[gbentley]

Thanks Janet. My previous settings where increased from default to accommodate up to 25 Mb [which was against the general advice given in the forums]

This has mostly worked over the years but in recent months has been causing delays as described. If an email is sitting in the users outbox for more than a few minutes this creates a support call. Often the only way to 'clear it' has been to restart the server and restarted Outlook.

Anyway, this is happening more and more often [from once every few weeks to several times a week]

I increased the server [XEON 5110 1.6GHZ 2 Cores, 2 Threads] RAM to 8 MB RAM & Dual 240GB SSD's

Whilst this has improved performance generally it hasn't really gone any way to relieving the above symptoms.

I need to at least implement a workaround for now as I am 22 miles away from the office and remote desktop isn't something that is always 'agreeable' when the user is under pressure / deadlines etc

[mmccarn]

email with several drawings attached i.e. 6-8 Mb can sit in outbox for ages

Messages stuck in the users' outbox would be at the client<->qpsmtpd stage; I would expect problems to apper in /var/log/sqpsmtpd/*

When I was using SME for spam filtering at work the qpsmtpd logs only covered about 90 minutes by default; If you need more time, increase the qpsmtpd log retention by changing the number of log files:

Code: [Select]

config setprop qpsmtpd KeepLogFiles 30
config setprop sqpsmtpd KeepLogFiles 30
sv t qpsmtpd
sv t sqpsmtpd


[Gary Douglas]

I had a similar issue with outlook, by default I now set outlook server timeout to 3 minutes, under advanced settings.