Topic: Meltdown-Spectre mitigation perfomance impacts?

[William R H]

Since installing the latest kernel my nightly incremental backup takes 70 mins, prior to update it was 10 minutes. Weekly full backup has gone from 3H30 to 6H15.

My SME Server is running as a guest os in a Proxmox/pve host

My Thunderbird email client, running in a Windows 10 VM on the same host as SME Server, now hangs periodically for 10 to 30 seconds - presumably when doing fetches that coincide with other users causing server activity.

What experiences are others having? Has anyone overcome any of these problems? Would giving SME Server more memory help?

I have yet to do the fixes on the PVE install - 115 Debian updates awaiting installation - year end tax stuff is looming and we cannot take the time out to fix and risk messing things up.

[root@mailserver ~]# uname -a
Linux mailserver 2.6.32-696.18.7.el6.x86_64 #1 SMP Thu Jan 4 17:31:22 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@mailserver ~]# rpm -qa kernel* | sort
kernel-2.6.32-696.13.2.el6.x86_64
kernel-2.6.32-696.16.1.el6.x86_64
kernel-2.6.32-696.18.7.el6.x86_64
kernel-firmware-2.6.32-696.18.7.el6.noarch
kernel-headers-2.6.32-696.18.7.el6.x86_64

[Daniel B.]

I doubt such slowdowns are because of the meltdown/spectre mitigation patches. Most likely there's something else slowing things down

[William R H]

The thing is nothing else has changed.


Before installing the mitigation kernel - the timings below are very typical and have been like that for several months. Backing up to a USB3 disk drive.

Incremental 0H18
==================================
DAILY BACKUP TO WORKSTATION REPORT
==================================
Backup of mailserver.xxx.xxx started at Wed Jan  3 22:15:01 2018
Destination //localhost/media/smeserver/mailserver.xxx.xxx/set4
Basename inc-006-20180103221501
Starting the backup with a timeout of 10 hours
 --------------------------------------------
 1827 inode(s) saved
   including 44 hard link(s) treated
 0 inode(s) changed at the moment of the backup and could not be saved properly
 0 byte(s) have been wasted in the archive to resave changing files
 309309 inode(s) not saved (no inode/file change)
 0 inode(s) failed to be saved (filesystem error)
 301 inode(s) ignored (excluded by filters)
 500 inode(s) recorded as deleted from reference backup
 --------------------------------------------
 Total number of inode(s) considered: 311937
 --------------------------------------------
 EA saved for 0 inode(s)
 --------------------------------------------
Destination disk usage 2.6T, 59% full, 1.8T available
Backup successfully terminated at Wed Jan  3 22:33:29 2018

Full - every Thursday - 3H21
==================================
DAILY BACKUP TO WORKSTATION REPORT
==================================
Backup of mailserver.xxx.xxx started at Thu Jan  4 22:15:01 2018
Destination //localhost/media/smeserver/mailserver.xxx.xxx/set5
Basename full-20180104221501
Starting the backup with a timeout of 24 hours
 --------------------------------------------
 309502 inode(s) saved
   including 44 hard link(s) treated
 0 inode(s) changed at the moment of the backup and could not be saved properly
 0 byte(s) have been wasted in the archive to resave changing files
 0 inode(s) not saved (no inode/file change)
 0 inode(s) failed to be saved (filesystem error)
 301 inode(s) ignored (excluded by filters)
 0 inode(s) recorded as deleted from reference backup
 --------------------------------------------
 Total number of inode(s) considered: 309803
 --------------------------------------------
 EA saved for 84 inode(s)
 --------------------------------------------
Destination disk usage 3.0T, 69% full, 1.4T available
Backup successfully terminated at Fri Jan  5 01:36:28 2018

After installing the updates, no changes to our daily operations, business carries on as usual but

Incremental that evening 1H1
==================================
DAILY BACKUP TO WORKSTATION REPORT
==================================
Backup of mailserver.xxx.xxx started at Fri Jan  5 22:15:09 2018
Destination //localhost/media/smeserver/mailserver.xxx.xxx/set5
Basename inc-001-20180105221509
Starting the backup with a timeout of 10 hours
 --------------------------------------------
 898 inode(s) saved
   including 44 hard link(s) treated
 0 inode(s) changed at the moment of the backup and could not be saved properly
 0 byte(s) have been wasted in the archive to resave changing files
 307668 inode(s) not saved (no inode/file change)
 0 inode(s) failed to be saved (filesystem error)
 301 inode(s) ignored (excluded by filters)
 1473 inode(s) recorded as deleted from reference backup
 --------------------------------------------
 Total number of inode(s) considered: 310340
 --------------------------------------------
 EA saved for 0 inode(s)
 --------------------------------------------
Destination disk usage 3.0T, 69% full, 1.4T available
Backup successfully terminated at Fri Jan  5 23:16:39 2018

Full the next Thursday 6H15
==================================
DAILY BACKUP TO WORKSTATION REPORT
==================================
Backup of mailserver.xxx.xxx started at Thu Jan 11 22:15:02 2018
Destination //localhost/media/smeserver/mailserver.xxx.xxx/set6
Basename full-20180111221502
Starting the backup with a timeout of 24 hours
 --------------------------------------------
 309683 inode(s) saved
   including 45 hard link(s) treated
 0 inode(s) changed at the moment of the backup and could not be saved properly
 0 byte(s) have been wasted in the archive to resave changing files
 0 inode(s) not saved (no inode/file change)
 0 inode(s) failed to be saved (filesystem error)
 301 inode(s) ignored (excluded by filters)
 0 inode(s) recorded as deleted from reference backup
 --------------------------------------------
 Total number of inode(s) considered: 309984
 --------------------------------------------
 EA saved for 84 inode(s)
 --------------------------------------------
Destination disk usage 3.4T, 79% full, 942G available
Backup successfully terminated at Fri Jan 12 04:30:16 2018

[Daniel B.]

Is the kernel the only update applied ? What if you reboot on the previous one ? Is the perf going back to normal ?

[William R H]

The only updates applied were the ones offered to us on the morning of 5/1/18 - see below.

Sorry, I am not experienced enough with Linux to be able to reboot with an earlier one. I will see if our support person can help.

From: root@xxx.xxx
Date: Fri, 05 Jan 2018 04:44:04 +0000
To: admin-yum@xxx.xxx
Subject: Updates available for mailserver
User-Agent: Heirloom mailx 12.4 7/29/08
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

===
=== yum reports available updates:
===

kernel.x86_64                         2.6.32-696.18.7.el6                updates
kernel-firmware.noarch                2.6.32-696.18.7.el6                updates
kernel-headers.x86_64                 2.6.32-696.18.7.el6                updates

[Daniel B.]

Just reboot, and at the grub prompt, select the 2 kernel in the list

[Stefano]

be aware that kernel 696.18 has many issues in VM.. for example, in XEN clients, it breaks the whole VM (there are many references out there about it)

I'm waiting for the next kernel update

[William R H]

OK. I've switched it back to the previous kernel by editing grub.conf and changing the default to 1

Now let's see what the timings are tonight...

[root@mailserver ~]# uname -a
Linux mailserver 2.6.32-696.16.1.el6.x86_64 #1 SMP Wed Nov 15 16:51:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[mmccarn]

For what it's worth, I heard a similar tale from a colleague today - but in his case it was the backup of a virtual Windows 2008 R2 box hosted on Windows HyperV that slowed down from 1 hr to 7 hrs after applying the meltdown/spectre patches on the host system.

[William R H]

My patches were on the guest.

What are the "firmware" changes? To be honest I have no idea how that works in a guest process. Are they real CPU firmware changes? Are they persistent once loaded into the CPU? Will they then apply to all guests running under the host?

I haven't attempted to patch the host yet.

[Daniel B.]

No, microcode updates won't affect guests. Only the kernel update might. What are the results after switching back to the previous kernel ?

[William R H]

Right - this is a lot better with the previous kernel - 0H33

==================================
DAILY BACKUP TO WORKSTATION REPORT
==================================
Backup of mailserver.xxx.xxx started at Wed Jan 17 22:15:03 2018
Destination //localhost/media/smeserver/mailserver.xxx.xxx/set6
Basename inc-006-20180117221503
Starting the backup with a timeout of 10 hours
 --------------------------------------------
 2362 inode(s) saved
   including 44 hard link(s) treated
 0 inode(s) changed at the moment of the backup and could not be saved properly
 0 byte(s) have been wasted in the archive to resave changing files
 308977 inode(s) not saved (no inode/file change)
 0 inode(s) failed to be saved (filesystem error)
 301 inode(s) ignored (excluded by filters)
 188 inode(s) recorded as deleted from reference backup
 --------------------------------------------
 Total number of inode(s) considered: 311828
 --------------------------------------------
 EA saved for 1 inode(s)
 --------------------------------------------
Destination disk usage 3.4T, 80% full, 910G available
Backup successfully terminated at Wed Jan 17 22:48:40 2018

[ReetP]

You mentioned above about memory - can you tell us how much the SME guest has (as a matter of interest)

Use one of these:

cat /proc/meminfo

free -h

vmstat -s

[William R H]

All three...

[root@mailserver ~]# cat /proc/meminfo
MemTotal:        3958156 kB
MemFree:         2022864 kB
Buffers:          140300 kB
Cached:           707040 kB
SwapCached:        73808 kB
Active:          1176676 kB
Inactive:         577688 kB
Active(anon):     656320 kB
Inactive(anon):   250744 kB
Active(file):     520356 kB
Inactive(file):   326944 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4128764 kB
SwapFree:        3794868 kB
Dirty:               140 kB
Writeback:             0 kB
AnonPages:        858984 kB
Mapped:            60952 kB
Shmem:                20 kB
Slab:             107984 kB
SReclaimable:      75788 kB
SUnreclaim:        32196 kB
KernelStack:        4032 kB
PageTables:        21252 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6107840 kB
Committed_AS:    1554516 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       19804 kB
VmallocChunk:   34359702468 kB
HardwareCorrupted:     0 kB
AnonHugePages:    653312 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8040 kB
DirectMap2M:     4087808 kB
[root@mailserver ~]# free -h
             total       used       free     shared    buffers     cached
Mem:          3.8G       1.8G       1.9G        20K       137M       690M
-/+ buffers/cache:       1.0G       2.7G
Swap:         3.9G       326M       3.6G
[root@mailserver ~]# vmstat -s
      3958156  total memory
      1926768  used memory
      1165544  active memory
       579264  inactive memory
      2031388  free memory
       140360  buffer memory
       707104  swap cache
      4128764  total swap
       333892  used swap
      3794872  free swap
       187518 non-nice user cpu ticks
          335 nice user cpu ticks
        39111 system cpu ticks
     10913971 idle cpu ticks
        55681 IO-wait cpu ticks
            9 IRQ cpu ticks
          880 softirq cpu ticks
         1998 stolen cpu ticks
      8971777 pages paged in
      8408670 pages paged out
       347004 pages swapped in
       405944 pages swapped out
      7127972 interrupts
      5544948 CPU context switches
   1516213991 boot time
        20112 forks

[ReetP]

Not sure if this is related....

https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_update_woes/