Koozali.org: home of the SME Server

Transfer speed, AOE, usbdisks

Offline sages

  • *
  • 182
  • +0/-0
    • http://www.sages.com.au
Transfer speed, AOE, usbdisks
« on: March 25, 2021, 05:16:29 AM »
I'm currently going through the process of upgrading my 9.2 box using a D525 motherboard with bios boot to a 10.x rc1(ish) box using a new (to me) J1800 motherboard and uefi boot.
Long story but basically I want to resuse the harddrives ie just change out the motherboard.
I don't have to go to uefi boot but why make it easy  :grin: although I'm not committed 100% at the moment.
I have backups (tested ok) to an OMV server for most of my 9.2 box. I do have some things that I don't backup regularly. Movies, music, etc. all up about ~800G worth of 'stuff'. Apart from the time and bandwidth having to download it again I don't love it enough to back it up regularly.
But I'd rather not. So currently I'm going down the path of following https://wiki.contribs.org/Backup_server_config and using that to xfer the noncritical data to my OMV box. It's a tad slow  :-( ~8-10MB/s ). My D525 box doesn't support USB3 so that's not an option.
Finally the question.  :-P Has anyone looked at ATA over ethernet for speeding up transfers? Following onto potential for backup to workstation? And the USB disks contrib integration in any way?
I'll give it a try once all of the updates/upgrades are sorted and report back but wondering if anyone else has gone down the ATA over Ethernet path and if so have they any observations to make?

(This may be better looked at with a VM on proxmox to see what happens)
Rod
...

Offline sages

  • *
  • 182
  • +0/-0
    • http://www.sages.com.au
Re: Transfer speed, AOE, usbdisks
« Reply #1 on: March 25, 2021, 07:20:11 AM »
Just to document this.
Pretty much followed this guide: https://www.howtoforge.com/tutorial/using-ata-over-ethernet-aoe-on-centos-7/

SME10rc1
aoe is available out of the box  but isn't loaded.
requires modprobe aoe
Had to locate an rpm that had aoetools in it (warning about centos 6 rpm's is noted).
(version change for rpm since above guide written)
rpm -i http://download.opensuse.org/repositories/system:/aoetools/CentOS_CentOS-6/x86_64/aoetools-36-1.4.x86_64.rpm
I setup aoe on my OMV box using one of the addon's from OMV extras. I had tested using a file for the block device and had to create and format as per guide above.

Results:
scp and/or tar over ssh between SME box and OMV on my setup is around 10MB/s.
Copy using mounted aoe drive and MC gives around 65MB/s. significantly faster.
Did observe several pauses at different times for approx 5-10sec. Might be due to cpu load on OMV box as concurrent copying occurring.

usbdisks contrib shows partial information regarding the aoe drive. No option to mount it nor device info. Maybe a future option if any interest.


 
...

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Transfer speed, AOE, usbdisks
« Reply #2 on: March 25, 2021, 01:05:40 PM »
Nice Wiki page.... :-)
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline sages

  • *
  • 182
  • +0/-0
    • http://www.sages.com.au
Re: Transfer speed, AOE, usbdisks
« Reply #3 on: March 29, 2021, 08:50:55 AM »
fwiw.
tested xfer rate between sme9.2 (intel d525) gb n/w to OMV (intel J1900).
CPU load ~50-60% on one cpu core. ie not to stressed.
Unfortunately xfer rate dropped after initial burst at ~65MB/s to longer term average for a 2.7G file to ~16MB/s
So I'm assuming that some form of buffering was occurring with the smaller file size I initially tested with.

Was pretty trivial to setup the connection and (according to the interweb) it's possible to take an ATA over ethernet drive and physically install it into a computer and read it direct.
Might be useful for someone with a system that doesn't have any spare HD connections available on a machine.

Anyway, documented for future reference.
...

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Transfer speed, AOE, usbdisks
« Reply #4 on: March 29, 2021, 09:27:10 AM »
Did you put it in the wiki?
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation