Koozali.org: home of the SME Server

affa makes no hardlinks

Offline adlaw

  • 13
  • +0/-0
affa makes no hardlinks
« on: July 19, 2013, 12:13:01 PM »
Hi,

I'm backuping one SME with affa (smeserver-affa-2.0.0-rc5) on a NFS share on a Linux Mint station (Mint 13 Maya, based on ubuntu 12.04 Precise Pangolin). But the problem is that affa doen't makes hardlinks, and the backup disk is quicly full as everyday all files are copied. And as I have almost 100 G of data, the backup takes a very long time !

I'm surprised, because I have another SME where affa is backuping on FreeNAS (NFS share also), and it's working well...

Here is the affa job config :
Code: [Select]
# db affa show
AffaGlobalDisable=no
DefaultAffaConfig=default
    sendStatus=weekly
    status=enabled
baksme=job
    AutoUnmount=yes
    AutomountDevice=192.168.1.53:/mnt/BAKSME/affa
    AutomountOptions=-t nfs -o nolock
    AutomountPoint=/var/affadevice
    ConnectionCheckTimeout=120
    Debug=yes
    Description=SME backup on local station
    DiskSpaceWarn=strict
    EmailAddresses=admin@mydomain.local
    Exclude[0]=
    Include[0]=
    RPMCheck=no
    RetryAfter=600
    RetryAttempts=3
    RetryNotification=yes
    RootDir=/var/affadevice
    SMEServer=yes
    SambaShare=no
    TimeSchedule=2030
    Watchdog=yes
    chattyOnSuccess=0
    dailyKeep=1
    doneDaily=2013199
    doneMonthly=201304
    doneWeekly=201328
    doneYearly=2013
    localNice=0
    monthlyKeep=0
    postJobCommand=
    preJobCommand=
    remoteHostName=localhost
    remoteNice=0
    rsync--inplace=yes
    rsync--modify-window=3600
    rsyncCompress=no
    rsyncTimeout=900
    scheduledKeep=1
    sshPort=22
    status=enabled
    weeklyKeep=1
    yearlyKeep=0

Where could be the problem ?


Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: affa makes no hardlinks
« Reply #1 on: July 19, 2013, 12:18:11 PM »
maybe a different NFS settings on the target machine?

Offline adlaw

  • 13
  • +0/-0
Re: affa makes no hardlinks
« Reply #2 on: July 19, 2013, 12:29:04 PM »
Hi Stefano,

Thank you for this so fast reply  :-)

Yes, maybe some NFS settings. I'll re-read the man, but I think that I already did and I don't see which setting could prevent affa to make the harlinks... Any idea ?

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: affa makes no hardlinks
« Reply #3 on: July 19, 2013, 12:30:03 PM »
yes.. take a look in the affa's logs ;-)

Offline adlaw

  • 13
  • +0/-0
Re: affa makes no hardlinks
« Reply #4 on: July 19, 2013, 03:45:50 PM »
Ooops ! I thought it was a problem with NFS only, as I recently changed the backup station. Affa seems to make correctly the backups if we except the hardlinks... But in fact, in the logs I have failed chown on each file ! (why affa is not sending a mail to warn about this ?)

The newly created folder with job name is owned by nobody:nogroup, with 755 rights. But it has been created by affa, and the rights of the parent folder are 777...

How must I do so that affa creates the folder with the good rights and owner ? Is it a problem with affa settings, or NFS settings ?

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: affa makes no hardlinks
« Reply #5 on: July 19, 2013, 05:11:16 PM »
I would start from here

HTH

Offline adlaw

  • 13
  • +0/-0
Re: affa makes no hardlinks
« Reply #6 on: July 20, 2013, 02:56:22 PM »
Hi,

Thanks for your help, Stephano !

Seems that the solution was to add no_root_squash option in /etc/exports on the station. I also added the sync option. I followed this tuto.

Before that, anything created in /var/affadevice had (seen from SME) the nfsnobody:nfsnogroup owner, and chown was impossible (operation not permitted).

After, a new file or directory has root:root owner  :-)

Now, I'll see what will happen with affa. SHould I do something to the already created archives, or just let affa do and replace them when it will be time (I have one scheduled and one daily archive) ?

Offline adlaw

  • 13
  • +0/-0
Re: affa makes no hardlinks
« Reply #7 on: July 22, 2013, 09:09:35 AM »
Hi,

Now, affa runs for 25 minutes, and stops with an error. Here is a part of the log :
Code: [Select]
Sun Jul 21 20:30:07[21999]: Running rsync...
Sun Jul 21 20:30:07[21999]: Exec Cmd: /usr/bin/rsync --archive --stats --delete-during --ignore-errors --delete-excluded --relative --partial  --modify-window=3600 --inplace --timeout=900  --numeric-ids --rsync-path='/usr/bin/rsync'  --link-dest='/var/affadevice/svgsme/daily.0'    "/home/e-smith" "/etc/e-smith/templates-custom" "/etc/e-smith/templates-user-custom" "/etc/ssh" "/root" "/etc/sudoers" "/etc/passwd" "/etc/shadow" "/etc/group" "/etc/gshadow" "/etc/samba/secrets.tdb" "/etc/samba/smbpasswd" /var/affadevice/svgsme/scheduled.running/
Sun Jul 21 20:55:01[21999]: Exec Out: rsync: writefd_unbuffered failed to write 7 bytes to socket [sender]: Broken pipe (32)
Sun Jul 21 20:55:01[21999]: Exec Out: [sender] io timeout after 30 seconds -- exiting
Sun Jul 21 20:55:01[21999]: Exec Out: rsync error: timeout in data send/receive (code 30) at io.c(140) [sender=3.0.9]
Sun Jul 21 20:55:01[21999]: Exec Out: 30
Sun Jul 21 20:55:01[21999]: Exec Out: rsync: writefd_unbuffered failed to write 316 bytes to socket [generator]: Broken pipe (32)
Sun Jul 21 20:55:01[21999]: Exec Out: exitstatus=rsync error: received SIGUSR1 (code 19) at main.c(1298) [generator=3.0.9]
Sun Jul 21 20:55:01[21999]: signaling post-backup event on localhost

Looks like a problem with the connection, but
- When I make other tests, the SME and the station communicate very well,
- Affa was not stopping on an error before I made the change in /etc/exports, and nothing else changed in the network
- Affa stops always at the same time, 25 mn after the begining

Did I do something wrong or missed something ? What can I do to solve that ?

Thanks.

Offline adlaw

  • 13
  • +0/-0
Re: affa makes no hardlinks
« Reply #8 on: July 25, 2013, 03:04:03 PM »
Hi,

I tried to add also the no_subtree_check option in /etc/exports, but the result is the same... The timeout occurs after a variable time, between 20 to 65 minutes, and seems not to depend on what is to be backed up.

Any idea on how I could solve that, or at least what I could do or try to locate the origin of the problem ?

Thanks.

Offline Brave Dave

  • *
  • 185
  • +0/-0
Re: affa makes no hardlinks
« Reply #9 on: July 26, 2013, 01:10:03 AM »
I don't think you can create hard links on an NFS share
.:DB:.

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: affa makes no hardlinks
« Reply #10 on: July 26, 2013, 02:11:17 AM »
adlaw

Quote
The timeout occurs after a variable time, between 20 to 65 minutes, and seems not to depend on what is to be backed up.

One possibility.
Similar issues with copying large files across network connections has been attributed to "flaky" NICs. Try a good quality NIC eg Intel which are usually well supported by Linux. Disable your onboard NIC & try a Intel plug in card.
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline adlaw

  • 13
  • +0/-0
Re: affa makes no hardlinks
« Reply #11 on: July 26, 2013, 11:45:45 PM »
Hi,

I don't think you can create hard links on an NFS share
Yes, it is possible. It works with my FreeNAS !

Disable your onboard NIC & try a Intel plug in card.
Thanks for this advice. But how do you explain that the timeout happens only with the no_root_squash option in /etc/exports ? I never had any timeout before I put this option, and the timeout disapears when I remove it...

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: affa makes no hardlinks
« Reply #12 on: July 27, 2013, 12:24:21 AM »
adlaw

Quote
Thanks for this advice. But how do you explain that the timeout happens only with the no_root_squash option in /etc/exports ? I never had any timeout before I put this option, and the timeout disapears when I remove it...

I was not trying to explain or answer your problem directly. You asked:
Quote
Any idea on how I could solve that, or at least what I could do or try to locate the origin of the problem ?
so I suggested a troubleshooting approach, if you swap the NIC to a known good plug in card, then you may find whether the original problem is resolved or not, ie by process of elimination.
This is based on previous examples where large file copying operations stopped for no apparent reason, which was eventually traced to a "flaky" NIC.

Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline adlaw

  • 13
  • +0/-0
Re: affa makes no hardlinks
« Reply #13 on: July 29, 2013, 08:24:56 AM »
Hi,

I suggested a troubleshooting approach [...] by process of elimination.
Yes, thanks  :-)

But :
1 - Put and remove the no_root_squash option in /etc/exports makes the "broken pipe" appear/desappear,
2 - The installation is far, and I have only ssh access.

So, do you think that it's really interresting to go there just to change the NIC ? I'm afraid that I will go there just to confirm that the problem is not the NIC... I should prefer to have others tries that I can do through ssh before, and go there only if really necessary.

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: affa makes no hardlinks
« Reply #14 on: July 29, 2013, 10:09:22 AM »
Hi,
Yes, it is possible. It works with my FreeNAS !

freenas is NOT linux and so you may likely have different versions/implementation of NFS with different startup parameters..

in any case I would say that the issue is not with SME.. it works with FreeNAS..

you should search on mint/ubuntu side


Offline adlaw

  • 13
  • +0/-0
Re: affa makes no hardlinks
« Reply #15 on: July 30, 2013, 09:20:24 AM »
Hi,

freenas is NOT linux and so you may likely have different versions/implementation of NFS with different startup parameters..
Sure ! But probably, there is a way to do the same things on two POSIX systems... More especially hard links certainly work on NFS/Linux, as well as on LINUX and NFS/BSD...

in any case I would say that the issue is not with SME.. it works with FreeNAS..

you should search on mint/ubuntu side
It's true. I was just asking here because I know that some people are backuping their SME on a linux station, and was hoping they could help me, even if the issue is certainly not affa or SME.

Thanks for your help anyway  :-)

Offline Gary Douglas

  • *
  • 68
  • +1/-0
Re: affa makes no hardlinks
« Reply #16 on: January 13, 2016, 04:21:20 PM »
Using the same NFS storage server, when moving from AFFA ver2 on SME8 to AFFA ver3 on SME9, the archives began using the full GB per archive.
Backup jobs would complete in the usual time, but storage was diminishing quickly.
When mounting the NFS share, SME9 server defaulted to nfsvers=4, whereas SME8 was using NFSVER 3.
I mounted the NFS server from SME9 using nfsvers=3, which has solved the issue.

# vim /etc/fstab
192.168.100.2:/affa  /affa  nfs   rsize=32768,wsize=32768,nfsvers=3,timeo=600,intr,soft,bg  0 0

I have now migrated from SME8 / AFFAv2  to SME9 / AFFAv3, and retained all previous archives.
Using du -sh * from the SME9 (or SME8) shows the correct byte count per archive, confirming hardlinks are working.

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: affa makes no hardlinks
« Reply #17 on: January 13, 2016, 04:35:43 PM »
Using the same NFS storage server, when moving from AFFA ver2 on SME8 to AFFA ver3 on SME9, the archives began using the full GB per archive.
Backup jobs would complete in the usual time, but storage was diminishing quickly.
When mounting the NFS share, SME9 server defaulted to nfsvers=4, whereas SME8 was using NFSVER 3.
I mounted the NFS server from SME9 using nfsvers=3, which has solved the issue.

# vim /etc/fstab
192.168.100.2:/affa  /affa  nfs   rsize=32768,wsize=32768,nfsvers=3,timeo=600,intr,soft,bg  0 0

I have now migrated from SME8 / AFFAv2  to SME9 / AFFAv3, and retained all previous archives.
Using du -sh * from the SME9 (or SME8) shows the correct byte count per archive, confirming hardlinks are working.

Can that go in the Wiki, or possibly a a bug against the affa contrib ?
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: affa makes no hardlinks
« Reply #18 on: January 13, 2016, 05:31:45 PM »
Can that go in the Wiki, or possibly a a bug against the affa contrib ?

in bugzilla, definitely.. it's a bug (unexpected behaviour) IMO

@gary: could you please open it? thank you

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Re: affa makes no hardlinks
« Reply #19 on: January 14, 2016, 01:32:14 AM »
When mounting the NFS share, SME9 server defaulted to nfsvers=4, whereas SME8 was using NFSVER 3.
I mounted the NFS server from SME9 using nfsvers=3, which has solved the issue.

So it seems that you have found an NFS4 implementation issue in Linux Mint.

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: affa makes no hardlinks
« Reply #20 on: January 14, 2016, 10:30:36 AM »
Charlie, there's no evidence that Gary Douglas is storing his data on a mint nfs storage, AFAICT

in any case, he said:

Quote
Using the same NFS storage server, when moving from AFFA ver2 on SME8 to AFFA ver3 on SME9, the archives began using the full GB per archive.

that means, IIUC, that affa3 changed its behaviour, giving unexpected results

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: affa makes no hardlinks
« Reply #21 on: January 14, 2016, 12:26:42 PM »
Stefano

Quote
Charlie, there's no evidence that Gary Douglas is storing his data on a mint nfs storage....

I would say that Charlie got confused with the OP adlaw, who was using Linux Mint
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline Gary Douglas

  • *
  • 68
  • +1/-0
Re: affa makes no hardlinks
« Reply #22 on: January 14, 2016, 12:34:07 PM »
Stefano, I filed a bug with the following detail, per your request, but it is not a bug.
This thread seemed to have gone unanswered, thought this may help others using AFFA to NFS storage.
I have completed my cutover today, final gotcha was excluding directories with spaces. Seems it requires a *  instead of a space.

i.e;  Exclude=/home/e-smith/files/users/rob/home/My*Documents/My*Music

-------------------------------------------------------------------------------------------
Bug 9186

Changing from SME8 AFFAv2 to SME9 AFFAv3 using the same NFS storage and taking over the AFFAv2 archives, caused all new archives to become the full archive size. i.e. hardlinks were not working.

I have run AFFAv2 on SME7/8 to NFS shared storage for many years without issues, originally from a SME7/8 server to FreeNAS. Since a few years ago my NFS Server is now a Proxmox/Debian with additional HP410/RAID5 storage shared using debian nfs-kernel-server. SME8/AFFAv2 and SME9/AFFAv3 are virtual on another proxmox box.

By default SME8 attaches to the NFS share using nfsvers=3 and hardlinks have never been a problem. SME9 by default connects to the same NFS share using nfsvers=4. The issue of hardlinks failing appears to be when using nfsvers=4. By setting SME9 to use nfsvers=3, hardlinks are maintained on the AFFA archives.

There is no bug with SME or AFFA (nor using NFS if configured correctly), bug only raised on request of Stefano.

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: affa makes no hardlinks
« Reply #23 on: January 14, 2016, 01:19:26 PM »
Gary Douglas

I think this would have worked

Exclude="/home/e-smith/files/users/rob/home/My Documents/My Music"

(or with single inverted commas)

Exclude='/home/e-smith/files/users/rob/home/My Documents/My Music'
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: affa makes no hardlinks
« Reply #24 on: January 14, 2016, 03:29:22 PM »
Stefano, I filed a bug with the following detail, per your request, but it is not a bug.
-------------------------------------------------------------------------------------------
Bug 9186

Changing from SME8 AFFAv2 to SME9 AFFAv3 using the same NFS storage and taking over the AFFAv2 archives, caused all new archives to become the full archive size. i.e. hardlinks were not working.

I have run AFFAv2 on SME7/8 to NFS shared storage for many years without issues, originally from a SME7/8 server to FreeNAS. Since a few years ago my NFS Server is now a Proxmox/Debian with additional HP410/RAID5 storage shared using debian nfs-kernel-server. SME8/AFFAv2 and SME9/AFFAv3 are virtual on another proxmox box.

By default SME8 attaches to the NFS share using nfsvers=3 and hardlinks have never been a problem. SME9 by default connects to the same NFS share using nfsvers=4. The issue of hardlinks failing appears to be when using nfsvers=4. By setting SME9 to use nfsvers=3, hardlinks are maintained on the AFFA archives.

There is no bug with SME or AFFA (nor using NFS if configured correctly), bug only raised on request of Stefano.

one of the main feature of Affa is the use of hardlinks.. maybe is not a "real" bug but just a documentation one
anyway,  since the main rule here is "if something doesn't work as expected out of the box is likely a bug", a bug in bugzilla is needed

than you for your time

Offline Gary Douglas

  • *
  • 68
  • +1/-0
Re: affa makes no hardlinks
« Reply #25 on: January 14, 2016, 04:05:58 PM »
Janet, I have tried

Exclude="/home/e-smith/files/users/rob/home/My Documents/My Music"
  and
Exclude='/home/e-smith/files/users/rob/home/My Documents/My Music'

  but both started fetching files + folders from /My Music

AFFAv2 worked with DB command, and stored Exclusions correctly if you said;  db affa setprop htsvr Exclude[0] /home/e-smith/files/users/rob/home/"My Documents"/"My Music"
 
  but that fails also on my AFFAv3.

I can confirm Exclude=/home/e-smith/files/users/rob/home/My*Documents/My*Music  works for Excluding folders with spaces


guest22

Re: affa makes no hardlinks
« Reply #26 on: January 14, 2016, 06:02:54 PM »
spaces in paths are never a good thing....

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: affa makes no hardlinks
« Reply #27 on: January 14, 2016, 06:46:21 PM »
spaces in paths are never a good thing....

So sayeth The Oracle  :lol: :lol:

Beers are on you mate.... FOSDEM here we come !
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

guest22

Re: affa makes no hardlinks
« Reply #28 on: January 14, 2016, 11:04:40 PM »
AFFAv2 worked with DB command, and stored Exclusions correctly if you said;  db affa setprop htsvr Exclude[0] /home/e-smith/files/users/rob/home/"My Documents"/"My Music"
 
  but that fails also on my AFFAv3.

I can confirm Exclude=/home/e-smith/files/users/rob/home/My*Documents/My*Music  works for Excluding folders with spaces


Have you filed a bug so the AFFA developers can take a look?