Koozali.org: home of the SME Server

SME8.0: Best Practice for Install on disks/arrays that are larger than 2TiB

Offline stabilys

  • *
  • 76
  • +0/-0
  • Ah din't do it!!
    • Stabilys Ltd
Hi all:

I have read http://forums.contribs.org/index.php/topic,48093.0.html and many other comments around this issue (see references at end and below).

I need to build a large array of 6 or 12 TiB on SME 8 (4x3TiB) onto which I need to move a retiring SME7 box

I am trying to draft a document for best practice while remaining as canonical as possible in SME terms, which I will make available once it becomes useful.

What I have done that works:

*NOTE* this is on new HP hardware of the G6 series with p410i Hardware RAID with BBU. However the best practice should apply to software RAID as well...

- I am unable to get SME 8 to recognise the >3TB available space at all in any way, it just produces the GPT error mentioned elsewhere. I don't think I can use the workround mentioned by Shad Lords in the first reference below.

- I have therefore added 2x250GiB drives in RAID1 as a boot and system array and installed SME8 onto that

- I have then created a 6TiB RAID6 array using 4x3TiB SATA disks (hence the RAID6 - SATA has a higher error and failure rate)

- I have then effectively followed the AddExtraHardDIsk instructions (howto AddExtraHardDisk: http://wiki.contribs.org/AddExtraHardDisk) with some variations, see below

- I have created an LVM pv, a vg, and a logical volume directly on the raw array

- This has been mounted as in "Alternative 1: Use /home/e-smith/files to mount the new disk"

Rationale: all user and ibay data on the large volume

- This gave arise to some quota errors due to the lack of quota db on that filesystem, I rebuilt it but am not sure I did it correctly.

- All of this has worked adequately. My concern is its long-term stability.

Some questions:

1. I am not used to working directly with LVM and no underlying partition, but even if I created a large partition using (say) parted or gfdisk then the GPT issue would still arise - I do not see an easy way to work around this in SME 8/CentOS5. Is there one?

2. I am not very happy about not having a partition as many data recovery tools rely on having such a partition. I am therefore being very careful to back up all the filesystem meta/creation data (location of superblocks etc) and the LVM recovery data from /etc/...

Is there anything else I should do?

3. I have taken note of the caveats concerning LVM at http://serverfault.com/questions/279571/lvm-dangers-and-caveats

The server in question does have a BBU on the RAID controller and I am backing up all data I can off the LVM, with nightly backups of the whole system.

But would this LVM on bare array approach work with a >2TiB array built with software RAID? Are there more risks? (the only time I ran into an unrecoverable data issue on SME was with a mixed failure of LVM on SW RAID!)

4. I still don't understand LVM and disk data allocation. Do I need to worry about data alignment on the array, and how best should I control it if I do?

5. How should I have rebuilt the quota dbs etc?

6. Finally, building a server in this way of course means that the bootup restore available on first boot will not be available as the large array will not be mounted under /home... at that time. I therefore intend to attempt to use the reset first boot option method mentioned here: http://wiki.contribs.org/Backup_server_config and then continue with a restore from backup; alternatively I might pull a disk and do a restore from disk (documented elsewhere).

Are there any reasons that this should not work in the circumstances I have laid out above?

---

Thanks for any input, I will try and test any suggestions and draft up a working document for further comment in due course (aka when I get a minute!). Sorry for the long post  :-)

MeJ

References:

http://bugs.contribs.org/show_bug.cgi?id=1963
http://forums.contribs.org/index.php?topic=34031.0
http://bugs.contribs.org/show_bug.cgi?id=6799
http://richardjh.org/blog/install-centos-onto-large-partitions-using-gpt-disk-layout/
« Last Edit: October 04, 2012, 06:31:17 PM by stabilys »
This, too, will pass ;)