Koozali.org: home of the SME Server

RAID possibilities - Linux Noob

Raulm

RAID possibilities - Linux Noob
« on: July 01, 2006, 05:38:17 AM »
Hi all

Completely new to Linux and trying to work out whether I should stick with SME or use something like Ubuntu/Centos server versions.

I really like the SME server model but at this point it's coming down to the ease of administering/maintaining hardware RAID arrays.

I wanted to install the OS on a single drive and use the RAID array as the file store but despite reading the manuals and lurking in the forums, I can't work out how to achieve this.

I have reached a tentative conclusion that I might be able to do it from root but alas my knowledge of Linux really doesn't extend beyond that.

Maybe my thinking about the disk/RAID approach is wrong for Linux/SME but if anybody has some advice on this, it would be greatly appreciated.

Thanks

Offline raem

  • *
  • 3,972
  • +4/-0
Re: RAID possibilities - Linux Noob
« Reply #1 on: July 01, 2006, 12:19:29 PM »
Raulm

> I wanted to install the OS on a single drive and use the RAID array as
> the file store but despite reading the manuals and lurking....

I don't think you have lurked or searched enough yet !
This is a fairly regular request.

The idea is to install the OS to the one disk only, then physically add in your (I assume) hardware RAID card and 2 additional drives, then mount the second drive (ie the hardware RAID array which is effectively seen by sme as one drive)

I don't know if the following is still applicable to sme7.
See FAQ's as they have a lot of useful tips,
from the sme6 FAQ
http://no.longer.valid/phpwiki/index.php/AdministrationFAQ#x32.HD
which has a link to this HOWTO
http://no.longer.valid/phpwiki/index.php/extra%20hard%20disk

See also the sme 7 Features, linked from the News page, as it explains how multiple drives are configured by default at OS install time depending on how many drives you have installed.
http://no.longer.valid/phpwiki/index.php/SME%207%20Features
...

Raulm

RAID possibilities - Linux Noob
« Reply #2 on: July 01, 2006, 11:43:24 PM »
Thanks RayMitchell - I hadn't seen the last link you posted before, and it looks exactly what I needed.

Now...the big question is...do I understand it    :hammer:

Thanks again.

Offline raem

  • *
  • 3,972
  • +4/-0
RAID possibilities - Linux Noob
« Reply #3 on: July 02, 2006, 04:07:27 AM »
Raulm

Don't conveniently overlook the requirement for sme 7, that ALL drives must be the same size for the "automatic" drive/RAID configuration to work.
...

Raulm

RAID possibilities - Linux Noob
« Reply #4 on: July 02, 2006, 08:58:42 AM »
Thanks for your replys Ray - much appreciated.

I guess my major issues relate to how RAID is managed - software vs hardware RAID.  I just don't understand enough about the implications of SW vs HW RAID and how SME handles it all and the potential impacts on data.

Based on the SME v7 final release and install I'm doing now, I get the impression that if I pug in my HW RAID array it will be have SW RAID layed over it - I'm not sure I want that from a performance or behavour perspective but I'm not clear at all, on any of it.  :-?

I know SME is intended to make it easy but I seem to want more control over the configuration than it allows for.

Anyways, I shall have a play and see what happens   :-D

Offline raem

  • *
  • 3,972
  • +4/-0
RAID possibilities - Linux Noob
« Reply #5 on: July 02, 2006, 10:02:50 AM »
Raulm

>  I just don't understand enough about the implications of SW vs HW RAID and how SME handles it all...

I think you are confused about when to use what.
It sounds like you do have a hardware RAID card, is that correct ?

In a simple installation with a hardware RAID card and 2 drives, you would select "single drive" during the install process, you would NOT select software RAID1.
The hardware card provides the RAID1 functionality and writes data appropriately to each drive. The sme server just sees the hardware card as if it is one drive.

If you did not have a hardware RAID card, then you would install the two drives on the motherboard drive connections, one as Primary master, the other as Secondary master. This gives better failure redundancy in the event a drive controller channel fails.
When you install the sme OS, then you DO select software RAID1.
The sme server OS will write to each drive appropriately in a mirror RAID1 configuration ie with identical data on each drive.

So do not mix up the usage of software RAID and hardware RAID, select one or the other, ie in sme parlance, software RAID or single drive

The sme server will configure the type of RAID array as per FAQ specification/manual depending how many drives you have installed, where they ALL must be the same size.

If you want to mix drive sizes and install the OS to the first IDE drive and the data etc to the hardware RAID card drives, then you will have to do this manually.
Personally I think you are unnecessarily complicating the matter.
Just install the 2 drives either as software RAID1 (which works very good) or hardware RAID using your RAID card, and install the OS fully to the array. All data will live on the array in that case.

Future upgrades will also be automatic too. If you have a unusual & not automatically supported drive setup, then you will need to remove drives when you do a OS upgrade & reconnect them afterwards etc.

I have sme servers running software RAID1 (2 disks) for 6 years now, without problems (apart from one drive failure which did not stop operation). The good part of software RAID1 is that you can power down, remove a drive (eg via removable caddie), insert a new one, power up, rebuild the array, and the server is offline for only a few minutes. It's fully functional too, as soon as the power is back up even while it is rebuilding the array. That way you have a fully configured backup available on the removed software RAID1 drive, the removed drive will function as a server in standalone broken array mode in an emergency, without needing to get a replacement RAID card (as you are using software RAID1).

If you want to persist with mixed drives and different OS & data locations, then install the OS to a single drive only first (remove all other drives first). Then add the hardware array or second single drive and add it to the system using the add extra drive howto. The hardware array is treated as a single drive by the system.

Hope that makes it more understandable.
...

Raulm

RAID possibilities - Linux Noob
« Reply #6 on: July 02, 2006, 11:23:31 AM »
Thanks Ray, your post has clarified things greatly for me.

I do indeed have a 3ware 7006-2 Hardware RAID card.  Works well so far with Linux - no recompile necessary.

My initial intention with the SME server was to create a home file server out of spare bits and pieces, which included a number of hdd's all of different sizes. I had intended to dedicate a disk to each user (3) with the HW array for backup.

Perhaps this is over complicating things but I do want to use these hdd's if I can.

Clearly I have some learning to do where SME/Linux/RAID/hdd management is concerned   :hammer:

Thanks again for your assistance, its much appreciated.

Offline raem

  • *
  • 3,972
  • +4/-0
RAID possibilities - Linux Noob
« Reply #7 on: July 02, 2006, 04:35:53 PM »
Raulm

> ...which included a number of hdd's all of different sizes.

> I had intended to dedicate a disk to each user (3)

You can set quotas in sme server, if disk space limiting per user is what you really want to achieve.


I read the release notes for sme7.0 final, which seem to indicate to me that you can put 3 disks in and have the system automatically configured.
A degraded RAID1 array on the first disk and a RAD array out of the other two. They all need to be the same size though.


Automated RAID installation

NOTE: All disks must be the same size for SME Server 7.0

- Single disk provides one-way RAID1
- Console screen allows simple addition of second disk to provide mirroring
- Two disks are configured as a RAID1 mirror
- Three to five disks are configured as RAID5 (/boot partition is RAID1)
- Six or more disks are configured as RAID6 (/boot partition is RAID1)
...

Raulm

RAID possibilities - Linux Noob
« Reply #8 on: July 02, 2006, 10:31:38 PM »
Hi Ray

Its that hdd same size requirement that is probably my main issue as all the drives I have are different sizes apart from the existing HW array.

I actually have 8 drives, IDE and SATA, all diff sizes, that I want re-use.  This really is my prime motivation for approaching it in the way that I am.

It would appear to me that at this stage, that if I want to use different size disks I will have to use a baseline distro that allows me to mount the drives individually. I'm also looking at the JBOD options, with the HW array as the backup.

Cheers

Offline raem

  • *
  • 3,972
  • +4/-0
RAID possibilities - Linux Noob
« Reply #9 on: July 03, 2006, 04:20:25 AM »
Raulm

> I actually have 8 drives, IDE and SATA, all diff sizes, that I want
> re-use.  This really is my prime motivation....
>...if I want to use different size disks I will have to use a baseline distro > that allows me to mount the drives individually.

sme 7 server is really a configuration of modules running on a standard  Linux distribution called Centos 4.3.
You can configure most Linux distributions to do anything you want, as long as you know how, Centos included.

I suggest you read up on how to install & configure multiple disks on Centos 4.3 and you'll find your answer, and still keep the benefits of using sme server.
Also look at
man mdadm
man fstab
and read that howto I mentioned earlier for pointers.

PS Using 8 drives just introduces 8 times the chance of a drive failing.
...

Raulm

RAID possibilities - Linux Noob
« Reply #10 on: July 03, 2006, 09:09:58 AM »
Hi Ray

I will have a look at the Centos documentation and see how I go.

In the meantime, thanks for your help so far.

Cheers

Offline jonroberts

  • ***
  • 111
  • +0/-0
    • http://www.westcountrybusiness.com
RAID possibilities - Linux Noob
« Reply #11 on: July 19, 2006, 10:54:18 AM »
Ray - Thanks for the clear explaination of RAID options - very useful.  So if Raulm will forgive me for hijacking his post, I'd just like to clarify one further question.

Almost all new motherboards now offer on-board SATA RAID - should this always be ignored in favour of the SME software RAID?

You make the point that with software RAID, the server is only down for the time it takes to replace the drive, whereas the hardware RAID may well want to rebuild the array before rebooting the server (Good point & one I'd not considered).

What about if the motherboard fails & the drives need to be moved to another server?  I guess the on-board RAID would be a problem (unless identical RAID chipset in use) but would the software array work in this case?

Finally is performance any consideration?  Again, with dedicated RAID cards this may be the case, but what about on-board RAID, how would this compare to SME Software RAID?

I'm just about to upgrade to 7.0 & then will think about upgrading clients too & so would like to be sure of the options.

Cheers
......

Offline raem

  • *
  • 3,972
  • +4/-0
RAID possibilities - Linux Noob
« Reply #12 on: July 19, 2006, 12:40:53 PM »
jonroberts

> Almost all new motherboards now offer on-board SATA RAID - should
> this always be ignored in favour of the SME software RAID?

First point is that the mb/chips/drivers need to be compatible with CentOS.
If the hardware is compatible, then it should be OK to use.
If you choose to use hardware RAID of any description (on board or separate cards) then you always select "single drive" installation when installing sme OS. You do not select software RAID1.
sme just sees the hardware RAID as if it were one drive. The hardware takes care of mirroring/striping to suit the type of RAID you configure the hardware for.
Personally I would say yes to ignore any on board hardware RAID, and just use software RAID1.

> You make the point that with software RAID, the server is only down for > the time it takes to replace the drive, whereas the hardware RAID may > well want to rebuild the array before rebooting the server (Good point & > one I'd not considered).

I have only used cheaper RAID cards in Windows, and they did require the drive to be replaced and the array to be fully rebuilt before the system would start the OS. That's a few hours wait for modern size drives eg 80Gb plus.
I'm not sure how this works in Linux, but as it is a hardware function, then I assume you have to wait also.
I think more expensive cards work diffently though, perhaps someone else who has used them could comment

Software RAID1 on sme rebuilds the array while you are using the server, without interruption to data. It's a bit slower during that time depending on your system.

> What about if the motherboard fails & the drives need to be moved to
> another server?  I guess the on-board RAID would be a problem (unless > identical RAID chipset in use) but would the software array work in this > case?

I don't have that experience to quote for sure, but with hardware RAID the drives are definitely dependant on the same hardware being available.
If a plugin board or motherboard fails, then I imagine you need an identical spare, which may not be so easy to get in a hurry.
There's a good reason to have a spare board !
In my Windows server (I have to use that due to some apps requiring Windows), I have a hot connected spare 3rd drive, to speed up the process of getting back on air in the event of a drive failure, but I still have to wait a few hours for the array to rebuild, even though I don't have to open the box.

By comparison if you use software RAID1, then the drives just plug into the normal hard drive connections on the motherboard. They are not dependant on the exact hardware make & model, they just need a Primary & Secondary channel connection on any motherboard, better to use similar processor types though.

I routinely swap a drive out of a software RAID1 sme server, and that drive functions perfectly well as a standalone server (in RAID1 degraded mode) if I plug it into the same channel on another computer elsewhere. A drive removed from a P4 even runs on a Celeron 333, with a few module errors here and there.
That's not ideal of course, just an example to show it does work on vastly different equipment.
That is a part of my backup strategy, as it neatly backs up all installed apps etc.


> Finally is performance any consideration?  Again, with dedicated RAID
> cards this may be the case, but what about on-board RAID, how would > this compare to SME Software RAID?

Again I don't have direct experince to quote you here, but I'm happy to belive Charlie Brady & his recommendations.
Software RAID1 is as capable as hardware RAID.
If you think about it, they are both software RAID ultimately, it's just a software program telling the bits where to write compared to a dedicated chip telling the bits where to write, and doesn't the dedicated chip use software anyway !

Charlie has posted numerous times in these forums various links to performance tests of hardware vs software RAID, which suggest there is little if no difference. Some posters here seem to disagree.
Hardware RAID via onboard the motherboard or via plugin card, I don't see that there could be any significant difference between them although you have an external bus to contend with, so therefore software RAID should still stand up well in comparison.
Perhaps a more knowledgeable user can also comment on performance.

I think that software RAID has a number of pluses when considering maintenance & simplicity & is cheaper to implement if you don't need to buy a card. It seems that possible performance differences would be a non issue for most of us, and any claims otherwise are likely to be incorrect.
...

Offline jonroberts

  • ***
  • 111
  • +0/-0
    • http://www.westcountrybusiness.com
RAID possibilities - Linux Noob
« Reply #13 on: July 19, 2006, 11:55:09 PM »
Ray,

Thanks for the comments - very interesting.

I tend to agree with everything you've said here & feel that software RAID gives the best recovery options.  If the performance issues are close enough to be debatable, then recovery options win for me every time!

I guess performance of software RAID will depend on the power of the server, whereas hardware RAID (with a dedicated card) should benefit from having some dedicated power (of course that will depend on the card).  

Just speculating here, but I'd guess on-board RAID may well also depend on getting a share of the onboard processor / resources.  If so, then its really only the equivalent of the software RAID.

Its academic really, but I'd be interested if anyone out there knows for sure.  In the mean time, it'll be software RAID for me.
......

Offline p-jones

  • *
  • 594
  • +0/-0
RAID possibilities - Linux Noob
« Reply #14 on: July 20, 2006, 12:56:10 AM »
What a great discussion thread. I have a couple of queries to add to this which Ray may or may not be able to answer staight away. My original plan as to just play till I got all the answers

I have just built a new V7 server. The Mobo has TWO SME compatible Sata Raid chips for a total of 4 SATA Chanels. I have 3 identical SATA Drives

Firstly, the on board raid only supports mode 1 & 2. By ignoring this I was able to achieve via Software Raid mode 5 and this really honks along. If I loose any one of these three drives do I still hav full redundancy ?

Secondly, If I were to add a fourth drive and create two stripped raids via hardware, it would seem feasible then to create a mirrored software raid from each pair - correct ? Any pitfalls here (other than perhaps performance). this is not really a true raid 5 as I understand it but it would seem to offer the best of both worlds with both stripping performance and mirrored redundancy.
...