Blivet & RAID10

[Jan Safranek]

Hi,

I noticed in F19 that blivet does not support RAID10 with 2 or 3
devices. What is the reason for this? Sure, RAID10 on 2 devices is equal
to RAID1. But if you add new device to RAID1, you get more mirrors,
while adding new device to RAID10 adds more space + speed.

I am really not system admin, is this valid use case or not?

Jan

[Jason L Tibbitts III]

> > > > > "JS" == Jan Safranek  writes:

JS> I am really not system admin, is this valid use case or not?

There is certainly a valid case for 3-drive RAID10; I have several
machines set up like that now, some under F18, where kickstart created
the array without issue:

md3 : active raid10 sda5[0] sdb5[1] sdc5[2]
      1284601856 blocks super 1.2 512K chunks 2 near-copies [3/3] [UUU]
      bitmap: 0/10 pages [0KB], 65536KB chunk

I do this because the hardware (Supermicro 2U Twin2) supports three
drives per node.

- J<

[David Lehman]

On Wed, 2013-07-03 at 10:34 +0200, Jan Safranek wrote:
> Hi,
>
> I noticed in F19 that blivet does not support RAID10 with 2 or 3
> devices. What is the reason for this? Sure, RAID10 on 2 devices is equal
> to RAID1. But if you add new device to RAID1, you get more mirrors,
> while adding new device to RAID10 adds more space + speed.
>
> I am really not system admin, is this valid use case or not?

I'm not a sysadmin either, nor am I a RAID expert, but my understanding
is that raid10 on <4 disks is actually RAID1E. Blivet doesn't support
RAID1E, regardless of what mdadm calls it.
> Jan
>
> Anaconda-devel-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/anaconda-devel-list

[Jan Safranek]

On 07/03/2013 05:08 PM, David Lehman wrote:
> On Wed, 2013-07-03 at 10:34 +0200, Jan Safranek wrote:
> > Hi,
> >
> > I noticed in F19 that blivet does not support RAID10 with 2 or 3
> > devices. What is the reason for this? Sure, RAID10 on 2 devices is equal
> > to RAID1. But if you add new device to RAID1, you get more mirrors,
> > while adding new device to RAID10 adds more space + speed.
> >
> > I am really not system admin, is this valid use case or not?
>
> I'm not a sysadmin either, nor am I a RAID expert, but my understanding
> is that raid10 on <4 disks is actually RAID1E. Blivet doesn't support
> RAID1E, regardless of what mdadm calls it.

You are right, with three drives layout of data on RAID10 = RAID1E. With
two drives, RAID10 = RAID1. But that's just coincidence, the data layout
is same, but MD header is different and will behave differently when new
drive is added.

In another reply, you have at least one user who uses RAID10 with 3 disks.

Jan

[David Lehman]

On Wed, 2013-07-03 at 17:31 +0200, Jan Safranek wrote:
> On 07/03/2013 05:08 PM, David Lehman wrote:
> > On Wed, 2013-07-03 at 10:34 +0200, Jan Safranek wrote:
> > > Hi,
> > >
> > > I noticed in F19 that blivet does not support RAID10 with 2 or 3
> > > devices. What is the reason for this? Sure, RAID10 on 2 devices is equal
> > > to RAID1. But if you add new device to RAID1, you get more mirrors,
> > > while adding new device to RAID10 adds more space + speed.
> > >
> > > I am really not system admin, is this valid use case or not?
> >
> > I'm not a sysadmin either, nor am I a RAID expert, but my understanding
> > is that raid10 on <4 disks is actually RAID1E. Blivet doesn't support
> > RAID1E, regardless of what mdadm calls it.
>
> You are right, with three drives layout of data on RAID10 = RAID1E. With
> two drives, RAID10 = RAID1. But that's just coincidence, the data layout
> is same, but MD header is different and will behave differently when new
> drive is added.

Blivet does not (and will not) aim to support every single configuration
allowed by, eg: mdadm. You can use some other means to create these
types of setups. If mdadm reports the array as usable, blivet will allow
you to use it. It just won't create such setups.
> In another reply, you have at least one user who uses RAID10 with 3 disks.

That user can create the array prior to installing the OS.
> Jan
>
> Anaconda-devel-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/anaconda-devel-list

[Jason L Tibbitts III]

> > > > > "DL" == David Lehman  writes:

DL> That user can create the array prior to installing the OS.

Boned again, I see.  Seems to be my lot in life.

- J<

[Adam Williamson]

On 2013-07-03 10:02, Jason L Tibbitts III wrote:
> > > > > > "DL" == David Lehman  writes:
>
> DL> That user can create the array prior to installing the OS.
>
> Boned again, I see.  Seems to be my lot in life.

I don't see how this is 'boned'. It is not practical to try and 
implement, in anaconda, a partitioning tool that can do every damn thing 
under the sun that the underlying tools are capable of. anaconda/blivet 
is already a *lot* more capable than most distro installers; at some 
point you just can't crowbar any more complexity in there without making 
it pretty much unusable.

It seems pretty reasonable to me to say 'look, there are some configs we 
just can't reasonably put in the UI, if you want to use them, pre-create 
them'. It's not a *life sentence*, or anything. It's not like it's 
particularly hard to pre-create your layout. Hell, some people prefer to 
do it that way.

[Jason L Tibbitts III]

> > > > > "AW" == Adam Williamson  writes:

AW> I don't see how this is 'boned'. It is not practical to try and
AW> implement, in anaconda, a partitioning tool that can do every damn
AW> thing under the sun that the underlying tools are capable
AW> of.

Well, it kind of was implemented, in F18 and before, as evidenced by the
fact that I've been installing (via kickstart) these machines this way
since F10.  Breaking this now seems capricious.  It's not as if the
procedure for creating a 3-drive RAID10 is any different than creating a
4-drive RAID10.

- J<

[David Lehman]

On Thu, 2013-07-04 at 00:20 -0500, Jason L Tibbitts III wrote:
> > > > > > "AW" == Adam Williamson  writes:
>
> AW> I don't see how this is 'boned'. It is not practical to try and
> AW> implement, in anaconda, a partitioning tool that can do every damn
> AW> thing under the sun that the underlying tools are capable
> AW> of.
>
> Well, it kind of was implemented, in F18 and before, as evidenced by the
> fact that I've been installing (via kickstart) these machines this way
> since F10.  Breaking this now seems capricious.  It's not as if the
> procedure for creating a 3-drive RAID10 is any different than creating a
> 4-drive RAID10.

There is more to what anaconda/blivet does than the procedure you'd use
from the command line. That's what makes them both useful.

As I said (or at least alluded to) before, the addition of support for
btrfs raid necessitated limiting our definition of the raid10 level so
that it would be supported by both backends (mdadm & btrfs).

A proper resolution to this matter is on my TODO list, but not at the
top of it. In the meantime, I will ask that you "pardon our progress"
and use one of the available workarounds to install to a 3-drive RAID10.

Thanks.

[Jan Safranek]

On 07/03/2013 09:59 PM, Adam Williamson wrote:
> On 2013-07-03 10:02, Jason L Tibbitts III wrote:
> > > > > > > "DL" == David Lehman  writes:
> >
> > DL> That user can create the array prior to installing the OS.
> >
> > Boned again, I see.  Seems to be my lot in life.
>
> I don't see how this is 'boned'. It is not practical to try and
> implement, in anaconda, a partitioning tool that can do every damn thing
> under the sun that the underlying tools are capable of. anaconda/blivet
> is already a *lot* more capable than most distro installers; at some
> point you just can't crowbar any more complexity in there without making
> it pretty much unusable.
>
> It seems pretty reasonable to me to say 'look, there are some configs we
> just can't reasonably put in the UI, if you want to use them, pre-create
> them'. It's not a *life sentence*, or anything. It's not like it's
> particularly hard to pre-create your layout. Hell, some people prefer to
> do it that way.

Blivet is not *only* installer. It's used also in OpenLMI, which aims to
administer storage stuff on running system. If Anaconda finds it too
confusing to offer RAID 10 on 2-3 drives in GUI, it's fine for me. But
blivet as generic storage administration library IMHO should allow this,
especially if it was working in the past.

Jan

[David Lehman]

On Wed, 2013-07-03 at 11:11 -0500, David Lehman wrote:
> On Wed, 2013-07-03 at 17:31 +0200, Jan Safranek wrote:
> > On 07/03/2013 05:08 PM, David Lehman wrote:
> > > On Wed, 2013-07-03 at 10:34 +0200, Jan Safranek wrote:
> > > > Hi,
> > > >
> > > > I noticed in F19 that blivet does not support RAID10 with 2 or 3
> > > > devices. What is the reason for this? Sure, RAID10 on 2 devices is equal
> > > > to RAID1. But if you add new device to RAID1, you get more mirrors,
> > > > while adding new device to RAID10 adds more space + speed.
> > > >
> > > > I am really not system admin, is this valid use case or not?
> > >
> > > I'm not a sysadmin either, nor am I a RAID expert, but my understanding
> > > is that raid10 on <4 disks is actually RAID1E. Blivet doesn't support
> > > RAID1E, regardless of what mdadm calls it.
> >
> > You are right, with three drives layout of data on RAID10 = RAID1E. With
> > two drives, RAID10 = RAID1. But that's just coincidence, the data layout
> > is same, but MD header is different and will behave differently when new
> > drive is added.
>
> Blivet does not (and will not) aim to support every single configuration
> allowed by, eg: mdadm. You can use some other means to create these
> types of setups. If mdadm reports the array as usable, blivet will allow
> you to use it. It just won't create such setups.

I suppose this warrants some explanation. The reason for this is that
all raid-like devices currently use the same code within blivet for
determining minimum members, &c for a given raid level. As a result, the
definitions have to be as standard as possible. At some point I may be
able to find time to make something more flexible that would allow
different definitions of a given raid level for different raid backends
(eg: btrfs). Once such a framework is in place I could potentially start
allowing things like mdadm raid10 with two or three members. Until then,
it isn't really feasible.

[Bryan J Smith]

On Wed, Jul 3, 2013 at 3:48 PM, David Lehman  wrote:
> I suppose this warrants some explanation. The reason for this is that
> all raid-like devices currently use the same code within blivet for
> determining minimum members, &c for a given raid level. As a result, the
> definitions have to be as standard as possible. At some point I may be
> able to find time to make something more flexible that would allow
> different definitions of a given raid level for different raid backends
> (eg: btrfs). Once such a framework is in place I could potentially start
> allowing things like mdadm raid10 with two or three members. Until then,
> it isn't really feasible.

This is 100% user (not maintainer) opinion here, but my "objective"
viewpoint is not to define things in what is possible, but to put them in
terms of "risk."

When it comes to any storage, my #1 concern when it comes to risk is
"recovery."  Not whether it will work, or whether the implementation (e.g.,
kernel DM/MD, mdadm, etc...) can make use of it, but can the storage be
recovered off-system, possibly by another solution.

So we have ...
 - SNIA DDF RAID-1E definitions
 - IBM-LSI (even infrequent Promise option) implementations
 - Linux mdadm implementations

So I would limit any "considerations" (i.e., just for consideration) where
they are common.  E.g., at a minimum I would require not just the fact that
mdadm offers a DDF metadata format option, but whether it implements a SNIA
compliant RAID-1E solution with it.  RAID-1E has the potential to confuse a
lot of sysadmins, whereas RAID-1 (or even 4-disc RAID-10) is fairly
straight-forward and understood.

But that's my 100% user opinion, not a maintainer viewpoint.  I'm more
concerned with people trying to "recover" things, and not always as they
should.  E.g., the operating system file system is lost, and maybe someone
is not aware it was Linux and try to discover what it is.  Rare, but I've
seen it happen.

--
Bryan J Smith - Professional, Technical Annoyance
b.j.smith at ieee.org - http://www.linkedin.com/in/bjsmith

[Bryan J Smith]

On Wed, Jul 3, 2013 at 11:08 AM, David Lehman  wrote:
> I'm not a sysadmin either, nor am I a RAID expert, but my understanding
> is that raid10 on <4 disks is actually RAID1E. Blivet doesn't support
> RAID1E, regardless of what mdadm calls it.

RAID-1E is one of those things that is kinda like EIDE.

EIDE was actually a[n] (unenforced? or little enforced?) trademark of
Western Digital used to describe certain set of register setup for
16/32-bit bus/data Programmed I/O (PIO) modes.  It then ended up being used
generically by most people, even though the ATA specifications created a
superset of EIDE which wasn't always 100% EIDE compatible either (let alone
added DMA).  Even EIDE was eventually used by the ATA Consortium itself to
describe it.

So as I understand it, IBM-LSI RAID-1E is more of the same.  RAID-1E on a
IBM or LSI card might be slightly different than the various modes in the
SNIA's Commmon RAID Disk Data Format (DDF) Specification.  The latter
really doesn't define "the standard" but takes a "technical position" on
layout (e.g., adjacent, off-set, etc...). [1]  As with most MD (or DM for
that matter), auto-detection with [Fake] RAID, one must take care to get
exactly, or not at all.

I like to call it "striped-mirrors," and explicitly differentiate from
"striping + mirroring," both of which is far more exacting of a statement.
> Blivet doesn't support RAID1E, regardless of what mdadm calls it.

Well, if there is one thing I've learned to understand about mdadm, it
doesn't try to call any RAID something that is not already in use.  When in
doubt, it defines formats without a name, and merely assigns a label for an
option.  This is not unlike most open source technology (e.g., Samba comes
to mind), it forces implementers and integrators to understand _more_
technical details that are far more _exacting_ than just a single name or
label.

Wikipedia has a good, quick intro of 'Non-Standard' "striped-mirrors"
between both mdadm [2] and RAID-1E [3].  Although I guess you could say
that any RAID-1E that follows DDF definition for RAID-1E is not exactly
'Non-Standard.'  The question is, what modes do and don't fit IBM-LSI
implementations?

-- bjs

[1] http://www.snia.org/tech_activities/standards/curr_standards/ddf
[2] http://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10
[3] en.wikipedia.org/wiki/Non-standard_RAID_levels#RAID_1E