Hi!
I tried to set up a system using four 3TB disks as a raid6. The disks are new, no old partitions laying around. I used a USB stick as install media (Fedora 20 x86_64 Gnome Live).
I chose manual partitioning to get everything as I wanted. When this was ready I got the error message that I also needed a BIOS boot partition (I gather that this due to being rather large disks). I tried to create one of those, but it seemed to only be created on one of the discs in the raid array, I wanted it created on all disks, just as the raid partitions. This to be able to boot from any of the disks in the array in case of a failure. As it is now, if the main disk dies, I can not start the system at all.
Actually I ended up having three BIOS boot partition on the first disk, being 2, 1, and 1 MB large. Which is quite annoying. But that's another problem...
Before submitting a bug report/RFE on this. Is there anyone out there that have done this, from anaconda? I.e. creating raid partitions and BIOS boot partitions on ALL disks? If so, how did you do it?
Lars
On 08/28/2014 10:48 AM, Lars E. Pettersson issued this missive:
Hi!
I tried to set up a system using four 3TB disks as a raid6. The disks are new, no old partitions laying around. I used a USB stick as install media (Fedora 20 x86_64 Gnome Live).
I chose manual partitioning to get everything as I wanted. When this was ready I got the error message that I also needed a BIOS boot partition (I gather that this due to being rather large disks). I tried to create one of those, but it seemed to only be created on one of the discs in the raid array, I wanted it created on all disks, just as the raid partitions. This to be able to boot from any of the disks in the array in case of a failure. As it is now, if the main disk dies, I can not start the system at all.
Actually I ended up having three BIOS boot partition on the first disk, being 2, 1, and 1 MB large. Which is quite annoying. But that's another problem...
Before submitting a bug report/RFE on this. Is there anyone out there that have done this, from anaconda? I.e. creating raid partitions and BIOS boot partitions on ALL disks? If so, how did you do it?
AFAIK, a BIOS boot partition has to be on the primary disk where it can be found. At boot time, the BIOS no idea about the RAID since it's a construct of the operating system. Thus the BIOS can't use a partition on the RAID to boot.
I think you need to reserve some small space on all the drives (and with 3TB drives you can afford to sacrifice a few MB), use the remainder as your RAID, and let the system put the boot partition in that reserved space on the primary drive (the one the BIOS sees as the boot drive).
I prefer hardware RAID for just this reason (e.g. if you lose the primary drive, it can still boot since the RAID is visible to the BIOS). Hardware SATA RAID controllers aren't that expensive ($350 US) for the peace of mind they bring.
This is just my opinion. Your mileage may vary. Void where prohibited. Batteries not included. Some assembly required. Yadda, yadda. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - grep me no patterns and I'll tell you no lines - ----------------------------------------------------------------------
On Thu, Aug 28, 2014 at 11:10:54 -0700, Rick Stevens ricks@alldigital.com wrote:
I think you need to reserve some small space on all the drives (and with 3TB drives you can afford to sacrifice a few MB), use the remainder as your RAID, and let the system put the boot partition in that reserved space on the primary drive (the one the BIOS sees as the boot drive).
I think using raid 1 (with the 1.0 header format) can work well for that. There can still grub issues with having a boot just work, but at least you have the stuff you need available.
On 08/28/14 20:21, Bruno Wolff III wrote:
On Thu, Aug 28, 2014 at 11:10:54 -0700, Rick Stevens ricks@alldigital.com wrote:
I think you need to reserve some small space on all the drives (and with 3TB drives you can afford to sacrifice a few MB), use the remainder as your RAID, and let the system put the boot partition in that reserved space on the primary drive (the one the BIOS sees as the boot drive).
I think using raid 1 (with the 1.0 header format) can work well for that. There can still grub issues with having a boot just work, but at least you have the stuff you need available.
Yes, that was my intention, raid1 for /boot and raid6 for / Accidentally I set /boot also to raid6 :) But that can be fixed.
On my old system I use 1TB disks, with /boot as raid1, and grub boot-loader installed on all disks. On that one I can boot the system from any of disks. Which is quite handy.
The problem here seem to be that due to the disks being large (larger than 1TB) they are setup as GPT (GUID Partition Table), and they then also need a BIOS boot partition to work on non UEFI based systems (if I have understood it correctly).
So, to be able to boot from any of the disks, I need a BIOS boot partition on all disks, but anaconda seem to only install it on one of the disks (i.e. I want the exactly identical partition tables on all disks).
Lars
On Aug 28, 2014, at 11:48 AM, Lars E. Pettersson lars@homer.se wrote:
Hi!
I tried to set up a system using four 3TB disks as a raid6. The disks are new, no old partitions laying around. I used a USB stick as install media (Fedora 20 x86_64 Gnome Live).
I chose manual partitioning to get everything as I wanted. When this was ready I got the error message that I also needed a BIOS boot partition (I gather that this due to being rather large disks). I tried to create one of those, but it seemed to only be created on one of the discs in the raid array, I wanted it created on all disks, just as the raid partitions. This to be able to boot from any of the disks in the array in case of a failure. As it is now, if the main disk dies, I can not start the system at all.
Precreate the GPT on each drive using fdisk, parted, or gdisk with a 1MB BIOS Boot partition. Anaconda won't delete it, and you won't have to worry about manually creating it, which is not easy to figure out. I've lobbied to make EFI System partition and BIOS boot always created automatically behind the scenes, even in manual partitioning. There's just no good reason for any user who uses a GUI installer to be responsible for creating such things. https://bugzilla.redhat.com/show_bug.cgi?id=1022316
Actually I ended up having three BIOS boot partition on the first disk, being 2, 1, and 1 MB large. Which is quite annoying. But that's another problem...
Before submitting a bug report/RFE on this. Is there anyone out there that have done this, from anaconda? I.e. creating raid partitions and BIOS boot partitions on ALL disks? If so, how did you do it?
Yes. I created the BIOS boot partition on each disk in advance.
Chris Murphy
On Aug 28, 2014, at 12:21 PM, Bruno Wolff III bruno@wolff.to wrote:
I think using raid 1 (with the 1.0 header format) can work well for that. There can still grub issues with having a boot just work, but at least you have the stuff you need available.
GRUB2 can locate vmlinuz/initramfs on md/mdadm raid0,1,10,5,6, even degraded. There is a BIOS specific limitation recognizing a certain number of drives. So you kinda have to watch out for that.
A possible problem with metadata format 0.9 (deprecated) and 1.0 is that the metadata is at the end of the partition; so in raid1 configurations, one of the partitions can be mounted normally as if there's no raid. The problem comes when modifying the volume, the raid metadata isn't updated. So when the raid is later assembled, md has no idea that one of the raid members is different than the other. This can quickly cause one or both members to become corrupted. So it's generally better (and recommended) to use metadata 1.2 format, which offsets the file system just beyond 4K from the start of the partition. That way a single raid member isn't mountable on its own, you have to first assemble the raid, then mount the file system.
Chris Murphy
On Aug 28, 2014, at 12:37 PM, Lars E. Pettersson lars@homer.se wrote:
On 08/28/14 20:21, Bruno Wolff III wrote:
On Thu, Aug 28, 2014 at 11:10:54 -0700, Rick Stevens ricks@alldigital.com wrote:
I think you need to reserve some small space on all the drives (and with 3TB drives you can afford to sacrifice a few MB), use the remainder as your RAID, and let the system put the boot partition in that reserved space on the primary drive (the one the BIOS sees as the boot drive).
I think using raid 1 (with the 1.0 header format) can work well for that. There can still grub issues with having a boot just work, but at least you have the stuff you need available.
Yes, that was my intention, raid1 for /boot and raid6 for / Accidentally I set /boot also to raid6 :) But that can be fixed.
Or do n-way raid1, which will cause /boot to have as many copies as you add member devices.
On my old system I use 1TB disks, with /boot as raid1, and grub boot-loader installed on all disks. On that one I can boot the system from any of disks. Which is quite handy.
That's still possible whether /boot is raid1 (n-way or mirror), or raid6. Anaconda will run grub2-install against all member devices that have a /boot member device (or at least it should, I have recently done 2 drive testing on this).
Post install you can grep -i grub2-install /var/log/anaconda/program.log
And it will show you the command used. It should list all member devices. So long as each member drive has a BIOS Boot partition, grub2-install will find it automatically and insert core.img there, as well as the specific jump code in the first 440 bytes of the (protective) MBR to find it.
The problem here seem to be that due to the disks being large (larger than 1TB) they are setup as GPT (GUID Partition Table), and they then also need a BIOS boot partition to work on non UEFI based systems (if I have understood it correctly).
MBR partition entries are 32-bit so they're limited to 2TB. There's a small advantage to using GPT in that there's a backup header and table. So long as your BIOS doesn't get fussy with GPT, it's fine to use it even on 1TB drives, but it's not necessary.
So, to be able to boot from any of the disks, I need a BIOS boot partition on all disks, but anaconda seem to only install it on one of the disks (i.e. I want the exactly identical partition tables on all disks).
It's messy. Again the installer should leave the user out of this entirely. You don't need a BIOS Boot that's exactly 1MB in size, so if you don't want to start from scratch but there's at least 100K free space on each member drive, you can create a BIOS boot there and then rerun grub2-install against all the drives, e.g.:
grub2-install /dev/sd[abcde]
To know for sure how big of a BIOS Boot you need:
ls -lh /boot/grub2/i386-pc/core.img
Mine's 26KB, as my boot fs is XFS, without LVM or raid. Yours will be a bit bigger to include mdraid1x.mod and raid6rec.mod and anything else that gets pulled in. I bet it's much less than 100K actually.
Chris Murphy
On 08/31/14 21:27, Chris Murphy wrote:
On Aug 28, 2014, at 12:37 PM, Lars E. Pettersson lars@homer.se wrote:
...
Yes, that was my intention, raid1 for /boot and raid6 for / Accidentally I set /boot also to raid6 :) But that can be fixed.
Or do n-way raid1, which will cause /boot to have as many copies as you add member devices.
Yes, that was what I ended up doing.
I partitioned /dev/sda as I wanted, with space for BIOS boot partition, and also an EFI partition if I ever would need one (more or less a setup as described in https://bugzilla.redhat.com/show_bug.cgi?id=1061478 I have not done anything to the EFI partiton yet though, it is just there at the moment). Raid1 for /boot and raid6 for /. I then copied this partition layout to all the other disks.
After that I started anaconda.
The trickiest part was to figure out how anaconda worked. It took me almost 3 hours and a lot of "googeling" to figure out how to use the "unknown" partitions in ananconda (click on the format). I would say that anaconda was not very intuitive... :)
Post install you can grep -i grub2-install /var/log/anaconda/program.log
# grep grub2-install /var/log/anaconda/anaconda.program.log 17:56:30,287 INFO program: Running... grub2-install --no-floppy /dev/sda 17:56:36,970 INFO program: Running... grub2-install --no-floppy /dev/sdb 17:56:43,012 INFO program: Running... grub2-install --no-floppy /dev/sdc 17:56:49,195 INFO program: Running... grub2-install --no-floppy /dev/sdd
Splendid! I actually did this myself manually, but good to know that anaconda also does it.
To know for sure how big of a BIOS Boot you need:
ls -lh /boot/grub2/i386-pc/core.img
Mine's 26KB, as my boot fs is XFS, without LVM or raid. Yours will be a bit bigger to include mdraid1x.mod and raid6rec.mod and anything else that gets pulled in. I bet it's much less than 100K actually.
Yep, mine was 31KB.
Well, my little home server is now fully functional, but not with the help of anaconda :)
Lars
-
Bruno Wolff III -
Chris Murphy -
Lars E. Pettersson -
Rick Stevens