8:47 p.m.
After long periods of waiting and hoping, etc I'm just about to do a
push of some changes that at least give us a good base to build on for
persistent live images. They're very very very heavily based off of
Douglas McClendon's patches that he posted here a month or two, but I've
done basically two things with them:
1) Integrated the findoverlay bits into mkliveinitrd directly and was
able to do a little bit of cleanup accordingly
2) Not put in the bits right now to do the snapshot teardown on shutdown
The former is pretty non-controversial. The latter, well, we're going
to need some testing to know how feasible this really is. I spent quite
a while this afternoon trying to cause any form of corruption without
the initscripts changes and never succeeded[1]. And realistically, if
it's going to cause corruption if we don't shutdown cleanly, then we
have problems if someone loses power or removes their USB key or any of
a whole bunch of other things occur. I _did_ have the journal get
recovered and as long as things were able to get to the journal, it was
good to go. But more testing is going to be needed. So if anyone here
wants to try building some images and putting them on USB drives and
testing them out, that'd be greatly appreciated. And then I'm going to
issue a larger call for testers with the Fedora 9 beta. If we find that
we need to integrate some of the shutdown bits, we'll make it happen.
But right now, I'm feeling pretty confident that we won't
The quick steps of how to take advantage of the functionality:
1) Build a live image using the newer mkinitrd (6.0.32) in your repos
2) Use the updated livecd-iso-to-disk.sh with an --overlay-size-mb=512
(or 1024 or whatever based on the capacity of your USB stick minus the
size of the ISO)
3) Boot the USB drive
You can also (although it's more complicated) use a USB stick with a CD
1) Build live image as in 1) above
2) Create /LiveOS/overlay-$v- on your USB stick with size how ever big
you want (can be a sparse file). $v is the label of the _iso_ (not of
the USB stick)
3) Find the label or UUID of the partition on the USB stick you're
using. Write it down.
4) When you boot the CD, have the USB stick plugged in and add to the
kernel command line
overlay=$dev
where $dev is what you found in step 3. If it's a label, do LABEL=foo;
similarly uuid ==> UUID=foo.
Big big big thanks and kudos to dmc for doing all of the legwork here
and letting me off the hook for so long on actually getting it
integrated. But once I finally got around to having a full afternoon to
spend on it, it was in great shape.
Jeremy
10:36 p.m.
Jeremy Katz wrote:
After long periods of waiting and hoping, etc I'm just about to
do a
push of some changes that at least give us a good base to build on for
persistent live images. They're very very very heavily based off of
Douglas McClendon's patches that he posted here a month or two, but I've
done basically two things with them:
1) Integrated the findoverlay bits into mkliveinitrd directly and was
able to do a little bit of cleanup accordingly
No general objections. As I look at it more, I'm sure I'll start
submitting some patches for mkinitrd.
2) Not put in the bits right now to do the snapshot teardown on
shutdown
No objections. I admit, that I also have not witnessed a failure, since
some kernel improvement maybe half a year ago seemed to make it
crash-proof. But of course it should do the right thing to shutdown
cleanly. But I agree that can be deferred for now.
The former is pretty non-controversial. The latter, well, we're
going
to need some testing to know how feasible this really is. I spent quite
a while this afternoon trying to cause any form of corruption without
the initscripts changes and never succeeded[1]. And realistically, if
it's going to cause corruption if we don't shutdown cleanly, then we
have problems if someone loses power or removes their USB key or any of
a whole bunch of other things occur. I _did_ have the journal get
recovered and as long as things were able to get to the journal, it was
good to go. But more testing is going to be needed. So if anyone here
wants to try building some images and putting them on USB drives and
testing them out, that'd be greatly appreciated. And then I'm going to
issue a larger call for testers with the Fedora 9 beta. If we find that
we need to integrate some of the shutdown bits, we'll make it happen.
But right now, I'm feeling pretty confident that we won't
The quick steps of how to take advantage of the functionality:
1) Build a live image using the newer mkinitrd (6.0.32) in your repos
2) Use the updated livecd-iso-to-disk.sh with an --overlay-size-mb=512
(or 1024 or whatever based on the capacity of your USB stick minus the
size of the ISO)
and minus whatever space you want for other stuff on the liveusb. I see
plenty of interesting uses of LiveCD/USBs with this feature, and the
writable area. I haven't yet looked at what Jeremy actually merged, but
the LiveUSB host filesystem (typically vfat) will be accessible under
/mnt/overlayfs. I think in the LiveUSB case it is also /mnt/live.
3) Boot the USB drive
You can also (although it's more complicated) use a USB stick with a CD
1) Build live image as in 1) above
2) Create /LiveOS/overlay-$v- on your USB stick with size how ever big
you want (can be a sparse file). $v is the label of the _iso_ (not of
the USB stick)
3) Find the label or UUID of the partition on the USB stick you're
using. Write it down.
4) When you boot the CD, have the USB stick plugged in and add to the
kernel command line
overlay=$dev
where $dev is what you found in step 3. If it's a label, do LABEL=foo;
similarly uuid ==> UUID=foo.
Big big big thanks and kudos to dmc for doing all of the legwork here
and letting me off the hook for so long on actually getting it
integrated. But once I finally got around to having a full afternoon to
spend on it, it was in great shape.
Thanks, and no problem. Of course, now I/someone really need to put
together a brief overview/faq page, to especially drill in the
mechanism, and set expectations appropriately given the mechanism (i.e.
as with ram for the livecd, creating and deleting files is not currently
handled as ideally efficient as one would like).
But given the appropriate expectations, I expect to see it end up being
useful in a lot of situations. And I am sure that in the future the
feature will improve over this initial functionality.
-dmc
11:40 a.m.
New subject: Unable to overide package/group selection
I have a kickstart that includes the xfce kickstart (which includes
fedora-8-base-desktop), I'm trying to add samba-client (which is removed
by fedora-8-base-desktop) and remove the language groups (added by the
xfce kickstart). For instance (pardon me if I mistype the package/group
names):
samba-client
-@zulu-languague-support
Those lines are ignored. The only way I've found to make the changes I
want is to actually modify the original kickstarts. The behavior I would
expect is that each kickstart could overide anything in an included
kickstart. I'm using the latest livecd-tools in fedora 8 updates rather
than the git version. So I'm either reporting a bug or reporting
something that's been fixed in git...
Tim
Douglas McClendon wrote:
Jeremy Katz wrote:
> After long periods of waiting and hoping, etc I'm just about to do a
> push of some changes that at least give us a good base to build on for
> persistent live images. They're very very very heavily based off of
> Douglas McClendon's patches that he posted here a month or two, but I've
> done basically two things with them:
> 1) Integrated the findoverlay bits into mkliveinitrd directly and was
> able to do a little bit of cleanup accordingly
No general objections. As I look at it more, I'm sure I'll start
submitting some patches for mkinitrd.
> 2) Not put in the bits right now to do the snapshot teardown on shutdown
No objections. I admit, that I also have not witnessed a failure,
since some kernel improvement maybe half a year ago seemed to make it
crash-proof. But of course it should do the right thing to shutdown
cleanly. But I agree that can be deferred for now.
> The former is pretty non-controversial. The latter, well, we're going
> to need some testing to know how feasible this really is. I spent quite
> a while this afternoon trying to cause any form of corruption without
> the initscripts changes and never succeeded[1]. And realistically, if
> it's going to cause corruption if we don't shutdown cleanly, then we
> have problems if someone loses power or removes their USB key or any of
> a whole bunch of other things occur. I _did_ have the journal get
> recovered and as long as things were able to get to the journal, it was
> good to go. But more testing is going to be needed. So if anyone here
> wants to try building some images and putting them on USB drives and
> testing them out, that'd be greatly appreciated. And then I'm going to
> issue a larger call for testers with the Fedora 9 beta. If we find that
> we need to integrate some of the shutdown bits, we'll make it happen.
> But right now, I'm feeling pretty confident that we won't
>
> The quick steps of how to take advantage of the functionality:
> 1) Build a live image using the newer mkinitrd (6.0.32) in your repos
> 2) Use the updated livecd-iso-to-disk.sh with an --overlay-size-mb=512
> (or 1024 or whatever based on the capacity of your USB stick minus the
> size of the ISO)
and minus whatever space you want for other stuff on the liveusb. I
see plenty of interesting uses of LiveCD/USBs with this feature, and
the writable area. I haven't yet looked at what Jeremy actually
merged, but the LiveUSB host filesystem (typically vfat) will be
accessible under /mnt/overlayfs. I think in the LiveUSB case it is
also /mnt/live.
> 3) Boot the USB drive
>
> You can also (although it's more complicated) use a USB stick with a CD
> 1) Build live image as in 1) above
> 2) Create /LiveOS/overlay-$v- on your USB stick with size how ever big
> you want (can be a sparse file). $v is the label of the _iso_ (not of
> the USB stick)
> 3) Find the label or UUID of the partition on the USB stick you're
> using. Write it down.
> 4) When you boot the CD, have the USB stick plugged in and add to the
> kernel command line
> overlay=$dev
> where $dev is what you found in step 3. If it's a label, do LABEL=foo;
> similarly uuid ==> UUID=foo.
>
> Big big big thanks and kudos to dmc for doing all of the legwork here
> and letting me off the hook for so long on actually getting it
> integrated. But once I finally got around to having a full afternoon to
> spend on it, it was in great shape.
Thanks, and no problem. Of course, now I/someone really need to put
together a brief overview/faq page, to especially drill in the
mechanism, and set expectations appropriately given the mechanism
(i.e. as with ram for the livecd, creating and deleting files is not
currently handled as ideally efficient as one would like).
But given the appropriate expectations, I expect to see it end up
being useful in a lot of situations. And I am sure that in the future
the feature will improve over this initial functionality.
-dmc
--
Fedora-livecd-list mailing list
Fedora-livecd-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/fedora-livecd-list
--
_________________________________
Tim Wood, CLP, RHCT
719.338.7484 (tel)
The Data Wranglers
Web, Database & more since since 1994
www.datawranglers.com
12:06 p.m.
New subject: Unable to overide package/group selection
On Wed, 2008-03-05 at 10:40 -0700, Tim Wood wrote:
I have a kickstart that includes the xfce kickstart (which includes
fedora-8-base-desktop), I'm trying to add samba-client (which is removed
by fedora-8-base-desktop) and remove the language groups (added by the
xfce kickstart). For instance (pardon me if I mistype the package/group
names):
Yeah, there's not support within the kickstart syntax for removal of a
group. I believe there's a bug filed against pykickstart, though
Jeremy
3:36 a.m.
New subject: Unable to overide package/group selection
Jeremy Katz wrote:
On Wed, 2008-03-05 at 10:40 -0700, Tim Wood wrote:
> I have a kickstart that includes the xfce kickstart (which includes
> fedora-8-base-desktop), I'm trying to add samba-client (which is removed
> by fedora-8-base-desktop) and remove the language groups (added by the
> xfce kickstart). For instance (pardon me if I mistype the package/group
> names):
Yeah, there's not support within the kickstart syntax for removal of a
group. I believe there's a bug filed against pykickstart, though
Yes. Filed this at
https://bugzilla.redhat.com/show_bug.cgi?id=428835
Chris Lumens said he will look into it.
Rahul
5:27 p.m.
Douglas McClendon wrote:
Jeremy Katz wrote:
> After long periods of waiting and hoping, etc I'm just about to do a
> push of some changes that at least give us a good base to build on for
> persistent live images. They're very very very heavily based off of
> Douglas McClendon's patches that he posted here a month or two, but I've
> done basically two things with them:
> 1) Integrated the findoverlay bits into mkliveinitrd directly and was
> able to do a little bit of cleanup accordingly
No general objections. As I look at it more, I'm sure I'll start
submitting some patches for mkinitrd.
> 2) Not put in the bits right now to do the snapshot teardown on shutdown
No objections. I admit, that I also have not witnessed a failure, since
some kernel improvement maybe half a year ago seemed to make it
crash-proof. But of course it should do the right thing to shutdown
cleanly. But I agree that can be deferred for now.
> The former is pretty non-controversial. The latter, well, we're going
> to need some testing to know how feasible this really is. I spent quite
> a while this afternoon trying to cause any form of corruption without
> the initscripts changes and never succeeded[1]. And realistically, if
> it's going to cause corruption if we don't shutdown cleanly, then we
> have problems if someone loses power or removes their USB key or any of
> a whole bunch of other things occur. I _did_ have the journal get
> recovered and as long as things were able to get to the journal, it was
> good to go. But more testing is going to be needed. So if anyone here
> wants to try building some images and putting them on USB drives and
> testing them out, that'd be greatly appreciated. And then I'm going to
> issue a larger call for testers with the Fedora 9 beta. If we find that
> we need to integrate some of the shutdown bits, we'll make it happen.
> But right now, I'm feeling pretty confident that we won't
>
> The quick steps of how to take advantage of the functionality:
> 1) Build a live image using the newer mkinitrd (6.0.32) in your repos
> 2) Use the updated livecd-iso-to-disk.sh with an --overlay-size-mb=512
> (or 1024 or whatever based on the capacity of your USB stick minus the
> size of the ISO)
and minus whatever space you want for other stuff on the liveusb. I see
plenty of interesting uses of LiveCD/USBs with this feature, and the
writable area. I haven't yet looked at what Jeremy actually merged, but
the LiveUSB host filesystem (typically vfat) will be accessible under
/mnt/overlayfs. I think in the LiveUSB case it is also /mnt/live.
Anybody else adding swap to their usb? That would be a nice option for
the live2usb script, IMHO, just not sure for the best/right way to add
that... I use a usb2ide adapter, and have a pre-existing swap partition
that "just works" on boot.
> 3) Boot the USB drive
>
> You can also (although it's more complicated) use a USB stick with a CD
> 1) Build live image as in 1) above
> 2) Create /LiveOS/overlay-$v- on your USB stick with size how ever big
> you want (can be a sparse file). $v is the label of the _iso_ (not of
> the USB stick)
> 3) Find the label or UUID of the partition on the USB stick you're
> using. Write it down.
> 4) When you boot the CD, have the USB stick plugged in and add to the
> kernel command line
> overlay=$dev
> where $dev is what you found in step 3. If it's a label, do LABEL=foo;
> similarly uuid ==> UUID=foo.
>
So lets just say that you boot your initrd, have the means to setup the
network, retrieving your "overlay file", could you then merge that into
the base livecd before your really mount sysroot?
Wow, I'm thinking updates as you boot or just before you install to the
hard-drive.... Or maybe grab my personal mod's from here... Or base live
+ overlay = spin... Just some random thoughts...
> Big big big thanks and kudos to dmc for doing all of the legwork
here
> and letting me off the hook for so long on actually getting it
> integrated. But once I finally got around to having a full afternoon to
> spend on it, it was in great shape.
Thanks, and no problem. Of course, now I/someone really need to put
together a brief overview/faq page, to especially drill in the
mechanism, and set expectations appropriately given the mechanism (i.e.
as with ram for the livecd, creating and deleting files is not currently
handled as ideally efficient as one would like).
But given the appropriate expectations, I expect to see it end up being
useful in a lot of situations. And I am sure that in the future the
feature will improve over this initial functionality.
Yes it will, thank you for the great start.
Jerry
10:32 a.m.
On Fri, 2008-03-07 at 17:27 -0600, Jerry Vonau wrote:
Anybody else adding swap to their usb? That would be a nice option
for
the live2usb script, IMHO, just not sure for the best/right way to add
that... I use a usb2ide adapter, and have a pre-existing swap partition
that "just works" on boot.
Swap on a USB stick tends to be less than ideal just due to the write
patterns for swap. There's nothing stopping it, though.
[snip]
So lets just say that you boot your initrd, have the means to setup
the
network, retrieving your "overlay file", could you then merge that into
the base livecd before your really mount sysroot?
Wow, I'm thinking updates as you boot or just before you install to the
hard-drive.... Or maybe grab my personal mod's from here... Or base live
+ overlay = spin... Just some random thoughts...
Yep, there's all sorts of things that start to be possible. Although
for a lot of things like network setup, etc we really need to get
further on merging the needs for live initrds into mkinitrd. What I've
done will make it transparent for when we do, but for those that haven't
looked, it's definitely a hack for now
Jeremy
2:20 p.m.
Jeremy Katz wrote:
The quick steps of how to take advantage of the functionality:
1) Build a live image using the newer mkinitrd (6.0.32) in your repos
2) Use the updated livecd-iso-to-disk.sh with an --overlay-size-mb=512
(or 1024 or whatever based on the capacity of your USB stick minus the
size of the ISO)
if [ -n "$overlaysizemb" ]; then
echo "Initializing persistent overlay file"
OVERFILE="overlay-$( /lib/udev/vol_id -l $USBDEV )-$(
/lib/udev/vol_id -u $USBDEV )"
dd if=/dev/null of=$USBMNT/LiveOS/$OVERFILE \
count=1 bs=1M seek=$overlaysizemb
Warning!
If your USB stick is vfat filesystem and $overlaysizemb is
some number larger than your RAM, your system will swap storm and die.
According to Ulrich Drepper this isn't dd using all RAM, but it is up to
the filesystem implementation. Since the OOM happens in kernel space it
cannot OOM kill anything and recover.
I just tried the same thing with the USB stick reformatted as ext3. It
successfully created the sparse file without any problem.
Perhaps this means we cannot support overlay images on vfat?
Warren Togami
wtogami(a)redhat.com
2:35 p.m.
Warren Togami wrote:
Jeremy Katz wrote:
>
> The quick steps of how to take advantage of the functionality:
> 1) Build a live image using the newer mkinitrd (6.0.32) in your repos
> 2) Use the updated livecd-iso-to-disk.sh with an --overlay-size-mb=512
> (or 1024 or whatever based on the capacity of your USB stick minus the
> size of the ISO)
if [ -n "$overlaysizemb" ]; then
echo "Initializing persistent overlay file"
OVERFILE="overlay-$( /lib/udev/vol_id -l $USBDEV )-$(
/lib/udev/vol_id -u $USBDEV )"
dd if=/dev/null of=$USBMNT/LiveOS/$OVERFILE \
count=1 bs=1M seek=$overlaysizemb
Warning!
If your USB stick is vfat filesystem and $overlaysizemb is
some number larger than your RAM, your system will swap storm and die.
According to Ulrich Drepper this isn't dd using all RAM, but it is up to
the filesystem implementation. Since the OOM happens in kernel space it
cannot OOM kill anything and recover.
I tried creating a 512 MB overlay file when my vfat usbkey had
only 300 MB left. The filesystem on the usbkey was no longer
useable and I had to reformat the usbkey to use it again.
--
Patrice
2:37 p.m.
On Thu, 2008-03-06 at 15:35 -0500, Patrice Guay wrote:
I tried creating a 512 MB overlay file when my vfat usbkey had
only 300 MB left. The filesystem on the usbkey was no longer
useable and I had to reformat the usbkey to use it again.
Yeah, this is an error case that I know we need to look for and catch,
just a matter of 'round tuits.
Jeremy
2:38 p.m.
On Thu, 2008-03-06 at 15:20 -0500, Warren Togami wrote:
Jeremy Katz wrote:
> The quick steps of how to take advantage of the functionality:
> 1) Build a live image using the newer mkinitrd (6.0.32) in your repos
> 2) Use the updated livecd-iso-to-disk.sh with an --overlay-size-mb=512
> (or 1024 or whatever based on the capacity of your USB stick minus the
> size of the ISO)
if [ -n "$overlaysizemb" ]; then
echo "Initializing persistent overlay file"
OVERFILE="overlay-$( /lib/udev/vol_id -l $USBDEV )-$(
/lib/udev/vol_id -u $USBDEV )"
dd if=/dev/null of=$USBMNT/LiveOS/$OVERFILE \
count=1 bs=1M seek=$overlaysizemb
Warning!
If your USB stick is vfat filesystem and $overlaysizemb is
some number larger than your RAM, your system will swap storm and die.
According to Ulrich Drepper this isn't dd using all RAM, but it is up to
the filesystem implementation. Since the OOM happens in kernel space it
cannot OOM kill anything and recover.
I just tried the same thing with the USB stick reformatted as ext3. It
successfully created the sparse file without any problem.
Perhaps this means we cannot support overlay images on vfat?
Or we may need to limit the size that we support. Warren and I are
looking a little closer now...
Jeremy
5900
days inactive
5905
days old
livecd@lists.fedoraproject.org
10 comments
7 participants
participants (7)
-
Douglas McClendon -
Jeremy Katz -
Jerry Vonau -
Patrice Guay -
Rahul Sundaram -
Tim Wood -
Warren Togami