1:12 p.m.
Hi!
I haven't grasped the persistance overlay usage yet.
I usually create/update a personal spin every week and install it to a
usb stick. Since F9, I've also been using the persistance flag. I don't
use my liveusb everyday but I like to keep it updated and in my pocket.
Last weekend, I updated my spin and installed on the usb stick. Halfway
through, I noticed I had forgotten the overlay flag in
livecd-iso-to-disk command. I thought, "well, i guess it will be
detected since it was created previously...".
Yesterday, I booted my usb stick at work and upon boot I can see a
message saying no persistance overlay found. Err... my best guess was
wrong.
So, how does the persistance work:
- Persistance is created with livecd-iso-to-disk, if I run it again
without the overlay flag, previous persistance overlay is deleted?
- Persistance is created with livecd-iso-to-disk, further runs of
livecd-iso-to-disk won't detect previous overlay?
I honestly looked for more info on the fedora wiki and man pages, but I
couldn't find a wiki page or a man page about it. If these don't exist,
I would be happy to help build one.
Thanks and best regards,
--
Pedro Silva
1:46 p.m.
On Wed, 2008-06-04 at 19:12 +0100, Pedro Silva wrote:
So, how does the persistance work:
- Persistance is created with livecd-iso-to-disk, if I run it again
without the overlay flag, previous persistance overlay is deleted?
Yes. Also, the overlay is specific to the *exact* filesystem image that
it is created for.
... and for the gory details. The way that persistence is implemented
is that the persistence file is used as the backing store for the device
mapper snapshot that we build on top of the "base" filesystem image.
This means that it contains just changed blocks from the base image and
that changes continue to build up over time, never reusing blocks from
the snapshot.
So, not ideal for more long-term cases as if you're updating with yum,
then you'll run out of space on your snapshot before you'd expect as we
overwrite things multiple times[1].
Which brings me to what I've been working on this week -- persistence
purely for /home[2]. So rather than using the persistence file for the
snapshot backing store, we instead mount it as /home[3]. At this point,
it's almost to where I'm happy enough with it to push, so stay tuned for
more details :-)
Jeremy
[1] Plus, your kernel will never get updated
[2] Actually, implemented such that you can have persistent changes for
the "OS" as well as a persistent /home. Then your /home sticks around
when you update the USB stick but any "OS" change disappear as with the
present bits
[3] Including support for encryption
2:03 p.m.
Qua, 2008-06-04 às 14:46 -0400, Jeremy Katz escreveu:
So, not ideal for more long-term cases as if you're updating with
yum,
then you'll run out of space on your snapshot before you'd expect as we
overwrite things multiple times[1].
I update my personal spin just by building it again but there maybe some
cases where yum maybe used on a livecd for quick and small stuff.
Which brings me to what I've been working on this week --
persistence
purely for /home[2]. So rather than using the persistence file for the
snapshot backing store, we instead mount it as /home[3]. At this point,
it's almost to where I'm happy enough with it to push, so stay tuned for
more details :-)
That's great news! :D
[2] Actually, implemented such that you can have persistent changes
for
the "OS" as well as a persistent /home. Then your /home sticks around
when you update the USB stick but any "OS" change disappear as with the
present bits
So, this means changes in /etc files (for example) are lost?
[3] Including support for encryption
Sweeeet!
Thanks for your info!
Best regards,
--
Pedro Silva
2:26 p.m.
On Wed, 2008-06-04 at 20:03 +0100, Pedro Silva wrote:
Qua, 2008-06-04 às 14:46 -0400, Jeremy Katz escreveu:
> [2] Actually, implemented such that you can have persistent changes for
> the "OS" as well as a persistent /home. Then your /home sticks around
> when you update the USB stick but any "OS" change disappear as with the
> present bits
So, this means changes in /etc files (for example) are lost?
Right. There's just not any good way as it stands to preserve them. On
the plus side, there should be less and less of this needing to be done
as more and more gets controlled from the user session rather than at a
system level. Stateless Linux here we come!
But at the same time, I am considering adding some form of hook to run a
script off of /home during the boot so that you could implement making
any changes you wanted. Just not sure if it's overkill.
Jeremy
2:28 p.m.
Why not put the hook in the fedora-live init script?
Tim
Jeremy Katz wrote:
On Wed, 2008-06-04 at 20:03 +0100, Pedro Silva wrote:
> Qua, 2008-06-04 às 14:46 -0400, Jeremy Katz escreveu:
>
>> [2] Actually, implemented such that you can have persistent changes for
>> the "OS" as well as a persistent /home. Then your /home sticks around
>> when you update the USB stick but any "OS" change disappear as with
the
>> present bits
>>
> So, this means changes in /etc files (for example) are lost?
>
Right. There's just not any good way as it stands to preserve them. On
the plus side, there should be less and less of this needing to be done
as more and more gets controlled from the user session rather than at a
system level. Stateless Linux here we come!
But at the same time, I am considering adding some form of hook to run a
script off of /home during the boot so that you could implement making
any changes you wanted. Just not sure if it's overkill.
Jeremy
--
Fedora-livecd-list mailing list
Fedora-livecd-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/fedora-livecd-list
--
_________________________________
Tim Wood, CLP, RHCT
719.338.7484 (tel)
The Data Wranglers
Web, Database & more since since 1994
www.datawranglers.com
2:46 p.m.
On Wed, 2008-06-04 at 13:28 -0600, Tim Wood wrote:
Jeremy Katz wrote:
> But at the same time, I am considering adding some form of hook to run a
> script off of /home during the boot so that you could implement making
> any changes you wanted. Just not sure if it's overkill.
Why not put the hook in the fedora-live init script?
That's actually where all of this magic lives (well init script +
livecd-iso-to-disk). Like I said, just not entirely sure if it's
overkill to try to support or not
Jeremy
2:58 p.m.
Qua, 2008-06-04 às 15:46 -0400, Jeremy Katz escreveu:
On Wed, 2008-06-04 at 13:28 -0600, Tim Wood wrote:
> Jeremy Katz wrote:
> > But at the same time, I am considering adding some form of hook to run a
> > script off of /home during the boot so that you could implement making
> > any changes you wanted. Just not sure if it's overkill.
>
> Why not put the hook in the fedora-live init script?
That's actually where all of this magic lives (well init script +
livecd-iso-to-disk). Like I said, just not entirely sure if it's
overkill to try to support or not
I'm in favor of a solution that would include /home and other system
folders. Since I have no real understanding of the inner workings of
livecd-iso-to-disk, I don't know how hard that would be. I find it
usefull to keep some changes in /etc files but there are other files
too. Would this have to include a "blacklist" for stuff
like /tmp, /var/cache ?
Regards,
--
Pedro Silva
3:46 p.m.
On Wed, 2008-06-04 at 20:58 +0100, Pedro Silva wrote:
Qua, 2008-06-04 às 15:46 -0400, Jeremy Katz escreveu:
> On Wed, 2008-06-04 at 13:28 -0600, Tim Wood wrote:
> > Jeremy Katz wrote:
> > > But at the same time, I am considering adding some form of hook to run a
> > > script off of /home during the boot so that you could implement making
> > > any changes you wanted. Just not sure if it's overkill.
> >
> > Why not put the hook in the fedora-live init script?
>
> That's actually where all of this magic lives (well init script +
> livecd-iso-to-disk). Like I said, just not entirely sure if it's
> overkill to try to support or not
I'm in favor of a solution that would include /home and other system
folders. Since I have no real understanding of the inner workings of
livecd-iso-to-disk, I don't know how hard that would be. I find it
usefull to keep some changes in /etc files but there are other files
too. Would this have to include a "blacklist" for stuff
like /tmp, /var/cache ?
The problem is there's no good way to define what is a "system file".
Not to mention that if you update a package, you may want to get its new
config files. Perhaps at some point, we'll be able to use something
other than dm-snapshot for our persistence, at which point, we'll have
some more options. But for now, it's what we've got
Jeremy
4:13 p.m.
I'm fond of using that script to wave a wand or three ;-)
if [[ -f /home/fedora/.init_script.sh ]]; then
sh /home/fedora/.init_script.sh
fi
Tim
Jeremy Katz wrote:
On Wed, 2008-06-04 at 13:28 -0600, Tim Wood wrote:
> Jeremy Katz wrote:
>
>> But at the same time, I am considering adding some form of hook to run a
>> script off of /home during the boot so that you could implement making
>> any changes you wanted. Just not sure if it's overkill.
>>
>
> Why not put the hook in the fedora-live init script?
>
That's actually where all of this magic lives (well init script +
livecd-iso-to-disk). Like I said, just not entirely sure if it's
overkill to try to support or not
Jeremy
--
Fedora-livecd-list mailing list
Fedora-livecd-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/fedora-livecd-list
--
_________________________________
Tim Wood, CLP, RHCT
719.338.7484 (tel)
The Data Wranglers
Web, Database & more since since 1994
www.datawranglers.com
1:03 a.m.
Jeremy Katz wrote:
On Wed, 2008-06-04 at 19:12 +0100, Pedro Silva wrote:
> So, how does the persistance work:
>
> - Persistance is created with livecd-iso-to-disk, if I run it again
> without the overlay flag, previous persistance overlay is deleted?
Yes. Also, the overlay is specific to the *exact* filesystem image that
it is created for.
... and for the gory details. The way that persistence is implemented
is that the persistence file is used as the backing store for the device
mapper snapshot that we build on top of the "base" filesystem image.
This means that it contains just changed blocks from the base image and
that changes continue to build up over time, never reusing blocks from
the snapshot.
So, not ideal for more long-term cases as if you're updating with yum,
then you'll run out of space on your snapshot before you'd expect as we
overwrite things multiple times[1].
Which brings me to what I've been working on this week -- persistence
purely for /home[2]. So rather than using the persistence file for the
snapshot backing store, we instead mount it as /home[3]. At this point,
it's almost to where I'm happy enough with it to push, so stay tuned for
more details :-)
Jeremy
[1] Plus, your kernel will never get updated
[2] Actually, implemented such that you can have persistent changes for
the "OS" as well as a persistent /home. Then your /home sticks around
when you update the USB stick but any "OS" change disappear as with the
present bits
[3] Including support for encryption
Sounds good Jeremy, you know I've always supported this idea because it
helps so many use cases.
FYI- the fixes to the current persistance implementation that I alluded
to before are basically this-
Have a defragmentation process which can merge the snapshot overlay
back. Even if it takes awhile for MarkMC's live kernel snapshot merging
patches to go upstream (which wouldn't help for the squashfs case),
there is actually a fairly straightforward alternative.
You can either have a simple boot choice to run a
'defragmentation/merge' pass while/before booting, or a more complex
version that does it to a running system (using the rootfs swap trick my
rebootless installer uses).
I.e. during the defrag/merge process, you just look at the combined
device (that is in a readonly/unchanging state) and then run mksquashfs
on it to build a new merged version. You'd have to have free space on
the liveusb or disk or ram to handle the transition time where you need
2 copies, but that's not so bad.
-dmc
1:17 a.m.
Jeremy Katz wrote:
On Wed, 2008-06-04 at 19:12 +0100, Pedro Silva wrote:
> So, how does the persistance work:
>
> - Persistance is created with livecd-iso-to-disk, if I run it again
> without the overlay flag, previous persistance overlay is deleted?
Yes. Also, the overlay is specific to the *exact* filesystem image that
it is created for.
... and for the gory details. The way that persistence is implemented
is that the persistence file is used as the backing store for the device
mapper snapshot that we build on top of the "base" filesystem image.
This means that it contains just changed blocks from the base image and
that changes continue to build up over time, never reusing blocks from
the snapshot.
This is not actually true. It's not that it never reuses blocks from
teh snapshot, it's that it doesn't try to.
If say you have a 256MB overlay, half used, and half of that wasted
(e.g. you copy two 128MB files to /home, then delete one of them). Then
currently there will be about 2G of free blocks on 4G of rootfs, and
128MB of those currently in a wasted state on the overlay. Assuming
ext3 is roughly random in its choice of free block to use, that means
that each new block written has about a 1/16 chance of actually
reclaiming overlay space, instead of using more space. Theoretically if
you have a 4G overlay, you'll never run out of space, though at some
point you'll run into the issues of why you didn't just install fedora
to a 4G partition on that device to start with.
-dmc
5798
days inactive
5799
days old
livecd@lists.fedoraproject.org
10 comments
4 participants
participants (4)
-
Douglas McClendon -
Jeremy Katz -
Pedro Silva -
Tim Wood