[Fedora-livecd-list] [PATCH] cleanupDeleted and container_size - saves 5.5% output size on f7livecd-i686

[Douglas McClendon]

Jeremy Katz wrote:
> On Fri, 2007-08-17 at 16:02 -0500, Douglas McClendon wrote:
>> Phillip Lougher wrote:
>>> Douglas McClendon-2 wrote:
>>>> Douglas McClendon wrote:
>>> Some stats based on the Fedora-8-Test-1-Live-i686.iso 
>>>
>>> original squashfs.img 712495104 (681M)
>>> sparse squashfs.img 709820416 (678M)
>>> sparse squashfs.img with 128Kbyte blocks 700100608 (669M)
>>>
>>> So, the compressed 0s used to take up about 2.5 Mbytes.  More significant is
>>> the data and seek time saving in not reading all these useless compressed
>>> 0s.
>>>
>>> time to do "dd if=os.img of=/dev/null" from CDROM
>>>
>>> original squashfs.img, 221 seconds
>>> sparse squashfs.img, 168 seconds
>>> sparse squashfs.img with 128byte blocks, 171 seconds
>>>
>>> on average about 20% speedup.
>> 20% speedup of reading data into /dev/null isn't all that useful.
>>
>> You didn't really mention what you were planning on using it for. 
>> Presumably you were also aware of my turboLiveInst patch, which 
>> accomplishes the 20% speedup in an actual copy situation.
>>
>> The sparse support could be used, perhaps with more code being written 
>> to support sparse copying to block devices with python, to gain the 
>> performance enhancements of turboLiveInst in an alternate manner.
> 
> I suspect that it can...

It can, but it doesn't solve the 2.1G->4.0G problem, which as mentioned, 
also scales if you want a +10G filesystem rather than +2G filesystem. 
(my prior statement using the word container could lead to confusion, as 
I meant filesystem).


> 
>> For reasons that I can only speculate to, Jeremy, and everyone else 
>> seems to have no interest in my turboLiveInst optimization approach. 
>> Perhaps this method will be more palatable for them for some reason.
> 
> It's a bit more palatable because it's not playing tricks with
> device-mapper and instead makes things know about sparse files and deal
> with them as sparse files.  Which then makes the solution more general
> and less dependent on knowing very very intimate details about how
> things are constructed.

Sorry, but I call bullshit.  The 'tricks' here, are an elegant solution 
to more aspects of the problem than the alternate solution.

It does not make the solution more general in any way whatsoever.

The turboLiveInst solution is not "dependent on knowing very intimate 
details about how things are constructed".

If by that statement you mean that people who read the code need to 
understand the very basic infrastructure of how the fedora livecd uses 
an ext3 image and a devicemapper snapshot to boot, and install from, 
then that is a GOOD thing.  The fact that just about nobody other than 
myself and davidz understands that well, is why the livecd wiping the 
root partition bug slipped through the cracks.

I will not argue this any further, but my stance remains that the 
turboLiveInst patch, is a _simple_ _elegant_ solution that fixes _more_ 
of the problem than an alternate sparse copy fix.

> 
>> This actually reopens the usefulness of the container-size option which 
>> I had included with the first version of the cleanupDeleted patch.  I.e. 
>> now it is no penalty whatsoever to go with a much larger container size. 
>>   This will remove the need to perform ugly device-mapper-zero tricks to 
>> expand the rootfs size at runtime, if you have enough memory in your 
>> overlay device to support that.  And assuming of course that either you 
>> are using the turboLiveInst patch, or the alternate method described 
>> above which requires a little more code to be written.
> 
> The big question is what is trying to be enabled by it and what the user
> experience of taking advantage of it is. If it's something that you end
> up passing lots of kernel command line options for, then I don't know if
> the added complexity is worth it.

Jeremy, you have this habit of throwing replies to my strategies, 
clearly before you understand them.  (see prior unanswered response 
regarding program= and eprogram= parameters to mayflower generated init).

In this case (hoping I didn't confuse the issue by bringing up kernel 
parameters), there are NO KERNEL PARAMETERS involved.  Not with 
turboLiveInst, nor with the larger container size.  I mean, where did 
that come from, bringing that into this conversation?


   If it's something that can be a
> benefit for all cases without requiring a lot of knowledge from the
> user, then maybe it is more useful.

I believe that sums up my attitude towards both turboLiveInst, and the 
larger container size feature.  Both are benefits for all cases, and 
require *NO* knowledge from the user.

-dmc