The continuous crippling of Fedora LiveCDs by removing usefull stuff for no apparent reason
Once upon a time, I believe it was Fedora 17 or Fedora 19, I was a happy camper.
I used to download a Fedora LiveCD, and it included everything I needed for functional work on a otherwise Windows netbook. I just booted off the USB Flash Drive, and off I went to do my daily online chores like E-Mail, twitter, and ocassional image editing via my favorite small image editor (Java Image Editor).
That was because OpenJDK 7 was included in the LiveCD image.
This served multiple purposes: 1. Leaving the Windows partition unchanged 2. Not losing space on the small HDD with a Linux install 3. Not leaving my files behind on the system 4. Making an effort to put my data files on the Cloud or LAN network attached storage, NOT the individual system's HDD.
Suddenly (I don't remember which release, but you can check the mailing list archives for my rant on the matter) OpenJDK binaries were removed from the live CD so I could no longer "java -jar JavaImageEditor.jar" nor could I use JNLP to launch desktop Java apps.
I came to the list at the time to rant, and if I remember correctly the answer was that OpenJDK was removed from the LiveCD due to space constraints, and that it was a real challenge to keep everything to still fit into a CD image.
Well, nowadays the Fedora 23 Mate-Compiz LiveCD I'm using is well above the 700MB CD-R limit, and I'm saving the LiveCD image to a 4GB pen drive anyway, yet not only OpenJDK 7 is not there -it should be OpenJDK 8 by now, by the way-, but also other useful stuff -native Linux binaries, command line utils that are a must-have- are also removed for no apparent reason.
Case in point: I booted off the LiveCD the other day needing to do a quick change to a LAN router I have at home, and which provides access to the LAN side via TELNET. Well, guess what? there's no telnet in the LiveCD. But it is in the repos. I did a yum install telnet and found it only uses 76 kbytes. So telnet is not there to save 76 kbytes off the livecd image size. Wow.
But the plot tickens. I wanted to download a mirrorr of a subsection of a web site to another usb flash drive using wget, using the wget -m -np -k -c syntax I know and love, and found .... WGET is not there either.
Since I have learned that the Fedora community is quite hostile to any kind of complaint ("it's as-is because we-know-better and you-re-surely-doing-things-wrong" or "why-do-you-want-to-do-that-in-the-first-place" or "why-dont-you-just-install-the-full-fedora-instead-of-booting-off-a-LiveCD-from-a-Flash-Drive"), my hopes of this being "fixed" (openJDK being reinstated into the LiveCD image, along with telnet and wget), I think I'll end up taking things into my own hands and "fixing" the LiveCD images.
So, is there an up-to-date tutorial out there on how to add stuff from the repos of a Fedora distro repos into the LiveCD image so the packages are available when LiveCD boots?
Thanks in advance. FC
On Fri, 18 Dec 2015 11:53:09 -0500 Fernando Cassia wrote:
So, is there an up-to-date tutorial out there on how to add stuff from the repos of a Fedora distro repos into the LiveCD image so the packages are available when LiveCD boots?
No tutorial, but if you use the liveusb-creator to transfer the livecd image to a usb stick, it has an option to create persistent storage, then when you boot the usb stick the first time, you just "dnf install" whatever is missing, and it hangs around. (Just don't try to install a new kernel - that is much trickier).
On Fri, Dec 18, 2015 at 12:46 PM, Tom Horsley horsley1953@gmail.com wrote:
No tutorial, but if you use the liveusb-creator to transfer the livecd image to a usb stick, it has an option to create persistent storage, then when you boot the usb stick the first time, you just "dnf install" whatever is missing, and it hangs around.
Hi Tom,
Thanks for the reply. I choose to stop using so-called "persistent storage" because it eventually goes FUBAR. Something about the loop device if I remember correctly. The issue being that all is well until you fill up all available storage space. Then the "persistent storage" partition becomes "damaged" and unmountable. I lost quite a few folders full of work that way, that's when I decided that was useless.
In fact if you google "loopback device persistent storage" and have autocomplete enabled, the next word that comes up after you type a space is "corrupted".
http://askubuntu.com/questions/18466/how-can-i-repair-casper-rw-file-system-...
Maybe it'd be time for the so-called "persistent storage" to be F2FS based rather than loopback?
Now *there's* an idea! FC
Allegedly, on or about 18 December 2015, Fernando Cassia sent:
Thanks for the reply. I choose to stop using so-called "persistent storage" because it eventually goes FUBAR. Something about the loop device if I remember correctly. The issue being that all is well until you fill up all available storage space. Then the "persistent storage" partition becomes "damaged" and unmountable. I lost quite a few folders full of work that way, that's when I decided that was useless.
That sounds like a problem I had on my old Amiga, more than a decade ago, where due to some programmer's maths errors, it'd try to fill a disc to 101% capacity.
Though, in this case, I suspect it's more to do with flash drives losing capacity, over time, as it wears out, and no file system being developed in mind of this (so that a reasonable large amount of space is always required to be kept free).
Personally, if I was to run a live OS from a USB stick. I'd be inclined to use two. One for the OS, another for my data.
I had tried running from a USB connected hard drive, but I would find that things would get stuffed up after a while. I've come to the conclusion that USB hardware is far from reliable. Apart from hard drives in cases getting hot, and bumped about, Linux seems rather good at creating data corruption when there are several drives in use (internal and/or external). I've experienced this over several different systems (motherboards, and drives, and types of drives).
You'd be doing things, then become aware that a drive wasn't available any more. Copying a bunch of files would stall, or would become a randomised mess (I learnt to always copy, never move files). Loading a program would stall or simply fail.
On 19 December 2015 at 05:39, Tim ignored_mailbox@yahoo.com.au wrote:
Allegedly, on or about 18 December 2015, Fernando Cassia sent:
Thanks for the reply. I choose to stop using so-called "persistent storage" because it eventually goes FUBAR. Something about the loop device if I remember correctly. The issue being that all is well until you fill up all available storage space. Then the "persistent storage" partition becomes "damaged" and unmountable. I lost quite a few folders full of work that way, that's when I decided that was useless.
That sounds like a problem I had on my old Amiga, more than a decade ago, where due to some programmer's maths errors, it'd try to fill a disc to 101% capacity.
Though, in this case, I suspect it's more to do with flash drives losing capacity, over time, as it wears out, and no file system being developed in mind of this (so that a reasonable large amount of space is always required to be kept free).
It's neither of these, https://fedoraproject.org/wiki/How_to_create_and_use_Live_USB#Data_persisten...
'currently implemented (as a Device-mapper copy-on-write snapshot), every single change to it (writes AND deletes) subtracts from its free space, so it will eventually be "used up"'
This is not the same as choosing to create a live image with an accompanying filesystem on a USB key, where the extra mounted filesystem is an ext4 or similar. But system files in a live image are located in the squashfs and are subject to the overlay limits.
On Sun, Dec 20, 2015 at 6:35 AM, Ian Malone ibmalone@gmail.com wrote:
It's neither of these,
https://fedoraproject.org/wiki/How_to_create_and_use_Live_USB#Data_persisten...
'currently implemented (as a Device-mapper copy-on-write snapshot), every single change to it (writes AND deletes) subtracts from its free space, so it will eventually be "used up"'
It confirms what I said: it's broken. How about changing the LiveCD creator to allow for "persistent storage" partition that is formatted with F2FS?
FC
On Sun, Dec 20, 2015 at 3:31 PM, Fernando Cassia fcassia@gmail.com wrote:
On Sun, Dec 20, 2015 at 6:35 AM, Ian Malone ibmalone@gmail.com wrote:
It's neither of these,
https://fedoraproject.org/wiki/How_to_create_and_use_Live_USB#Data_persisten...
'currently implemented (as a Device-mapper copy-on-write snapshot), every single change to it (writes AND deletes) subtracts from its free space, so it will eventually be "used up"'
It confirms what I said: it's broken.
It's not broken. It's working as designed and documented. From livecd-iso-to-disk --help
*Note well* that deletion of any original files in the read-only root filesystem does not recover any storage space on your LiveOS device. Storage in the persistent /LiveOS/overlay-<device_id> file is allocated as needed, but the system will crash *without warning* and fail to boot once the overlay has been totally consumed. If significant changes or updates to the root filesystem are to be made, carefully watch the fraction of space allocated in the overlay by issuing the 'dmsetup status' command at a command line of the running LiveOS image. Some consumption of root filesystem and overlay space can be avoided by specifying a persistent home filesystem for user files, see --home-size-mb below.
How about changing the LiveCD creator to allow for "persistent storage" partition that is formatted with F2FS?
Someone needs to volunteer to do the research why that's a better option, and include some patches and kickstart scripts so that others can test. F2FS does not produce better results just by using it. It's highly customizeable/tunable, and it assumes you know things about your flash based drive that it can't know (because manufacturer's hide this information) so you can tune it. If you don't tune it, you can get worse results than just using ext4/XFS/Btrfs or heck even FAT or NTFS because the FTL in especially USB flash storage is very well tuned for FAT.
Agree with Chris.
Cheers, Sylvia
On 20/12/2015, Chris Murphy lists@colorremedies.com wrote:
On Sun, Dec 20, 2015 at 3:31 PM, Fernando Cassia fcassia@gmail.com wrote:
On Sun, Dec 20, 2015 at 6:35 AM, Ian Malone ibmalone@gmail.com wrote:
It's neither of these,
https://fedoraproject.org/wiki/How_to_create_and_use_Live_USB#Data_persisten...
'currently implemented (as a Device-mapper copy-on-write snapshot), every single change to it (writes AND deletes) subtracts from its free space, so it will eventually be "used up"'
It confirms what I said: it's broken.
It's not broken. It's working as designed and documented. From livecd-iso-to-disk --help
*Note well* that deletion of any original files in the read-only root filesystem does not recover any storage space on your LiveOS device. Storage in the persistent /LiveOS/overlay-<device_id> file is allocated as needed, but the system will crash *without warning* and fail to boot once the overlay has been totally consumed. If significant changes or updates to the root filesystem are to be made, carefully watch the fraction of space allocated in the overlay by issuing the 'dmsetup status' command at a command line of the running LiveOS image. Some consumption of root filesystem and overlay space can be avoided by specifying a persistent home filesystem for user files, see --home-size-mb below.
How about changing the LiveCD creator to allow for "persistent storage" partition that is formatted with F2FS?
Someone needs to volunteer to do the research why that's a better option, and include some patches and kickstart scripts so that others can test. F2FS does not produce better results just by using it. It's highly customizeable/tunable, and it assumes you know things about your flash based drive that it can't know (because manufacturer's hide this information) so you can tune it. If you don't tune it, you can get worse results than just using ext4/XFS/Btrfs or heck even FAT or NTFS because the FTL in especially USB flash storage is very well tuned for FAT.
-- Chris Murphy -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
On Mon, Dec 21, 2015 at 10:34 AM, Sylvia Sánchez lailahfsf@gmail.com wrote:
Agree with Chris. Cheers, Sylvia
Thanks for letting us know your thoughts, Sylv.
FC
On 21 December 2015 at 01:17, Chris Murphy lists@colorremedies.com wrote:
On Sun, Dec 20, 2015 at 3:31 PM, Fernando Cassia fcassia@gmail.com wrote:
On Sun, Dec 20, 2015 at 6:35 AM, Ian Malone ibmalone@gmail.com wrote:
It's neither of these,
https://fedoraproject.org/wiki/How_to_create_and_use_Live_USB#Data_persisten...
'currently implemented (as a Device-mapper copy-on-write snapshot), every single change to it (writes AND deletes) subtracts from its free space, so it will eventually be "used up"'
It confirms what I said: it's broken.
It's not broken. It's working as designed and documented. From livecd-iso-to-disk --help
*Note well* that deletion of any original files in the read-only root filesystem does not recover any storage space on your LiveOS device. Storage in the persistent /LiveOS/overlay-<device_id> file is allocated as needed, but the system will crash *without warning* and fail to boot once the overlay has been totally consumed. If significant changes or updates to the root filesystem are to be made, carefully watch the fraction of space allocated in the overlay by issuing the 'dmsetup status' command at a command line of the running LiveOS image. Some consumption of root filesystem and overlay space can be avoided by specifying a persistent home filesystem for user files, see --home-size-mb below.
How about changing the LiveCD creator to allow for "persistent storage" partition that is formatted with F2FS?
Someone needs to volunteer to do the research why that's a better option, and include some patches and kickstart scripts so that others can test. F2FS does not produce better results just by using it. It's highly customizeable/tunable, and it assumes you know things about your flash based drive that it can't know (because manufacturer's hide this information) so you can tune it. If you don't tune it, you can get worse results than just using ext4/XFS/Btrfs or heck even FAT or NTFS because the FTL in especially USB flash storage is very well tuned for FAT.
More than that, the live images are built on squashfs as a compressed filesystem (would direct Fernando to https://en.wikipedia.org/wiki/SquashFS), which is where the requirement to use an overlay to allow persistent storage on root comes from. There is the option (as I mentioned, removed from the quote of course) to use a persistent storage image on a normal filesystem alongside it for home (and whether this is ext4 or f2fs has nothing to do with the price of fish), but that cannot be done for the squashfs. So far as I know, you would need to rebuild the image when creating the live device to allow changing the root filesystem.
If you want a live image with certain packages installed it's not too hard to roll your own, just create a kickstart file that includes whatever base kickstart you want and add packages.
On Mon, Dec 21, 2015 at 8:11 PM, Ian Malone ibmalone@gmail.com wrote:
More than that, the live images are built on squashfs as a compressed filesystem (would direct Fernando to https://en.wikipedia.org/wiki/SquashFS), which is where the requirement to use an overlay to allow persistent storage on root comes from. There is the option (as I mentioned, removed from the quote of course) to use a persistent storage image on a normal filesystem alongside it for home (and whether this is ext4 or f2fs has nothing to do with the price of fish), but that cannot be done for the squashfs. So far as I know, you would need to rebuild the image when creating the live device to allow changing the root filesystem.
If you want a live image with certain packages installed it's not too hard to roll your own, just create a kickstart file that includes whatever base kickstart you want and add packages.
I don't follow why not just install to a 2nd USB stick normally, and use that stick to boot from on various computers. The live is for rather short duration usage, like evaluation, installation, and troubleshooting. It's not exactly well suited for being fully updated, e.g. you can't update the kernel or the initramfs. So the ability to keep the system really up to date is inherently limited with the overlay, and it's slower using squashfs+overlay than any of the other available options.
Happy solstice.
On Fri, Dec 18, 2015 at 11:53:09AM -0500, Fernando Cassia wrote:
Since I have learned that the Fedora community is quite hostile to any kind of complaint ("it's as-is because we-know-better and
Well, I admit to being a little hostile to _this_ kind of complaint, which is pretty hard to even respond to in a helpful way because it's so full of unsubstantiated aggressive statements. (Starting right in the subject.)
If you'd start instead with "Hey, people are mostly putting these on bigger USB drives these days, so could we raise the size limit and include more useful stuff"?, you _might_ get better results.
On the other hand, you might _not_, because your use case for the live USB is not necessarily everyone's, and in fact it certainly is not the main reason we produce these images today.
The primary thing is: the main point of the live CD is to provide an easy environment for installs of the Workstation edition and the various desktop spins. Not much work has been put into making it really useful as a thing in itself beyond a tool to use to make sure everything works with your hardware and as a convenience in a pinch. Lots of the functionality — like the overlay support — to do more than that has bitrotted because no one who cares has worked on it.
If you care, or know someone who cares, _that's_ how you can improve the situation.
On Fri, Dec 18, 2015 at 12:57 PM, Matthew Miller mattdm@fedoraproject.org wrote:
Well, I admit to being a little hostile to _this_ kind of complaint, which is pretty hard to even respond to in a helpful way because it's so full of unsubstantiated aggressive statements. (Starting right in the subject.)
If you'd start instead with "Hey, people are mostly putting these on bigger USB drives these days, so could we raise the size limit and include more useful stuff"?, you _might_ get better results.
Hi Matthew,
I must learn to write rants in a more constructive style, I give you that point. I do in in the spirit that some day developers learn to ask users before making drastic changes like removing stuff.
In my view, every such change should first ask two questions: -What will break by removing this stuff? -Are we annoying users by removing this? -Have we asked them for feedback?.
My view from this side is that often the devs just do the changes on their small echo chamber and don't even consider the above points.
:-[ FC
On Fri, Dec 18, 2015 at 12:57 PM, Matthew Miller mattdm@fedoraproject.org wrote:
On the other hand, you might _not_, because your use case for the live USB is not necessarily everyone's, and in fact it certainly is not the main reason we produce these images today.
The issue will go away the day I can boot a LiveCD image and install Fedora on another USB flash drive using Samsung F2FS. But as far as I know that is still way off, even Ubuntu still doesn't support it, and that is a real pity because F2FS has been out for the last three years or more.
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1261175
FC
On 12/18/2015 10:38 AM, Fernando Cassia wrote:
The issue will go away the day I can boot a LiveCD image and install Fedora on another USB flash drive using Samsung F2FS. But as far as I know that is still way off, even Ubuntu still doesn't support it, and that is a real pity because F2FS has been out for the last three years or more.
I've been told that you can use a Live image on one USB to make a regular installation on another USB, but I've never tried it.
On 12/18/2015 10:38 AM, Fernando Cassia wrote:
On Fri, Dec 18, 2015 at 12:57 PM, Matthew Miller <mattdm@fedoraproject.org mailto:mattdm@fedoraproject.org> wrote:
On the other hand, you might _not_, because your use case for the live USB is not necessarily everyone's, and in fact it certainly is not the main reason we produce these images today.The issue will go away the day I can boot a LiveCD image and install Fedora on another USB flash drive using Samsung F2FS. But as far as I know that is still way off, even Ubuntu still doesn't support it, and that is a real pity because F2FS has been out for the last three years or more.
I don't know about the LiveCD images, but F2FS is part of the kernel and has been for quite a while. I'm running F22 and it's in there:
[root@prophead ~]# ls /lib/modules/`uname -r`/kernel/fs | grep f2fs f2fs
---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 226437340 Yahoo: origrps2 - - - - Microsoft Windows: Proof that P.T. Barnum was right - ----------------------------------------------------------------------
On Fri, Dec 18, 2015 at 2:44 PM, Rick Stevens ricks@alldigital.com wrote:
I don't know about the LiveCD images, but F2FS is part of the kernel and has been for quite a while. I'm running F22 and it's in there:
[root@prophead ~]# ls /lib/modules/`uname -r`/kernel/fs | grep f2fs f2fs
If I remember correctly, Anaconda didnt support installing to F2FS partitions - or partitioning F2FS in the first place.
But that must have been at around F21 or F22. Will have to check now...
FC
On Fri, Dec 18, 2015 at 3:06 PM, Fernando Cassia fcassia@gmail.com wrote:
If I remember correctly, Anaconda didnt support installing to F2FS partitions - or partitioning F2FS in the first place.
This was my exchange, two years ago. https://lists.fedoraproject.org/pipermail/users/2013-October/441707.html
FC
Well, before writing a rant you should check that your facts are up to date. You see, F2FS is supported and you didn't know.
Cheers, Sylvia
On 18/12/2015, Fernando Cassia fcassia@gmail.com wrote:
On Fri, Dec 18, 2015 at 2:44 PM, Rick Stevens ricks@alldigital.com wrote:
I don't know about the LiveCD images, but F2FS is part of the kernel and has been for quite a while. I'm running F22 and it's in there:
[root@prophead ~]# ls /lib/modules/`uname -r`/kernel/fs | grep f2fs f2fs
If I remember correctly, Anaconda didnt support installing to F2FS partitions - or partitioning F2FS in the first place.
But that must have been at around F21 or F22. Will have to check now...
FC
-- During times of Universal Deceit, telling the truth becomes a revolutionary act Durante épocas de Engaño Universal, decir la verdad se convierte en un Acto Revolucionario
- George Orwell
On Fri, Dec 18, 2015 at 3:51 PM, Sylvia Sánchez lailahfsf@gmail.com wrote:
Well, before writing a rant you should check that your facts are up to date. You see, F2FS is supported and you didn't know.
F2FS is supported in the kernel. Is is supported by Anaconda as an installation target/option?
My rant, if you re-read it, was about the removal of binaries that were previously included, from the LiveCD.
Please re-read it. Thanks. FC
Allegedly, on or about 18 December 2015, Matthew Miller sent:
The primary thing is: the main point of the live CD is to provide an easy environment for installs of the Workstation edition and the various desktop spins. Not much work has been put into making it really useful as a thing in itself beyond a tool to use to make sure everything works with your hardware and as a convenience in a pinch.
I thought the *main* point of a live <anything>, was to make a runnable OS without doing any installation.
I can't say that I like doing installs from them, they're such a dead slow thing to boot up, and run. And then they give you little to no installation options. You may as well just make a bootable command line environment disc, that just installs without asking you questions, it'd run faster and easier. That'd certainly be more useful for labs who'd like to just "insert and boot," to set up multiple systems, and don't want to sit at each terminal answering questions, or don't have something available to automate it over a network.
I always prefer the old style install disc, where you get a basic graphical environment to set up your disc drives, then perhaps some options about your installation, then just let it install.
I've tried package selection at install time, in the past, but it'd often bomb out, usually took ages, or it'd reselect things I'd deselected, that I didn't want or need, because I didn't know, and couldn't tell, that they were requirements.
-----Original Message----- From: fcassia@gmail.com Sent: Fri, 18 Dec 2015 11:53:09 -0500 To: users@lists.fedoraproject.org Subject: The continuous crippling of Fedora LiveCDs by removing usefull stuff for no apparent reason
Once upon a time, I believe it was Fedora 17 or Fedora 19, I was a happy camper.
That was because OpenJDK 7 was included in the LiveCD image.
Suddenly (I don't remember which release, but you can check the mailing list archives for my rant on the matter) OpenJDK binaries were removed from the live CD so I could no longer "java -jar JavaImageEditor.jar" nor could I use JNLP to launch desktop Java apps.
I came to the list at the time to rant, and if I remember correctly the answer was that OpenJDK was removed from the LiveCD due to space constraints, and that it was a real challenge to keep everything to still fit into a CD image.
On Fri, Dec 18, 2015 at 5:51 PM, Antonio Olivares wingators@inbox.com wrote:
A bigger Specialized spin called Scientific has OpenJDK :) Since nowadays we still need a DVD since nothing fits on a CD anymore :(
https://labs.fedoraproject.org/en/scientific/
It should contain JAVA as it is listed here:
Thanks Antonio.
Will check it out. Wasn't aware of such Scientific spin. Does the "labs" subdomain mean it is somewhat different from other Fedora respins?
Since I run this on a netbook with low resources (Intel Atom, 32-bit CPU, only 2 GB RAM), I'd prefer if I could get openJDK retrofitted into the somewhat lightweight Mate-Compiz spin. (I tried Fedora XFCE but hate its browser, Midori).
FC
On Fri, Dec 18, 2015 at 05:58:55PM -0500, Fernando Cassia wrote:
Will check it out. Wasn't aware of such Scientific spin. Does the "labs" subdomain mean it is somewhat different from other Fedora respins?
We've split it so the "spins" are focused on presenting desktop environments (KDE, Xfce, etc.), while "labs" are collections of software for a given purpose.
Since I run this on a netbook with low resources (Intel Atom, 32-bit CPU, only 2 GB RAM), I'd prefer if I could get openJDK retrofitted into the somewhat lightweight Mate-Compiz spin. (I tried Fedora XFCE but hate its browser, Midori).
With these needs, it seems like you're really _way_ into "making your own custom Fedora Remix". See https://fedoraproject.org/wiki/Remix
-
Antonio Olivares -
Chris Murphy -
Fernando Cassia -
Ian Malone -
Joe Zeff -
Matthew Miller -
Rick Stevens -
Sylvia Sánchez -
Tim -
Tom Horsley