"install devtools" in F22 live boot, device-mapper implosion

[Chris Murphy]

User over on users@ list reports booting Fedora 22 Workstation (live
media), and doing:

# dnf install "Developer Tools"

Implodes.

I've reproduced the problem, and the gist is that there's a dm
snapshot that's too small for this task, it gets full quickly, the
file system face plants, and the kernel oopses. No recovery is
possible. Long version [1].

Question 1: Should this work better and be a supported use case?

Question 2: If so how to fix it? Change the configuration for a bigger
snapshot? Or use thin provisioning snapshot? Or overlayfs?

I haven't tested livecd-iso-to-disk --overlay-size-mb option, but I
give it better than even odds that for the baremetal case it will work
and might be a suitable work around. Thoughts?



[1]

If a bug should be filed, let me know what to file it against. I have
no idea what's responsible for the configuration of the install
media's live environment itself.

# dmsetup status
live-base: 0 12582912 linear
live-osimg-min: 0 12582912 snapshot 3664/3664 24
live-rw: 0 12582912 snapshot 230544/1048576 912

# dnf group install "Development Tools"

## I allow metadata to download. Total metadata downloaded 41MB
(stable branch) and 13MB (updates branch).

# dmsetup status
[root@localhost ~]# dmsetup status
live-base: 0 12582912 linear
live-osimg-min: 0 12582912 snapshot 3664/3664 24
live-rw: 0 12582912 snapshot 484016/1048576 1896

So the snapshot is filling up. Continue with download and install of
packages, while continue to check dmsetup status and eventually I get
to this:

[root@localhost ~]# dmsetup status
live-base: 0 12582912 linear
live-osimg-min: 0 12582912 snapshot 3664/3664 24
live-rw: 0 12582912 snapshot 691048/1048576 2704
[root@localhost ~]# dmsetup status
live-base: 0 12582912 linear
live-osimg-min: 0 12582912 snapshot 3664/3664 24
live-rw: 0 12582912 snapshot 703536/1048576 2752
[root@localhost ~]# dmsetup status
live-base: 0 12582912 linear
live-osimg-min: 0 12582912 snapshot 3664/3664 24
live-rw: 0 12582912 snapshot 716320/1048576 2800
[root@localhost ~]# dmsetup status
live-base: 0 12582912 linear
live-osimg-min: 0 12582912 snapshot 3664/3664 24
live-rw: 0 12582912 snapshot 716400/1048576 2800
[root@localhost ~]# dmsetup status
live-base: 0 12582912 linear
live-osimg-min: 0 12582912 snapshot 3664/3664 24
live-rw: 0 12582912 snapshot 716464/1048576 2800
[root@localhost ~]# dmsetup status
live-base: 0 12582912 linear
live-osimg-min: 0 12582912 snapshot 3664/3664 24
live-rw: 0 12582912 snapshot Invalid
[root@localhost ~]# dmesg
-bash: /usr/bin/dmesg: Input/output error

Separately I had a journalctl -f running and it captures

[  928.709182] localhost kernel: device-mapper: snapshots:
Invalidating snapshot: Unable to allocate exception.

And then a long pile of ext4 errors.



-- 
Chris Murphy

[Adam Williamson]

On Thu, 2015-07-30 at 21:11 -0600, Chris Murphy wrote:
>  User over on users@ list reports booting Fedora 22 Workstation (live
>  media), and doing:
>  
>  # dnf install "Developer Tools"
>  
>  Implodes.
>  
>  I've reproduced the problem, and the gist is that there's a dm
>  snapshot that's too small for this task, it gets full quickly, the
>  file system face plants, and the kernel oopses. No recovery is
>  possible. Long version [1].
>  
>  Question 1: Should this work better and be a supported use case? 
My understanding has always been that installing packages on the live
media is an unsupported, 'if you're lucky it might work for a few small
ones' kind of thing.

If someone wants to try and make it work better that's great, of
course.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Fabian Deutsch]

On Fri, Jul 31, 2015 at 5:28 AM, Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  On Thu, 2015-07-30 at 21:11 -0600, Chris Murphy wrote:
> > User over on users@ list reports booting Fedora 22 Workstation (live
> > media), and doing:
> >
> > # dnf install "Developer Tools"
> >
> > Implodes.
> >
> > I've reproduced the problem, and the gist is that there's a dm
> > snapshot that's too small for this task, it gets full quickly, the
> > file system face plants, and the kernel oopses. No recovery is
> > possible. Long version [1].
> >
> > Question 1: Should this work better and be a supported use case?
>
>  My understanding has always been that installing packages on the live
>  media is an unsupported, 'if you're lucky it might work for a few small
>  ones' kind of thing.
>
>  If someone wants to try and make it work better that's great, of
>  course. 
We could try to also use rd.live.fsimg=1 in addition to root=live:.
This will not give us more RAM, but can help sometimes.

IIRC root=live: will create loop mounts for the squashfs and create a
dm-snapshot to provide a writable fs.
The problem with this method is that it does not free memory after it
has been claimed. That means the memory consumption is determined by
the number of changes, not by the actual requirements.
In other words: Creating a 500MB file will acquire 500MB more, but
deleting it does not give that memory back.
This behavior can slowly eat up the memory over time.

If rd.live.fsimg=1 is used in addition, then the fs within the
squashfs will be copied to a tmpfs and going from there.
The benefit of this method is, that unused memory will be given back
to the system.
When we take the example from above, then the 500MB will be freed
again, once the file got removed.
The drawback is that you need enough memory to at least hold the
complete uncompressed liveimg.

- fabian

-- 
Fabian Deutsch <fdeutsch(a)redhat.com&gt;
RHEV Hypervisor
Red Hat