On Fri, Mar 8, 2013 at 6:55 PM, Frederick Grose <span dir="ltr"><<a href="mailto:fgrose@gmail.com" target="_blank">fgrose@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div><font color="#000099"><font><font face="trebuchet ms,sans-serif"></font></font></font>On Thu, Mar 7, 2013 at 10:41 AM, <span dir="ltr"><<a href="mailto:John.Florian@dart.biz" target="_blank">John.Florian@dart.biz</a>></span> wrote:<br>
</div></div><div class="gmail_quote"><div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><tt><font>> From: Frederick Grose <<a href="mailto:fgrose@gmail.com" target="_blank">fgrose@gmail.com</a>></font></tt>
<br><div><tt><font>> On Wed, Mar 6, 2013 at 3:59 PM, <<a href="mailto:John.Florian@dart.biz" target="_blank">John.Florian@dart.biz</a>>
wrote:</font></tt>
<br></div><tt><font><snip></font></tt>
<br><div><div><tt><font>> root@aos-61:46 # # Lets now make it all go wonky:
<br>
> root@aos-61:46 # time dd if=/dev/zero of=/foo <br>
> Bus error <br>
> <br>
> real 1m15.775s <br>
> user 0m2.818s <br>
> sys 0m24.129s <br>
> root@aos-61:46 # <br>
> root@aos-61:46 # ls /root <br>
> -bash: /bin/ls: Input/output error <br>
> root@aos-61:46 # df -h <br>
> -bash: /usr/bin/df: Input/output error
<br>
>
<br>
> root@aos-61:46 # mount
<br>
>
<br>
> -bash: /usr/bin/mount: Input/output error
<br>
>
<br>
> root@aos-61:46 # cat /proc/meminfo
<br>
>
<br>
> -bash: /usr/bin/cat: Input/output error
<br>
>
<br>
> <br>
> Is this expected? Is there anything I can do, e.g., configuration-<br>
> wise, that can prevent this? Ideally this would fail much like
any <br>
> other full disk situation. I understand that the overlay consumes
<br>
> space, i.e., memory, for this file growth, including file removals,
<br>
> but I'd at least like to be able to remotely reboot a system when
in<br>
> this state, however I can't even do that because the reboot command
<br>
> will either return the same I/O error or it may succeed but get the
<br>
> I/O error when systemd tries to read /usr/lib/systemd/system/reboot.target.
<br>
> <br>
> I dug around in bugzilla, but found nothing there. I can file
a <br>
> bug, but which package is likely at fault here?<br>
> --<br>
> John Florian</font></tt>
<br><tt><font>> <br>
> See <a href="https://fedoraproject.org/wiki/LiveOS_image" target="_blank">https://fedoraproject.org/wiki/LiveOS_image</a> for some
background <br>
> and potential workarounds.</font></tt>
<br></div></div><tt><font>> <br>
> --Fred --</font></tt>
<br>
<br>
<br><tt><font>There's really not much on that page that helps me
here. I'm trying to use Live images for a mostly-stateless embedded
appliance OS deployed to hundreds or thousands of devices. I realize
that the COW design is always going to be limited, but a more graceful
failure mode is really needed, somehow. For our use, the biggest
gain in stability here actually comes from systemd's journal with its trim-before-write
approach instead of the legacy write now, trim asynchronously approach
we used to have. However, that only covers one specific use case:
logging. Writing to proper persistent storage allows me to avoid
the root file system overlay, but most of these embedded devices use CF
or SD cards for storage, which have limited write cycles that must be respected.</font></tt>
<br>
<br><tt><font>Is there a way to implement an artificial capacity
limit that would prevent processes from exhausting the overlay so that
the reserve might be used for recording the event and rebooting back to
a safer state?</font></tt>
<br>
<br><tt><font>At the very least, I think this page could benefit
from a little stronger, more explicit wording of this failure case. While
it talks a little about some work-arounds, it actually says very little
about why they are needed. Only in the "Overlay Recovery"
section does it hint at the crash potential.</font></tt>
<br><font face="sans-serif"><br>
--<br>
John Florian</font><br></blockquote></div></div><div><br>Thank you for the review! I've updated the wiki page based on your comments,<br><a href="https://fedoraproject.org/wiki/LiveOS_image" target="_blank">https://fedoraproject.org/wiki/LiveOS_image</a><br>
<br>Documenting that a temporary overlay is a 0.5 GiB sparse file in a RAM filesystem gave me the idea to try using an overlay size greater than available memory, and hope that kernel out-of-memory warnings would intervene before the device-mapper filesystem invalidation.<br>
<br>I modified <span style="font-family:courier new,monospace">/usr/sbin/dmsquash-live-root</span> in the initramfs to create a temporary 500 GiB sparse overlay:<br><br><span style="font-family:courier new,monospace">dd if=/dev/null of=/overlay bs=1024 count=1 seek=$((512*1024*1024)) 2> /dev/null</span><br>
<br>Then after booting an updated, Fedora 18 Live desktop, LiveUSB read only and running your failure demo,<br><br><span style="font-family:courier new,monospace"><font>time dd if=/dev/zero of=/foo</font></span><br><br>I got out-of-memory warnings after a file of about 450 MiB was written and the command returned--no crash!<br>
<br>Some post test output:<br><br><span style="font-family:courier new,monospace">[root@localhost ~]# dmsetup status<br>live-osimg-min: 0 8388608 snapshot 2584/2584 24<br>live-rw: 0 8388608 snapshot 921720/1073741824 3600<br>
</span><br><span style="font-family:courier new,monospace">top - 18:11:53 up 17 min, 3 users, load average: 0.68, 0.75, 0.57<br>Tasks: 182 total, 2 running, 180 sleeping, 0 stopped, 0 zombie<br>%Cpu(s): 1.6 us, 1.6 sy, 0.0 ni, 96.5 id, 0.0 wa, 0.2 hi, 0.0 si, 0.0 st<br>
KiB Mem: 3339812 total, 3260284 used, 79528 free, 316384 buffers<br>KiB Swap: 3341308 total, 0 used, 3341308 free, 1948108 cached</span><br><br>You might test this method in your systems and let us know how it works.<br>
<br> --Fred<br></div></div></blockquote><div><br></div><div>Pardon my bad observations, my above conclusion IS WRONG and unsupported by the above test.</div><div><br></div><div>I deceived myself with an unfamiliar error message, and actually seem to have tested James Heather's method in my last test.</div>
<div><br></div><div>My root filesystem size was 4 GiB with about 450 MiB free. An out-of-disc-space warning is what actually popped up and caused the test command to exit before another failure or crash.</div><div><br></div>
<div>To retest the oversized overlay hypothesis, I resized the LiveUSB root filesystem to 12 GiB and repeated the test on it as an attached LiveOS filesystem, <font face="courier new, monospace">/dev/mapper/dm-PCBV6p </font>(mounted at <font face="courier new, monospace">/mnt/a</font>).</div>
<div><br></div><div><div><font face="courier new, monospace">[root@localhost a]# dmsetup status</font></div>
<div><font face="courier new, monospace">dm-PCBV6p</font><span style="font-family:'courier new',monospace">: 0 25165824 snapshot 24/1073741824 16</span></div><div><font face="courier new, monospace"><br></font></div>
<div><font face="courier new, monospace"><div>[root@localhost ~]# dmsetup table</div><div>dm-PCBV6p: 0 25165824 snapshot 7:9 7:10 P 8</div></font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><div>
[root@localhost ~]# losetup -a</div><div>/dev/loop8: [2081]:1833 (/run/media/fgrose/LIVE/LiveOS/squashfs.img)</div><div>/dev/loop9: [1800]:3 (/run/media/livemnt-squash-mRJNIA/LiveOS/rootfs.img)</div><div>/dev/loop10: [0017]:58601 (/run/media/tmpvJjuX7)</div>
<div>/dev/loop11: [2081]:1832 (/run/media/fgrose/LIVE/LiveOS/home.img)</div><div><br></div></font></div><div><div><font face="courier new, monospace">[root@localhost ~]# df -Th</font></div><div><font face="courier new, monospace">Filesystem Type Size Used Avail Use% Mounted on</font></div>
<div><font face="courier new, monospace">devtmpfs devtmpfs 1.6G 0 1.6G 0% /dev</font></div><div><font face="courier new, monospace">tmpfs tmpfs 1.6G 152K 1.6G 1% /dev/shm</font></div>
<div><font face="courier new, monospace">tmpfs tmpfs 1.6G 3.3M 1.6G 1% /run</font></div><div><font face="courier new, monospace">tmpfs tmpfs 1.6G 0 1.6G 0% /sys/fs/cgroup</font></div>
<div><font face="courier new, monospace">/dev/sda1 ext4 18G 8.8G 7.7G 54% /</font></div><div><font face="courier new, monospace">tmpfs tmpfs 1.6G 28K 1.6G 1% /tmp</font></div>
<div><span style="font-family:'courier new',monospace">/dev/sdc1 vfat 15G 8.8G 6.2G 59% /run/media/fgrose/LIVE</span></div><div><font face="courier new, monospace">/dev/loop8 squashfs 929M 929M 0 100% /run/media/livemnt-squash</font><span style="font-family:'courier new',monospace">-mRJNIA</span></div>
<div><font face="courier new, monospace">/dev/mapper/dm-PCBV6p</font><span style="font-family:'courier new',monospace"> ext4 12G 3.4G 8.2G 30% /mnt/a</span></div><div><font face="courier new, monospace">/dev/loop11 ext4 380M 35M 325M 10% /mnt/a/home</font></div>
</div><div><br></div><div><div style="font-family:'courier new',monospace">[root@localhost ~]# mount</div><div style="font-family:'courier new',monospace">proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)</div>
<div style="font-family:'courier new',monospace">sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)</div><div style="font-family:'courier new',monospace">devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=1652304k,nr_inodes=413076,mode=755)</div>
<div style="font-family:'courier new',monospace">securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)</div><div style="font-family:'courier new',monospace">selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)</div>
<div style="font-family:'courier new',monospace">tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)</div><div style="font-family:'courier new',monospace">devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)</div>
<div style="font-family:'courier new',monospace">tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)</div><div style="font-family:'courier new',monospace">tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,seclabel,mode=755)</div>
<div style="font-family:'courier new',monospace">cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)</div><div style="font-family:'courier new',monospace">
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)</div><div style="font-family:'courier new',monospace">cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)</div>
<div style="font-family:'courier new',monospace">cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)</div><div style="font-family:'courier new',monospace">cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)</div>
<div style="font-family:'courier new',monospace">cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)</div><div style="font-family:'courier new',monospace">cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)</div>
<div style="font-family:'courier new',monospace">cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)</div><div style="font-family:'courier new',monospace">cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)</div>
<div style="font-family:'courier new',monospace">/dev/sda1 on / type ext4 (rw,relatime,seclabel,data=ordered)</div><div style="font-family:'courier new',monospace">rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)</div>
<div style="font-family:'courier new',monospace">systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=37,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)</div><div style="font-family:'courier new',monospace">
configfs on /sys/kernel/config type configfs (rw,relatime)</div><div style="font-family:'courier new',monospace">mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)</div><div style="font-family:'courier new',monospace">
debugfs on /sys/kernel/debug type debugfs (rw,relatime)</div><div style="font-family:'courier new',monospace">tmpfs on /tmp type tmpfs (rw,seclabel)</div><div style="font-family:'courier new',monospace">hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)</div>
<div style="font-family:'courier new',monospace">gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)</div><div style="font-family:'courier new',monospace">
/dev/sdc1 on /run/media/fgrose/LIVE type vfat (rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2)</div><div style="font-family:'courier new',monospace">
/dev/sdb2 on /var/cache/yum type ext4 (rw,relatime,seclabel,data=ordered)</div><div style="font-family:'courier new',monospace">/dev/loop8 on /run/media/livemnt-squash-mRJNIA type squashfs (ro,relatime,seclabel)</div>
<div style="font-family:'courier new',monospace">/dev/mapper/dm-PCBV6p on /mnt/a type ext4 (rw,relatime,seclabel,data=ordered)</div><div style="font-family:'courier new',monospace">/dev/loop11 on /mnt/a/home type ext4 (rw,relatime,seclabel,data=ordered)</div>
<div style="font-family:'courier new',monospace">/dev/sdc1 on /mnt/a/run/initramfs/live type vfat (rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,showexec,utf8,flush,errors=remount-ro)</div>
<div style="font-family:'courier new',monospace"><br></div><div style="font-family:'courier new',monospace"><br></div><div><font face="arial, helvetica, sans-serif">The target filesystem, </font><font face="courier new, monospace">/dev/mapper/dm-PCBV6p</font><font face="arial, helvetica, sans-serif">, did go invalid, was changed to ro, which led to this test command output:</font></div>
</div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="courier new, monospace">[root@localhost a]# time dd if=/dev/zero of=foo</font></div>
<div><font face="courier new, monospace">dd: writing to ‘foo’: Read-only file system</font></div><div><font face="courier new, monospace">4029694+0 records in</font></div><div><font face="courier new, monospace">4029693+0 records out</font></div>
<div><font face="courier new, monospace"><a href="tel:2063202816" value="+12063202816" target="_blank">2063202816</a> bytes (2.1 GB) copied, 40.0696 s, 51.5 MB/s</font></div><div><font face="courier new, monospace"><br></font></div>
<div><font face="courier new, monospace">real<span style="white-space:pre-wrap"> </span>0m40.079s</font></div>
<div><font face="courier new, monospace">user<span style="white-space:pre-wrap"> </span>0m3.799s</font></div><div><font face="courier new, monospace">sys<span style="white-space:pre-wrap"> </span>0m32.422s</font></div>
</div><div><br></div><div>In a separate terminal I manually monitored the dmsetup status:</div><div><br></div><div><div><div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div><div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 184/1073741824 16</font></div>
<div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div><div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 192/1073741824 16</font></div><div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div>
<div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 192/1073741824 16</font></div><div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div><div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 263440/1073741824 1040</font></div>
<div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div><div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 366360/1073741824 1440</font></div><div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div>
<div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 526608/1073741824 2064</font></div><div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div><div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 703280/1073741824 2752</font></div>
<div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div><div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 904600/1073741824 3528</font></div><div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div>
<div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 1131568/1073741824 4416</font></div><div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div><div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 1383288/1073741824 5392</font></div>
<div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div><div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 1579432/1073741824 6160</font></div><div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div>
<div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 1579432/1073741824 6160</font></div><div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div><div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 1731568/1073741824 6752</font></div>
<div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div><div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 2191040/1073741824 8536</font></div><div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div>
<div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 2420840/1073741824 9432</font></div><div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div><div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 2632232/1073741824 10256</font></div>
<div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div><div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 2632288/1073741824 10256</font></div><div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div>
<div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 2964616/1073741824 11544</font></div><div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div><div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot 3208632/1073741824 12496</font></div>
<div><font face="courier new, monospace">[root@localhost ~]# dmsetup status</font></div><div><font face="courier new, monospace">dm-PCBV6p: 0 25165824 snapshot Invalid</font></div></div><div><br></div><div><font face="courier new, monospace">[root@localhost ~]# df -Th</font></div>
<div><font face="courier new, monospace">Filesystem Type Size Used Avail Use% Mounted on</font></div>
<div><font face="courier new, monospace">devtmpfs devtmpfs 1.6G 0 1.6G 0% /dev</font></div><div><font face="courier new, monospace">tmpfs tmpfs 1.6G 504K 1.6G 1% /dev/shm</font></div>
<div><font face="courier new, monospace">tmpfs tmpfs 1.6G 1000M 632M 62% /run</font></div><div><font face="courier new, monospace">tmpfs tmpfs 1.6G 0 1.6G 0% /sys/fs/cgroup</font></div>
<div><font face="courier new, monospace">/dev/sda1 ext4 18G 8.7G 7.7G 54% /</font></div><div><font face="courier new, monospace">tmpfs tmpfs 1.6G 68K 1.6G 1% /tmp</font></div>
<div><span style="font-family:'courier new',monospace">/dev/sdc1 vfat 15G 8.8G 6.2G 59% /run/media/fgrose/LIVE</span></div><div><font face="courier new, monospace">/dev/loop8 squashfs 929M 929M 0 100% /run/media/livemnt-squash</font><span style="font-family:'courier new',monospace">-mRJNIA</span></div>
<div><font face="courier new, monospace">/dev/mapper/dm-PCBV6p</font><span style="font-family:'courier new',monospace"> ext4 12G 4.6G 7.1G 40% /mnt/a</span></div><div><font face="courier new, monospace">/dev/loop11 ext4 380M 35M 325M 10% /mnt/a/home</font></div>
<div><br></div><div><font face="arial, helvetica, sans-serif">The invalidation occurred at the 1.6 GB size limit applied to the /run tmpfs where the overlay, /dev/loop10, was mounted,</font></div><div><div><div><font face="courier new, monospace">[root@localhost ~]# losetup /dev/loop10</font></div>
<div><font face="courier new, monospace">/dev/loop10: [0017]:58601 (/run/media/tmpvJjuX7)</font></div><div style="font-family:'courier new',monospace"><br></div></div></div><div><br></div>
<div><font face="courier new, monospace">[root@localhost ~]# ls /mnt/a</font></div><div><font face="courier new, monospace">ls: cannot access /mnt/a/.readahead: Input/output error</font></div></div><div><font face="courier new, monospace"><br>
</font></div><div><div style="font-family:'courier new',monospace">top - 00:26:06 up 13 min, 4 users, load average: 0.55, 0.68, 0.44</div><div style="font-family:'courier new',monospace">Tasks: 204 total, 2 running, 202 sleeping, 0 stopped, 0 zombie</div>
<div style="font-family:'courier new',monospace">%Cpu(s): 4.3 us, 1.6 sy, 0.0 ni, 92.4 id, 1.3 wa, 0.3 hi, 0.0 si, 0.0 st</div><div style="font-family:'courier new',monospace">KiB Mem: 3339812 total, 3256176 used, 83636 free, 68956 buffers</div>
<div style="font-family:'courier new',monospace">KiB Swap: 3341308 total, 0 used, 3341308 free, 2312664 cached</div><div style="font-family:'courier new',monospace"><br></div><div><font face="arial, helvetica, sans-serif">Notice that Swap was not activated, but free memory got down to ~83 MiB.</font></div>
<div><font face="arial, helvetica, sans-serif">When I tested the above on the booted LiveUSB, 2-3 GiB of swap was activated before the fatal crash.</font></div><div><br></div><div>So an oversized overlay DOES NOT prevent device-mapper invalidation by the above test method.</div>
<div><br></div><div><font face="arial, helvetica, sans-serif"> --Fred</font></div><div style="font-family:'courier new',monospace"><br></div></div><div><br></div></div><br>