OK instead of hijacking some other thread...(sorta too late).
# lsblk -f NAME FSTYPE LABEL UUID MOUNTPOINT sda =E2=94=9C=E2=94=80sda1 vfat 5956-63D8 /boot/efi =E2=94=9C=E2=94=80sda2 btrfs f23a 8b0c4840-4fc7-4782-a4c0-25fec8a40dd4 /sysroot =E2=94=9C=E2=94=80sda3 ext4 908cb4df-410b-47e4-afb1-872255bd1244 /boot =E2=94=9C=E2=94=80sda4 LVM2_member 9WkBEW-XwcN-8f99-nQOP-E1w8-QiBs-6Xio7a =E2=94=94=E2=94=80sda5 swap 53f95874-06db-4433-a74b-8dc31105c528 [SWAP]
-bash-4.3# pvs PV VG Fmt Attr PSize PFree /dev/sda4 VG lvm2 a-- 426.51g 126.36g -bash-4.3# vgs VG #PV #LV #SN Attr VSize VFree VG 1 1 0 wz--n- 426.51g 126.36g -bash-4.3# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert docker-pool VG twi-a-tz-- 300.00g 0.00 0.44
What I'd like to know how to do, is get docker to use the thin pool setup here as backing, instead of Btrfs.
What I tried:
# cp /usr/lib/docker-storage-setup/docker-storage-setup /etc/sysconfig/docker-storage-setup
I then edited that file making one change such that VG=3DVG since that's the name of the VG. So it still says to use STORAGE_DRIVER=3Ddevicemapper.
# systemctl status docker-storage-setup =E2=97=8F docker-storage-setup.service - Docker Storage Setup Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; enabled; vendor preset: disabled) Active: inactive (dead) since Thu 2015-12-10 01:09:06 MST; 45s ago Process: 8008 ExecStart=3D/usr/bin/docker-storage-setup (code=3Dexited, status=3D0/SUCCESS) Main PID: 8008 (code=3Dexited, status=3D0/SUCCESS)
Dec 10 01:09:05 f23a.localdomain systemd[1]: Starting Docker Storage Setup.= .. Dec 10 01:09:05 f23a.localdomain docker-storage-setup[8008]: INFO: Volume group backing root filesystem could not be determined Dec 10 01:09:06 f23a.localdomain docker-storage-setup[8008]: Logical volume "docker-pool" changed. Dec 10 01:09:06 f23a.localdomain systemd[1]: Started Docker Storage Setup.
# systemctl status docker =E2=97=8F docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/docker.service.d =E2=94=94=E2=94=80flannel.conf Active: failed (Result: exit-code) since Thu 2015-12-10 01:09:06 MST; 48= s ago Docs: http://docs.docker.com Process: 8032 ExecStart=3D/usr/bin/docker daemon $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $INSECURE_REGISTRY (code=3Dexited, status=3D1/FAILURE) Main PID: 8032 (code=3Dexited, status=3D1/FAILURE)
Dec 10 01:09:06 f23a.localdomain systemd[1]: Starting Docker Application Container Engine... Dec 10 01:09:06 f23a.localdomain docker[8032]: time=3D"2015-12-10T01:09:06.240354137-07:00" level=3Dinfo msg=3D"[graphdriver] using prior storage driver "btrfs"" Dec 10 01:09:06 f23a.localdomain docker[8032]: time=3D"2015-12-10T01:09:06.249931298-07:00" level=3Dfatal msg=3D"Error starting daemon: SELinux is not supporte...h driver" Dec 10 01:09:06 f23a.localdomain systemd[1]: docker.service: Main process exited, code=3Dexited, status=3D1/FAILURE Dec 10 01:09:06 f23a.localdomain systemd[1]: Failed to start Docker Application Container Engine. Dec 10 01:09:06 f23a.localdomain systemd[1]: docker.service: Unit entered failed state. Dec 10 01:09:06 f23a.localdomain systemd[1]: docker.service: Failed with result 'exit-code'. Hint: Some lines were ellipsized, use -l to show in full.
So there's a disconnect between docker-storage-setup, which is set to use devicemapper and the volume group named VG, and docker.service which continues to use the btrfs driver (and then fails for the known reason that --selinux-enabled is set by default and that's not yet working until the automatic chcon patch lands).
In the meantime I can just remove --selinux-enabled. But I'd like to know how to redirect docker to use lvm thinp if that's possible.
On 12/10/2015 03:39 AM, Chris Murphy wrote:
OK instead of hijacking some other thread...(sorta too late).
# lsblk -f NAME FSTYPE LABEL UUID MOUNTPOINT sda =E2=94=9C=E2=94=80sda1 vfat 5956-63D8 /boot/efi =E2=94=9C=E2=94=80sda2 btrfs f23a 8b0c4840-4fc7-4782-a4c0-25fec8a40dd4 /sysroot =E2=94=9C=E2=94=80sda3 ext4 908cb4df-410b-47e4-afb1-872255bd1244 /boot =E2=94=9C=E2=94=80sda4 LVM2_member 9WkBEW-XwcN-8f99-nQOP-E1w8-QiBs-6Xio7a =E2=94=94=E2=94=80sda5 swap 53f95874-06db-4433-a74b-8dc31105c528 [SWAP]
-bash-4.3# pvs PV VG Fmt Attr PSize PFree /dev/sda4 VG lvm2 a-- 426.51g 126.36g -bash-4.3# vgs VG #PV #LV #SN Attr VSize VFree VG 1 1 0 wz--n- 426.51g 126.36g -bash-4.3# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert docker-pool VG twi-a-tz-- 300.00g 0.00 0.44
So you created the VG VG and the docker-pool LV on your own before docker-storage-setup is run?
What I would do is leave sda4 blank and then put the following in your config file:
DEVS=/dev/sda4 VG=vgdocker
What I think this will do is create a PV out of /dev/sda4 and create a VG (named vgdocker) on top of it. It will then create the docker-pool LV for you and have the docker daemon use that as the backing store.
Let me know if this is what you were looking for or not!
Dusty
On Thu, Dec 10, 2015 at 9:54 AM, Dusty Mabe dusty@dustymabe.com wrote:
So you created the VG VG and the docker-pool LV on your own before docker-storage-setup is run?
Yes.
What I would do is leave sda4 blank and then put the following in your config file:
DEVS=/dev/sda4 VG=vgdocker
Fails with
Dec 10 11:10:13 f23a.localdomain docker-storage-setup[1135]: Partition specification unsupported at this time.
What I think this will do is create a PV out of /dev/sda4 and create a VG (named vgdocker) on top of it. It will then create the docker-pool LV for you and have the docker daemon use that as the backing store.
Let me know if this is what you were looking for or not!
Yes.
----- Original Message -----
From: "Chris Murphy" lists@colorremedies.com To: "Fedora Cloud SIG" cloud@lists.fedoraproject.org Sent: Thursday, December 10, 2015 12:39:26 AM Subject: root on btrfs and lvmthinp for docker backing confusion
OK instead of hijacking some other thread...(sorta too late).
# lsblk -f NAME FSTYPE LABEL UUID MOUNTPOINT sda =E2=94=9C=E2=94=80sda1 vfat 5956-63D8 /boot/efi =E2=94=9C=E2=94=80sda2 btrfs f23a 8b0c4840-4fc7-4782-a4c0-25fec8a40dd4 /sysroot =E2=94=9C=E2=94=80sda3 ext4 908cb4df-410b-47e4-afb1-872255bd1244 /boot =E2=94=9C=E2=94=80sda4 LVM2_member 9WkBEW-XwcN-8f99-nQOP-E1w8-QiBs-6Xio7a =E2=94=94=E2=94=80sda5 swap 53f95874-06db-4433-a74b-8dc31105c528 [SWAP]
-bash-4.3# pvs PV VG Fmt Attr PSize PFree /dev/sda4 VG lvm2 a-- 426.51g 126.36g -bash-4.3# vgs VG #PV #LV #SN Attr VSize VFree VG 1 1 0 wz--n- 426.51g 126.36g -bash-4.3# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert docker-pool VG twi-a-tz-- 300.00g 0.00 0.44
What I'd like to know how to do, is get docker to use the thin pool setup here as backing, instead of Btrfs.
What I tried:
# cp /usr/lib/docker-storage-setup/docker-storage-setup /etc/sysconfig/docker-storage-setup
I then edited that file making one change such that VG=3DVG since that's the name of the VG. So it still says to use STORAGE_DRIVER=3Ddevicemapper.
# systemctl status docker-storage-setup =E2=97=8F docker-storage-setup.service - Docker Storage Setup Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; enabled; vendor preset: disabled) Active: inactive (dead) since Thu 2015-12-10 01:09:06 MST; 45s ago Process: 8008 ExecStart=3D/usr/bin/docker-storage-setup (code=3Dexited, status=3D0/SUCCESS) Main PID: 8008 (code=3Dexited, status=3D0/SUCCESS)
Dec 10 01:09:05 f23a.localdomain systemd[1]: Starting Docker Storage Setup.= .. Dec 10 01:09:05 f23a.localdomain docker-storage-setup[8008]: INFO: Volume group backing root filesystem could not be determined Dec 10 01:09:06 f23a.localdomain docker-storage-setup[8008]: Logical volume "docker-pool" changed. Dec 10 01:09:06 f23a.localdomain systemd[1]: Started Docker Storage Setup.
# systemctl status docker =E2=97=8F docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/docker.service.d =E2=94=94=E2=94=80flannel.conf Active: failed (Result: exit-code) since Thu 2015-12-10 01:09:06 MST; 48= s ago Docs: http://docs.docker.com Process: 8032 ExecStart=3D/usr/bin/docker daemon $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $INSECURE_REGISTRY (code=3Dexited, status=3D1/FAILURE) Main PID: 8032 (code=3Dexited, status=3D1/FAILURE)
Dec 10 01:09:06 f23a.localdomain systemd[1]: Starting Docker Application Container Engine... Dec 10 01:09:06 f23a.localdomain docker[8032]: time=3D"2015-12-10T01:09:06.240354137-07:00" level=3Dinfo msg=3D"[graphdriver] using prior storage driver "btrfs"" Dec 10 01:09:06 f23a.localdomain docker[8032]: time=3D"2015-12-10T01:09:06.249931298-07:00" level=3Dfatal msg=3D"Error starting daemon: SELinux is not supporte...h driver" Dec 10 01:09:06 f23a.localdomain systemd[1]: docker.service: Main process exited, code=3Dexited, status=3D1/FAILURE Dec 10 01:09:06 f23a.localdomain systemd[1]: Failed to start Docker Application Container Engine. Dec 10 01:09:06 f23a.localdomain systemd[1]: docker.service: Unit entered failed state. Dec 10 01:09:06 f23a.localdomain systemd[1]: docker.service: Failed with result 'exit-code'. Hint: Some lines were ellipsized, use -l to show in full.
So there's a disconnect between docker-storage-setup, which is set to use devicemapper and the volume group named VG, and docker.service which continues to use the btrfs driver (and then fails for the known reason that --selinux-enabled is set by default and that's not yet working until the automatic chcon patch lands).
In the meantime I can just remove --selinux-enabled. But I'd like to know how to redirect docker to use lvm thinp if that's possible.
You can ignore docker-storage-setup and edit /etc/sysconfig/docker-storage yourself.
Here's what it looks like from the f23 vagrant box:
DOCKER_STORAGE_OPTIONS=-s devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/atomicos-docker--pool --storage-opt dm.use_deferred_removal=true
Jason
-- Chris Murphy _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/cloud@lists.fedoraproject.org
On Thu, Dec 10, 2015 at 11:47 AM, Jason Brooks jbrooks@redhat.com wrote:
You can ignore docker-storage-setup and edit /etc/sysconfig/docker-storage yourself.
OK I've done "systemctl disable docker-storage-setup"
Here's what it looks like from the f23 vagrant box:
DOCKER_STORAGE_OPTIONS=-s devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/atomicos-docker--pool --storage-opt dm.use_deferred_removal=true
Short version: This works. Docker service starts, no errors, and it's not using a loopback device.
Followup question: Does Docker directly use a thin pool without creating (virtual size) logical volumes? Because I don't see any other LV's created, and no XFS filesystem appears on the host using the mount command. And yet I see XFS mount and umount kernel messages on the host. This is sort of an esoteric question. However, I have no access to container files from the host like I can see inside each btrfs subvolume when btrfs is the backing method. And that suggests possibly rather different backup strategies depending on the backing.
Long version of the question:
# systemctl stop docker
Copied generic docker-storage and edited as follows:
DOCKER_STORAGE_OPTIONS=-s devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/vgfedora-docker--pool --storage-opt dm.use_deferred_removal=true
# systemctl start docker Dec 10 13:05:02 f23a.localdomain systemd[1]: Starting Docker Application Container Engine... Dec 10 13:05:06 f23a.localdomain docker[1695]: time="2015-12-10T13:05:06.377269077-07:00" level=info msg="Firewalld running: false" Dec 10 13:05:06 f23a.localdomain docker[1695]: time="2015-12-10T13:05:06.791189262-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.1/16. Daemon option --bip can be used to set a preferred IP address" Dec 10 13:05:07 f23a.localdomain docker[1695]: time="2015-12-10T13:05:07.572415354-07:00" level=info msg="Loading containers: start." Dec 10 13:05:07 f23a.localdomain docker[1695]: ...... Dec 10 13:05:07 f23a.localdomain docker[1695]: time="2015-12-10T13:05:07.576094399-07:00" level=info msg="Loading containers: done." Dec 10 13:05:07 f23a.localdomain docker[1695]: time="2015-12-10T13:05:07.576663930-07:00" level=info msg="Daemon has completed initialization" Dec 10 13:05:07 f23a.localdomain docker[1695]: time="2015-12-10T13:05:07.576770908-07:00" level=info msg="Docker daemon" commit=f7c1d52-dirty execdriver=native-0.2 graphdriver=devicemapper version=1.9.1-fc23 Dec 10 13:05:07 f23a.localdomain docker[1695]: time="2015-12-10T13:05:07.577004141-07:00" level=info msg="API listen on /var/run/docker.sock" Dec 10 13:05:07 f23a.localdomain systemd[1]: Started Docker Application Container Engine.
host# docker pull fedora host# docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE fedora latest 597717fc21bd 2 weeks ago 204 MB
So that all works, and then also the thin pool data% is growing after each step, according to lvs. But there is no logical volume, no file system.
host# docker run -i -t fedora /bin/bash
host# mount | grep xfs selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime) host# mount | grep ext /dev/sda3 on /boot type ext4 (rw,relatime,seclabel,stripe=4,data=ordered)
It's working. But is Docker directly using the thin pool without creating a thin logical volume and file system? That's unexpected.
host# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert docker-pool vgfedora twi-aotz-- 250.00g 0.11 0.53 host# docker pull ubuntu [...snip pull output...] -bash-4.3# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert docker-pool vgfedora twi-aotz-- 250.00g 0.20 0.63
Yeah, it seems to directly use the thin pool without create a virtual size LV first. Kernel messages show a bunch of items like this:
[ 7818.790685] XFS (dm-4): Mounting V5 Filesystem [ 7819.038211] XFS (dm-4): Ending clean mount [ 7834.219132] XFS (dm-4): Unmounting Filesystem
So it's using XFS on something, that isn't appearing in mount on the host or in the container. And multiple containers all appear to use XFS on the same device mapper device (dm-4) which does not appear on the host in the /dev/ directory. So this is really... obscured.
On 12/10/2015 08:35 PM, Chris Murphy wrote:
On Thu, Dec 10, 2015 at 11:47 AM, Jason Brooks jbrooks@redhat.com wrote:
You can ignore docker-storage-setup and edit /etc/sysconfig/docker-storage yourself.
OK I've done "systemctl disable docker-storage-setup"
Here's what it looks like from the f23 vagrant box:
DOCKER_STORAGE_OPTIONS=-s devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/atomicos-docker--pool --storage-opt dm.use_deferred_removal=true
Short version: This works. Docker service starts, no errors, and it's not using a loopback device.
Followup question: Does Docker directly use a thin pool without creating (virtual size) logical volumes? Because I don't see any other LV's created, and no XFS filesystem appears on the host using the mount command. And yet I see XFS mount and umount kernel messages on the host. This is sort of an esoteric question. However, I have no access to container files from the host like I can see inside each btrfs subvolume when btrfs is the backing method. And that suggests possibly rather different backup strategies depending on the backing.
I believe it chops it up using low level device mapper stuff. I think you don't see the mounts on your host because they are in a different mount namespace (part of the magic behind containers).
For more info on docker + device mapper look at slides 37-44 of [1]
[1] - http://www.slideshare.net/Docker/docker-storage-drivers
On Thu, Dec 10, 2015 at 9:18 PM, Dusty Mabe dusty@dustymabe.com wrote:
On 12/10/2015 08:35 PM, Chris Murphy wrote:
Followup question: Does Docker directly use a thin pool without creating (virtual size) logical volumes? Because I don't see any other LV's created, and no XFS filesystem appears on the host using the mount command. And yet I see XFS mount and umount kernel messages on the host. This is sort of an esoteric question. However, I have no access to container files from the host like I can see inside each btrfs subvolume when btrfs is the backing method. And that suggests possibly rather different backup strategies depending on the backing.
I believe it chops it up using low level device mapper stuff. I think you don't see the mounts on your host because they are in a different mount namespace (part of the magic behind containers).
For more info on docker + device mapper look at slides 37-44 of [1]
[1] - http://www.slideshare.net/Docker/docker-storage-drivers
I read all the slides. That is really helpful, there's quite a bit of detail considering they're slides.
It's definitely more devicemapper than LVM based (makes sense, the driver is "devicemapper" after all). The most that appears in LVM's view is the thin pool, and once Docker owns it, LVM can't make virtual LV's from that pool. As to the obscurity, on the one hand it's a perception because while I'm quite comfortable with LVM tools, I'm not that comfortable with dmsetup; and on the other hand the local backing should probably be considered disposable, without warning, in a production setup anyway. So some regular sweep of container states (if that's important) should be made into images and put elsewhere. Seriously, if the backing store were to faceplant, it's simply faster to start from the most recent image than attempt repairs.
-
Chris Murphy -
Dusty Mabe -
Jason Brooks