2:14 p.m.
In the cloud meeting today I brought up overlayfs and F25. After
discussing with the engineers closer to the technology they recommend
waiting to move to overlayfs as the default in F26.
I think this will work well because it will give us some time to allow
people to "try" overlayfs in F25 (we should provide good docs on this)
and then give us feedback before we go with it as default in F26. If
the feedback is bad then maybe we wouldn't even go with it in F26, but
hopefully that won't be the case.
Thoughts?
Dusty
3:45 p.m.
On Wed, Sep 14, 2016 at 12:14 PM, Dusty Mabe <dusty(a)dustymabe.com> wrote:
In the cloud meeting today I brought up overlayfs and F25. After
discussing with the engineers closer to the technology they recommend
waiting to move to overlayfs as the default in F26.
I think this will work well because it will give us some time to allow
people to "try" overlayfs in F25 (we should provide good docs on this)
and then give us feedback before we go with it as default in F26. If
the feedback is bad then maybe we wouldn't even go with it in F26, but
hopefully that won't be the case.
Thoughts?
Sounds good to me.
>
> Dusty
> _______________________________________________
> cloud mailing list
> cloud(a)lists.fedoraproject.org
> https://lists.fedoraproject.org/admin/lists/cloud@lists.fedoraproject.org
4:55 p.m.
On Wed, Sep 14, 2016 at 2:45 PM, Jason Brooks <jbrooks(a)redhat.com> wrote:
On Wed, Sep 14, 2016 at 12:14 PM, Dusty Mabe
<dusty(a)dustymabe.com> wrote:
>
> In the cloud meeting today I brought up overlayfs and F25. After
> discussing with the engineers closer to the technology they recommend
> waiting to move to overlayfs as the default in F26.
>
> I think this will work well because it will give us some time to allow
> people to "try" overlayfs in F25 (we should provide good docs on this)
> and then give us feedback before we go with it as default in F26. If
> the feedback is bad then maybe we wouldn't even go with it in F26, but
> hopefully that won't be the case.
>
> Thoughts?
Sounds good to me.
I'm uncertain if this is current or needs an update:
Evaluate overlayfs with docker
https://github.com/kubernetes/kubernetes/issues/15867
If the way forward is a non-duplicating cache then I see a major
advantage gone. But that alone isn't enough to promote something else,
I'd just say, hedge your bets. Pretty much all the reasons why CoreOS
switched from Btrfs to overlay have been fixed, although there's a
asstrometric ton of enospc rework landing in kernel 4.8 [1] that will
need time to shake out, and if anyone's able to break it, one of the
best ways of getting it fixed and avoiding regressions is to come up
with an xfstests [2] for it to be cleanly reproduced. The Facebook
devs consistently report finding hardware (even enterprise stuff that
they use) doing batshit things that Btrfs catches and corrects that
other filesystems aren't seeing. And then on the slow downs mainly due
to fragmentation when creating and destroying many snapshots over a
short period of time, this probably could be mitigated with garbage
collection optimization, and I've had some ideas about that if anyone
wants to futz around with it.
The more conservative change is probably XFS + overlayfs though, since
now XFS checksums fs metadata and the journal, which helps catch
problems before they get worse.
[1]
http://www.spinics.net/lists/linux-btrfs/msg53410.html
[2] semi random example
http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/cmds/xfstests.git;a=blob;f=te...
--
Chris Murphy
4:32 p.m.
On Wed, Sep 14, 2016 at 2:14 PM, Dusty Mabe <dusty(a)dustymabe.com> wrote:
In the cloud meeting today I brought up overlayfs and F25. After
discussing with the engineers closer to the technology they recommend
waiting to move to overlayfs as the default in F26.
I think this will work well because it will give us some time to allow
people to "try" overlayfs in F25 (we should provide good docs on this)
and then give us feedback before we go with it as default in F26. If
the feedback is bad then maybe we wouldn't even go with it in F26, but
hopefully that won't be the case.
Thoughts?
Seems a little conservative, but I'm not opposed.
I've been under the impression that part of the point of the Two Week
Release cycle was to be able to deliver new stuff faster and fix
issues faster but playing it safe isn't inherently a bad approach
either.
-AdamM
>
> Dusty
> _______________________________________________
> cloud mailing list
> cloud(a)lists.fedoraproject.org
> https://lists.fedoraproject.org/admin/lists/cloud@lists.fedoraproject.org
4:47 p.m.
On 09/14/2016 02:32 PM, Adam Miller wrote:
On Wed, Sep 14, 2016 at 2:14 PM, Dusty Mabe
<dusty(a)dustymabe.com> wrote:
>
> In the cloud meeting today I brought up overlayfs and F25. After
> discussing with the engineers closer to the technology they recommend
> waiting to move to overlayfs as the default in F26.
>
> I think this will work well because it will give us some time to allow
> people to "try" overlayfs in F25 (we should provide good docs on this)
> and then give us feedback before we go with it as default in F26. If
> the feedback is bad then maybe we wouldn't even go with it in F26, but
> hopefully that won't be the case.
>
> Thoughts?
Seems a little conservative, but I'm not opposed.
I've been under the impression that part of the point of the Two Week
Release cycle was to be able to deliver new stuff faster and fix
issues faster but playing it safe isn't inherently a bad approach
either.
When is F26 out?
--
--
Josh Berkus
Project Atomic
Red Hat OSAS
8:51 a.m.
On Wed, Sep 14, 2016 at 02:47:09PM -0700, Josh Berkus wrote:
> Seems a little conservative, but I'm not opposed.
> I've been under the impression that part of the point of the Two Week
> Release cycle was to be able to deliver new stuff faster and fix
> issues faster but playing it safe isn't inherently a bad approach
> either.
When is F26 out?
Next June.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
3:25 a.m.
On 14/09/16, Adam Miller wrote:
On Wed, Sep 14, 2016 at 2:14 PM, Dusty Mabe
<dusty(a)dustymabe.com> wrote:
>
> In the cloud meeting today I brought up overlayfs and F25. After
> discussing with the engineers closer to the technology they recommend
> waiting to move to overlayfs as the default in F26.
>
> I think this will work well because it will give us some time to allow
> people to "try" overlayfs in F25 (we should provide good docs on this)
> and then give us feedback before we go with it as default in F26. If
> the feedback is bad then maybe we wouldn't even go with it in F26, but
> hopefully that won't be the case.
>
> Thoughts?
Seems a little conservative, but I'm not opposed.
I've been under the impression that part of the point of the Two Week
Release cycle was to be able to deliver new stuff faster and fix
issues faster but playing it safe isn't inherently a bad approach
either.
For two week atomic we are not tied with the Fedora 25 release cycle. We
can enable it in our release when we think it is ready for the
consumers. It does not have to wait F26 release. For example we see it
is in good condition after one week of F25 release, we can then enable
it default in the next 2WA release.
Kushal
--
Fedora Cloud Engineer
CPython Core Developer
https://kushaldas.in
https://dgplug.org
8:51 a.m.
On Thu, Sep 15, 2016 at 01:55:16PM +0530, Kushal Das wrote:
For two week atomic we are not tied with the Fedora 25 release cycle.
We
can enable it in our release when we think it is ready for the
consumers. It does not have to wait F26 release. For example we see it
is in good condition after one week of F25 release, we can then enable
it default in the next 2WA release.
+1
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
8:57 a.m.
On 09/15/2016 04:25 AM, Kushal Das wrote:
On 14/09/16, Adam Miller wrote:
> On Wed, Sep 14, 2016 at 2:14 PM, Dusty Mabe <dusty(a)dustymabe.com> wrote:
>>
>> In the cloud meeting today I brought up overlayfs and F25. After
>> discussing with the engineers closer to the technology they recommend
>> waiting to move to overlayfs as the default in F26.
>>
>> I think this will work well because it will give us some time to allow
>> people to "try" overlayfs in F25 (we should provide good docs on this)
>> and then give us feedback before we go with it as default in F26. If
>> the feedback is bad then maybe we wouldn't even go with it in F26, but
>> hopefully that won't be the case.
>>
>> Thoughts?
>
> Seems a little conservative, but I'm not opposed.
>
> I've been under the impression that part of the point of the Two Week
> Release cycle was to be able to deliver new stuff faster and fix
> issues faster but playing it safe isn't inherently a bad approach
> either.
For two week atomic we are not tied with the Fedora 25 release cycle. We
can enable it in our release when we think it is ready for the
consumers. It does not have to wait F26 release. For example we see it
is in good condition after one week of F25 release, we can then enable
it default in the next 2WA release.
That is correct, but changing a default like that might be a bad idea.
My opinion is that it should happen on a major release boundary.
The user still has the option to choose to user overlayfs if they
want.
Dusty
8:02 a.m.
On Thu, Sep 15, 2016, at 09:57 AM, Dusty Mabe wrote:
That is correct, but changing a default like that might be a bad
idea.
My opinion is that it should happen on a major release boundary.
One thing this impacts is the AH partitioning - it no longer makes
sense by default with overlayfs. I think we should probably do exactly
the same thing as the Server SIG (and consider doing it for Workstation
too), which actually argues for just fixing the Anaconda defaults.
Server thread:
https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject....
11:56 a.m.
On Fri, Sep 16, 2016 at 7:02 AM, Colin Walters <walters(a)verbum.org> wrote:
On Thu, Sep 15, 2016, at 09:57 AM, Dusty Mabe wrote:
> That is correct, but changing a default like that might be a bad idea.
> My opinion is that it should happen on a major release boundary.
One thing this impacts is the AH partitioning - it no longer makes
sense by default with overlayfs. I think we should probably do exactly
the same thing as the Server SIG (and consider doing it for Workstation
too), which actually argues for just fixing the Anaconda defaults.
Server thread:
https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject....
The genesis of that was me pointing to Cloud Atomic ISO's handling;
since Server went with pretty much the identical layout, it managed to
get slipped in for Alpha. It was a proven layout. [1]
For an Atomic Host overlayfs based layout, there's nothing within
Fedora that's a proven layout. For starters, it could be something
much simpler than what CoreOS is doing [2]. If the target installation
is VM, then dropping LVM stuff makes sense. If it's going to include
baremetal, keeping LVM makes sense. I'm a bit unclear on this point,
but with a handwave it sorta feels like Cloud->Container WG is far
less interested in the baremetal case, where Server is about as
interested in baremetal as VM and container cases. If that's true,
then the CoreOS layout is a decent starting point, and just needs some
simplification to account for ostree deployments rather than partition
priority flipping.
Inode exhaustion?
If the installer is going to create the file system used for overlayfs
backing storage with ext4, that probably means mkfs.ext4 -i 4096 will
need to be used; so how does that get propagated to only AH installs,
for both automatic and custom partitioning? Or figure out a way to
drop custom/manual partitioning from the UI. Or does using XFS
mitigate this issue? A simple search turns up no inode exhaustion
reports with XFS. The work around for ext4 is at mkfs time, it's not
something that can be changed later.
Release blocking and custom partitioning?
Upon AH image becoming release blocking, then "The installer must be
able to create and install to any workable partition layout using any
file system and/or container format combination offered in a default
installer configuration. " applies. Example bug [3] where this fails
right now. Does it make sense for AH installations to somehow be
exempt from custom partitioning resulting in successful installations?
And what would that look like criterion wise (just grant an
exception?) or installer wise (drop the custom UI or put up warnings
upon entering?)
[1]
https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject....
[2]
https://coreos.com/os/docs/latest/sdk-disk-partitions.html
Trivial pursuit is this "the GPT priority attribute" which I can find
no where else, but I rather like this idea of using an xattr on a
directory as the hint for which fs tree the bootloader should use
rather than writing out new bootloader configurations.
[3]
https://bugzilla.redhat.com/show_bug.cgi?id=1289752
--
Chris Murphy
3:15 p.m.
On Fri, Sep 16, 2016 at 10:56 AM, Chris Murphy <lists(a)colorremedies.com> wrote:
Inode exhaustion?
If the installer is going to create the file system used for overlayfs
backing storage with ext4, that probably means mkfs.ext4 -i 4096 will
need to be used; so how does that get propagated to only AH installs,
for both automatic and custom partitioning? Or figure out a way to
drop custom/manual partitioning from the UI. Or does using XFS
mitigate this issue? A simple search turns up no inode exhaustion
reports with XFS. The work around for ext4 is at mkfs time, it's not
something that can be changed later.
I just did some more digging, and also chatted with Eric Sandeen about
this. Here's what I've learned:
- Inode exhaustion with mkfs.ext4 defaults can be a real thing with
overlayfs [1]
- mkfs.ext4 -i 4096 will make 1 inode per 4096 byte block, so 1:1,
which is a metric shittonne of inodes
- a different -i value might be more practical most of the time, but
if the maximum aren't created at mkfs time and are exhausted the fs
basically face plants and no more files can be created; and it's only
fixable by a.) deleting a bunch of files or b.) creating a new file
system to have more inodes preallocated.
- mkfs.ext4 hands off the actual creation of the inodes to lazy init
at first mount time, it's a lot of metadata being written to do this
- XFS doesn't have this issue, its inode allocation is dynamic (there
are limits but can be changed with xfs_growfs)
- XFS now defaults to -m crc=1, and by extension -n flags=1 which
overlayfs wants for putting filetype in the directory entry; Fedora 24
had a sufficiently new xfsprogs for this from the get go.
I don't know what the workflow is for creating the persistent storage
for the host, whether this will be Anaconda's role or something else?
If Anaconda, my experience has been the Anaconda team are reluctant to
use non-default mkfs unless there's a UI toggle for it. You'd need to
run all this by them and see if there's a way to do a mkfs.ext4 -i
4096 for just Atomic Host installations, there's no point doing that
for workstation installations. Or just use XFS.
[1]
https://github.com/coreos/bugs/issues/264
https://github.com/boot2docker/boot2docker/issues/992
--
Chris Murphy
4:17 p.m.
On Fri, Sep 16, 2016 at 2:15 PM, Chris Murphy <lists(a)colorremedies.com> wrote:
You'd need to
run all this by them and see if there's a way to do a mkfs.ext4 -i
4096 for just Atomic Host installations, there's no point doing that
for workstation installations. Or just use XFS.
Another possibility is an AH specific /etc/mke2fs.conf file on the
installation media only.
[defaults]
base_features =
sparse_super,large_file,filetype,resize_inode,dir_index,ext_attr
default_mntopts = acl,user_xattr
enable_periodic_fsck = 0
blocksize = 4096
inode_size = 256
inode_ratio = 16384
By changing inode_ratio = 4096, it achieves the same outcome as -i
4096 without having to pass that flag at mkfs time. And it'd only
affect installation time file systems (including /boot and / as well
as the persistent storage for overlayfs and containers). So... yeah.
FWIW, you're basically already using XFS with the dm-thin
docker-storage-setup you've got going on right now. It doesn't get
mounted anywhere, but
$ docker info
[chris@localhost ~]$ sudo docker info
[sudo] password for chris:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.10.3
Storage Driver: devicemapper
Pool Name: fedora--atomic-docker--pool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
So, just use XFS across the board (plus overlayfs on the persistent
storage for containers).
As for Workstation changing file systems, that's another ball of wax.
I'd just say use XFS + overlayfs there too to keep it simple across
the various products in the near term. And then presumably the
Workstation folks will want Btrfs when it sufficiently stable that the
kernel team won't freak out if there's still no Btrfs specific kernel
dev on the team or at Red Hat.
--
Chris Murphy
4:12 p.m.
Just in case this poor horse isn't suitably beaten yet.
1. Create 4 qcow2 files per
qemu-img create -f qcow2 *.qcow2 120g
Each qcow2 starts out 194K (not preallocated).
q
2. Format each qcow2
mkfs.ext4 <dev>
mkfs.ext4 -i 4096 <dev>
mkfs.xfs <dev>
mkfs.btrfs <dev>
3. mount each fs (mainly to be fair since ext4 does lazy init) and
wait until the qcow2 stops growing.
5.5M -rw-r--r--. 1 qemu qemu 5.9M Sep 19 20:40 bios_btrfs.qcow2
2.1G -rw-r--r--. 1 root root 2.1G Sep 19 20:27 bios_ext4_default.qcow2
7.7G -rw-r--r--. 1 root root 7.7G Sep 19 20:33 bios_ext4_i4096.qcow2
62M -rw-r--r--. 1 qemu qemu 62M Sep 19 20:40 bios_xfs.qcow2
Btrfs and XFS take seconds to completely initialize. Ext4 defaults
took 6 minutes, and with -i 4096 it took 8 minutes to complete lazy
init.
--
Chris Murphy
2442
days inactive
2448
days old
14 comments
8 participants
participants (8)
-
Adam Miller -
Chris Murphy -
Colin Walters -
Dusty Mabe -
Jason Brooks -
Josh Berkus -
Kushal Das -
Matthew Miller