F32, enable fstrim.timer by default

[Chris Murphy]

Hi,

This desktop@ thread [1] about a slow device restored by enabling
fstrim.service, got me thinking about enabling fstrim.timer [2] by
default in Fedora Workstation. But I'm curious if it might be
desirable in other Fedora Editions, and making it a system-wide
change?

I've checked recent versions of openSUSE and Ubuntu, and they have it
enabled. Therefore I estimate the likelihood of running into cons
(below) is pretty remote. Most people won't notice anything.

The pros:
+ when passed down to flash drives that support trim, it provides a
hint to the drive firmware about erase blocks ready for erasure. Some
devices will have improved wear leveling and performance as a result,
but this is firmware specific.
LVM [3] and dm-crypt [4] passdown appears to be enabled by default on Fedora.
+ with LVM thin provisioning, it will cause unused LV extents to be
returned to the thin pool for use by other LVs, kind of a nifty work
around for XFS not supporting fs shrink resize.

The gotchas:
+  Few, but highly visible, reports of buggy SSDs that corrupt or lose
data soon after trim being issued. By now, most have been blacklisted
in the kernel, and/or have manufacturer firmware updates. We shouldn't
run into this problem unless someone has older hardware that hasn't
been update and for some reason also hasn't been blacklisted in the
kernel.
+ Older SSD's have only non-queued trim support, which also can result
in a brief hang while the command is processed. This is highly
variable based on the device firmware, and the workload. But using
weekly fstrim is preferred for these devices, instead of using the
discard mount option in /etc/fstab.
+ Possible exposure of fs locality pattern may be a security risk for
some workflows. [4] [5]


[1]
https://lists.fedoraproject.org/archives/list/[e-mail]...

[2]
fstrim.timer, if enabled, runs fstrim.service weekly, specifically
Monday at midnight local time; and if the system isn't available at
that time, it runs during or very soon after the next startup. The
command:
ExecStart=/usr/sbin/fstrim --fstab --verbose --quiet

fstab means only file systems in fstab are included; verbose reports
the mount point and bytes potentially discarded and is recorded in the
systemd journal; quiet suppresses errors which is typical for file
systems and devices that don't support fstrim, e.g. the EFI System
partition, which is FAT16/32; and USB flash "stick" drives, and hard
drives.

[3]
/etc/lvm/lvm.conf, if I'm reading it correctly, file system discards
are passed down:
    # This configuration option has an automatic default value.
    # thin_pool_discards = "passdown"

Due to this Fedora 27 feature; trim is passed down by dm-crypt as well
for LUKS volumes. Curiously because Fedora neither sets the discard
mount option for any file system, nor enables fstrim.timer, this
feature isn't being taken advantage of.

[4]
https://fedoraproject.org/wiki/Changes/EnableTrimOnDmCrypt

[5]
Trim on LUKS/dm-crypt note from upstream, section 5.19
https://gitlab.com/cryptsetup/cryptsetup/-/wikis/FrequentlyAskedQuestions...

-- 
Chris Murphy

[Martin Pitt]

Hello Chris,

Chris Murphy [2019-12-17 22:23 -0700]:
>  This desktop@ thread [1] about a slow device restored by enabling
>  fstrim.service, got me thinking about enabling fstrim.timer [2] by
>  default in Fedora Workstation. But I'm curious if it might be
>  desirable in other Fedora Editions, and making it a system-wide
>  change? 
This is a function/property of hardware, so it's IMO not desktop specific at
all. Servers suffer just as well from hard disks becoming slower.

>  I've checked recent versions of openSUSE and Ubuntu, and they
> have it
>  enabled. 
Incidentally I worked on that on the Ubuntu side 6 years ago:
https://blueprints.launchpad.net/ubuntu/+spec/core-1311-ssd-trimming This still
has the "discard vs. cron job" pros and cons and benchmarks, so might still be
useful.

Thanks!

Martin

[Martin Kolman]

On Wed, 2019-12-18 at 13:11 +0100, Martin Pitt wrote:
>  Hello Chris,
>  
>  Chris Murphy [2019-12-17 22:23 -0700]:
>  > This desktop@ thread [1] about a slow device restored by enabling
>  > fstrim.service, got me thinking about enabling fstrim.timer [2] by
>  > default in Fedora Workstation. But I'm curious if it might be
>  > desirable in other Fedora Editions, and making it a system-wide
>  > change?
>  
>  This is a function/property of hardware, so it's IMO not desktop specific at
>  all. Servers suffer just as well from hard disks becoming slower. 
This will also
trim thin LVs on thin pools (if any), right ? 

So not just hardware, it can even make "software" storage layouts faster
& potentially even avoid pool exhaustion in some cases. :)

>  
>  > I've checked recent versions of openSUSE and Ubuntu, and they have it
>  > enabled.
>  
>  Incidentally I worked on that on the Ubuntu side 6 years ago:
>  https://blueprints.launchpad.net/ubuntu/+spec/core-1311-ssd-trimming This still
>  has the "discard vs. cron job" pros and cons and benchmarks, so might still be
>  useful.
>  
>  Thanks!
>  
>  Martin
>  _______________________________________________
>  server mailing list -- server(a)lists.fedoraproject.org
>  To unsubscribe send an email to server-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Chris Murphy]

On Wed, Dec 18, 2019 at 5:12 AM Martin Pitt <mpitt(a)redhat.com&gt; wrote:
>  Hello Chris,
>
>  Chris Murphy [2019-12-17 22:23 -0700]:
>  > This desktop@ thread [1] about a slow device restored by enabling
>  > fstrim.service, got me thinking about enabling fstrim.timer [2] by
>  > default in Fedora Workstation. But I'm curious if it might be
>  > desirable in other Fedora Editions, and making it a system-wide
>  > change?
>
>  This is a function/property of hardware, so it's IMO not desktop specific at
>  all. Servers suffer just as well from hard disks becoming slower. 
True, although I think the problem is far less likely to happen on
server and cloud hardware, which tend to be suitably overprovisioned
and have better wear leveling algorithms. So for any workload, there's
always erased cells available to accept data at full device speeds,
without either discard mount option or scheduled fstrim. Perhaps
fstrim.timer makes this more efficient? Seems speculative, but for
sure it's safe. At worse, the drive ignores the hint and it's a no op.

>  > I've checked recent versions of openSUSE and Ubuntu, and they have it
>  > enabled.
>
>  Incidentally I worked on that on the Ubuntu side 6 years ago:
>  https://blueprints.launchpad.net/ubuntu/+spec/core-1311-ssd-trimming This still
>  has the "discard vs. cron job" pros and cons and benchmarks, so might still be
>  useful. 
Yes it is. Since that time, the market now has queued trim devices,
and the discard mount option works fine with those. But fstrim.timer
would work fine for all cases, both queued and non-queued.

I suppose it's possible a narrow range of use cases, very heavy writes
and deletions, with queued trim supporting drives, where the discard
mount option is better than fstrim.timer, in that it provides very up
to date hinting of no longer used cells. But it doesn't hurt them to
have both, in the case where the user/admin sets discard mount option
themselves, and are unaware that fstrim.timer is already enabled.


-- 
Chris Murphy

[Chris Murphy]

readd cloud@ list

On Wed, Dec 18, 2019 at 5:32 AM Martin Kolman <mkolman(a)redhat.com&gt; wrote:
>  On Wed, 2019-12-18 at 13:11 +0100, Martin Pitt wrote:
>  > Hello Chris,
>  >
>  > Chris Murphy [2019-12-17 22:23 -0700]:
>  > > This desktop@ thread [1] about a slow device restored by enabling
>  > > fstrim.service, got me thinking about enabling fstrim.timer [2] by
>  > > default in Fedora Workstation. But I'm curious if it might be
>  > > desirable in other Fedora Editions, and making it a system-wide
>  > > change?
>  >
>  > This is a function/property of hardware, so it's IMO not desktop specific at
>  > all. Servers suffer just as well from hard disks becoming slower. 
>  This will also trim thin LVs on thin pools (if any), right ? 
Correct.

>  So not just hardware, it can even make "software" storage
> layouts faster
>  & potentially even avoid pool exhaustion in some cases. :) 
Maybe. I'm not sure either way if someone would actually notice
performance improvements, but it wouldn't make them worse. And yes,
potentially avoid pool exhaustion, in particular in under provisioned
cases.

I forgot to mention: with qemu/kvm/libvirt VM's, the trim would not
get passed down to the backing storage due to default settings. The
discard mode "unmap" is supported with a SCSI disk using virtio SCSI
controller; I see some curious works/doesn't work with "plain" virtio
disk. But when it works, it does pass down to underlying thinp LV and
raw files. If anyone is doing raw file backups, or otherwise paying
for storage, it could save them some coins. And when it doesn't work,
literally nothing happens: the file doesn't get holes punched out, but
there's also no corruption (I test these things with a Btrfs scrub; it
will consistently always complain if a single metadata or data
checksum mismatches).

Anyway, it's an optimization. Pretty well tested elsewhere at this
point. And offhand not aware of any liabilities, but thought I'd ask
about it before writing up a system wide change proposal.

One thing I see in the Ubuntu fstrim.service unit file that I'm not
seeing in the Fedora fstrim.service unit file, is a conditional for
containers (line 4). I'm not sure where to ask about that. Maybe
upstream systemd?

$ cat /lib/systemd/system/fstrim.service
[Unit]
Description=Discard unused blocks on filesystems from /etc/fstab
Documentation=man:fstrim(8)
ConditionVirtualization=!container

[Service]
Type=oneshot
ExecStart=/sbin/fstrim --fstab --verbose --quiet
ProtectSystem=strict
ProtectHome=yes
PrivateDevices=no
PrivateNetwork=yes
PrivateUsers=no
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
MemoryDenyWriteExecute=yes
SystemCallFilter=@default @file-system @basic-io @system-service
chris@chris-Standard-PC-Q35-ICH9-2009:~$



-- 
Chris Murphy

[Chris Murphy]

On Wed, Dec 18, 2019 at 10:56 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  One thing I see in the Ubuntu fstrim.service unit file that I'm not
>  seeing in the Fedora fstrim.service unit file, is a conditional for
>  containers (line 4). I'm not sure where to ask about that. Maybe
>  upstream systemd? 
Found it. I'm not sure if util-linux would typically be found in a
container base image? Probably no point in calling fstrim in that
case, but also doesn't hurt.

$ sudo dnf provides /usr/lib/systemd/system/fstrim.timer
util-linux-2.34-3.fc31.x86_64 : A collection of basic system utilities
Repo        : @System
Matched from:
Filename    : /usr/lib/systemd/system/fstrim.timer



-- 
Chris Murphy

[Neal Gompa]

On Wed, Dec 18, 2019 at 12:08 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  On Wed, Dec 18, 2019 at 10:56 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  >
>  > One thing I see in the Ubuntu fstrim.service unit file that I'm not
>  > seeing in the Fedora fstrim.service unit file, is a conditional for
>  > containers (line 4). I'm not sure where to ask about that. Maybe
>  > upstream systemd?
>
>  Found it. I'm not sure if util-linux would typically be found in a
>  container base image? Probably no point in calling fstrim in that
>  case, but also doesn't hurt.
>
>  $ sudo dnf provides /usr/lib/systemd/system/fstrim.timer
>  util-linux-2.34-3.fc31.x86_64 : A collection of basic system utilities
>  Repo        : @System
>  Matched from:
>  Filename    : /usr/lib/systemd/system/fstrim.timer
>  
Yeah, util-linux is pretty common in some types of containers. It
probably makes sense to send a PR to add that conditional.



--
真実はいつも一つ！/ Always, there's only one truth!

[Chris Murphy]

On Wed, Dec 18, 2019 at 11:10 AM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Wed, Dec 18, 2019 at 12:08 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  >
>  > On Wed, Dec 18, 2019 at 10:56 AM Chris Murphy <lists(a)colorremedies.com&gt;
> wrote:
>  > >
>  > > One thing I see in the Ubuntu fstrim.service unit file that I'm not
>  > > seeing in the Fedora fstrim.service unit file, is a conditional for
>  > > containers (line 4). I'm not sure where to ask about that. Maybe
>  > > upstream systemd?
>  >
>  > Found it. I'm not sure if util-linux would typically be found in a
>  > container base image? Probably no point in calling fstrim in that
>  > case, but also doesn't hurt.
>  >
>  > $ sudo dnf provides /usr/lib/systemd/system/fstrim.timer
>  > util-linux-2.34-3.fc31.x86_64 : A collection of basic system utilities
>  > Repo        : @System
>  > Matched from:
>  > Filename    : /usr/lib/systemd/system/fstrim.timer
>  >
>
>  Yeah, util-linux is pretty common in some types of containers. It
>  probably makes sense to send a PR to add that conditional. 
Appears to be in the upstream version already.

-- 
Chris Murphy

[Chris Murphy]

On Wed, Dec 18, 2019 at 5:32 AM Martin Kolman <mkolman(a)redhat.com&gt; wrote:

>  This will also trim thin LVs on thin pools (if any), right ?
>
>  So not just hardware, it can even make "software" storage layouts faster
>  & potentially even avoid pool exhaustion in some cases. :) 
Just a reminder, the underlying unit, fstrim.service, uses 'fstrim
--fstab' so only fstab file systems are affected. The user would need
to change the unit file to use --all instead of --fstab to affect all
mounted file systems. I'll include that info in the change wiki. I
imagine the best practice is to copy the original unit file, edit it,
and use it as a drop in unit file in /etc ?

Unfinished change, still in progress...
https://fedoraproject.org/wiki/Changes/EnableFSTrimTimer




-- 
Chris Murphy

[Karel Zak]

On Wed, Dec 18, 2019 at 02:42:42PM -0700, Chris Murphy wrote:
>  Just a reminder, the underlying unit, fstrim.service, uses
> 'fstrim
>  --fstab' so only fstab file systems are affected. The user would need
>  to change the unit file to use --all instead of --fstab to affect all 
We (upstream) originally had --all in the service file, but later
--fstab was requested by Debian guys. It seems better to default to
explicitly configured filesystems than blindly try all stuff
including removable media.

>  mounted file systems. I'll include that info in the change wiki.
> I
>  imagine the best practice is to copy the original unit file, edit it,
>  and use it as a drop in unit file in /etc ? 
Yes, if you need something else that the default.

    Karel

-- 
 Karel Zak  <kzak(a)redhat.com&gt;
 http://karelzak.blogspot.com