stratis: fs exhausted?

[aanno]

Hello,

I'm using stratis for many years now (for my /home and /opt mount 
points). It looks like this:

$ stratis pool
Name                         Total / Used / Free    Properties 
                                   UUID   Alerts
stratis_hdd   1.43 TiB / 819.51 GiB / 645.32 GiB    Ca,~Cr, Op 
   093c8d42-21b8-46a2-a7e8-5d35f458fa58   WS001

$ stratis fs
Pool          Filesystem   Total / Used / Free / Limit 
              Created             Device                          UUID

stratis_hdd   home         2 TiB / 708.92 GiB / 1.31 TiB / None     Apr 
28 2019 12:29   /dev/stratis/stratis_hdd/home   1715
5095-e225-4fb0-b020-ec2ffa6a5e4d
stratis_hdd   opt          1 TiB / 109.11 GiB / 914.89 GiB / None   Apr 
28 2019 12:30   /dev/stratis/stratis_hdd/opt    fb19
a29e-ab39-4b41-8d37-0dc6d222a2b9

$ stratis blockdev
Pool Name     Device Node   Physical Size    Tier   UUID
stratis_hdd   /dev/dm-6        292.95 GiB   CACHE 
   770d0190-c17b-4349-a9f9-0968036e6b2c
stratis_hdd   /dev/dm-7          1.43 TiB    DATA 
   9a5b1c4d-8014-4155-990e-eedfd27803a6

However, since a few weeks my /home has become unstable. Newly created 
(or altered) files are (sometimes) corrupted after a reboot of the system.

For both mount points I'm using a SSD cache in front of a HDD. It feels 
a bit like the caching layer is all right, but the HDD layer is on error 
(exhausted?). For me, it looks like that would explain the observed 
behaviour.

I wonder if

* I could somehow 'disable' (or remove) the SSD caching layer?
* I could get some debug informations?

Kind regards,

aanno

[Bryan Gurney]

Hi,

Can you please provide some information on the following:

- The version of the stratisd and stratis-cli packages being used
- The version of the kernel running ("uname -r")
- The version of the distribution (e.g.: "Fedora 38")
- The output of "lsblk -i" that details the block devices in the Stratis pool

Also, please look through the kernel logs (i.e.: "journalctl -kb"),
and check for any error messages from the following sources:

"kernel device-mapper: "
"kernel: XFS ...:"
"kernel: sd..." or "kernel: ata..." (for the driver handling the data
and cache drives on the system)


Thanks,

Bryan


On Thu, Dec 7, 2023 at 1:37 PM aanno <aannoaanno(a)gmail.com&gt; wrote:
>  Hello,
>
>  I'm using stratis for many years now (for my /home and /opt mount points). It looks
> like this:
>
>  $ stratis pool
>  Name                         Total / Used / Free    Properties                           
>        UUID   Alerts
>  stratis_hdd   1.43 TiB / 819.51 GiB / 645.32 GiB    Ca,~Cr, Op  
> 093c8d42-21b8-46a2-a7e8-5d35f458fa58   WS001
>
>  $ stratis fs
>  Pool          Filesystem   Total / Used / Free / Limit              Created            
> Device                          UUID
>
>  stratis_hdd   home         2 TiB / 708.92 GiB / 1.31 TiB / None     Apr 28 2019 12:29  
> /dev/stratis/stratis_hdd/home   1715
>  5095-e225-4fb0-b020-ec2ffa6a5e4d
>  stratis_hdd   opt          1 TiB / 109.11 GiB / 914.89 GiB / None   Apr 28 2019 12:30  
> /dev/stratis/stratis_hdd/opt    fb19
>  a29e-ab39-4b41-8d37-0dc6d222a2b9
>
>  $ stratis blockdev
>  Pool Name     Device Node   Physical Size    Tier   UUID
>  stratis_hdd   /dev/dm-6        292.95 GiB   CACHE   770d0190-c17b-4349-a9f9-0968036e6b2c
>  stratis_hdd   /dev/dm-7          1.43 TiB    DATA   9a5b1c4d-8014-4155-990e-eedfd27803a6
>
>  However, since a few weeks my /home has become unstable. Newly created (or altered) files
> are (sometimes) corrupted after a reboot of the system.
>
>  For both mount points I'm using a SSD cache in front of a HDD. It feels a bit like
> the caching layer is all right, but the HDD layer is on error (exhausted?). For me, it
> looks like that would explain the observed behaviour.
>
>  I wonder if
>
>  * I could somehow 'disable' (or remove) the SSD caching layer?
>  * I could get some debug informations?
>
>  Kind regards,
>
>  aanno
>
>  --
>  _______________________________________________
>  stratis-devel mailing list -- stratis-devel(a)lists.fedorahosted.org
>  To unsubscribe send an email to stratis-devel-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[John Baublitz]

Hi aanno,
I would also like to just mention that when you say the physical space is
exhausted, I'm assuming you're referring to alert WS001, correct? We may
need to document this a little bit better because this alert just means
that all of the space in the pool is fully allocated. This alert
specifically pops up when:

1. The pool extends to take up all available physical space (this will not
set the warning you are seeing).
2. You fill up the remaining space to the point where only 15 GiB is left.
3. The pool attempts to extend the data device further and determines there
is no space left to extend (this is when the warning is set).

Now practically what this means is that when the warning is set, you have
15GiB left of space. If you fill up the 15GiB, the pool will not be able to
be extended further to help you out. What I'm mostly curious about is,
looking at the space you reported in stratis pool/fs it looks like the
physical space on the pool is not full enough to warrant the alert. Is this
output after deleting a significant amount of data from the pool? From what
I can see, you shouldn't have this alert set unless your usage spiked
significantly and went back down. The warning doesn't clear out unless you
add more space so the state you're in is not impossible or even unlikely,
but it does make me wonder a little bit about your usage pattern. Are you
still seeing this instability even with the lower usage you pasted here?

On Thu, Dec 7, 2023 at 1:37 PM aanno <aannoaanno(a)gmail.com&gt; wrote:

>  Hello,
>
>  I'm using stratis for many years now (for my /home and /opt mount points).
>  It looks like this:
>
>  $ stratis pool
>  Name                         Total / Used / Free    Properties
>                                    UUID   Alerts
>  stratis_hdd   1.43 TiB / 819.51 GiB / 645.32 GiB    Ca,~Cr, Op
>    093c8d42-21b8-46a2-a7e8-5d35f458fa58   WS001
>
>  $ stratis fs
>  Pool          Filesystem   Total / Used / Free / Limit
>               Created             Device                          UUID
>
>  stratis_hdd   home         2 TiB / 708.92 GiB / 1.31 TiB / None     Apr 28
>  2019 12:29   /dev/stratis/stratis_hdd/home   1715
>  5095-e225-4fb0-b020-ec2ffa6a5e4d
>  stratis_hdd   opt          1 TiB / 109.11 GiB / 914.89 GiB / None   Apr 28
>  2019 12:30   /dev/stratis/stratis_hdd/opt    fb19
>  a29e-ab39-4b41-8d37-0dc6d222a2b9
>
>  $ stratis blockdev
>  Pool Name     Device Node   Physical Size    Tier   UUID
>
>  stratis_hdd   /dev/dm-6        292.95 GiB   CACHE
>    770d0190-c17b-4349-a9f9-0968036e6b2c
>  stratis_hdd   /dev/dm-7          1.43 TiB    DATA
>    9a5b1c4d-8014-4155-990e-eedfd27803a6
>
>  However, since a few weeks my /home has become unstable. Newly created (or
>  altered) files are (sometimes) corrupted after a reboot of the system.
>
>  For both mount points I'm using a SSD cache in front of a HDD. It feels a
>  bit like the caching layer is all right, but the HDD layer is on error
>  (exhausted?). For me, it looks like that would explain the observed
>  behaviour.
>
>  I wonder if
>
>  * I could somehow 'disable' (or remove) the SSD caching layer?
>  * I could get some debug informations?
>
>  Kind regards,
>
>  aanno
>
>  --
>  _______________________________________________
>  stratis-devel mailing list -- stratis-devel(a)lists.fedorahosted.org
>  To unsubscribe send an email to stratis-devel-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>  

-- 
John
he/him
Principal Software Engineer, Stratis team