Hi, I've noticed when trying remediate performance issues in F34 under a vm, that fedora does not have a swap specification in fstab anymore, but is using, in my case, and 8GB swap partition in /dev/zram0. Does this mean that if I create a swap partition of a bigger size and specify it in fstab it will be ignored? Alternatively, given the fedora is using it's own rules for determining the swap size based on the amount of memory available, is there any way to increase the size of /dev/zram0 so that there is more swap space available to the system?
regards, Steve
On 19/06/2021 12:45, Stephen Morris wrote:
Hi, I've noticed when trying remediate performance issues in F34 under a vm, that fedora does not have a swap specification in fstab anymore, but is using, in my case, and 8GB swap partition in /dev/zram0. Does this mean that if I create a swap partition of a bigger size and specify it in fstab it will be ignored?
Ignored? No. But not primary. e.g.
[egreshko@meimei ~]$ swapon NAME TYPE SIZE USED PRIO /dev/sda2 partition 16.9G 0B -2 /dev/zram0 partition 4.7G 0B 100
So, it may or may not be used.
Alternatively, given the fedora is using it's own rules for determining the swap size based on the amount of memory available, is there any way to increase the size of /dev/zram0 so that there is more swap space available to the system?
I think
man zram-generator
will help in that.
On 19/6/21 15:31, Ed Greshko wrote:
On 19/06/2021 12:45, Stephen Morris wrote:
Hi, I've noticed when trying remediate performance issues in F34 under a vm, that fedora does not have a swap specification in fstab anymore, but is using, in my case, and 8GB swap partition in /dev/zram0. Does this mean that if I create a swap partition of a bigger size and specify it in fstab it will be ignored?
Ignored? No. But not primary. e.g.
[egreshko@meimei ~]$ swapon NAME TYPE SIZE USED PRIO /dev/sda2 partition 16.9G 0B -2 /dev/zram0 partition 4.7G 0B 100
So, it may or may not be used.
Alternatively, given the fedora is using it's own rules for determining the swap size based on the amount of memory available, is there any way to increase the size of /dev/zram0 so that there is more swap space available to the system?
I think
man zram-generator
will help in that.
Thanks Ed, I'll check that man data out. In my case with an auto storage configuration at F34 install time there isn't any swap entry in fstab or swap partition. I think I need to expand this in virtualbox as I think that virtualbox's lack of video memory is causing performance issues in F34.
regards, Steve
On 19/06/2021 21:44, Stephen Morris wrote:
On 19/6/21 15:31, Ed Greshko wrote:
On 19/06/2021 12:45, Stephen Morris wrote:
Hi, I've noticed when trying remediate performance issues in F34 under a vm, that fedora does not have a swap specification in fstab anymore, but is using, in my case, and 8GB swap partition in /dev/zram0. Does this mean that if I create a swap partition of a bigger size and specify it in fstab it will be ignored?
Ignored? No. But not primary. e.g.
[egreshko@meimei ~]$ swapon NAME TYPE SIZE USED PRIO /dev/sda2 partition 16.9G 0B -2 /dev/zram0 partition 4.7G 0B 100
So, it may or may not be used.
Alternatively, given the fedora is using it's own rules for determining the swap size based on the amount of memory available, is there any way to increase the size of /dev/zram0 so that there is more swap space available to the system?
I think
man zram-generator
will help in that.
Thanks Ed, I'll check that man data out. In my case with an auto storage configuration at F34 install time there isn't any swap entry in fstab or swap partition. I think I need to expand this in virtualbox as I think that virtualbox's lack of video memory is causing performance issues in F34.
My system has been upgraded from versions without ZRAM. That is the reason my system has a defined swap partition on disk.
I don't see the connection between Video Memory and swap.
VirtualBox has a per-VM setting for its display and video memory. At least that is the case for VirtualBox running as a host on linux. I seem to recall you're using VirtualBox on HW running Windows?
One of the first things I did after installing F34 is disable swap-on-zram: touch /etc/systemd/zram-generator.conf and define a swap partition in fstab.
Bill
On 6/19/2021 12:45 AM, Stephen Morris wrote:
Hi, I've noticed when trying remediate performance issues in F34 under a vm, that fedora does not have a swap specification in fstab anymore, but is using, in my case, and 8GB swap partition in /dev/zram0. Does this mean that if I create a swap partition of a bigger size and specify it in fstab it will be ignored? Alternatively, given the fedora is using it's own rules for determining the swap size based on the amount of memory available, is there any way to increase the size of /dev/zram0 so that there is more swap space available to the system?
regards, Steve _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
The server is running on Raid-1 SSDs with 64GB of RAM
Bill
On 6/21/2021 3:41 AM, Samuel Sieb wrote:
On 6/20/21 7:25 PM, Bill Shirley wrote:
One of the first things I did after installing F34 is disable swap-on-zram: touch /etc/systemd/zram-generator.conf and define a swap partition in fstab.
Why? _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On 21/06/2021 16:05, Bill Shirley wrote:
The server is running on Raid-1 SSDs with 64GB of RAM
I suppose the follow-up question would be are you seeing the swap partition actually being used?
Does "swapon" show it has been used?
[0 07:20:34 root@yoda33 ~]$ swapon NAME TYPE SIZE USED PRIO /dev/sdc1 partition 64G 9.7M 300 /dev/sdd1 partition 64G 9.4M 300 [0 03:06:33 root@yoda33 ~]$ uptime 03:08:30 up 23 days, 18:33, 2 users, load average: 0.01, 0.01, 0.00
On 6/21/2021 4:18 AM, Ed Greshko wrote:
On 21/06/2021 16:05, Bill Shirley wrote:
The server is running on Raid-1 SSDs with 64GB of RAM
I suppose the follow-up question would be are you seeing the swap partition actually being used?
Does "swapon" show it has been used?
The SSDs are a lot slower than compressing a page into RAM.
There was extensive discussion on the Fedora Devel list when this change was proposed.
Personally I was convinced that this change is an improvement for any system that is under memory pressure. I'm not going to try to recall the discussion as I may get some details wrong.
I switched it on in F33 and have for over a year seen no down side for my work loads. My work loads are file+email server, firewall, KDE desktops, Kodi music server.
At my work once we get on to Centos 8 I'm planning to performance test with zram swap. We have a work load that is very sensitive to disk I/O spikes and there is some sad code that uses swap for 10s to 15s every 20mins of so that I want to make go away. We have RAID-10 SSD where we see issues.
Barry
On 21 Jun 2021, at 09:05, Bill Shirley bill@ShirleyFamily.net wrote:
The server is running on Raid-1 SSDs with 64GB of RAM
Bill
On 6/21/2021 3:41 AM, Samuel Sieb wrote:
On 6/20/21 7:25 PM, Bill Shirley wrote:
One of the first things I did after installing F34 is disable swap-on-zram: touch /etc/systemd/zram-generator.conf and define a swap partition in fstab.
Why? _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On 2021-06-21 1:05 a.m., Bill Shirley wrote:
The server is running on Raid-1 SSDs with 64GB of RAM
Bill
On 6/21/2021 3:41 AM, Samuel Sieb wrote:
On 6/20/21 7:25 PM, Bill Shirley wrote:
One of the first things I did after installing F34 is disable swap-on-zram: touch /etc/systemd/zram-generator.conf and define a swap partition in fstab.
Why?
I don't see how that's an answer to why you would disable zram. Especially when your later reply shows that you're not really even using the disk swap anyway.
On Mon, Jun 21, 2021 at 10:51 AM Barry Scott barry@barrys-emacs.org wrote:
The SSDs are a lot slower than compressing a page into RAM.
There was extensive discussion on the Fedora Devel list when this change was proposed.
Personally I was convinced that this change is an improvement for any system that is under memory pressure. I'm not going to try to recall the discussion as I may get some details wrong.
At a high level, zram is a ram disk that has transparent compression. You can format it with mkswap or any other file system, use it as a block device.
But nuts and bolts memory management, reclaim, paging in and out, it's quite complicated. There's work happening since kernel 5.8 to make swap a lot more effective. And on going work to make mm and zswap do the right thing. And zswap is a different thing altogether, it's a front cache that uses a compressed memory pool as a cache for a conventional swap file or partition. And it works on an least recently used basis. So it has a way of determining what's stale and pushing that out to disk, while keeping recent things in the (memory) cache. In this case we don't have the concerns with priority inversions that can happen when a particular sequence of events happens:
1. zram based swap has higher priority 2. conventional swap has lower priority 3. early workloads fill up zram with stale things not used again later 4. the general workload ends up using disk based swap
So this is not really any worse than before at this point except it is consuming some memory, just to keep stale things available in case they get used. And if they do get used, it'll be quite fast. That's not obviously a bad thing, except it is taking a limited resource off the table. That's atypical for desktop workloads. But you can imagine that the more resources a system has the more variable the workload can be, and you could see early swap fill up a zram device, and then it can't be used again until the programs that created those anonymous pages are quit, in the extreme case.
Anyway, how to optimize was the whole point of moving to zram based swap. And it won't stop there. There is still more work happening to get zswap cgroups aware. Neither zram nor zswap are at the moment so for resource control purposes, we actually need a plain swap partition or swap file, it can't even be on dm-crypt at the moment. And one of the nice things about zram based swap is, it's volatile, so we have less security concerns about questionable things ending up on persistent storage that don't even go in a user's ~/home. Anything could be evicted to swap.
I switched it on in F33 and have for over a year seen no down side for my work loads. My work loads are file+email server, firewall, KDE desktops, Kodi music server.
At my work once we get on to Centos 8 I'm planning to performance test with zram swap. We have a work load that is very sensitive to disk I/O spikes and there is some sad code that uses swap for 10s to 15s every 20mins of so that I want to make go away. We have RAID-10 SSD where we see issues.
With centos 8 kernels you'd probably use zram based swap because it's more mature in the older kernels. If you are able to use an elrepo kernel you could try changing nothing else and see if the kernel 5.8+ changes help your workload all by themselves. And if not you could look at either zram based swap or zswap.
On Mon, Jun 21, 2021 at 1:25 PM Samuel Sieb samuel@sieb.net wrote:
On 2021-06-21 1:05 a.m., Bill Shirley wrote:
The server is running on Raid-1 SSDs with 64GB of RAM
Bill
On 6/21/2021 3:41 AM, Samuel Sieb wrote:
On 6/20/21 7:25 PM, Bill Shirley wrote:
One of the first things I did after installing F34 is disable swap-on-zram: touch /etc/systemd/zram-generator.conf and define a swap partition in fstab.
Why?
I don't see how that's an answer to why you would disable zram. Especially when your later reply shows that you're not really even using the disk swap anyway.
Lots of folks don't realize that zram devices don't use any memory (small amount of overhead based on the size of the zram device, and driver; less than 0.1% of the zram device size), and that it's dynamically allocated. But it's true that swap efficacy as a percentage cannot be 100% like it is with disk or file based swap. That it's so much faster makes up for the lower efficacy. By that I mean, a 4 KiB page being swapped out to disk means you free 4 KiB RAM and consume 4 KiB on disk. With zram based swap, it's still memory, but it's compressed. So you free up 4 KiB uncompressed memory for other things; and you consume ~2 KiB RAM for that compressed page in the zram device. The efficacy is related to the compression ratio you get, which is anywhere 2:1 to 3:1. So it's an efficacy of 50% to 75%.
For sure it's better than no swap which has an efficacy of 0% :) In fact that's misleading because when a system can't evict dirty pages at all, it's forced to do file page reclaim, i.e. libraries, executables, configurations that exist as files on disk, can be removed from memory via reclaim, because they're already on disk and can just be read back in. But when under memory pressure, reclaim can look a lot like swap thrashing and even compete with it. So some swap is better, and also due to SSDs, we're probably better off with a higher swappiness value, i.e. give equal weight to page out of anonymous pages as reclaim.
But some workloads are different and you can actually get a kind of in-memory swap thrashing same as on disk. It's so fast that it's normally not a problem. Until it is. So we definitely want to keep an eye on reports of folks having issues that sound like hangs or lockups, any time the desktop becomes unresponsive we want to find out what's going on
First thing to check in those cases is if the system is running uresourced. It's only enabled by default on GNOME right now. But it's considered safe to run as an opt in for other desktops, though we want to keep an eye on possible regressions.
There is still more work to do in this area, in particular wiring up the IO isolation. Any time there's memory pressure it'll quickly lead to IO pressure: reclaim and swap.
On 20/6/21 02:22, Ed Greshko wrote:
On 19/06/2021 21:44, Stephen Morris wrote:
On 19/6/21 15:31, Ed Greshko wrote:
On 19/06/2021 12:45, Stephen Morris wrote:
Hi, I've noticed when trying remediate performance issues in F34 under a vm, that fedora does not have a swap specification in fstab anymore, but is using, in my case, and 8GB swap partition in /dev/zram0. Does this mean that if I create a swap partition of a bigger size and specify it in fstab it will be ignored?
Ignored? No. But not primary. e.g.
[egreshko@meimei ~]$ swapon NAME TYPE SIZE USED PRIO /dev/sda2 partition 16.9G 0B -2 /dev/zram0 partition 4.7G 0B 100
So, it may or may not be used.
Alternatively, given the fedora is using it's own rules for determining the swap size based on the amount of memory available, is there any way to increase the size of /dev/zram0 so that there is more swap space available to the system?
I think
man zram-generator
will help in that.
Thanks Ed, I'll check that man data out. In my case with an auto storage configuration at F34 install time there isn't any swap entry in fstab or swap partition. I think I need to expand this in virtualbox as I think that virtualbox's lack of video memory is causing performance issues in F34.
My system has been upgraded from versions without ZRAM. That is the reason my system has a defined swap partition on disk.
I don't see the connection between Video Memory and swap.
As I understand it, because I'm not using a dedicated graphics card the video/graphics memory used by the vm is being sourced from real memory, and I assume as part of the memory allocation being given to the vm, hence would potentially increase the requirement for memory paging.
VirtualBox has a per-VM setting for its display and video memory. At least that is the case for VirtualBox running as a host on linux. I seem to recall you're using VirtualBox on HW running Windows?
I am running Virtualbox on a Windows 10 host as I am running on a raid 10 motherboard supplied raid environment and Fedora workstation won't install to raid (it can't see any devices), but having said that though, Windows 10 can't see any devices to install to, but the motherboard bios provides facilities to generate the necessary raid drivers to specify at windows install time, but unfortunately it doesn't generate linux drivers. The possible issue with video memory in virtualbox is the windows version of virtualbox only allows a maximum of 128MB to be allocated, which I think is no where near enough, hence the performance issues. This is also why I'm still using Vmware player, as it allows significantly more video memory allocation, and I was giving it 2GB of video memory, but with that I was getting performance issues (I was allocating 16GB of memory to the vm) and when I dropped the video memory allocation back to the recommended of 768MB performance improved.
regards, Steve
On 2021-06-21 4:29 p.m., Stephen Morris wrote:
On 20/6/21 02:22, Ed Greshko wrote:
On 19/06/2021 21:44, Stephen Morris wrote: My system has been upgraded from versions without ZRAM. That is the reason my system has a defined swap partition on disk.
I don't see the connection between Video Memory and swap.
As I understand it, because I'm not using a dedicated graphics card the video/graphics memory used by the vm is being sourced from real memory, and I assume as part of the memory allocation being given to the vm, hence would potentially increase the requirement for memory paging.
The system graphics card is irrelevant. The VM display is virtual anyway, so I expect any memory used by the virtual graphics card comes from system RAM.
VirtualBox has a per-VM setting for its display and video memory. At least that is the case for VirtualBox running as a host on linux. I seem to recall you're using VirtualBox on HW running Windows?
I am running Virtualbox on a Windows 10 host as I am running on a raid 10 motherboard supplied raid environment and Fedora workstation won't install to raid (it can't see any devices), but having said that though, Windows 10 can't see any devices to install to, but the motherboard bios provides facilities to generate the necessary raid drivers to specify at windows install time, but unfortunately it doesn't generate linux drivers. The possible issue with video memory in virtualbox is the windows version of virtualbox only allows a maximum of 128MB to be allocated, which I think is no where near enough, hence the performance issues. This is also why I'm still using Vmware player, as it allows significantly more video memory allocation, and I was giving it 2GB of video memory, but with that I was getting performance issues (I was allocating 16GB of memory to the vm) and when I dropped the video memory allocation back to the recommended of 768MB performance improved.
You don't mention how much RAM the computer has, but 2GB of video memory for a VM seems extremely excessive. Even the 768MB seems far beyond anything useful. There's a reason virtualbox has a max of 128MB.
On 22/6/21 11:06, Samuel Sieb wrote:
On 2021-06-21 4:29 p.m., Stephen Morris wrote:
On 20/6/21 02:22, Ed Greshko wrote:
On 19/06/2021 21:44, Stephen Morris wrote: My system has been upgraded from versions without ZRAM. That is the reason my system has a defined swap partition on disk.
I don't see the connection between Video Memory and swap.
As I understand it, because I'm not using a dedicated graphics card the video/graphics memory used by the vm is being sourced from real memory, and I assume as part of the memory allocation being given to the vm, hence would potentially increase the requirement for memory paging.
The system graphics card is irrelevant. The VM display is virtual anyway, so I expect any memory used by the virtual graphics card comes from system RAM.
VirtualBox has a per-VM setting for its display and video memory. At least that is the case for VirtualBox running as a host on linux. I seem to recall you're using VirtualBox on HW running Windows?
I am running Virtualbox on a Windows 10 host as I am running on a raid 10 motherboard supplied raid environment and Fedora workstation won't install to raid (it can't see any devices), but having said that though, Windows 10 can't see any devices to install to, but the motherboard bios provides facilities to generate the necessary raid drivers to specify at windows install time, but unfortunately it doesn't generate linux drivers. The possible issue with video memory in virtualbox is the windows version of virtualbox only allows a maximum of 128MB to be allocated, which I think is no where near enough, hence the performance issues. This is also why I'm still using Vmware player, as it allows significantly more video memory allocation, and I was giving it 2GB of video memory, but with that I was getting performance issues (I was allocating 16GB of memory to the vm) and when I dropped the video memory allocation back to the recommended of 768MB performance improved.
You don't mention how much RAM the computer has, but 2GB of video memory for a VM seems extremely excessive. Even the 768MB seems far beyond anything useful. There's a reason virtualbox has a max of 128MB.
My system has 32GB of memory. I found that 768MB for the video memory (which is the recommended amount) was the optimal amount of memory for performance as decreasing the memory below that started producing performance issues in F34 as well, so I've left it as that.
regards, Steve
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-
Barry Scott -
Bill Shirley -
Chris Murphy -
Ed Greshko -
Samuel Sieb -
Stephen Morris