Hello Fedora musicians, I've been lurking this list for a little bit and
this is my first time chiming in on something.
I think it is important to pursue an official realtime kernel for Fedora. I think a distribution focused on audio without a realtime kernel would have a serious bug, that IMO, would be worth delaying publication for. ...
Real Time Kernels are available from PlanetCCRMA:
http://ccrma.stanford.edu/planetccrma/mirror/fedora/linux/planetcore/20/x86_...
There are concerns about the implementation of Real Time Kernel as expressed in the Musician's Guide:
http://docs.fedoraproject.org/en-US/Fedora/15/html/Musicians_Guide/sect-Musi...
I am not a systems programmer so I can't speak to these concerns. Some of the names of the real time kernel developers listed on the PlanetCCRMA kernel-rt page are Red Hat employees.
I would like to see Fedora be the premier linux distribution for music. But until we can overcome the concerns listed in the Musician's Guide we will probably not have a real time kernel in the Fedora repositories. Maybe a Fedora "Re-Mix", or Fedora.Next and Workstation with the works with Fedora software library may break the ice. -- Jeff
One of the reasons I'd like to see a realtime (and/or low-latency) patched version of Linux in Fedora is to keep it more up-to-date than Planet CCRMA. The latest release in the CCRMA repo at the moment is 3.14 whereas the current stable version of Linux in Fedora Updates is 3.16 and 3.17 is in testing. I don't think being behind a week or two in updates would be a big deal, but being behind 3 releases of Linux is. I have no experience with RPM or creating & maintaining other packages, but I have been considering learning so I can contribute to Fedora. I am willing to help with maintaining a realtime and/or low-latency kernel package, but I think someone with more experience maintaining packages should be at the lead of this. I have also been thinking about packaging a Linux-libre realtime kernel for Freedora (http://www.fsfla.org/ikiwiki/selibre/linux-libre/freed-ora.en.html ). Having an up-to-date realtime kernel in the Fedora repos would facilitate that. Regarding the security and stability concerns, yes, those are real concerns. Albeit rarely, I have had a few issues with some misbehaving program (I think Firefox or Chromimum, not sure though) locking the whole system when running the Planet CCRMA kernel for normal desktop use, so I only boot into the Planet CCRMA kernel when I need to. I am not sure, but I don't think the low-latency kernel that is default in Ubuntu Studio has the same issues. Hopefully someone more knowledgable can chime in. If no one knows, we could test out how Ubuntu Studio runs, although I probably won't have much time to do so for a few weeks. We could do what Ubuntu Studio does and package a low-latency kernel as default in Fedora Jam and also have a realtime kernel package in the repositories. I'm not sure what exactly is different between the generic and low-latency kernels, but I don't think it would be too difficult to figure that out by looking at the Ubuntu packages. They have a wiki page explaining the different kernels they package: https://help.ubuntu.com/community/UbuntuStudio/RealTimeKernel
Sent: Thursday, November 13, 2014 at 2:28 PM From: "Jeff Sandys" jpsandys@gmail.com To: music@lists.fedoraproject.org Subject: Re: [Fedora-music-list] Low Latency vs. Real Time Kernel - actual latencies ?
Hello Fedora musicians, I've been lurking this list for a little bit and this is my first time chiming in on something.
I think it is important to pursue an official realtime kernel for Fedora. I think a distribution focused on audio without a realtime kernel would have a serious bug, that IMO, would be worth delaying publication for. ... Real Time Kernels are available from PlanetCCRMA: http://ccrma.stanford.edu/planetccrma/mirror/fedora/linux/planetcore/20/x86_... There are concerns about the implementation of Real Time Kernel as expressed in the Musician's Guide: http://docs.fedoraproject.org/en-US/Fedora/15/html/Musicians_Guide/sect-Musi...] I am not a systems programmer so I can't speak to these concerns. Some of the names of the real time kernel developers listed on the PlanetCCRMA kernel-rt page are Red Hat employees. I would like to see Fedora be the premier linux distribution for music. But until we can overcome the concerns listed in the Musician's Guide we will probably not have a real time kernel in the Fedora repositories. Maybe a Fedora "Re-Mix", or Fedora.Next and Workstation with the works with Fedora software library may break the ice. -- Jeff _______________________________________________ music mailing list music@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/music%5Bhttps://admin.fedor...]
Sent: Thursday, November 13, 2014 at 10:59 AM From: "Brian Monroe" briancmonroe@gmail.com To: "Be Ing" be.0@gmx.com, music@lists.fedoraproject.org Subject: Re: [Fedora-music-list] Low Latency vs. Real Time Kernel - actual latencies ?I think so too, thanks for chiming in.
I'm still waiting to get into the packagers group, but I have a koji account and theoretically could compile an rt kernel. I think the standard naming schema in other distros is kernel-rt. It should be only adding a few lines to the spec file to enable the rt kernel, but when you look at how many kernel update there are for Fedora every week, I'm not sure as to how up to date we'll be able to keep up due to the work load. We're already are down on developers, and people like Brandon are keeping us afloat.
Are we going to be ok as a project to be behind a week or two in Kernel releases? Personally I'm for more stable kernels when it comes to music production vs. having the latest and greatest, but I also think that should be a clearly indicated as a feature
That being said, I feel strongly as though others should take this task on, if not me, then someone else or better yet, a few of us. I'm looking into the Ubuntu Studio and turns out they dropped the RT kernel as default. They're using a "lowlatency" kernel instead of a rt kernel (though they do still supply an rt kernel but not by default). I do know that users are able to get 1.5 ms latency with zero xruns so I'm guessing they're doing something other than real-time scheduling, I just don't know what. Thoughts?
hi all, it sounds interesting, i'll be glad to contribute in some way, even if i'm not a developer. personally i guess that a low latency kernel fits most of the use cases, but it would be definitely interesting working on this.. according to my limited knowledge on this topics, the current kernel should be "near" lowlatency:
here's my (default) config
grep CONFIG_PREEMPT /boot/config-$(uname -r) # CONFIG_PREEMPT_RCU is not set CONFIG_PREEMPT_NOTIFIERS=y # CONFIG_PREEMPT_NONE is not set CONFIG_PREEMPT_VOLUNTARY=y # CONFIG_PREEMPT is not set
grep CONFIG_HZ /boot/config-$(uname -r) # CONFIG_HZ_PERIODIC is not set # CONFIG_HZ_100 is not set # CONFIG_HZ_250 is not set # CONFIG_HZ_300 is not set CONFIG_HZ_1000=y CONFIG_HZ=1000
probably, as a first step, it would be good to compile the kernel with CONFIG_PREEMP=y
i've applied the recommendations given by QuickScan, but still having "not good" for limits.conf even though the new limits should be fetched from /etc/security/limits.d/95-jack.conf . i'm not finding the cpugovernors, probably they're not compiled in the kernel and it would be worth to give it a try (AVLinux gives the chance to set up those from the panel)
As Brian stated, in the early UbuntuStudio releases a rt kernel was delivered by default, but then they moved to lowlatency
according to the kernel archives, the latest rt patch is for 3.14...
https://www.kernel.org/pub/linux/kernel/projects/rt/3.14/
2014-11-13 22:39 GMT+01:00 Be Ing be.0@gmx.com:
One of the reasons I'd like to see a realtime (and/or low-latency) patched version of Linux in Fedora is to keep it more up-to-date than Planet CCRMA. The latest release in the CCRMA repo at the moment is 3.14 whereas the current stable version of Linux in Fedora Updates is 3.16 and 3.17 is in testing. I don't think being behind a week or two in updates would be a big deal, but being behind 3 releases of Linux is.
I have no experience with RPM or creating & maintaining other packages, but I have been considering learning so I can contribute to Fedora. I am willing to help with maintaining a realtime and/or low-latency kernel package, but I think someone with more experience maintaining packages should be at the lead of this.
I have also been thinking about packaging a Linux-libre realtime kernel for Freedora ( http://www.fsfla.org/ikiwiki/selibre/linux-libre/freed-ora.en.html ). Having an up-to-date realtime kernel in the Fedora repos would facilitate that.
Regarding the security and stability concerns, yes, those are real concerns. Albeit rarely, I have had a few issues with some misbehaving program (I think Firefox or Chromimum, not sure though) locking the whole system when running the Planet CCRMA kernel for normal desktop use, so I only boot into the Planet CCRMA kernel when I need to. I am not sure, but I don't think the low-latency kernel that is default in Ubuntu Studio has the same issues. Hopefully someone more knowledgable can chime in. If no one knows, we could test out how Ubuntu Studio runs, although I probably won't have much time to do so for a few weeks. We could do what Ubuntu Studio does and package a low-latency kernel as default in Fedora Jam and also have a realtime kernel package in the repositories. I'm not sure what exactly is different between the generic and low-latency kernels, but I don't think it would be too difficult to figure that out by looking at the Ubuntu packages. They have a wiki page explaining the different kernels they package: https://help.ubuntu.com/community/UbuntuStudio/RealTimeKernel
Sent: Thursday, November 13, 2014 at 2:28 PM From: "Jeff Sandys" jpsandys@gmail.com To: music@lists.fedoraproject.org Subject: Re: [Fedora-music-list] Low Latency vs. Real Time Kernel -
actual latencies ?
Hello Fedora musicians, I've been lurking this list for a little bit and
this is my first time chiming in on something.
I think it is important to pursue an official realtime kernel for Fedora.
I think a distribution focused on audio without a realtime kernel would have a serious bug, that IMO, would be worth delaying publication for.
...
Real Time Kernels are available from PlanetCCRMA:
http://ccrma.stanford.edu/planetccrma/mirror/fedora/linux/planetcore/20/x86_...
There are concerns about the implementation of Real Time Kernel as
expressed in the Musician's Guide:
http://docs.fedoraproject.org/en-US/Fedora/15/html/Musicians_Guide/sect-Musi...]
I am not a systems programmer so I can't speak to these concerns. Some
of the names of the real time kernel developers listed on the PlanetCCRMA kernel-rt page are Red Hat employees.
I would like to see Fedora be the premier linux distribution for music.
But until we can overcome the concerns listed in the Musician's Guide we will probably not have a real time kernel in the Fedora repositories. Maybe a Fedora "Re-Mix", or Fedora.Next and Workstation with the works with Fedora software library may break the ice.
-- Jeff _______________________________________________ music mailing list
music@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/music%5Bhttps://admin.fedor...]
Sent: Thursday, November 13, 2014 at 10:59 AM From: "Brian Monroe" briancmonroe@gmail.com To: "Be Ing" be.0@gmx.com, music@lists.fedoraproject.org Subject: Re: [Fedora-music-list] Low Latency vs. Real Time Kernel -
actual latencies ?I think so too, thanks for chiming in.
I'm still waiting to get into the packagers group, but I have a koji
account and theoretically could compile an rt kernel. I think the standard naming schema in other distros is kernel-rt. It should be only adding a few lines to the spec file to enable the rt kernel, but when you look at how many kernel update there are for Fedora every week, I'm not sure as to how up to date we'll be able to keep up due to the work load. We're already are down on developers, and people like Brandon are keeping us afloat.
Are we going to be ok as a project to be behind a week or two in Kernel
releases? Personally I'm for more stable kernels when it comes to music production vs. having the latest and greatest, but I also think that should be a clearly indicated as a feature
That being said, I feel strongly as though others should take this task
on, if not me, then someone else or better yet, a few of us.
I'm looking into the Ubuntu Studio and turns out they dropped the RT
kernel as default. They're using a "lowlatency" kernel instead of a rt kernel (though they do still supply an rt kernel but not by default). I do know that users are able to get 1.5 ms latency with zero xruns so I'm guessing they're doing something other than real-time scheduling, I just don't know what. Thoughts? _______________________________________________ music mailing list music@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/music
On 15/11/14 13:05, The name of my band is 1984 wrote: <snip>
i've applied the recommendations given by QuickScan, but still having "not good" for limits.conf even though the new limits should be fetched from /etc/security/limits.d/95-jack.conf . i'm not finding the cpugovernors, probably they're not compiled in the kernel and it would be worth to give it a try (AVLinux gives the chance to set up those from the panel)
You need to install kernel-tools to get the governors. Some info here:
https://fedoraproject.org/wiki/JACK_Audio_Connection_Kit#CPU_Performance_Gov...
i've already installed the kernel-tools...anyway, just solved. i thought the modules for my cpu as well as cpufreq_stats were loaded by default but actually they weren't. my mistake. i've got the governors now.
thanx
2014-11-16 9:58 GMT+01:00 Brendan Jones brendan.jones.it@gmail.com:
On 15/11/14 13:05, The name of my band is 1984 wrote:
<snip>
i've applied the recommendations given by QuickScan, but still having "not good" for limits.conf even though the new limits should be fetched from /etc/security/limits.d/95-jack.conf . i'm not finding the cpugovernors, probably they're not compiled in the kernel and it would be worth to give it a try (AVLinux gives the chance to set up those from the panel)
You need to install kernel-tools to get the governors. Some info here:
https://fedoraproject.org/wiki/JACK_Audio_Connection_ Kit#CPU_Performance_Governing
music mailing list music@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/music
On 13/11/14 22:39, Be Ing wrote:
One of the reasons I'd like to see a realtime (and/or low-latency) patched version of Linux in Fedora is to keep it more up-to-date than Planet CCRMA. The latest release in the CCRMA repo at the moment is 3.14 whereas the current stable version of Linux in Fedora Updates is 3.16 and 3.17 is in testing. I don't think being behind a week or two in updates would be a big deal, but being behind 3 releases of Linux is.
The latest realtime kernel _is_ 3.14.23. I think CCRMA has been doing a great job [1].
Some good points have been raised in this thread by you and others so I'll just vent ...
If you want a little bit of background into what goes in to creating the patch set read this article [2].
A lot of what was provided for earlier realtime kernels has been merged into the mainline kernel and for the majority of users this is more than sufficient (turn on thread irqs as a kernel parameter).
If you are suffering from latency problems, make sure to look to other sources such as wifi (blacklist it / remove from your initramfs if not required), GPU (eg. set your nvidia card to GPU/PowerMizer/Preferred Mode = MAx Performance) and do some research about the appropriate settings for your audio interface.
I can absolutely guarantee that only members of the kernel team would end up maintaining a realtime kernel for Fedora. That is why they are asking for empirical test data on why they should consider adopting one. Those problems/race conditions you've noticed will surely end up as bug reports that they have to action.
You could perform your own testing of an RT kernel starting with using CCRMA's kernels and compare them with the latest Fedora kernels (or even better grab the latest rawhide-no-debug kernel), and provide a link to the SRPM so that they can build the kernels themselves to verify.
For them to accept your data at all, you would need to do these tests without any third party drivers or software (ie. only software from Fedora mainline repositories). In terms of the test data they will be asking for, you need to show a marked improvement between the fedora kernels and realtime one.
If you want to learn what's involved I'd suggest downloading CCRMA's source RPM's and building a kernel yourself, and also follow the instructions here [3]. Its not as hard as it looks at first glance. Just time consuming.
[1] https://www.kernel.org/pub/linux/kernel/projects/rt/3.14/ [2] http://lwn.net/Articles/617140/ [3] https://fedoraproject.org/wiki/Building_a_custom_kernel
On Thu Nov 13 2014 at 12:28:27 PM Jeff Sandys jpsandys@gmail.com wrote:
Real Time Kernels are available from PlanetCCRMA:
http://ccrma.stanford.edu/planetccrma/mirror/fedora/linux/planetcore/20/x86_...
There are concerns about the implementation of Real Time Kernel as expressed in the Musician's Guide:
http://docs.fedoraproject.org/en-US/Fedora/15/html/Musicians_Guide/sect-Musi...
I am not a systems programmer so I can't speak to these concerns. Some of the names of the real time kernel developers listed on the PlanetCCRMA kernel-rt page are Red Hat employees.
I would like to see Fedora be the premier linux distribution for music. But until we can overcome the concerns listed in the Musician's Guide we will probably not have a real time kernel in the Fedora repositories. Maybe a Fedora "Re-Mix", or Fedora.Next and Workstation with the works with Fedora software library may break the ice. -- Jeff
Well, I think those concerns are why Ubuntu moved to their low-latency kernel instead of their kernel-rt. According to their documentation (wiki) the lowlatency kernel is a "soft" preempt kernel and the rt kernel is "hard' but there's no docs on what make them different, and people in channel don't seem to know.
Following their lead may resolve some of those concerns, and also may indicate the time to move away from KDE for potential stability reasons.
On 11/19/2014 08:01 AM, Brian Monroe wrote:
On Thu Nov 13 2014 at 12:28:27 PM Jeff Sandys <jpsandys@gmail.com mailto:jpsandys@gmail.com> wrote:
Real Time Kernels are available from PlanetCCRMA: http://ccrma.stanford.edu/planetccrma/mirror/fedora/linux/planetcore/20/x86_64/repoview/kernel-rt.html There are concerns about the implementation of Real Time Kernel as expressed in the Musician's Guide: http://docs.fedoraproject.org/en-US/Fedora/15/html/Musicians_Guide/sect-Musicians_Guide-CCRMA_Security_and_Stability.html I am not a systems programmer so I can't speak to these concerns. Some of the names of the real time kernel developers listed on the PlanetCCRMA kernel-rt page are Red Hat employees. I would like to see Fedora be the premier linux distribution for music. But until we can overcome the concerns listed in the Musician's Guide we will probably not have a real time kernel in the Fedora repositories. Maybe a Fedora "Re-Mix", or Fedora.Next and Workstation with the works with Fedora software library may break the ice.Well, I think those concerns are why Ubuntu moved to their low-latency kernel instead of their kernel-rt. According to their documentation (wiki) the lowlatency kernel is a "soft" preempt kernel and the rt kernel is "hard' but there's no docs on what make them different, and people in channel don't seem to know.
(sorry for the very late response to this thread - busy busy busy, I think it is still worth to comment on this)
Brian, there is no "hard realtime" kernel for Linux. There are only degrees of latency you can achieve with different combinations of compile-time options, patching and runtime optimization. All of them are "soft". You cannot guarantee deadlines on Linux - but you can get pretty close with a fully optimized system and the RT patch, and if you have the right hardware.
I don't know what options Ubunutu uses to build their "low latency" kernel, I imagine they just enable full preemption and irq threading. This gives you better latency than a "non-preempt" kernel, but in my experience that is not enough for realtime audio work with small buffers (by that I mean jack running at 128x2 or 64x2). YMMV.
To get really low latencies you need the RT patch (which has stability issues as has been mentioned in the thread - there are as many testers as for mainline). To fix those we would need more testers and proper bug reports that are followed up. But of course that needs kernel gurus that can respond.
If you read this article you will see that the future of the RT patch is not good at this point:
http://lwn.net/Articles/617140/
Let's hope this gets better (it is not the first time that the rt patch has lagged way behind mainline, I think a major rewrite happened in the Fedora 1 or 2 timeframe and for a while there was nothing new). And BTW, this is why we are currently stuck at 3.14.x in the Planet CCRMA kernels, there is nothing newer...
Sigh... -- Fernando
There's no such thing as a hard kernel. [sarcasm]But that means Ubuntu was wrong, or they lied! I'm shocked! Everything on the internets is true![/sarcasm]
Thanks for straitening that out for me. I had emailed the maintainer at Ubuntu but never got a response. (maybe my emails give offensive order because I haven't heard from anyone at RedHat lately either) Sigh...
I'm bummed to hear that progress on this may be halting. I wish there was someone to bully to get some traction or funding for this.
Sigh...
On Tue Dec 30 2014 at 10:07:47 PM Fernando Lopez-Lezcano < nando@ccrma.stanford.edu> wrote:
On 11/19/2014 08:01 AM, Brian Monroe wrote:
On Thu Nov 13 2014 at 12:28:27 PM Jeff Sandys <jpsandys@gmail.com mailto:jpsandys@gmail.com> wrote:
Real Time Kernels are available from PlanetCCRMA: http://ccrma.stanford.edu/planetccrma/mirror/fedora/linux/planetcore/20/x86_64/repoview/kernel-rt.html
There are concerns about the implementation of Real Time Kernel as expressed in the Musician's Guide: http://docs.fedoraproject.org/en-US/Fedora/15/html/Musicians_Guide/sect-Musicians_Guide-CCRMA_Security_and_Stability.html
I am not a systems programmer so I can't speak to these concerns. Some of the names of the real time kernel developers listed on the PlanetCCRMA kernel-rt page are Red Hat employees. I would like to see Fedora be the premier linux distribution for music. But until we can overcome the concerns listed in the Musician's Guide we will probably not have a real time kernel in the Fedora repositories. Maybe a Fedora "Re-Mix", or Fedora.Next and Workstation with the works with Fedora software library may break the ice.Well, I think those concerns are why Ubuntu moved to their low-latency kernel instead of their kernel-rt. According to their documentation (wiki) the lowlatency kernel is a "soft" preempt kernel and the rt kernel is "hard' but there's no docs on what make them different, and people in channel don't seem to know.
(sorry for the very late response to this thread - busy busy busy, I think it is still worth to comment on this)
Brian, there is no "hard realtime" kernel for Linux. There are only degrees of latency you can achieve with different combinations of compile-time options, patching and runtime optimization. All of them are "soft". You cannot guarantee deadlines on Linux - but you can get pretty close with a fully optimized system and the RT patch, and if you have the right hardware.
I don't know what options Ubunutu uses to build their "low latency" kernel, I imagine they just enable full preemption and irq threading. This gives you better latency than a "non-preempt" kernel, but in my experience that is not enough for realtime audio work with small buffers (by that I mean jack running at 128x2 or 64x2). YMMV.
To get really low latencies you need the RT patch (which has stability issues as has been mentioned in the thread - there are as many testers as for mainline). To fix those we would need more testers and proper bug reports that are followed up. But of course that needs kernel gurus that can respond.
If you read this article you will see that the future of the RT patch is not good at this point:
http://lwn.net/Articles/617140/
Let's hope this gets better (it is not the first time that the rt patch has lagged way behind mainline, I think a major rewrite happened in the Fedora 1 or 2 timeframe and for a while there was nothing new). And BTW, this is why we are currently stuck at 3.14.x in the Planet CCRMA kernels, there is nothing newer...
Sigh... -- Fernando
-
Be -
Brendan Jones -
Brian Monroe -
Fernando Lopez-Lezcano -
Jeff Sandys -
The name of my band is 1984