======================== Apologies and So Forth ========================
First, I would like to apologize for the delay in getting this post done. I really didn't realize the amount of energy the trip would take from me and how fuzzy brained I was for a week afterwords. Second, I would like to thank the Open Source and Standards (OSAS) group at Red Hat for sponsoring me that week. I believe I got a lot of things done and that we can work on getting Extra Packages for Enterprise Linux (EPEL) moving forward in some direction. Third I would like to thank everyone who took the time to talk to me at one of these places to tell me about what they were able to do with EPEL and what they were no longer able to use it for.
One of the items that people brought up a lot was that they felt conversations about fixing/growing/changing EPEL have become a broken record. Various things are said at many conferences by various people, but nothing ever gets done on this to the point that they feel they have been lied to. After succumbing to the fuzzy headed "what did I say yesterday?" for a week after flying back, I think that most of these broken promises have come up from jet-lag and people overload. If it hadn't been that I had people from IBM, CERN, and various other groups bring this up over and over again.. I may have forgotten just as well. In any case, I am going to try and outline every item that I wrote down on many pages of notes in this blog. If I have forgotten some point that you wanted me to bring up, please send me an email so that I can make sure it is dealt with.
=========================== Things EPEL is doing well ===========================
The primary user of EPEL packages are on Red Hat Enterprise Linux (or a clone like CentOS or Scientific Linux) version 6 with a slowly shrinking set of EL-5 users. The audiences that use EPEL are very diverse. Everything from large organizations with strict audit controls to fast moving startups which have to work with the large organizations as their customers to the university trying to do some experiments to match things done in other organizations.
Many users brought up the fact that the win of EPEL is that it is a low energy move to get packages from Fedora rebuilt for an Enterprise environment. The fact that the package spec files had been vetted and tested in Fedora made for less chances of conflicts or configuration problems that crept into their own spec files or ones they got off the internet from various groups. Things EPEL is not doing well in
This is going to be a rather long list of items that came up. Some of them conflict with each other but that is mainly due to the fact that the customers are so diverse.
* The packaging guidelines for each EPEL version are not clear. This is mainly due to the fact that they are usually based off of older versions of Fedora (Fedora-6 for EPEL-5, Fedora-12 for EPEL-6, and Fedora-18 for EPEL-7) that may conflict with each other or with how things are being done in current EPEL. * EPEL does not have a regular release structure like Fedora and RHEL have. There isn't an EPEL-5.11 channel with an epel-updates-5.11 where updates are available. Because of various repository limitations, this means that directories aren't able to keep multiple old copies so downgrades when things do break aren't easy. * EPEL promises to keep things stable and only update for fixes, but this is only done on a few packages where others get upgraded to and fro. There does not seem to be much "steering" or "release engineering" of what is in the trees. * EPEL only covers part of Enterprise Linux (the Server product) but a lot of packages are for the Workstation but there is no way to see when things replace/conflict with them. [People believe that we build against the equivalent of CentOS-5/6/7 versus a subchannel.] * EPEL sometimes has weird breaks between releases. The git in EPEL-5 is newer than what is in EL-6 in a way that was breaking repositories when pushes were done from EL-5 systems. [People believe there is a promise that such changes are tested against.] * EPEL packages disappear. If there is no maintainer packages are retired but if someone is re-building out the EL-5 hosts.. they need that package to be available. [People believe there is a promise that packages will be always available.] * From various people who were (or former) EPEL maintainers: packages in EPEL are the most complained about. If they can't update the package in EPEL, they get complaints about it being too old and they may end up having a newer version available for what they need to do anyway. If they do update the version, they get complaints that they broke someone because they needed some old version. Most of the complaints usually are the worst kinds ( off-hand death threats (why don't you die in a fire) etc etc).
============================== Things people wanted in EPEL ==============================
There were various things that people were wanting in EPEL that weren't really breakages because they don't exist yet.
* Enable 'alternative' architectures. There were requests from people about CentOS-i386 and CentOS-arm (for the Raspberry Pi2 in schools) with no vision on how those would be enabled.
* Enable more packages with shorter lifetimes. One of the things that multiple sites were doing was rebuilding all of a Fedora release for their internal mirrors to supplement all of the things that EPEL didn't have in it. Why can't there be an EPEL-rawhide where all of these packages are built but no 'promises'?
* Is it possible to have a branch management system where packages get updated regularly? Say either whenever Fedora moves to a new release or Red Hat updates to the next branch (6.7->6.8, 7.2->7.3)
* Could packages in EPEL be tested in the CentOS continuous integration infrastructure as part of the autokarma testing?
============= Conclusions =============
I don't think anything above is new to people who have been contributing to EPEL in the last ~10 years. A lot of the problems are ones that were brought up in the beginning as we tried to square the circle of differing use cases. However, I wanted to catalogue them here and then make a promise that I will do my best to figure out ways to solve them by FOSDEM 2017 in some form or another.
As I said at the beginning of the post, I believe that people had other complaints and suggestions. If I forgot or miswrote, my apologies and I will correct.
"SJS" == Stephen John Smoogen smooge@gmail.com writes:
SJS> * The packaging guidelines for each EPEL version are not SJS> clear. This is mainly due to the fact that they are usually based SJS> off of older versions of Fedora (Fedora-6 for EPEL-5, Fedora-12 SJS> for EPEL-6, and Fedora-18 for EPEL-7) that may conflict with each SJS> other or with how things are being done in current EPEL.
I've been trying to help with this. I strongly believe that Fedora must be allowed to move forward (and be cleaned up) without having to worry about what EPEL is going to do, but even then there are plenty of opportunities for EPEL to get some of those advancements as well. Now that we have epel-rpm-macros we can try to "backport" some of the packaging progress that Fedora is making, though of course we'll never be able to do everything. And there is an issue about EPEL packaging differing from RHEL packaging, or packaging for other add-on repositories. So it's a complex issue.
Also note that the EPEL guidelines can be edited by anyone, so if something is missing or unclear, then anyone can fix it.
- J<
On 16 February 2016 at 21:51, Jason L Tibbitts III tibbs@math.uh.edu wrote:
"SJS" == Stephen John Smoogen smooge@gmail.com writes:
SJS> * The packaging guidelines for each EPEL version are not SJS> clear. This is mainly due to the fact that they are usually based SJS> off of older versions of Fedora (Fedora-6 for EPEL-5, Fedora-12 SJS> for EPEL-6, and Fedora-18 for EPEL-7) that may conflict with each SJS> other or with how things are being done in current EPEL.
I've been trying to help with this. I strongly believe that Fedora must be allowed to move forward (and be cleaned up) without having to worry about what EPEL is going to do, but even then there are plenty of
Thank you for your help here. I think most of the problems were either people getting complained at because package X from 2008 in EL-5 didn't conform to current Fedora standards, or vice versa. Most of the time they would like to have a clear set of guidelines they can point to know what the henry they are supposed to do or not do in a package. Being able to use the epel-rpm-macros I believe will help this a lot.
opportunities for EPEL to get some of those advancements as well. Now that we have epel-rpm-macros we can try to "backport" some of the packaging progress that Fedora is making, though of course we'll never be able to do everything. And there is an issue about EPEL packaging differing from RHEL packaging, or packaging for other add-on repositories. So it's a complex issue.
Also note that the EPEL guidelines can be edited by anyone, so if something is missing or unclear, then anyone can fix it.
I would dispute that assertion. Mainly because most of the people who think something is missing or unclear don't know what the answer should be. There are also a lot of burned bridges in the current documentation where people did a lot of work to change things only to get complaints and reverts (several times). This is due to various people thinking the charter of EPEL means different things and that there are "promises" that EPEL is supposed to do even if no one has ever accomplished it.
- J<
On Tue, 16 Feb 2016 21:42:18 -0700 Stephen John Smoogen smooge@gmail.com wrote:
======================== Apologies and So Forth ========================
First, I would like to apologize for the delay in getting this post done. I really didn't realize the amount of energy the trip would take from me and how fuzzy brained I was for a week afterwords. Second, I would like to thank the Open Source and Standards (OSAS) group at Red Hat for sponsoring me that week. I believe I got a lot of things done and that we can work on getting Extra Packages for Enterprise Linux (EPEL) moving forward in some direction. Third I would like to thank everyone who took the time to talk to me at one of these places to tell me about what they were able to do with EPEL and what they were no longer able to use it for.
Thanks for talking to folks and writing this up!
...snip...
- The packaging guidelines for each EPEL version are not clear. This is mainly due to the fact that they are usually based off of older versions of Fedora (Fedora-6 for EPEL-5, Fedora-12 for EPEL-6, and Fedora-18 for EPEL-7) that may conflict with each other or with how things are being done in current EPEL.
I've always gone with: "Fedora guidelines apply except for these small changes", but yeah, we could be better about those changes. Tibb's recent macro works might reduce these, but I don't think we can ever get to 0.
- EPEL does not have a regular release structure like Fedora and RHEL have. There isn't an EPEL-5.11 channel with an epel-updates-5.11 where updates are available. Because of various repository limitations, this means that directories aren't able to keep multiple old copies so downgrades when things do break aren't easy.
Yeah. I think we could include 2 versions of everything (at the cost of 2x of the mirror space and bandwith), but then you have things like foo-1.0 has a major security bug and foo-1.1 came out to fix it, and you trick someone into downgrading or installing the old one and exploit them. ;(
- EPEL promises to keep things stable and only update for fixes, but this is only done on a few packages where others get upgraded to and fro. There does not seem to be much "steering" or "release engineering" of what is in the trees.
Yeah, I think mostly this is due to the fact that there is not anyone who works on EPEL full time. Everyone who does things does them in their "spare" time, so having some kind of micro scrutiny isn't in the cards. ;(
I think we could improve this with more feedback... when an incompatible upgrade goes out, ask people to note it so we can talk to the maintainer and ask them not to do that kind of thing.
- EPEL only covers part of Enterprise Linux (the Server product) but a lot of packages are for the Workstation but there is no way to see when things replace/conflict with them. [People believe that we build against the equivalent of CentOS-5/6/7 versus a subchannel.]
Yeah, not sure how to fix that without a second workstation branch. :(
- EPEL sometimes has weird breaks between releases. The git in EPEL-5 is newer than what is in EL-6 in a way that was breaking repositories when pushes were done from EL-5 systems. [People believe there is a promise that such changes are tested against.]
Huh, first I have heard of that one..
- EPEL packages disappear. If there is no maintainer packages are retired but if someone is re-building out the EL-5 hosts.. they need that package to be available. [People believe there is a promise that packages will be always available.]
Yeah. Again, continuing to ship an old vulnerable version of something seems bad, but I wish we had a better way to communicate this. ;(
- From various people who were (or former) EPEL maintainers: packages in EPEL are the most complained about. If they can't update the package in EPEL, they get complaints about it being too old and they may end up having a newer version available for what they need to do anyway. If they do update the version, they get complaints that they broke someone because they needed some old version. Most of the complaints usually are the worst kinds ( off-hand death threats (why don't you die in a fire) etc etc).
Yeah, sad. It's the old Enterprise linux mantra: "I want everything to be 100% stable, secure and never change, except for this one thing I care about, that should be todays git checkout every day"
============================== Things people wanted in EPEL ==============================
There were various things that people were wanting in EPEL that weren't really breakages because they don't exist yet.
- Enable 'alternative' architectures. There were requests from people about CentOS-i386 and CentOS-arm (for the Raspberry Pi2 in schools) with no vision on how those would be enabled.
We talked about this in the last meeting. There's apparently 3-4 ways we could do this, all of which require work. I hope we can get Dennis to post to the list these options and try and decide which we want to go with and who will step up to help work on it.
- Enable more packages with shorter lifetimes. One of the things that multiple sites were doing was rebuilding all of a Fedora release for their internal mirrors to supplement all of the things that EPEL didn't have in it. Why can't there be an EPEL-rawhide where all of these packages are built but no 'promises'?
We could, but again someone would need to step up to work on it. This would be a lot of work as Fedora has vastly more packages that you might think. ;)
- Is it possible to have a branch management system where packages get updated regularly? Say either whenever Fedora moves to a new release or Red Hat updates to the next branch (6.7->6.8, 7.2->7.3)
We could mass rebuild at every release... but that also brings up the time after a rhel release but before a centos release. ;(
- Could packages in EPEL be tested in the CentOS continuous integration infrastructure as part of the autokarma testing?
No idea, but it would be lovely if it could work.
I don't think anything above is new to people who have been contributing to EPEL in the last ~10 years. A lot of the problems are ones that were brought up in the beginning as we tried to square the circle of differing use cases. However, I wanted to catalogue them here and then make a promise that I will do my best to figure out ways to solve them by FOSDEM 2017 in some form or another.
It sounds to me a lot of it is communication. Perhaps we could figure out some way to more directly communicate with users.
kevin
On 18 February 2016 at 12:46, Kevin Fenzi kevin@scrye.com wrote:
On Tue, 16 Feb 2016 21:42:18 -0700 Stephen John Smoogen smooge@gmail.com wrote:
======================== Apologies and So Forth ========================
First, I would like to apologize for the delay in getting this post done. I really didn't realize the amount of energy the trip would take from me and how fuzzy brained I was for a week afterwords. Second, I would like to thank the Open Source and Standards (OSAS) group at Red Hat for sponsoring me that week. I believe I got a lot of things done and that we can work on getting Extra Packages for Enterprise Linux (EPEL) moving forward in some direction. Third I would like to thank everyone who took the time to talk to me at one of these places to tell me about what they were able to do with EPEL and what they were no longer able to use it for.
Thanks for talking to folks and writing this up!
...snip...
- The packaging guidelines for each EPEL version are not clear. This is mainly due to the fact that they are usually based off of older versions of Fedora (Fedora-6 for EPEL-5, Fedora-12 for EPEL-6, and Fedora-18 for EPEL-7) that may conflict with each other or with how things are being done in current EPEL.
I've always gone with: "Fedora guidelines apply except for these small changes", but yeah, we could be better about those changes. Tibb's recent macro works might reduce these, but I don't think we can ever get to 0.
One of the requests was to have snapshots of the guidelines that we worked each channel against so that it was clearer what 5 wanted versus 6 wanted.
- EPEL does not have a regular release structure like Fedora and RHEL have. There isn't an EPEL-5.11 channel with an epel-updates-5.11 where updates are available. Because of various repository limitations, this means that directories aren't able to keep multiple old copies so downgrades when things do break aren't easy.
Yeah. I think we could include 2 versions of everything (at the cost of 2x of the mirror space and bandwith), but then you have things like foo-1.0 has a major security bug and foo-1.1 came out to fix it, and you trick someone into downgrading or installing the old one and exploit them. ;(
If we don't delete them from koji we aren't fixing anything because if I can trick you to downgrade, I can trick you to go to the version in koji because it has the fix needed. [Since I have seen people talk about their systems getting broken into after they did exactly that.. I think it isn't going too far in assumptions :)]
- EPEL promises to keep things stable and only update for fixes, but this is only done on a few packages where others get upgraded to and fro. There does not seem to be much "steering" or "release engineering" of what is in the trees.
Yeah, I think mostly this is due to the fact that there is not anyone who works on EPEL full time. Everyone who does things does them in their "spare" time, so having some kind of micro scrutiny isn't in the cards. ;(
I think we could improve this with more feedback... when an incompatible upgrade goes out, ask people to note it so we can talk to the maintainer and ask them not to do that kind of thing.
Or not promise it at all. I think the underlying issue is that people think we do have full-time people working on EPEL with the same controls (if not more) than we have in Fedora.
- EPEL only covers part of Enterprise Linux (the Server product) but a lot of packages are for the Workstation but there is no way to see when things replace/conflict with them. [People believe that we build against the equivalent of CentOS-5/6/7 versus a subchannel.]
Yeah, not sure how to fix that without a second workstation branch. :(
The only monstrosities I have thought of were: epel-server-N epel-workstation-N epel-combined-N
which sounded like a ton of work for little benefit.
- EPEL sometimes has weird breaks between releases. The git in EPEL-5 is newer than what is in EL-6 in a way that was breaking repositories when pushes were done from EL-5 systems. [People believe there is a promise that such changes are tested against.]
Huh, first I have heard of that one..
Me too. It took up half the short meeting because it broke CERN and a couple other places.
I don't think anything above is new to people who have been contributing to EPEL in the last ~10 years. A lot of the problems are ones that were brought up in the beginning as we tried to square the circle of differing use cases. However, I wanted to catalogue them here and then make a promise that I will do my best to figure out ways to solve them by FOSDEM 2017 in some form or another.
It sounds to me a lot of it is communication. Perhaps we could figure out some way to more directly communicate with users.
OH yeah.. that was one of the items.. why is the website so old and dead. I told them your story about trying to fix it up and finding parts reverted over and over again. Someone recommended : Just start from scratch and kill the old stuff. Which I think was part of the "recharter" talks.
kevin
epel-devel mailing list epel-devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/epel-devel@lists.fedoraproject.or...
On Thu, 18 Feb 2016 14:42:12 -0700 Stephen John Smoogen smooge@gmail.com wrote:
One of the requests was to have snapshots of the guidelines that we worked each channel against so that it was clearer what 5 wanted versus 6 wanted.
Sure, we have it divided into 5 and 6 here (I assume we don't have too many changes from 7): https://fedoraproject.org/wiki/EPEL:Packaging
Yeah. I think we could include 2 versions of everything (at the cost of 2x of the mirror space and bandwith), but then you have things like foo-1.0 has a major security bug and foo-1.1 came out to fix it, and you trick someone into downgrading or installing the old one and exploit them. ;(
If we don't delete them from koji we aren't fixing anything because if I can trick you to downgrade, I can trick you to go to the version in koji because it has the fix needed. [Since I have seen people talk about their systems getting broken into after they did exactly that.. I think it isn't going too far in assumptions :)]
Well, it becomes a great deal harder.
1. Hey, you should 'yum downgrade foo' because the newest one isn't good.
vs
2. Hey, you should download this https://kojipkgs.fedoraproject.org/blah/blah/blah/foo.rpm and 'yum --nogpgcheck localinstall foo.rpm' because the new one is broken.
The first one sounds a lot more legit. I think not having it in enabled repos makes it a good deal more clear.
Or not promise it at all. I think the underlying issue is that people think we do have full-time people working on EPEL with the same controls (if not more) than we have in Fedora.
Could be, yeah.
- EPEL only covers part of Enterprise Linux (the Server product)
but a lot of packages are for the Workstation but there is no way to see when things replace/conflict with them. [People believe that we build against the equivalent of CentOS-5/6/7 versus a subchannel.]
Yeah, not sure how to fix that without a second workstation branch. :(
The only monstrosities I have thought of were: epel-server-N epel-workstation-N epel-combined-N
which sounded like a ton of work for little benefit.
Yes.
OH yeah.. that was one of the items.. why is the website so old and dead. I told them your story about trying to fix it up and finding parts reverted over and over again. Someone recommended : Just start from scratch and kill the old stuff. Which I think was part of the "recharter" talks.
I'd fullly support someone working over the wiki... always good. ;)
Not sure I have the cycles to do it myself tho.
kevin
On 19 February 2016 at 08:35, Kevin Fenzi kevin@scrye.com wrote:
On Thu, 18 Feb 2016 14:42:12 -0700 Stephen John Smoogen smooge@gmail.com wrote:
One of the requests was to have snapshots of the guidelines that we worked each channel against so that it was clearer what 5 wanted versus 6 wanted.
Sure, we have it divided into 5 and 6 here (I assume we don't have too many changes from 7): https://fedoraproject.org/wiki/EPEL:Packaging
Yeah. I think we could include 2 versions of everything (at the cost of 2x of the mirror space and bandwith), but then you have things like foo-1.0 has a major security bug and foo-1.1 came out to fix it, and you trick someone into downgrading or installing the old one and exploit them. ;(
If we don't delete them from koji we aren't fixing anything because if I can trick you to downgrade, I can trick you to go to the version in koji because it has the fix needed. [Since I have seen people talk about their systems getting broken into after they did exactly that.. I think it isn't going too far in assumptions :)]
Well, it becomes a great deal harder.
- Hey, you should 'yum downgrade foo' because the newest one isn't
good.
vs
- Hey, you should download this
https://kojipkgs.fedoraproject.org/blah/blah/blah/foo.rpm and 'yum --nogpgcheck localinstall foo.rpm' because the new one is broken.
The first one sounds a lot more legit. I think not having it in enabled repos makes it a good deal more clear.
Well currently people are having to be told that for every package we remove even if it isn't a security problem just because it was removed without maintainer. Once that becomes the 'norm' it becomes easier to pull off the other type.
Or not promise it at all. I think the underlying issue is that people think we do have full-time people working on EPEL with the same controls (if not more) than we have in Fedora.
Could be, yeah.
- EPEL only covers part of Enterprise Linux (the Server product)
but a lot of packages are for the Workstation but there is no way to see when things replace/conflict with them. [People believe that we build against the equivalent of CentOS-5/6/7 versus a subchannel.]
Yeah, not sure how to fix that without a second workstation branch. :(
The only monstrosities I have thought of were: epel-server-N epel-workstation-N epel-combined-N
which sounded like a ton of work for little benefit.
Yes.
OH yeah.. that was one of the items.. why is the website so old and dead. I told them your story about trying to fix it up and finding parts reverted over and over again. Someone recommended : Just start from scratch and kill the old stuff. Which I think was part of the "recharter" talks.
I'd fullly support someone working over the wiki... always good. ;)
Not sure I have the cycles to do it myself tho.
I believe we are going to need to do so anyway as one of the proposals coming out of the DevConf things was basically moving the wiki to be a whiteboard versus a place where actual documentation for subprojects go. However that is me listening to talks while trying to do my homework so I am not sure how I am reading that to be "what we are looking to do" to something else.
kevin
epel-devel mailing list epel-devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/epel-devel@lists.fedoraproject.or...
epel-devel@lists.fedoraproject.org
-
Jason L Tibbitts III -
Kevin Fenzi -
Stephen John Smoogen