Blue Sky Discussion: EPEL-next or EPIC

[Stephen John Smoogen]

                             EPIC Planning Document
     __________________________________________________________________

History / Background

   Since 2007, Fedora Extra Packages for Enterprise Linux (EPEL) has
   been rebuilding Fedora Project Linux packages for Red Hat
   Enterprise Linux and its clones. Originally the goal was to compile
   packages that RHEL did not ship but were useful in the running of
   Fedora Infrastructure and other sites. Packages would be forked
   from the nearest Fedora release (Fedora 3 for EPEL-4, Fedora 6 for
   EPEL-5) with little updating or moving of packages in order to give
   similar lifetimes as the EL packages. Emphasis was made on
   backporting fixes versus upgrading, and also not making large
   feature changes which would cause confusion. If a package could not
   longer be supported, it would be removed from the repository to
   eliminate security concerns. At the time RHEL lifetimes were
   thought to be only 5-6 years so backporting did not look like a
   large problem.

   As RHEL and its clones became more popular, Red Hat began to extend
   the lifetime of the Enterprise Linux releases from 6 years to 10
   years of "active" support. This made trying to backport fixes
   harder and many packages in EPEL would be "aged" out and
   removed. This in turn caused problems for consumers who had tied
   kickstarts and other scripts to having access to those
   packages. Attempts to fix this by pushing for release upgrade
   policies have run into resistance from packagers who find focusing
   on the main Fedora releases a full time job already and only build
   EPEL packages as one-offs. Other attempts to update policies have
   run into needing major updates and changes to build tools and
   scripting but no time to do so. Finally, because EPEL has not
   majorly changed in 10 years, conversations about changing fall into
   "well EPEL has always done it like this" from consumers, packagers,
   and engineering at different places.

   In order to get around many of these resistance points with
   changing EPEL, I suggest that we frame the problems around a new
   project called Extra Packages for Inter Communities. The goal of
   this project would be to build packages from FedoraProject Linux
   releases to various Enterprise Linux whether they are Red Hat
   Enterprise Linux, CentOS, Scientific Linux or Oracle Enterprise
   Linux.
   __________________________________________________________________

Problems and Proposals

  Composer Limitations:

   Problem

      Currently EPEL uses the Fedora build system to compose a release
      of packages every couple of days. Because each day creates a new
      compose, the only channels are the various architectures and a
      testing where future packages can be tested. Updates are not in
      a seperate because EPEL does not track releases.

   EPEL packagers currently have to support a package for the 10 year
   lifetime of an RHEL release. If they have to update a release, all
   older versions are no longer available. If they no longer want to
   support a package it is completely removed. While this sounds like
   it increases security of consumers, Fedora does not remove old
   packages from older releases.

   Proposed Solution

          EPIC will match the Enterprise Linux major/minor numbers for
          releases. This means that a set of packages will be built
          for say EL5 subrelease 11 (aka 5.11). Those packages would
          populate for each supported architecture a release, updates
          and updates-testing directory. This will allow for a set of
          packages to be composed when the subrelease occurs and then
          stay until the release is ended.

  /pub/epic/releases/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
  /pub/epic/updates/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
  /pub/epic/updates/testing/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/

   Once a minor release is done, the old tree will be hard linked to
   an appropriate archive directory.

  /pub/archives/epic/releases/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
  /pub/archives/epic/updates/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
  /pub/archives/epic/updates/testing/5/5.11/{x86_64,source,i386,aarch64,arm,ppc6
4}/

   A new one will be built and placed in appropriate sub
   directories. Hard links to the latest will point to the new one,
   and after some time the oldtree will be removed from the active
   directory tree.

  Channel Limitations:

   Problem

          EPEL is built against a subset of channels that Red Hat
          Enterprise Linux has for customers, namely the Server, High
          Availability, Optional, and some sort of Extras. Effort is
          made to make sure that EPEL does not replace with newer
          packages anything in those channels. However this does not
          extend to packages which are in the Workstation, Desktop,
          and similar channels. This can cause problems where EPEL’s
          packages replace something in those channels.

   Proposed Solution

          EPIC will be built against the latest released CentOS minor
          release using the channels which are enabled by default in
          the CentOS-Base.repo. These packages are built from source
          code that Red Hat delivers via a git mechanism to the CentOS
          project in order to rebuild them for mass
          consumption. Packages will not be allowed to replace/update
          according to the standard RPM Name-Epoch-Version-Release
          (NEVR) mechanism. This will allow EPIC to actually service
          more clients

  Build System Limitations

   Problem

          EPEL is built against Red Hat Enterprise Linux. Because
          these packages are not meant for general consumption, the
          Fedora Buildsystem does not import them but builds them
          similarly to a hidden buildroot. This causes multiple
          problems:

          + If EPEL has a package with the same name, it supercedes the
            RHEL one even if the NEVR is newer. This means old packages
            may get built against and constant pruning needs to be done.
          + If the EPEL package has a newer NEVR, it will replace the RHEL
            one which may not be what the consumer intended. This may
            break other software requirements.
          + Because parts of the build are hidden the package build may
            not be as auditable as some consumers would like.

   Proposed Solution

          EPIC will import into the build system the CentOS build it
          is building against. With this the build is not hidden from
          view.  It also makes it easier to put in rules that an EPIC
          package will never replace/remove a core build
          package. Audits of how a build is done can be clearly shown.

  Greater Frequency Rebasing

   Problem

          Red Hat Enterprise Linux have been split between competing
          customer needs. Customers wish to have some packages stay
          steady for 10 years with only some updates to them, but they
          have also found that they need rapidly updated software. In
          order to bridge this, recent RHEL releases have rebased many
          software packages during a minor release. This has caused
          problems because EPEL packages were built against older
          software ABI’s which no longer work with the latest
          RHEL. This requires the EPEL software to be rebased and
          rebuilt regularly. Conversely, because of how the Fedora
          build system sees Red Hat Enterprise Linux packages, it only
          knows about the latest packages. In the 2-4 weeks between
          various community rebuilds getting their minor release
          packages built, EPEL packages may be built against API’s
          which are not available.

   Proposed Solution

          The main EPIC releases will be built against specific CentOS
          releases versus the Continual Release (CR) channel. When the
          next RHEL minor is announced, the EPIC releng will create
          new git branch from the current minor version (aka 5.10 →
          5.11).  Packagers can then make major updates to versions or
          other needs done. When the CentOS CR is populated with the
          new rpms, packages will be built in the new tree using those
          packages.  After 2 weeks, the EPIC minor release will be
          frozen and any new packages or fixes will occur in the
          updates tree.
          __________________________________________________________________


Guidelines

  Packaging

    EL-4

   This release is no longer supported by CentOS and will not be supported
   by EPIC.

    EL-5

   This release is no longer supported by CentOS and will not be supported
   by EPIC.

    EL-6

   This release is supported until Nov 30 2020 (2020-11-30). The base
   packaging rules for any package would be those used by the Fedora
   Project during its 12 and 13 releases. Where possible, EPIC will make
   macros to keep packaing more in line with current packaging rules.

    EL-7

   This release is supported until Jun 30 2024 (2024-06-30). The base
   packaging rules for any package would be those used by the Fedora
   Project during its 18 and 19 releases. Because EL7 has seen major
   updates in certain core software, newer packaging rules from newer
   releases is possible to follow.

    EL-next

   Red Hat has not publicly announced what its next release will be, when
   it will be released, or what its lifetime is. When that occurs, it will
   be clearer which Fedora release packaging will be based off of.

  GIT structure

   Currently EPEL uses only 1 branch for every major RHEL release. In
   order to better match how current RHEL releases contain major
   differences, EPIC will have a branch for every major.minor release.
   This is to allow for people who need older versions for their usage to
   better snapshot and build their own software off of it.

   There are several naming patterns which need to be researched:
  /<package_name>/epic/6/10/
  /<package_name>/epic/6/11/
  /<package_name>/epic/7/6/
  /<package_name>/epic/7/7/
  /<package_name>/epic-6/6.10/
  /<package_name>/epic-6/6.11/
  /<package_name>/epic-7/7.6/
  /<package_name>/epic-7/7.7/
  /<package_name>/epic-6.10/
  /<package_name>/epic-6.11/
  /<package_name>/epic-7.6/
  /<package_name>/epic-7.7/

   Git module patterns will need to match what upstream delivers for any
   future EL.

  Continous Integration (CI) Gating

    EPIC-6

   The EL-6 lifecycle is reaching its final sub releases with more
   focus and growth in EL-7 and the future. Because of this gating
   will be turned off EPIC-6. Testing of packages can be done at the
   packagers discretion but is not required.

    EPIC-7

   The EL-7 lifecycle is midstream with 1-2 more minor releases with
   major API changes. Due to this, it makes sense to research if
   gating can be put in place for the next minor release. If the time
   and energy to retrofit tools to the older EL are possible then it
   can be turned on.

    EPIC-next

   Because gating is built into current Fedora releases, there should
   be no problem with turning it on for a future release. Packages
   which do not pass testing will be blocked just as they will be in
   Fedora 29+ releases.

  Modules

    EPIC-6

   Because EL-6’s tooling is locked at this point, it does not make
   sense to investigate modules.

    EPIC-7

   Currently EL-7 does not support Fedora modules and would require
   updates to yum, rpm and other tools in order to do so. If these
   show up in some form in a future minor release, then trees for
   modules can be created and builds done.

    EPIC-next

   The tooling for modules can match how Fedora approaches it. This
   means that rules for module inclusion will be similar to package
   inclusion.  EPIC-next modules must not replace/conflict with CentOS
   modules. They may use their own namespace to offer newer versions
   than what is offered and those modules may be removed in the next
   minor release if CentOS offers them then.

  Build/Update Policy

    Major Release

   In the past, Red Hat has released a public beta before it finalizes
   its next major version. If possible, the rebuilders have come out
   with their versions of this release in order to learn what gotcha’s
   they will have when the .0 release occurs. Once the packages for
   the beta are built, EPIC will make a public call for packages to be
   released to it. Because packagers may not want to support a beta or
   they know that there will be other problems, these packages will
   NOT be auto branched from Fedora.

    Minor Release

   The current method CentOS uses to build a minor release is to begin
   rebuilding packages, patching problems and then when ready put
   those packages in their /cr/ directory. These are then tested for
   by people while updates are built and ISOs for the final minor
   release is done.

   The steps for EPIC release engineering will be the following:

    1. Branch all current packages from X.Y to X.Y+1
    2. Make any bugzilla updates needed
    3. Rebuild all branched packages against CR
    4. File FTBFS against any packages.
    5. Packagers will announce major updates to mailing list
    6. Packagers will build updates against CR.
    7. 2 weeks in, releng will cull any packages which are still FTBFS
    8. 2 weeks in, releng will compose and lock the X.Y+1 release
    9. symlinks will point to the new minor release.
   10. 4 weeks in, releng will finish archiving off the X.Y release

    Between Releases

   Updates and new packages between releases will be pushed to the
   appropriate /updates/X.Y/ tree. Packagers will be encouraged to
   only make minor non-api breaking updates during this time. Major
   changes are possible, but need to follow this work flow:

    1. Announce to the epel list that a change is required and why
    2. Open a ticket to EPIC steering committee on this change
    3. EPIC steering committee approves/disapproves change
    4. If approved change happens but packages are in updates
    5. If not approved it can be done next minor release.

  Build System

    Build in Fedora

   Currently EPEL is built in Fedora using the Fedora Build system
   which integrates koji, bodhi, greenwave, other tools together. This
   could be still used with EPIC.

    Build in CentOS

   EPIC could be built in the CentOS BuildSystem (CBS) which also uses
   koji and has some integration to the CentOS Jenkins CI system.

    Build in Cloud

   Instead of using existing infrastructure, EPIC is built with newly
   stood up builders in Amazon or similar cloud environments. The
   reasoning behind this would be to see if other build systems can
   transition there eventually.
   __________________________________________________________________

Definitions

   Blue Sky Project
          A project with a different name to help eliminate preconceptions
          with the existing project.

   Customer
          A person who pays for a service either in money, time or goods.

   Consumer
          Sometimes called a user. A person who is consuming the service
          without work put into it.

   EPEL
          Extra Packages for Enterprise Linux. A product name which was to
          be replaced years ago, but no one came up with a better one.

   EPIC
          Extra Packages Inter Community.

   RHEL
          Red Hat Enterprise Linux
     __________________________________________________________________

   Last updated 2018-05-15 14:58:37 EDT


-- 
Stephen J Smoogen.

[Jim Perrin]

On 05/15/2018 12:06 PM, Stephen John Smoogen wrote:

<Big Snip>

>    Modules
>  
>      EPIC-6
>  
>     Because EL-6’s tooling is locked at this point, it does not make
>     sense to investigate modules.
>  
>      EPIC-7
>  
>     Currently EL-7 does not support Fedora modules and would require
>     updates to yum, rpm and other tools in order to do so. If these
>     show up in some form in a future minor release, then trees for
>     modules can be created and builds done.
>   

I *think* the version of rpm has been bumped enough as of EL 7.5 to
support the tooling needed for modules. This would clearly need to be
verified, but if this is the case then modern version of dnf may be all
that's required.


>      EPIC-next
>  
>     The tooling for modules can match how Fedora approaches it. This
>     means that rules for module inclusion will be similar to package
>     inclusion.  EPIC-next modules must not replace/conflict with CentOS
>     modules. They may use their own namespace to offer newer versions
>     than what is offered and those modules may be removed in the next
>     minor release if CentOS offers them then.
>   

-- 
Jim Perrin
The CentOS Project | http://www.centos.org
twitter: @BitIntegrity | GPG Key: FA09AD77

[Neal Gompa]

On Tue, May 15, 2018 at 9:12 PM Jim Perrin <jperrin(a)centos.org&gt; wrote:

>  I *think* the version of rpm has been bumped enough as of EL 7.5 to
>  support the tooling needed for modules. This would clearly need to be
>  verified, but if this is the case then modern version of dnf may be all
>  that's required. 
Modularity is an abstraction on top of RPMs that requires a separate repo
and enhanced DNF for it.

I currently provide an EPEL-safe version of DNF in my COPR[1] that I'm
considering putting into EPEL, but it doesn't have the modularity stuff
because it's too much of a mess right now.

[1]: https://copr.fedorainfracloud.org/coprs/ngompa/dnf2-el7/



--
真実はいつも一つ！/ Always, there's only one truth!

[Matthew Miller]

On Tue, May 15, 2018 at 09:31:07PM -0400, Neal Gompa wrote:
>  I currently provide an EPEL-safe version of DNF in my COPR[1] that
> I'm
>  considering putting into EPEL, but it doesn't have the modularity stuff
>  because it's too much of a mess right now.
>  [1]: https://copr.fedorainfracloud.org/coprs/ngompa/dnf2-el7/ 
Have you seen this?
https://blog.centos.org/2018/04/yum4-dnf-for-centos-7-updates/

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Neal Gompa]

On Wed, May 16, 2018 at 1:20 PM Matthew Miller <mattdm(a)fedoraproject.org&gt;
wrote:

>  On Tue, May 15, 2018 at 09:31:07PM -0400, Neal Gompa wrote:
>  > I currently provide an EPEL-safe version of DNF in my COPR[1] that I'm
>  > considering putting into EPEL, but it doesn't have the modularity stuff
>  > because it's too much of a mess right now.
>  > [1]: https://copr.fedorainfracloud.org/coprs/ngompa/dnf2-el7/ 
>  Have you seen this?
>  https://blog.centos.org/2018/04/yum4-dnf-for-centos-7-updates/ 

I did. The problem with that version is that some people can't test it
because it replaces system `rpm` and `gpgme` packages, among other things.
My version is designed to be able to ship in EPEL per the existing policy,
which allows a wider array of people to try it out.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Kevin Fenzi]

There's been a ton of talk about this over the years, but thanks a lot
to Smooge for making a more formal proposal we can look at and adjust
and see if we can get something we actually want to do. :)


On 05/15/2018 12:06 PM, Stephen John Smoogen wrote:
>                               EPIC Planning Document
>       __________________________________________________________________
>  
>  History / Background 
...snip...
>     Proposed Solution
>  
>            EPIC will match the Enterprise Linux major/minor numbers for
>            releases. This means that a set of packages will be built
>            for say EL5 subrelease 11 (aka 5.11). Those packages would
>            populate for each supported architecture a release, updates
>            and updates-testing directory. This will allow for a set of
>            packages to be composed when the subrelease occurs and then
>            stay until the release is ended.
>  
>    /pub/epic/releases/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
>    /pub/epic/updates/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
>    /pub/epic/updates/testing/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
>  
>     Once a minor release is done, the old tree will be hard linked to
>     an appropriate archive directory. 
When is a minor release considered done? When the next minor release is
released?

...snip....

>     Proposed Solution
>  
>            The main EPIC releases will be built against specific CentOS
>            releases versus the Continual Release (CR) channel. When the
>            next RHEL minor is announced, the EPIC releng will create
>            new git branch from the current minor version (aka 5.10 →
>            5.11).  Packagers can then make major updates to versions or
>            other needs done. When the CentOS CR is populated with the
>            new rpms, packages will be built in the new tree using those
>            packages.  After 2 weeks, the EPIC minor release will be
>            frozen and any new packages or fixes will occur in the
>            updates tree. 
So, the branching and build against CR and freezing the release would
all be done by epel releng, or would it be a opt-in thing by maintainers?

...snip...

>     There are several naming patterns which need to be researched:
>    /<package_name>/epic/6/10/
>    /<package_name>/epic/6/11/
>    /<package_name>/epic/7/6/
>    /<package_name>/epic/7/7/
>    /<package_name>/epic-6/6.10/
>    /<package_name>/epic-6/6.11/
>    /<package_name>/epic-7/7.6/
>    /<package_name>/epic-7/7.7/
>    /<package_name>/epic-6.10/
>    /<package_name>/epic-6.11/
>    /<package_name>/epic-7.6/
>    /<package_name>/epic-7.7/
>  
>     Git module patterns will need to match what upstream delivers for any
>     future EL. 
This will of course need tooling changes if it's done in Fedora
Infrastructure, but doable.

>    Continous Integration (CI) Gating
>  
>      EPIC-6
>  
>     The EL-6 lifecycle is reaching its final sub releases with more
>     focus and growth in EL-7 and the future. Because of this gating
>     will be turned off EPIC-6. Testing of packages can be done at the
>     packagers discretion but is not required.
>  
>      EPIC-7
>  
>     The EL-7 lifecycle is midstream with 1-2 more minor releases with
>     major API changes. Due to this, it makes sense to research if
>     gating can be put in place for the next minor release. If the time
>     and energy to retrofit tools to the older EL are possible then it
>     can be turned on.
>  
>      EPIC-next
>  
>     Because gating is built into current Fedora releases, there should
>     be no problem with turning it on for a future release. Packages
>     which do not pass testing will be blocked just as they will be in
>     Fedora 29+ releases. 
Provided all the people doing that are ok with adding another
target/more builds/more artifacts, etc. ie, communication would need to
be made and acks from the various groups.

...snip...

>     The steps for EPIC release engineering will be the following:
>  
>      1. Branch all current packages from X.Y to X.Y+1
>      2. Make any bugzilla updates needed
>      3. Rebuild all branched packages against CR
>      4. File FTBFS against any packages.
>      5. Packagers will announce major updates to mailing list
>      6. Packagers will build updates against CR.
>      7. 2 weeks in, releng will cull any packages which are still FTBFS
>      8. 2 weeks in, releng will compose and lock the X.Y+1 release
>      9. symlinks will point to the new minor release.
>     10. 4 weeks in, releng will finish archiving off the X.Y release 
All very doable, but also a fair bit of work. :)

Basically this is the same process we used for major release bringup in
epel. It's basically like rawhide at first, all builds go in every day,
then things that are broken are culled, etc, then finally updates turned
on. (But in this case epic would have a seperate update repo).

>      Between Releases
>  
>     Updates and new packages between releases will be pushed to the
>     appropriate /updates/X.Y/ tree. Packagers will be encouraged to
>     only make minor non-api breaking updates during this time. Major
>     changes are possible, but need to follow this work flow:
>  
>      1. Announce to the epel list that a change is required and why
>      2. Open a ticket to EPIC steering committee on this change
>      3. EPIC steering committee approves/disapproves change
>      4. If approved change happens but packages are in updates
>      5. If not approved it can be done next minor release. 
Yeah, we kinda have this now with the existing EPEL exception process,
but the trick is what is a major change? some people don't realize. Its
hard to have a easy rule...

>  
>    Build System
>  
>      Build in Fedora
>  
>     Currently EPEL is built in Fedora using the Fedora Build system
>     which integrates koji, bodhi, greenwave, other tools together. This
>     could be still used with EPIC.
>  
>      Build in CentOS
>  
>     EPIC could be built in the CentOS BuildSystem (CBS) which also uses
>     koji and has some integration to the CentOS Jenkins CI system. 
I think from a tools/work point of view it would make more sense to just
add it to the existing Fedora infrastructure, but from a organizational
point of view it would make more sense to just be in CentOS. If it was
in Fedora there would need to be a lot of importing of CentOS builds,
while if it was in CentOS that wouldn't be needed. So, not sure there,
tradeoffs.

>  
>      Build in Cloud
>  
>     Instead of using existing infrastructure, EPIC is built with newly
>     stood up builders in Amazon or similar cloud environments. The
>     reasoning behind this would be to see if other build systems can
>     transition there eventually. 
I don't think this is likely a winner: No other arch support, just
x86_64. Network and logistics problems possible, etc.

Anyhow, I think the proposal is doable, but the epic releng work would
be substantial. I think we need a dedicated person for all that, or
enough part time people to make it viable. Also, a few things
logistically to decide.. and then finally: is there enough interest by
users in this?

Thanks again for writing this up smooge!

kevin

[Stephen John Smoogen]

On 16 May 2018 at 13:14, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  There's been a ton of talk about this over the years, but thanks
> a lot
>  to Smooge for making a more formal proposal we can look at and adjust
>  and see if we can get something we actually want to do. :)
>
>
>  On 05/15/2018 12:06 PM, Stephen John Smoogen wrote:
> >                              EPIC Planning Document
> >      __________________________________________________________________
> >
> > History / Background
>  ...snip...
> >    Proposed Solution
> >
> >           EPIC will match the Enterprise Linux major/minor numbers for
> >           releases. This means that a set of packages will be built
> >           for say EL5 subrelease 11 (aka 5.11). Those packages would
> >           populate for each supported architecture a release, updates
> >           and updates-testing directory. This will allow for a set of
> >           packages to be composed when the subrelease occurs and then
> >           stay until the release is ended.
> >
> >   /pub/epic/releases/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
> >   /pub/epic/updates/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
> >   /pub/epic/updates/testing/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
> >
> >    Once a minor release is done, the old tree will be hard linked to
> >    an appropriate archive directory.
>
>  When is a minor release considered done? When the next minor release is
>  released?
>  
I would expect it would be set by policy but dictated when the next
minor release is released. In the case of the 'last' like 5.11 we may
need to have some additional policy for say 6.20 (making this up)
where packages are put in which EOL the package in the repo.
[something like xmms-3.10.2-9eol-unmaintained.i386.rpm] it would
basically be a rebuild of the last package in the repo with some
blanket file saying "This package is no longer maintained by the EPEL
project. It is recommended that you either remove this package or find
a different repository which still maintains it. You are on your own."

>  ...snip....
>
> >    Proposed Solution
> >
> >           The main EPIC releases will be built against specific CentOS
> >           releases versus the Continual Release (CR) channel. When the
> >           next RHEL minor is announced, the EPIC releng will create
> >           new git branch from the current minor version (aka 5.10 →
> >           5.11).  Packagers can then make major updates to versions or
> >           other needs done. When the CentOS CR is populated with the
> >           new rpms, packages will be built in the new tree using those
> >           packages.  After 2 weeks, the EPIC minor release will be
> >           frozen and any new packages or fixes will occur in the
> >           updates tree.
>
>  So, the branching and build against CR and freezing the release would
>  all be done by epel releng, or would it be a opt-in thing by maintainers?
>  
I figure it would be done by EPEL releng. I think if it were opt-in
then it would be basically be 'meh' except for that 1 person who can't
stand anyone touching their package.

>  ...snip...
>
> >    There are several naming patterns which need to be researched:
> >   /<package_name>/epic/6/10/
> >   /<package_name>/epic/6/11/
> >   /<package_name>/epic/7/6/
> >   /<package_name>/epic/7/7/
> >   /<package_name>/epic-6/6.10/
> >   /<package_name>/epic-6/6.11/
> >   /<package_name>/epic-7/7.6/
> >   /<package_name>/epic-7/7.7/
> >   /<package_name>/epic-6.10/
> >   /<package_name>/epic-6.11/
> >   /<package_name>/epic-7.6/
> >   /<package_name>/epic-7.7/
> >
> >    Git module patterns will need to match what upstream delivers for any
> >    future EL.
>
>  This will of course need tooling changes if it's done in Fedora
>  Infrastructure, but doable.
>
> >   Continous Integration (CI) Gating
> >
> >     EPIC-6
> >
> >    The EL-6 lifecycle is reaching its final sub releases with more
> >    focus and growth in EL-7 and the future. Because of this gating
> >    will be turned off EPIC-6. Testing of packages can be done at the
> >    packagers discretion but is not required.
> >
> >     EPIC-7
> >
> >    The EL-7 lifecycle is midstream with 1-2 more minor releases with
> >    major API changes. Due to this, it makes sense to research if
> >    gating can be put in place for the next minor release. If the time
> >    and energy to retrofit tools to the older EL are possible then it
> >    can be turned on.
> >
> >     EPIC-next
> >
> >    Because gating is built into current Fedora releases, there should
> >    be no problem with turning it on for a future release. Packages
> >    which do not pass testing will be blocked just as they will be in
> >    Fedora 29+ releases.
>
>  Provided all the people doing that are ok with adding another
>  target/more builds/more artifacts, etc. ie, communication would need to
>  be made and acks from the various groups.
>  
Agreed. From the fact that we have arbitrary branching, anyone can
make a module and a bunch of other things.. I figured adding a couple
of defined targets wasn't too much to ask.

>  ...snip...
>
> >    The steps for EPIC release engineering will be the following:
> >
> >     1. Branch all current packages from X.Y to X.Y+1
> >     2. Make any bugzilla updates needed
> >     3. Rebuild all branched packages against CR
> >     4. File FTBFS against any packages.
> >     5. Packagers will announce major updates to mailing list
> >     6. Packagers will build updates against CR.
> >     7. 2 weeks in, releng will cull any packages which are still FTBFS
> >     8. 2 weeks in, releng will compose and lock the X.Y+1 release
> >     9. symlinks will point to the new minor release.
> >    10. 4 weeks in, releng will finish archiving off the X.Y release
>
>  All very doable, but also a fair bit of work. :)
>
>  Basically this is the same process we used for major release bringup in
>  epel. It's basically like rawhide at first, all builds go in every day,
>  then things that are broken are culled, etc, then finally updates turned
>  on. (But in this case epic would have a seperate update repo).
>
> >     Between Releases
> >
> >    Updates and new packages between releases will be pushed to the
> >    appropriate /updates/X.Y/ tree. Packagers will be encouraged to
> >    only make minor non-api breaking updates during this time. Major
> >    changes are possible, but need to follow this work flow:
> >
> >     1. Announce to the epel list that a change is required and why
> >     2. Open a ticket to EPIC steering committee on this change
> >     3. EPIC steering committee approves/disapproves change
> >     4. If approved change happens but packages are in updates
> >     5. If not approved it can be done next minor release.
>
>  Yeah, we kinda have this now with the existing EPEL exception process,
>  but the trick is what is a major change? some people don't realize. Its
>  hard to have a easy rule...
>  
At the moment, I am going to say it will have to be a duck test. Too
much software uses weird 'oh today I felt it should be 4.0 when
yesterday was 1.2' to have a solid test. If we had a robust CI for
every package then it would be 'if the new version breaks the old
tests.. its a major change :)' but I think it will be best to start
with a 'If it smells like a major change, and it feels like a major
change, it is probably one'

> >
> >   Build System
> >
> >     Build in Fedora
> >
> >    Currently EPEL is built in Fedora using the Fedora Build system
> >    which integrates koji, bodhi, greenwave, other tools together. This
> >    could be still used with EPIC.
> >
> >     Build in CentOS
> >
> >    EPIC could be built in the CentOS BuildSystem (CBS) which also uses
> >    koji and has some integration to the CentOS Jenkins CI system.
>
>  I think from a tools/work point of view it would make more sense to just
>  add it to the existing Fedora infrastructure, but from a organizational
>  point of view it would make more sense to just be in CentOS. If it was
>  in Fedora there would need to be a lot of importing of CentOS builds,
>  while if it was in CentOS that wouldn't be needed. So, not sure there,
>  tradeoffs.
>
> >
> >     Build in Cloud
> >
> >    Instead of using existing infrastructure, EPIC is built with newly
> >    stood up builders in Amazon or similar cloud environments. The
> >    reasoning behind this would be to see if other build systems can
> >    transition there eventually.
>
>  I don't think this is likely a winner: No other arch support, just
>  x86_64. Network and logistics problems possible, etc.
>
>  Anyhow, I think the proposal is doable, but the epic releng work would
>  be substantial. I think we need a dedicated person for all that, or
>  enough part time people to make it viable. Also, a few things
>  logistically to decide.. and then finally: is there enough interest by
>  users in this?
>
>  Thanks again for writing this up smooge!
>
>  kevin
>
>
>  _______________________________________________
>  epel-devel mailing list -- epel-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to epel-devel-leave(a)lists.fedoraproject.org
>  


-- 
Stephen J Smoogen.

[Matthew Miller]

On Tue, May 15, 2018 at 03:06:43PM -0400, Stephen John Smoogen wrote:
>    Modules 
[...]
>      EPIC-next
>  
>     The tooling for modules can match how Fedora approaches it. This
>     means that rules for module inclusion will be similar to package
>     inclusion.  EPIC-next modules must not replace/conflict with CentOS
>     modules. They may use their own namespace to offer newer versions
>     than what is offered and those modules may be removed in the next
>     minor release if CentOS offers them then. 
I think we should do this part differently. Each Fedora module has its
own lifespan, separate from that of the base OS. It automatically gets
built across all currently-supported base OSes. That way, I can
maintain, say, "calc-2.x" in just one stream (where stream = git
branch), and it automatically gets built everywhere. Rather than having
EPIC modules be entirely separate, I'd like to just include them in
this: by default, build all modules against both the Fedora _and_ EL
bases.

This has a further implication: these modules _may_ replace or conflict
with CentOS (and possibly RHEL) modules. But you'd have to opt-in to
those streams explicitly to do so. With non-modular Yum/DNF, having some
packages update base software would be horrific because enabling the
repo and running `yum update` would do who-knows-what. But with
modules, that's not a problem.


>     EPIC
>            Extra Packages Inter Community. 
Extra Packages for Impassioned Community?
Extra Packages Included by Community?
Extra Packages Introduced by Community?
Extra Packages Including Community?
Extra Packages for Innnnnterprise Community? (Or "EPEC"?)



-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Stephen John Smoogen]

On 17 May 2018 at 15:31, Matthew Miller <mattdm(a)fedoraproject.org&gt; wrote:
>  On Tue, May 15, 2018 at 03:06:43PM -0400, Stephen John Smoogen
> wrote:
> >   Modules
>  [...]
> >     EPIC-next
> >
> >    The tooling for modules can match how Fedora approaches it. This
> >    means that rules for module inclusion will be similar to package
> >    inclusion.  EPIC-next modules must not replace/conflict with CentOS
> >    modules. They may use their own namespace to offer newer versions
> >    than what is offered and those modules may be removed in the next
> >    minor release if CentOS offers them then.
>
>  I think we should do this part differently. Each Fedora module has its
>  own lifespan, separate from that of the base OS. It automatically gets
>  built across all currently-supported base OSes. That way, I can
>  maintain, say, "calc-2.x" in just one stream (where stream = git
>  branch), and it automatically gets built everywhere. Rather than having
>  EPIC modules be entirely separate, I'd like to just include them in
>  this: by default, build all modules against both the Fedora _and_ EL
>  bases.
>
>  This has a further implication: these modules _may_ replace or conflict
>  with CentOS (and possibly RHEL) modules. But you'd have to opt-in to
>  those streams explicitly to do so. With non-modular Yum/DNF, having some
>  packages update base software would be horrific because enabling the
>  repo and running `yum update` would do who-knows-what. But with
>  modules, that's not a problem.
>  
I have been worried about a negative feedback loop here on packagers.
This is a new technology and packagers aren't usually people who care
about EPEL. Having them to deal with multiple OS's they don't use
makes it more likely they don't want to put stuff in modules in the
first place. I would prefer to be looser on the uptake of modules to
get people comfortable with them in Fedora first. I also don't expect
a large demand for modules in EPIC.

Most enterprise system administrators are from Missouri. They are
going to want you to prove that it isn't going to eat their systems
before they are going to want to try it in development. They are also
going to want to make sure that it is still being developed in F3X
versus something that just showed up in F28. As 2-4 releases show up
with modules in them, then it is going to be wanted more in EPIC and
by then the procedures and plans for how they are done in Fedora
should be 'solid' enough for the enterprise admin.



> >    EPIC
> >           Extra Packages Inter Community.
>
>  Extra Packages for Impassioned Community?
>  Extra Packages Included by Community?
>  Extra Packages Introduced by Community?
>  Extra Packages Including Community?
>  Extra Packages for Innnnnterprise Community? (Or "EPEC"?)
>  
Extra Packages for Introverted Communities

>  --
>  Matthew Miller
>  <mattdm(a)fedoraproject.org&gt;
>  Fedora Project Leader
>  _______________________________________________
>  epel-devel mailing list -- epel-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to epel-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...


-- 
Stephen J Smoogen.

[Matthew Miller]

On Thu, May 17, 2018 at 04:36:17PM -0400, Stephen John Smoogen wrote:
>  I have been worried about a negative feedback loop here on
> packagers.
>  This is a new technology and packagers aren't usually people who care
>  about EPEL. Having them to deal with multiple OS's they don't use
>  makes it more likely they don't want to put stuff in modules in the
>  first place. I would prefer to be looser on the uptake of modules to
>  get people comfortable with them in Fedora first. I also don't expect
>  a large demand for modules in EPIC. 
Modules solve two problems for EPIC:

First, since each module has a lifecycle definition, they make it easy
to maintain packages for EPIC without the 10-year commitment (or,
indeed, any more commitment then the packager makes in Fedora). And
that works both ways: as a consumer, I can know up front what lifecycle
I can (at least nominally) expect from a given module stream.

Second, I *do* think there are a lot of cases where different EPIC
users will want different versions. The Django module we have in Fedora
now would be immediately useful.

I do also worry about the possible negative feedback loop. I suggest we
start _building_ for EL by default but not tagging into the repo. Make
_that_ opt-in.


>  >>           Extra Packages Inter Community.
>  > Extra Packages for Impassioned Community?
>  > Extra Packages Included by Community?
>  > Extra Packages Introduced by Community?
>  > Extra Packages Including Community?
>  > Extra Packages for Innnnnterprise Community? (Or "EPEC"?)
>  Extra Packages for Introverted Communities 
Extroverted Packages for Introverted Communities!

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Stephen John Smoogen]

On 17 May 2018 at 18:11, Matthew Miller <mattdm(a)fedoraproject.org&gt; wrote:
>  On Thu, May 17, 2018 at 04:36:17PM -0400, Stephen John Smoogen
> wrote:
> > I have been worried about a negative feedback loop here on packagers.
> > This is a new technology and packagers aren't usually people who care
> > about EPEL. Having them to deal with multiple OS's they don't use
> > makes it more likely they don't want to put stuff in modules in the
> > first place. I would prefer to be looser on the uptake of modules to
> > get people comfortable with them in Fedora first. I also don't expect
> > a large demand for modules in EPIC.
>
>  Modules solve two problems for EPIC:
>
>  First, since each module has a lifecycle definition, they make it easy
>  to maintain packages for EPIC without the 10-year commitment (or,
>  indeed, any more commitment then the packager makes in Fedora). And
>  that works both ways: as a consumer, I can know up front what lifecycle
>  I can (at least nominally) expect from a given module stream.
>
>  Second, I *do* think there are a lot of cases where different EPIC
>  users will want different versions. The Django module we have in Fedora
>  now would be immediately useful.
>
>  I do also worry about the possible negative feedback loop. I suggest we
>  start _building_ for EL by default but not tagging into the repo. Make
>  _that_ opt-in.
>
>  
First, I am not disagreeing with what you are saying. I wrote the
proposal as things that are inside EPEL's circle of influence with
most of the other items going to take a LOT of release engineering
work to accomplish. I am also writing it for a risk averse consumer
base who want assurances that they can get old stuff even if it is
past its sell by date and that it won't eat their children if they
fat-finger a command. If Fedora wants to build it across all
platforms, I don't have a problem with it.. I just didn't see it as
inside of what we influenced.

> > >>           Extra Packages Inter Community.
> > > Extra Packages for Impassioned Community?
> > > Extra Packages Included by Community?
> > > Extra Packages Introduced by Community?
> > > Extra Packages Including Community?
> > > Extra Packages for Innnnnterprise Community? (Or "EPEC"?) 
I think with our horrible history of naming this project EPEC is what
we go with. I just want the new logo not to look like a horse's butt
with tail. [I know that isn't what it was meant to be.. it is a sail
on a boat.. but once someone describes it like that.. its like a
Tyranosaurus Rex fighting a Triceratops but not as fun]

> > Extra Packages for Introverted Communities
>
>  Extroverted Packages for Introverted Communities!
>
>  --
>  Matthew Miller
>  <mattdm(a)fedoraproject.org&gt;
>  Fedora Project Leader
>  _______________________________________________
>  epel-devel mailing list -- epel-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to epel-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...


-- 
Stephen J Smoogen.

[Ian Pilcher]

On 05/17/2018 06:53 PM, Stephen John Smoogen wrote:
>  I think with our horrible history of naming this project EPEC is
> what
>  we go with. I just want the new logo not to look like a horse's butt
>  with tail. 
Already taken.

 
https://www.openstack.org/themes/openstack/images/project-mascots/Cinder/...

-- 
========================================================================
Ian Pilcher                                         arequipeno(a)gmail.com
-------- "I grew up before Mark Zuckerberg invented friendship" --------
========================================================================

[Matthew Miller]

On Thu, May 17, 2018 at 07:53:18PM -0400, Stephen John Smoogen wrote:
>  First, I am not disagreeing with what you are saying. I wrote the
>  proposal as things that are inside EPEL's circle of influence with
>  most of the other items going to take a LOT of release engineering
>  work to accomplish. I am also writing it for a risk averse consumer
>  base who want assurances that they can get old stuff even if it is
>  past its sell by date and that it won't eat their children if they
>  fat-finger a command. If Fedora wants to build it across all
>  platforms, I don't have a problem with it.. I just didn't see it as
>  inside of what we influenced. 
That's all fair too. I just don't want to undershoot, either. :)

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

good idea, 
EPIC release --> rhel release, centOS release
keep the old RPMS when EPIC package upgrade.
no compatibility issue any more.

[Martin Jackson]

I agree that it makes sense to associate EPIC releases with EL "point" 
or "y" releases.

Some orgs (like us) download new content on timers, and while it's not 
wrong to say that people should read release notes and updates, there is 
currently a *lot* of content in EPEL etc to keep track of, and I think 
people find it really valuable to have a repo that makes the "don't 
overwrite/upgrade core" promise.  I like the differentiation that is 
starting to develop in the CentOS ecosystem where there are some more 
"experimental" repos (like YUM4) that *do* overwrite, and say so 
explicitly.  The ecosystem also has good tools to mix and match these 
repositories with core content (pulp/Satellite).  Generally, we sync all 
of EPEL but only use the command-line bits; I'm unaware of any user 
reporting problems due to a GTK rebase in an EL minor release.

IUS used to talk a bit about repo safety on their web page, does this 
need to be something more explicit for repos in general?  Or something 
that might be reflected in metadata somehow?

Thanks,

Marty

On 5/17/2018 10:01 PM, heelsleeh(a)163.com wrote:
>  good idea,
>  EPIC release --> rhel release, centOS release
>  keep the old RPMS when EPIC package upgrade.
>  no compatibility issue any more.
>  _______________________________________________
>  epel-devel mailing list -- epel-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to epel-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...

[Neal Gompa]

On Fri, May 18, 2018 at 8:27 AM Martin Jackson <mhjacks(a)swbell.net&gt; wrote:

>  I agree that it makes sense to associate EPIC releases with EL
> "point"
>  or "y" releases. 
>  Some orgs (like us) download new content on timers, and while
> it's not
>  wrong to say that people should read release notes and updates, there is
>  currently a *lot* of content in EPEL etc to keep track of, and I think
>  people find it really valuable to have a repo that makes the "don't
>  overwrite/upgrade core" promise.  I like the differentiation that is
>  starting to develop in the CentOS ecosystem where there are some more
>  "experimental" repos (like YUM4) that *do* overwrite, and say so
>  explicitly.  The ecosystem also has good tools to mix and match these
>  repositories with core content (pulp/Satellite).  Generally, we sync all
>  of EPEL but only use the command-line bits; I'm unaware of any user
>  reporting problems due to a GTK rebase in an EL minor release. 
>  IUS used to talk a bit about repo safety on their web page, does
> this
>  need to be something more explicit for repos in general?  Or something
>  that might be reflected in metadata somehow? 

One of the things that I've been talking to the DNF team about is making
DNF/YUM4 more repo-aware so that it follows user tracks on where packages
come from unless the user specifically requests it to be switched. This
behavior would help resolve a long-standing issue of user expectations on
following certain update paths (w.r.t. package update sources). Things like
EPIC would be generally less dangerous because the risk of switching
between sources mid-stream (as current Yum/DNF behavior is now) would be
substantially lower.

This feature has always been available to us, we just never wired it up. So
we would get one of the principal benefits of Modularity for virtually all
packages for free.

-- 
真実はいつも一つ！/ Always, there's only one truth!