F39 Proposal: Make Toolbx a release-blocking deliverable and have release-blocking test criteria (System-Wide Change)

[Aoife Moloney]

https://fedoraproject.org/wiki/Changes/ToolbxReleaseBlocker

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.


== Summary ==

Up to date [https://src.fedoraproject.org/container/fedora-toolbox
fedora-toolbox] [https://opencontainers.org/ OCI] images must be
published on [https://registry.fedoraproject.org/
registry.fedoraproject.org] as release-blocking deliverables, and
there must be release-blocking test criteria to ensure usable
[https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs.

== Owner ==

* Name: [[User:Rishi|Debarshi Ray]], [[User:Sumantrom|Sumantro Mukherjee]]
* Email: debarshir(a)redhat.com, sumukher(a)redhat.com



== Detailed Description ==
Currently, there is no formal requirement for
[https://containertoolbx.org/ Toolbx] to be usable when a new Fedora
is released.  This is a problem for a tool that's so popular and
provides something as fundamental as an interactive command line
environment for software development and troubleshooting the host
operating system.  This Change aims to address this.

Toolbx has two parts — an [https://opencontainers.org/ OCI] image,
which defaults to
[https://src.fedoraproject.org/container/fedora-toolbox
fedora-toolbox] on Fedora hosts, and the
[https://src.fedoraproject.org/rpms/toolbox toolbox] RPM.  The OCI
image is pulled by the RPM to set up a containerized interactive CLI
environment.

First, we want to ensure that there are up to date
[https://src.fedoraproject.org/container/fedora-toolbox
fedora-toolbox] OCI images published on
[https://registry.fedoraproject.org/ registry.fedoraproject.org] as
release-blocking deliverables at critical points in the development
schedule, just like the installation ISOs for the Editions from
[https://download.fedoraproject.org/pub/fedora/linux/releases/
download.fedoraproject.org].  This must at least happen when an
upcoming Fedora release is branched from Rawhide, and for the Beta and
Final release candidates.  If possible, they should be updated more
frequently as part of the nightly composes.  We do not expect this to
happen after a Fedora release has gone GA.

Second, we want to have release-blocking test criteria to ensure
usable [https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs at
critical points in the development schedule.  This must be used to
ensure that both changes in the Toolbx stack, and future
[https://docs.fedoraproject.org/en-US/program_management/changes_policy/
Changes] in other parts of the OS do not break Toolbx, and at least
cover the Beta and Final release candidates.

Examples of changes in the Toolbx stack causing breakage can be
[https://bugzilla.redhat.com/show_bug.cgi?id=2183034 FUSE] preventing
RPMs with file capabilities from being installed inside Toolbx
containers, Toolbx [https://github.com/containers/toolbox/issues/643
bind mounts] preventing RPMs with <code>%attr()</code> from being
installed or causing
[https://bugzilla.redhat.com/show_bug.cgi?id=2188304
systemd-tmpfiles(8)] to throw errors, etc..  Examples of changes in
other parts of the OS can be changes to Fedora's Kerberos stack
causing Kerberos to stop working inside Toolbx containers,
[[Changes/EnableSysctlPingGroupRange|changes]] to the
<code>sysctl(8)</code>
[https://github.com/systemd/systemd/commit/90ce7627dfe824ff6e7c0ca5f96350f...
configuration] breaking ping(8), changes in
[https://github.com/containers/toolbox/issues/562 Mutter] breaking
graphical applications, etc..

Note that having release-blocking test criteria for the
<code>toolbox</code> RPM will also implicitly test the
<code>fedora-toolbox</code> image.

== Feedback ==


== Benefit to Fedora ==

This Change improves the quality of the containerized interactive
command line [https://containertoolbx.org/ Toolbx] environments on
Fedora by adding formal requirements to ensure that they are usable
when a new Fedora is released.  This will bring them closer to the
reliability of similar environments running directly on the host
operating system.

Toolbx is installed by default on CoreOS, Silverblue and Workstation.
It is indispensable for software developers using Silverblue to bypass
the difficulty of setting up a development environment in the usual
way. It is widely used, even on Workstation, by those who don't want
to pollute their host OS, or want to access a CLI environment that's
different from the host's without installing a virtual machine, or
want a pre-configured development environment.

It will be beneficial to consider the
[https://src.fedoraproject.org/container/fedora-toolbox
fedora-toolbox] images as release-blocking deliverables because
Fedora's [https://opencontainers.org/ OCI] infrastructure is often
broken.  Here are [https://pagure.io/releng/issue/11092 two]
[https://pagure.io/releng/issue/11367 recent] examples of <code>fedpkg
container-build</code> not working.  In the second case, it was
preventing the images from being rebuilt to pull in an
[https://bugzilla.redhat.com/show_bug.cgi?id=2170878 important]
bug-fix.  The broken infrastructure prevents regular Fedora
contributors from jumping in to rebuild and publish the images at
critical points in the development schedule.  Making them
release-blocking deliverables would attract greater attention and
scrutiny from release engineering and ensure that a Fedora development
cycle does not proceed with broken or outdated or missing
<code>fedora-toolbox</code> images.

At the moment, the branching of an upcoming Fedora release from
Rawhide is a particularly chaotic time.  Since the
<code>fedora-toolbox</code> images for Fedora Branched and Rawhide are
not rebuilt as part of the branching process, there is a lot of
confusion for end-users until someone manually rebuilds the images and
gets them published, which can take some time as described above.
Having the images built and published by release engineering as part
of the branching will avoid this.

If someone installs the Beta or the Final on their host, and creates a
Toolbx container using the default <code>fedora-toolbox</code> image,
then, barring exceptions, the host and the container will have the
same RPM versions.  Just like Workstation and Silverblue are released
with the same versions.  This will ensure greater consistency in terms
of bug-fixes, features and pending updates.

Last but not the least, we cannot have release-blocking test criteria
for the [https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs
unless the images are also release-blocking deliverables, because the
RPMs depend on the images.  This brings us to the benefits of the
second part of this Change.

It will be beneficial to have release-blocking test criteria for the
[https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs because they
interact with different parts of the OS to set up the containerized
interactive CLI environments.  These environments have seamless access
to the user’s home directory, the Wayland and X11 sockets, networking
(including Avahi), removable devices (like USB sticks), systemd
journal, SSH agent, D-Bus, ulimits, /dev and the udev database, etc.,
and any unexpected change in another part of the OS can break the
environment.

The release-blocking test criteria will be a way to co-ordinate
several disparate groups of developers to ensure that the
containerized interactive CLI Toolbx environments on Fedora are just
as reliable as those running directly on the host OS.

== Scope ==

* Proposal owners: Write down the release-blocking test criteria for
the [https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs that
will be used for testing and the blocker bugs process.

* Other developers: The release-blocking test criteria for Toolbx will
require others to debug and fix bugs, possibly blockers, and be
mindful of making changes to other parts of the operating system that
break the Toolbx environment.


* Release engineering: [https://pagure.io/releng/issue/11399 #11399]

* Policies and guidelines: N/A (not needed for this Change)

* Trademark approval: [https://pagure.io/Fedora-Council/tickets/issue/449 #449]


* Alignment with Community Initiatives:


== Upgrade/compatibility impact ==

This is only a change to the Fedora release processes.  Therefore,
systems with a previous version of Fedora won't need any manual
intervention.

There should not be any user-visible change, other than, barring
exceptions, the
[https://src.fedoraproject.org/container/fedora-toolbox
fedora-toolbox] image having the same RPM versions as the host at
critical points in the development schedule, fewer bugs and a more
reliable Toolbx.

== How To Test ==

Up to date [https://src.fedoraproject.org/container/fedora-toolbox
fedora-toolbox] images must be available from
[https://registry.fedoraproject.org/ registry.fedoraproject.org] when
an upcoming Fedora release is branched from Rawhide, and for the Beta
and Final release candidates.  The following steps can be used to test
this:
* Check the Toolbx containers and images present locally using
<code>toolbox list</code>.
* Remove any Toolbx containers and images for the Fedora release under
development using <code>toolbox rm</code> and <code>toolbox
rmi</code>.
* Create a new Toolbx container with <code>toolbox create</code> and
enter it with <code>toolbox enter</code>.
* Use <code>sudo dnf update</code> inside the Toolbx container.
Barring exceptions, the container should have the same RPM versions as
the host.

The [https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs must
satisfy the test criteria for the Beta and Final release candidates.
Writing the test criteria is part of this Change, so they don't exist
at the moment, but likely examples can be:
* Graphical Apps running inside Toolbx container should be accessible outside
* Graphical Apps (text editors) should retain their state even when
the virtual terminal is closed

== User Experience ==

This Change improves the quality of the containerized interactive
command line Toolbx environments on Fedora by ensuring that they are
just as reliable as those running directly on the host operating
system.

If someone installs the Beta or the Final on their host, and creates a
Toolbx container using the default
[https://src.fedoraproject.org/container/fedora-toolbox
fedora-toolbox] image, then, barring exceptions, the host and the
container should have the same RPM versions. Just like Workstation and
Silverblue are released with the same versions. This will ensure
greater consistency in terms of bug-fixes, features and pending
updates.

== Dependencies ==

N/A (not needed for this Change)

== Contingency Plan ==

* Contingency mechanism: If there are no up to date
[https://src.fedoraproject.org/container/fedora-toolbox
fedora-toolbox] images published on
[https://registry.fedoraproject.org/ registry.fedoraproject.org] as
release-blocking deliverables, then the release-blocking test criteria
for the [https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs
cannot be put into production.  In that case this Change cannot be
implemented and status quo will be maintained.  If the images get
published, but the test criteria is absent, then only the first half
of the Change will be implemented, and users can still benefit from
the more predictably updated images.
* Contingency deadline: We need this by the Change completion deadline
or before Fedora 39 is branched from Rawhide, whichever is earlier. As
per the [https://fedorapeople.org/groups/schedule/f-39/f-39-key-tasks.html
schedule], both of those are currently set to happen on the 8th of
August 2023.
* Blocks release? No

== Documentation ==

* Toolbx basics: https://containertoolbx.org/install/
* Toolbx manuals: https://github.com/containers/toolbox/tree/main/doc




-- 

Aoife Moloney

Product Owner

Community Platform Engineering Team

Red Hat EMEA

Communications House

Cork Road

Waterford

[Barry]

>  On 8 May 2023, at 14:51, Aoife Moloney <amoloney(a)redhat.com&gt;
> wrote:
>  
>  https://fedoraproject.org/wiki/Changes/ToolbxReleaseBlocker
>  
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>  
>  
>  == Summary ==
>  
>  Up to date [https://src.fedoraproject.org/container/fedora-toolbox
>  fedora-toolbox] [https://opencontainers.org/ OCI] images must be 
This url does not exist.

>  published on [https://registry.fedoraproject.org/
>  registry.fedoraproject.org] as release-blocking deliverables, and
>  there must be release-blocking test criteria to ensure usable
>  [https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs.
>  
>  == Owner ==
>  
>  * Name: [[User:Rishi|Debarshi Ray]], [[User:Sumantrom|Sumantro Mukherjee]]
>  * Email: debarshir(a)redhat.com, sumukher(a)redhat.com
>  
>  
>  
>  == Detailed Description ==
>  Currently, there is no formal requirement for
>  [https://containertoolbx.org/ Toolbx] to be usable when a new Fedora 
Nor this one.

Barry

>  is released.  This is a problem for a tool that's so popular and
>  provides something as fundamental as an interactive command line
>  environment for software development and troubleshooting the host
>  operating system.  This Change aims to address this.
>  
>  Toolbx has two parts — an [https://opencontainers.org/ OCI] image,
>  which defaults to
>  [https://src.fedoraproject.org/container/fedora-toolbox
>  fedora-toolbox] on Fedora hosts, and the
>  [https://src.fedoraproject.org/rpms/toolbox toolbox] RPM.  The OCI
>  image is pulled by the RPM to set up a containerized interactive CLI
>  environment.
>  
>  First, we want to ensure that there are up to date
>  [https://src.fedoraproject.org/container/fedora-toolbox
>  fedora-toolbox] OCI images published on
>  [https://registry.fedoraproject.org/ registry.fedoraproject.org] as
>  release-blocking deliverables at critical points in the development
>  schedule, just like the installation ISOs for the Editions from
>  [https://download.fedoraproject.org/pub/fedora/linux/releases/
>  download.fedoraproject.org].  This must at least happen when an
>  upcoming Fedora release is branched from Rawhide, and for the Beta and
>  Final release candidates.  If possible, they should be updated more
>  frequently as part of the nightly composes.  We do not expect this to
>  happen after a Fedora release has gone GA.
>  
>  Second, we want to have release-blocking test criteria to ensure
>  usable [https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs at
>  critical points in the development schedule.  This must be used to
>  ensure that both changes in the Toolbx stack, and future
>  [https://docs.fedoraproject.org/en-US/program_management/changes_policy/
>  Changes] in other parts of the OS do not break Toolbx, and at least
>  cover the Beta and Final release candidates.
>  
>  Examples of changes in the Toolbx stack causing breakage can be
>  [https://bugzilla.redhat.com/show_bug.cgi?id=2183034 FUSE] preventing
>  RPMs with file capabilities from being installed inside Toolbx
>  containers, Toolbx [https://github.com/containers/toolbox/issues/643
>  bind mounts] preventing RPMs with <code>%attr()</code> from being
>  installed or causing
>  [https://bugzilla.redhat.com/show_bug.cgi?id=2188304
>  systemd-tmpfiles(8)] to throw errors, etc..  Examples of changes in
>  other parts of the OS can be changes to Fedora's Kerberos stack
>  causing Kerberos to stop working inside Toolbx containers,
>  [[Changes/EnableSysctlPingGroupRange|changes]] to the
>  <code>sysctl(8)</code>
>  [https://github.com/systemd/systemd/commit/90ce7627dfe824ff6e7c0ca5f96350f...
>  configuration] breaking ping(8), changes in
>  [https://github.com/containers/toolbox/issues/562 Mutter] breaking
>  graphical applications, etc..
>  
>  Note that having release-blocking test criteria for the
>  <code>toolbox</code> RPM will also implicitly test the
>  <code>fedora-toolbox</code> image.
>  
>  == Feedback ==
>  
>  
>  == Benefit to Fedora ==
>  
>  This Change improves the quality of the containerized interactive
>  command line [https://containertoolbx.org/ Toolbx] environments on
>  Fedora by adding formal requirements to ensure that they are usable
>  when a new Fedora is released.  This will bring them closer to the
>  reliability of similar environments running directly on the host
>  operating system.
>  
>  Toolbx is installed by default on CoreOS, Silverblue and Workstation.
>  It is indispensable for software developers using Silverblue to bypass
>  the difficulty of setting up a development environment in the usual
>  way. It is widely used, even on Workstation, by those who don't want
>  to pollute their host OS, or want to access a CLI environment that's
>  different from the host's without installing a virtual machine, or
>  want a pre-configured development environment.
>  
>  It will be beneficial to consider the
>  [https://src.fedoraproject.org/container/fedora-toolbox
>  fedora-toolbox] images as release-blocking deliverables because
>  Fedora's [https://opencontainers.org/ OCI] infrastructure is often
>  broken.  Here are [https://pagure.io/releng/issue/11092 two]
>  [https://pagure.io/releng/issue/11367 recent] examples of <code>fedpkg
>  container-build</code> not working.  In the second case, it was
>  preventing the images from being rebuilt to pull in an
>  [https://bugzilla.redhat.com/show_bug.cgi?id=2170878 important]
>  bug-fix.  The broken infrastructure prevents regular Fedora
>  contributors from jumping in to rebuild and publish the images at
>  critical points in the development schedule.  Making them
>  release-blocking deliverables would attract greater attention and
>  scrutiny from release engineering and ensure that a Fedora development
>  cycle does not proceed with broken or outdated or missing
>  <code>fedora-toolbox</code> images.
>  
>  At the moment, the branching of an upcoming Fedora release from
>  Rawhide is a particularly chaotic time.  Since the
>  <code>fedora-toolbox</code> images for Fedora Branched and Rawhide are
>  not rebuilt as part of the branching process, there is a lot of
>  confusion for end-users until someone manually rebuilds the images and
>  gets them published, which can take some time as described above.
>  Having the images built and published by release engineering as part
>  of the branching will avoid this.
>  
>  If someone installs the Beta or the Final on their host, and creates a
>  Toolbx container using the default <code>fedora-toolbox</code> image,
>  then, barring exceptions, the host and the container will have the
>  same RPM versions.  Just like Workstation and Silverblue are released
>  with the same versions.  This will ensure greater consistency in terms
>  of bug-fixes, features and pending updates.
>  
>  Last but not the least, we cannot have release-blocking test criteria
>  for the [https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs
>  unless the images are also release-blocking deliverables, because the
>  RPMs depend on the images.  This brings us to the benefits of the
>  second part of this Change.
>  
>  It will be beneficial to have release-blocking test criteria for the
>  [https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs because they
>  interact with different parts of the OS to set up the containerized
>  interactive CLI environments.  These environments have seamless access
>  to the user’s home directory, the Wayland and X11 sockets, networking
>  (including Avahi), removable devices (like USB sticks), systemd
>  journal, SSH agent, D-Bus, ulimits, /dev and the udev database, etc.,
>  and any unexpected change in another part of the OS can break the
>  environment.
>  
>  The release-blocking test criteria will be a way to co-ordinate
>  several disparate groups of developers to ensure that the
>  containerized interactive CLI Toolbx environments on Fedora are just
>  as reliable as those running directly on the host OS.
>  
>  == Scope ==
>  
>  * Proposal owners: Write down the release-blocking test criteria for
>  the [https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs that
>  will be used for testing and the blocker bugs process.
>  
>  * Other developers: The release-blocking test criteria for Toolbx will
>  require others to debug and fix bugs, possibly blockers, and be
>  mindful of making changes to other parts of the operating system that
>  break the Toolbx environment.
>  
>  
>  * Release engineering: [https://pagure.io/releng/issue/11399 #11399]
>  
>  * Policies and guidelines: N/A (not needed for this Change)
>  
>  * Trademark approval: [https://pagure.io/Fedora-Council/tickets/issue/449 #449]
>  
>  
>  * Alignment with Community Initiatives:
>  
>  
>  == Upgrade/compatibility impact ==
>  
>  This is only a change to the Fedora release processes.  Therefore,
>  systems with a previous version of Fedora won't need any manual
>  intervention.
>  
>  There should not be any user-visible change, other than, barring
>  exceptions, the
>  [https://src.fedoraproject.org/container/fedora-toolbox
>  fedora-toolbox] image having the same RPM versions as the host at
>  critical points in the development schedule, fewer bugs and a more
>  reliable Toolbx.
>  
>  == How To Test ==
>  
>  Up to date [https://src.fedoraproject.org/container/fedora-toolbox
>  fedora-toolbox] images must be available from
>  [https://registry.fedoraproject.org/ registry.fedoraproject.org] when
>  an upcoming Fedora release is branched from Rawhide, and for the Beta
>  and Final release candidates.  The following steps can be used to test
>  this:
>  * Check the Toolbx containers and images present locally using
>  <code>toolbox list</code>.
>  * Remove any Toolbx containers and images for the Fedora release under
>  development using <code>toolbox rm</code> and <code>toolbox
>  rmi</code>.
>  * Create a new Toolbx container with <code>toolbox create</code> and
>  enter it with <code>toolbox enter</code>.
>  * Use <code>sudo dnf update</code> inside the Toolbx container.
>  Barring exceptions, the container should have the same RPM versions as
>  the host.
>  
>  The [https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs must
>  satisfy the test criteria for the Beta and Final release candidates.
>  Writing the test criteria is part of this Change, so they don't exist
>  at the moment, but likely examples can be:
>  * Graphical Apps running inside Toolbx container should be accessible outside
>  * Graphical Apps (text editors) should retain their state even when
>  the virtual terminal is closed
>  
>  == User Experience ==
>  
>  This Change improves the quality of the containerized interactive
>  command line Toolbx environments on Fedora by ensuring that they are
>  just as reliable as those running directly on the host operating
>  system.
>  
>  If someone installs the Beta or the Final on their host, and creates a
>  Toolbx container using the default
>  [https://src.fedoraproject.org/container/fedora-toolbox
>  fedora-toolbox] image, then, barring exceptions, the host and the
>  container should have the same RPM versions. Just like Workstation and
>  Silverblue are released with the same versions. This will ensure
>  greater consistency in terms of bug-fixes, features and pending
>  updates.
>  
>  == Dependencies ==
>  
>  N/A (not needed for this Change)
>  
>  == Contingency Plan ==
>  
>  * Contingency mechanism: If there are no up to date
>  [https://src.fedoraproject.org/container/fedora-toolbox
>  fedora-toolbox] images published on
>  [https://registry.fedoraproject.org/ registry.fedoraproject.org] as
>  release-blocking deliverables, then the release-blocking test criteria
>  for the [https://src.fedoraproject.org/rpms/toolbox toolbox] RPMs
>  cannot be put into production.  In that case this Change cannot be
>  implemented and status quo will be maintained.  If the images get
>  published, but the test criteria is absent, then only the first half
>  of the Change will be implemented, and users can still benefit from
>  the more predictably updated images.
>  * Contingency deadline: We need this by the Change completion deadline
>  or before Fedora 39 is branched from Rawhide, whichever is earlier. As
>  per the [https://fedorapeople.org/groups/schedule/f-39/f-39-key-tasks.html
>  schedule], both of those are currently set to happen on the 8th of
>  August 2023.
>  * Blocks release? No
>  
>  == Documentation ==
>  
>  * Toolbx basics: https://containertoolbx.org/install/
>  * Toolbx manuals: https://github.com/containers/toolbox/tree/main/doc
>  
>  
>  
>  
>  -- 
>  
>  Aoife Moloney
>  
>  Product Owner
>  
>  Community Platform Engineering Team
>  
>  Red Hat EMEA
>  
>  Communications House
>  
>  Cork Road
>  
>  Waterford
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Zbigniew Jędrzejewski-Szmek]

On Mon, May 08, 2023 at 04:49:09PM +0100, Barry wrote:
>  > fedora-toolbox] [https://opencontainers.org/ OCI] images must
> be
>  This url does not exist. 
>  > [https://containertoolbx.org/ Toolbx] to be usable when a new
> Fedora
>  Nor this one. 
Both work fine here. Maybe some temporary network glitch?

Zbyszek

[Kevin Fenzi]

I'm broadly in favor here, some comments in line...

...snip...
>  First, we want to ensure that there are up to date
>  [https://src.fedoraproject.org/container/fedora-toolbox
>  fedora-toolbox] OCI images published on
>  [https://registry.fedoraproject.org/ registry.fedoraproject.org] as
>  release-blocking deliverables at critical points in the development
>  schedule, just like the installation ISOs for the Editions from
>  [https://download.fedoraproject.org/pub/fedora/linux/releases/
>  download.fedoraproject.org].  This must at least happen when an
>  upcoming Fedora release is branched from Rawhide, and for the Beta and
>  Final release candidates.  If possible, they should be updated more
>  frequently as part of the nightly composes.  We do not expect this to
>  happen after a Fedora release has gone GA. 
In order to do this at branch point, we will need to move building this
image into the compose process and mark it blocking. Which hopefully we
can do. ;) 

...snip...
>  It will be beneficial to consider the
>  [https://src.fedoraproject.org/container/fedora-toolbox
>  fedora-toolbox] images as release-blocking deliverables because
>  Fedora's [https://opencontainers.org/ OCI] infrastructure is often
>  broken.  Here are [https://pagure.io/releng/issue/11092 two]
>  [https://pagure.io/releng/issue/11367 recent] examples of <code>fedpkg
>  container-build</code> not working.  In the second case, it was
>  preventing the images from being rebuilt to pull in an
>  [https://bugzilla.redhat.com/show_bug.cgi?id=2170878 important]
>  bug-fix.  The broken infrastructure prevents regular Fedora
>  contributors from jumping in to rebuild and publish the images at
>  critical points in the development schedule.  Making them
>  release-blocking deliverables would attract greater attention and
>  scrutiny from release engineering and ensure that a Fedora development
>  cycle does not proceed with broken or outdated or missing
>  <code>fedora-toolbox</code> images. 
I'd like to note that making this blocking doesn't waive any kind of
magic wand that makes our infrastructure more reliable. ;) 
The container build pipeline is a long collection of fragile things. 
It may well result in us slipping more based on things not working. ;( 

kevin

[Zbigniew Jędrzejewski-Szmek]

On Mon, May 08, 2023 at 09:24:08AM -0700, Kevin Fenzi wrote:
>  I'm broadly in favor here, some comments in line...
>  
>  ...snip...
>  > First, we want to ensure that there are up to date
>  > [https://src.fedoraproject.org/container/fedora-toolbox
>  > fedora-toolbox] OCI images published on
>  > [https://registry.fedoraproject.org/ registry.fedoraproject.org] as
>  > release-blocking deliverables at critical points in the development
>  > schedule, just like the installation ISOs for the Editions from
>  > [https://download.fedoraproject.org/pub/fedora/linux/releases/
>  > download.fedoraproject.org].  This must at least happen when an
>  > upcoming Fedora release is branched from Rawhide, and for the Beta and
>  > Final release candidates.  If possible, they should be updated more
>  > frequently as part of the nightly composes.  We do not expect this to
>  > happen after a Fedora release has gone GA. 
I think we need some clarity wrt. to the dependency order here.
Let's say we:
>  In order to do this at branch point, we will need to move building
> this
>  image into the compose process and mark it blocking. 
OK, so we build things, but
then we need to publish to registry.fp.o,
which is asynchrounous (?). When we test the newly built ISOs, do
we test them also with the latest image that we get from registry.fp.o?
And if we find a bug anywhere in this pipeline, we respin everything?

>  ...snip...
>  > It will be beneficial to consider the
>  > [https://src.fedoraproject.org/container/fedora-toolbox
>  > fedora-toolbox] images as release-blocking deliverables because
>  > Fedora's [https://opencontainers.org/ OCI] infrastructure is often
>  > broken.  Here are [https://pagure.io/releng/issue/11092 two]
>  > [https://pagure.io/releng/issue/11367 recent] examples of <code>fedpkg
>  > container-build</code> not working.  In the second case, it was
>  > preventing the images from being rebuilt to pull in an
>  > [https://bugzilla.redhat.com/show_bug.cgi?id=2170878 important]
>  > bug-fix.  The broken infrastructure prevents regular Fedora
>  > contributors from jumping in to rebuild and publish the images at
>  > critical points in the development schedule.  Making them
>  > release-blocking deliverables would attract greater attention and
>  > scrutiny from release engineering and ensure that a Fedora development
>  > cycle does not proceed with broken or outdated or missing
>  > <code>fedora-toolbox</code> images.
>  
>  I'd like to note that making this blocking doesn't waive any kind of
>  magic wand that makes our infrastructure more reliable. ;) 
>  The container build pipeline is a long collection of fragile things. 
>  It may well result in us slipping more based on things not working. ;( 
Hmm, quoting from https://pagure.io/releng/issue/11092:
> > Also the aarch64 cluster is running on Fedora 33 boxes, so we
> > should probably try to do a full redeploy :-(
>  We can't upgrade it from f33 because docker is no longer in f34+ and
>  openshift origin / 3.11 doesn't support any newer either. 
Is this still true? I don't think we want to make the Fedora release
process contingent on something that requires F33.

Zbyszek

[Stephen Smoogen]

On Mon, 8 May 2023 at 15:58, Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt;
wrote:

>  On Mon, May 08, 2023 at 09:24:08AM -0700, Kevin Fenzi wrote:
>
>  > I'd like to note that making this blocking doesn't waive any kind of
>  > magic wand that makes our infrastructure more reliable. ;)
>  > The container build pipeline is a long collection of fragile things.
>  > It may well result in us slipping more based on things not working. ;(
>
>  Hmm, quoting from https://pagure.io/releng/issue/11092:
>  >> Also the aarch64 cluster is running on Fedora 33 boxes, so we
>  >> should probably try to do a full redeploy :-(
>  > We can't upgrade it from f33 because docker is no longer in f34+ and
>  > openshift origin / 3.11 doesn't support any newer either.
>
>  Is this still true? I don't think we want to make the Fedora release
>  process contingent on something that requires F33.
>
>  ``` 
$ sudo -i ssh osbs-aarch64-node01.iad2.fedoraproject.org cat
/etc/system-release
Fedora release 33 (Thirty Three)
```

My memory of this is that this is not an easy thing to 'fix'.



-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren

[Demi Marie Obenour]

On 5/8/23 16:10, Stephen Smoogen wrote:
>  On Mon, 8 May 2023 at 15:58, Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl&gt;
>  wrote:
>  
> > On Mon, May 08, 2023 at 09:24:08AM -0700, Kevin Fenzi wrote:
> >
> >> I'd like to note that making this blocking doesn't waive any kind of
> >> magic wand that makes our infrastructure more reliable. ;)
> >> The container build pipeline is a long collection of fragile things.
> >> It may well result in us slipping more based on things not working. ;(
> >
> > Hmm, quoting from https://pagure.io/releng/issue/11092:
> >>> Also the aarch64 cluster is running on Fedora 33 boxes, so we
> >>> should probably try to do a full redeploy :-(
> >> We can't upgrade it from f33 because docker is no longer in f34+ and
> >> openshift origin / 3.11 doesn't support any newer either.
> >
> > Is this still true? I don't think we want to make the Fedora release
> > process contingent on something that requires F33.
> >
> > ```
>  $ sudo -i ssh osbs-aarch64-node01.iad2.fedoraproject.org cat
>  /etc/system-release
>  Fedora release 33 (Thirty Three)
>  ```
>  
>  My memory of this is that this is not an easy thing to 'fix'. 
Podman instead of Docker?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Kevin Fenzi]

On Mon, May 08, 2023 at 07:57:30PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  
>  I think we need some clarity wrt. to the dependency order here.
>  Let's say we:
>  > In order to do this at branch point, we will need to move building this
>  > image into the compose process and mark it blocking.
>  OK, so we build things, but then we need to publish to registry.fp.o,
>  which is asynchrounous (?). When we test the newly built ISOs, do 
No, it happens at the end of the compose (if no blocking deliverables
failed causing the compose to fail)

>  we test them also with the latest image that we get from
> registry.fp.o?
>  And if we find a bug anywhere in this pipeline, we respin everything? 
Good question. I'll note that currently we do not do any specific
testing after branch point. We freeze things to get a compose to
complete, but then we move on. It's not like Beta or Final.

>  > I'd like to note that making this blocking doesn't waive
> any kind of
>  > magic wand that makes our infrastructure more reliable. ;) 
>  > The container build pipeline is a long collection of fragile things. 
>  > It may well result in us slipping more based on things not working. ;(
>  
>  Hmm, quoting from https://pagure.io/releng/issue/11092:
>  >> Also the aarch64 cluster is running on Fedora 33 boxes, so we
>  >> should probably try to do a full redeploy :-(
>  > We can't upgrade it from f33 because docker is no longer in f34+ and
>  > openshift origin / 3.11 doesn't support any newer either.
>  
>  Is this still true? I don't think we want to make the Fedora release
>  process contingent on something that requires F33. 
yes, it's still true. Note thats the aarch64 osbs cluster.
The x86_64 one is rhel7.

So, perhaps it would make sense to only make the x86_64 one blocking?

kevin

[Sebastian Crane]

Dear Kevin,

>  > Hmm, quoting from https://pagure.io/releng/issue/11092:
>  > >> Also the aarch64 cluster is running on Fedora 33 boxes, so we
>  > >> should probably try to do a full redeploy :-(
>  > > We can't upgrade it from f33 because docker is no longer in f34+ and
>  > > openshift origin / 3.11 doesn't support any newer either.
>  >
>  > Is this still true? I don't think we want to make the Fedora release
>  > process contingent on something that requires F33.
>
>  yes, it's still true. Note thats the aarch64 osbs cluster.
>  The x86_64 one is rhel7. 
Might it be possible to replace Docker with CRI-O on the OpenShift
cluster?

Best wishes,

Sebastian

[Kevin Fenzi]

On Mon, May 08, 2023 at 09:29:02PM +0100, Sebastian Crane wrote:
>  Dear Kevin,
>  
>  > > Hmm, quoting from https://pagure.io/releng/issue/11092:
>  > > >> Also the aarch64 cluster is running on Fedora 33 boxes, so we
>  > > >> should probably try to do a full redeploy :-(
>  > > > We can't upgrade it from f33 because docker is no longer in f34+ and
>  > > > openshift origin / 3.11 doesn't support any newer either.
>  > >
>  > > Is this still true? I don't think we want to make the Fedora release
>  > > process contingent on something that requires F33.
>  >
>  > yes, it's still true. Note thats the aarch64 osbs cluster.
>  > The x86_64 one is rhel7.
>  
>  Might it be possible to replace Docker with CRI-O on the OpenShift
>  cluster? 
Nope. openshift 3 / osbs1 uses docker. :(

kevin

[Demi Marie Obenour]

On 5/8/23 18:49, Kevin Fenzi wrote:
>  On Mon, May 08, 2023 at 09:29:02PM +0100, Sebastian Crane wrote:
> > Dear Kevin,
> >
> >>> Hmm, quoting from https://pagure.io/releng/issue/11092:
> >>>>> Also the aarch64 cluster is running on Fedora 33 boxes, so we
> >>>>> should probably try to do a full redeploy :-(
> >>>> We can't upgrade it from f33 because docker is no longer in f34+ and
> >>>> openshift origin / 3.11 doesn't support any newer either.
> >>>
> >>> Is this still true? I don't think we want to make the Fedora release
> >>> process contingent on something that requires F33.
> >>
> >> yes, it's still true. Note thats the aarch64 osbs cluster.
> >> The x86_64 one is rhel7.
> >
> > Might it be possible to replace Docker with CRI-O on the OpenShift
> > cluster?
>  
>  Nope. openshift 3 / osbs1 uses docker. :(
>  
>  kevin 
alias docker=podman
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Neal Gompa]

On Mon, May 8, 2023 at 7:05 PM Demi Marie Obenour <demiobenour(a)gmail.com&gt; wrote:
>  On 5/8/23 18:49, Kevin Fenzi wrote:
>  > On Mon, May 08, 2023 at 09:29:02PM +0100, Sebastian Crane wrote:
>  >> Dear Kevin,
>  >>
>  >>>> Hmm, quoting from https://pagure.io/releng/issue/11092:
>  >>>>>> Also the aarch64 cluster is running on Fedora 33 boxes, so we
>  >>>>>> should probably try to do a full redeploy :-(
>  >>>>> We can't upgrade it from f33 because docker is no longer in f34+
> and
>  >>>>> openshift origin / 3.11 doesn't support any newer either.
>  >>>>
>  >>>> Is this still true? I don't think we want to make the Fedora
> release
>  >>>> process contingent on something that requires F33.
>  >>>
>  >>> yes, it's still true. Note thats the aarch64 osbs cluster.
>  >>> The x86_64 one is rhel7.
>  >>
>  >> Might it be possible to replace Docker with CRI-O on the OpenShift
>  >> cluster?
>  >
>  > Nope. openshift 3 / osbs1 uses docker. :(
>  >
>  > kevin
>
>  alias docker=podman 
Using Podman with it through Podman's implementation of the Docker
Host protocol *may* work, but a version of Podman that has it is not
currently available for RHEL 7 (which is what OpenShift 3.11 runs on).



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Demi Marie Obenour]

On 5/8/23 19:09, Neal Gompa wrote:
>  On Mon, May 8, 2023 at 7:05 PM Demi Marie Obenour
> <demiobenour(a)gmail.com&gt; wrote:
> >
> > On 5/8/23 18:49, Kevin Fenzi wrote:
> >> On Mon, May 08, 2023 at 09:29:02PM +0100, Sebastian Crane wrote:
> >>> Dear Kevin,
> >>>
> >>>>> Hmm, quoting from https://pagure.io/releng/issue/11092:
> >>>>>>> Also the aarch64 cluster is running on Fedora 33 boxes, so
> we
> >>>>>>> should probably try to do a full redeploy :-(
> >>>>>> We can't upgrade it from f33 because docker is no longer in
> f34+ and
> >>>>>> openshift origin / 3.11 doesn't support any newer either.
> >>>>>
> >>>>> Is this still true? I don't think we want to make the Fedora
> release
> >>>>> process contingent on something that requires F33.
> >>>>
> >>>> yes, it's still true. Note thats the aarch64 osbs cluster.
> >>>> The x86_64 one is rhel7.
> >>>
> >>> Might it be possible to replace Docker with CRI-O on the OpenShift
> >>> cluster?
> >>
> >> Nope. openshift 3 / osbs1 uses docker. :(
> >>
> >> kevin
> >
> > alias docker=podman
>  
>  Using Podman with it through Podman's implementation of the Docker
>  Host protocol *may* work, but a version of Podman that has it is not
>  currently available for RHEL 7 (which is what OpenShift 3.11 runs on). 
Time for an OpenShift upgrade?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Stephen Smoogen]

On Mon, 8 May 2023 at 19:35, Demi Marie Obenour <demiobenour(a)gmail.com&gt;
wrote:

>  On 5/8/23 19:09, Neal Gompa wrote:
>  > On Mon, May 8, 2023 at 7:05 PM Demi Marie Obenour <demiobenour(a)gmail.com&gt;
>  wrote:
>  >>
>  >> On 5/8/23 18:49, Kevin Fenzi wrote:
>  >>> On Mon, May 08, 2023 at 09:29:02PM +0100, Sebastian Crane wrote:
>  >>>> Dear Kevin,
>  >>>>
>  >>>>>> Hmm, quoting from https://pagure.io/releng/issue/11092:
>  >>>>>>>> Also the aarch64 cluster is running on Fedora 33 boxes,
> so we
>  >>>>>>>> should probably try to do a full redeploy :-(
>  >>>>>>> We can't upgrade it from f33 because docker is no longer
> in f34+
>  and
>  >>>>>>> openshift origin / 3.11 doesn't support any newer
> either.
>  >>>>>>
>  >>>>>> Is this still true? I don't think we want to make the Fedora
> release
>  >>>>>> process contingent on something that requires F33.
>  >>>>>
>  >>>>> yes, it's still true. Note thats the aarch64 osbs cluster.
>  >>>>> The x86_64 one is rhel7.
>  >>>>
>  >>>> Might it be possible to replace Docker with CRI-O on the OpenShift
>  >>>> cluster?
>  >>>
>  >>> Nope. openshift 3 / osbs1 uses docker. :(
>  >>>
>  >>> kevin
>  >>
>  >> alias docker=podman
>  >
>  > Using Podman with it through Podman's implementation of the Docker
>  > Host protocol *may* work, but a version of Podman that has it is not
>  > currently available for RHEL 7 (which is what OpenShift 3.11 runs on).
>
>  Time for an OpenShift upgrade?
>
>  
I realize people are trying to help with suggestions, but the problem is
that this is not a simple solution or it would have been done years ago.

The Fedora Build System is a very complicated set of applications all tied
together with custom scripting, schema, and various updates over time.
There are somewhere around 40 core services and 30 more subsidiary ones
which need to be continually working to allow for the daily builds for
multiple releases. Many of those services have been added in at different
times with different operating systems and rules. In order to work with
each other a lot of 'glue' scripting, libraries and such are needed.

You upgrade podman/docker in one section and you find that the message bus
isn't sending in another because an API call isn't there anymore. You try
putting in something which says it has a koji plugin and you find it
doesn't work with our koji and then you need to extend the plugin 40 ways
to work with bodhi, pdc, fedmsg and fedora-messaging, and a dozen other
things which all are the actual build system. Once those get fixed, you
find that it is now causing other glue parts to fail in odd ways no one
remembers why without some archaeological coding.

When dealing with the Build system side of things there is currently 1
senior system administrator and 1 senior release engineer. There are some
volunteers who can work on subparts but mainly have day jobs doing other
things. Looking at IRC, most of the 10-12 hour work day is dealing with
compose problems, build issues, hardware failures, and similar things. Most
of the 40 component subsystems like the Open Shift Build System (OSBS) are
brought in by a dedicated team who have a budget time to get it working and
into place. Once that time is done, any fixes are on the Fedora staff or
working with available time from whatever team.

tl;dr: The Fedora Build System is in total a very complicated set of tools
where changing or upgrading any one part tends to cascade to fixing lots of
glue throughout the system. Instead of looking to upgrade things to add
more deliverables, it may be better to look at other ways to get those
deliverables built.


-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren

[Demi Marie Obenour]

On 5/9/23 07:53, Stephen Smoogen wrote:
>  On Mon, 8 May 2023 at 19:35, Demi Marie Obenour
> <demiobenour(a)gmail.com&gt;
>  wrote:
>  
> > On 5/8/23 19:09, Neal Gompa wrote:
> >> On Mon, May 8, 2023 at 7:05 PM Demi Marie Obenour <demiobenour(a)gmail.com&gt;
> > wrote:
> >>>
> >>> On 5/8/23 18:49, Kevin Fenzi wrote:
> >>>> On Mon, May 08, 2023 at 09:29:02PM +0100, Sebastian Crane wrote:
> >>>>> Dear Kevin,
> >>>>>
> >>>>>>> Hmm, quoting from https://pagure.io/releng/issue/11092:
> >>>>>>>>> Also the aarch64 cluster is running on Fedora 33
> boxes, so we
> >>>>>>>>> should probably try to do a full redeploy :-(
> >>>>>>>> We can't upgrade it from f33 because docker is no
> longer in f34+
> > and
> >>>>>>>> openshift origin / 3.11 doesn't support any newer
> either.
> >>>>>>>
> >>>>>>> Is this still true? I don't think we want to make the
> Fedora release
> >>>>>>> process contingent on something that requires F33.
> >>>>>>
> >>>>>> yes, it's still true. Note thats the aarch64 osbs cluster.
> >>>>>> The x86_64 one is rhel7.
> >>>>>
> >>>>> Might it be possible to replace Docker with CRI-O on the OpenShift
> >>>>> cluster?
> >>>>
> >>>> Nope. openshift 3 / osbs1 uses docker. :(
> >>>>
> >>>> kevin
> >>>
> >>> alias docker=podman
> >>
> >> Using Podman with it through Podman's implementation of the Docker
> >> Host protocol *may* work, but a version of Podman that has it is not
> >> currently available for RHEL 7 (which is what OpenShift 3.11 runs on).
> >
> > Time for an OpenShift upgrade?
> >
> >
>  I realize people are trying to help with suggestions, but the problem is
>  that this is not a simple solution or it would have been done years ago.
>  
>  The Fedora Build System is a very complicated set of applications all tied
>  together with custom scripting, schema, and various updates over time.
>  There are somewhere around 40 core services and 30 more subsidiary ones
>  which need to be continually working to allow for the daily builds for
>  multiple releases. Many of those services have been added in at different
>  times with different operating systems and rules. In order to work with
>  each other a lot of 'glue' scripting, libraries and such are needed.
>  
>  You upgrade podman/docker in one section and you find that the message bus
>  isn't sending in another because an API call isn't there anymore. You try
>  putting in something which says it has a koji plugin and you find it
>  doesn't work with our koji and then you need to extend the plugin 40 ways
>  to work with bodhi, pdc, fedmsg and fedora-messaging, and a dozen other
>  things which all are the actual build system. Once those get fixed, you
>  find that it is now causing other glue parts to fail in odd ways no one
>  remembers why without some archaeological coding. 
Having ancient versions of OpenShift is almost certainly a security problem.
If there is stuff that cannot be kept on a supported version it should be
removed.

>  When dealing with the Build system side of things there is currently
> 1
>  senior system administrator and 1 senior release engineer. There are some
>  volunteers who can work on subparts but mainly have day jobs doing other
>  things. Looking at IRC, most of the 10-12 hour work day is dealing with
>  compose problems, build issues, hardware failures, and similar things. Most
>  of the 40 component subsystems like the Open Shift Build System (OSBS) are
>  brought in by a dedicated team who have a budget time to get it working and
>  into place. Once that time is done, any fixes are on the Fedora staff or
>  working with available time from whatever team. 
This is the actual problem IMO.  What will it take to fix it?

>  tl;dr: The Fedora Build System is in total a very complicated set of
> tools
>  where changing or upgrading any one part tends to cascade to fixing lots of
>  glue throughout the system. Instead of looking to upgrade things to add
>  more deliverables, it may be better to look at other ways to get those
>  deliverables built. 
Do you mean outside of the Fedora Build System?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Kevin Fenzi]

Just a general answer/info here at the bottom of the thread...

I realize our container build pipeline is not great, but it's currently
working and I will keep it working until we replace it.

I agree we should replace it, and there's lots of options, but I don't
think this thread is the place to go back and forth about them. 

I know of at least kiwi, osbuild, some other build systems that don't
fully exist yet, switching to use quay.io, osbs2 (based on openshift4),
and probibly others.

kevin

[Clement Verna]

On Tue, 9 May 2023, 17:51 Kevin Fenzi, <kevin(a)scrye.com&gt; wrote:

>  Just a general answer/info here at the bottom of the thread...
>
>  I realize our container build pipeline is not great, but it's currently
>  working and I will keep it working until we replace it.
>
>  I agree we should replace it, and there's lots of options, but I don't
>  think this thread is the place to go back and forth about them.
>
>  I know of at least kiwi, osbuild, some other build systems that don't
>  fully exist yet, switching to use quay.io, osbs2 (based on openshift4),
>  and probibly others.
>  
I also agree that we should replace it but that's not an easy task
unfortunately.
My understanding of osbuild is that it focuses on base images and not
layered images so that probably would not be a good candidate for replacing
OSBS. I might also be wrong :-)

>  kevin
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>

[Owen Taylor]

On Tue, May 9, 2023 at 11:52 AM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:

>  Just a general answer/info here at the bottom of the thread...
>
>  I realize our container build pipeline is not great, but it's currently
>  working and I will keep it working until we replace it.
>
>  I agree we should replace it, and there's lots of options, but I don't
>  think this thread is the place to go back and forth about them.
>
>  I know of at least kiwi, osbuild, some other build systems that don't
>  fully exist yet, switching to use quay.io, osbs2 (based on openshift4),
>  and probibly others.
>  
What if we made the Toolbox container image just one more base image and
built it with ImageFactory?

 - Integrated into the compose process
 - Across all architectures
 - No OSBS dependency

The main disadvantage is that it is no longer layered, so *if* you happen
to have the exact same Fedora image version around for some other reason (a
big if), you save a fraction of space:

Fedora 38 container - 71M compressed, 201M uncompressed
Toolbox add-on layer -  232M compressed, 753M uncompressed
Toolbox squashed  - 291M compressed, 884M uncompressed

But generally seems like it would be a win. osbuild/kiwi/whatever can be
left as a separate project.

- Owen

[Kevin Fenzi]

On Wed, May 24, 2023 at 01:50:21PM -0400, Owen Taylor wrote:
>  
>  What if we made the Toolbox container image just one more base image and
>  built it with ImageFactory?
>  
>   - Integrated into the compose process
>   - Across all architectures
>   - No OSBS dependency
>  
>  The main disadvantage is that it is no longer layered, so *if* you happen
>  to have the exact same Fedora image version around for some other reason (a
>  big if), you save a fraction of space:
>  
>  Fedora 38 container - 71M compressed, 201M uncompressed
>  Toolbox add-on layer -  232M compressed, 753M uncompressed
>  Toolbox squashed  - 291M compressed, 884M uncompressed
>  
>  But generally seems like it would be a win. osbuild/kiwi/whatever can be
>  left as a separate project. 
I would support this fully. ;) 

It would make it a lot easier and then it wouldn't depend on the module
build pipeline.

kevin

[Debarshi Ray]

Hey Owen,

On Wed, 2023-05-24 at 13:50 -0400, Owen Taylor wrote:
>  
>  What if we made the Toolbox container image just one more base image
>  and built it with ImageFactory?
>  
>   - Integrated into the compose process
>   - Across all architectures
>   - No OSBS dependency
>   
>  The main disadvantage is that it is no longer layered, so *if* you
>  happen to have the exact same Fedora image version around for some
>  other reason (a big if), you save a fraction of space:
>                        
>  Fedora 38 container - 71M compressed, 201M uncompressed      
>  Toolbox add-on layer -  232M compressed, 753M uncompressed
>  Toolbox squashed  - 291M compressed, 884M uncompressed 
I am not too attached to the bandwidth or space savings due to the
fedora-toolbox image being a layered image.

[ In future, I would like to  explore if we can ship the fedora-toolbox
image as part of the Silverblue and Workstation ISOs, to avoid the need
for a sizable download just to set up the default Toolbx.  That could
unlock things like defaulting to a Toolbx shell or generally help
promoting Toolbx as a primary interactive CLI environment.  Anyway,
that's a big 'if'.  ]

My main concern, which I had brought up in the Release Engineering
tickets before [1,2] is whether the fedora-toolbox images would
continue to be defined as a Docker/Containerfile.  I am asking because
the fedora base images are defined as kickstart files [3].

There's some value in continuing to use a Docker/Containerfile because
it's widely known and leads to a common workflow.  It is used for the
official Toolbx images for other distributions like Arch Linux, RHEL
and Ubuntu, and a growing list of third-party Toolbx images [4].  A
Docker/Containerfile is easy to use with 'podman build' as part of the
upstream CI, which makes it easier to test all the different parts.

Currently the Fedora and RHEL images that are shipped to users aren't
built straight from the upstream Git repository, but from the separate
copies on Fedora and Red Hat dist-git.  Even then, it's a lot easier to
manage two copies of the sources in the same format, as opposed to one
using Docker/Containerfiles and the other something else like kickstart
files.

Cheers,
Rishi

[1] https://pagure.io/releng/issue/11399#comment-853047
    Attached to this Change proposal

[2] https://pagure.io/releng/issue/11189#comment-840689
    Past discussion that led to this Change

[3] https://pagure.io/fedora-kickstarts

[4] https://github.com/toolbx-images/images

[Owen Taylor]

On Mon, May 29, 2023 at 8:16 AM Debarshi Ray via devel <
devel(a)lists.fedoraproject.org&gt; wrote:

>  Hey Owen,
>
>  On Wed, 2023-05-24 at 13:50 -0400, Owen Taylor wrote:
>  >
>  > What if we made the Toolbox container image just one more base image
>  > and built it with ImageFactory?
>  >
>  >  - Integrated into the compose process
>  >  - Across all architectures
>  >  - No OSBS dependency
>  >
>  > The main disadvantage is that it is no longer layered, so *if* you
>  > happen to have the exact same Fedora image version around for some
>  > other reason (a big if), you save a fraction of space:
>  >
>  > Fedora 38 container - 71M compressed, 201M uncompressed
>  > Toolbox add-on layer -  232M compressed, 753M uncompressed
>  > Toolbox squashed  - 291M compressed, 884M uncompressed
>
>  I am not too attached to the bandwidth or space savings due to the
>  fedora-toolbox image being a layered image.
>
>  [ In future, I would like to  explore if we can ship the fedora-toolbox
>  image as part of the Silverblue and Workstation ISOs, to avoid the need
>  for a sizable download just to set up the default Toolbx.  That could
>  unlock things like defaulting to a Toolbx shell or generally help
>  promoting Toolbx as a primary interactive CLI environment.  Anyway,
>  that's a big 'if'.  ]
>
>  My main concern, which I had brought up in the Release Engineering
>  tickets before [1,2] is whether the fedora-toolbox images would
>  continue to be defined as a Docker/Containerfile.  I am asking because
>  the fedora base images are defined as kickstart files [3].
>
>  There's some value in continuing to use a Docker/Containerfile because
>  it's widely known and leads to a common workflow.  It is used for the
>  official Toolbx images for other distributions like Arch Linux, RHEL
>  and Ubuntu, and a growing list of third-party Toolbx images [4].  A
>  Docker/Containerfile is easy to use with 'podman build' as part of the
>  upstream CI, which makes it easier to test all the different parts. 

Unfortunately, if we switched to using ImageFactory or pretty much anything
other than OSBS, we'd no longer be able to define the Fedora toolbox image
as a Dockerfile. It certainly is a disadvantage that it's no longer
connected to the way that the other toolbox images are defined, but maybe
that's bearable if we can substantially improve the workflow?

With some refactoring of the kickstarts for Fedora containers, we could
also remove some of the hackiness in the Dockerfile where it is undoing
choices in the base image.

When I ask this question, I do have a hidden agenda - I'm thinking that it
might be possible to move Flatpak generation to a Koji plugin that runs
directly on the builders, but that would leave the toolbox as the sole user
of the OSBS cluster, which seems
dubiously sustainable. So maybe we can take Toolbox off of OSBS as well and
retire OSBS?

Now, if we could instead move to OSBS 2 and have that actively maintained
and across architecture, that could be better, but I don't really see where
the resources are going to come from to do that.

- Owen

[Debarshi Ray]

Hey Owen,

On Mon, 2023-05-29 at 12:39 -0400, Owen Taylor wrote:
>  On Mon, May 29, 2023 at 8:16 AM Debarshi Ray via devel
>  <devel(a)lists.fedoraproject.org&gt; wrote:
>  > 
>  > My main concern, which I had brought up in the Release Engineering
>  > tickets before [1,2] is whether the fedora-toolbox images would
>  > continue to be defined as a Docker/Containerfile.  I am asking
>  > because the fedora base images are defined as kickstart files [3].
>  > 
>  > There's some value in continuing to use a Docker/Containerfile
>  > because it's widely known and leads to a common workflow.  It is
>  > used for the official Toolbx images for other distributions like
>  > Arch Linux, RHEL and Ubuntu, and a growing list of third-party
>  > Toolbx images [4].  A Docker/Containerfile is easy to use with
>  > 'podman build' as part of the upstream CI, which makes it easier to
>  > test all the different parts.
>  
>  Unfortunately, if we switched to using ImageFactory or pretty much
>  anything other than OSBS, we'd no longer be able to define the Fedora
>  toolbox image as a Dockerfile. It certainly is a disadvantage that
>  it's no longer connected to the way that the other toolbox images are
>  defined, 
It's an advantage to have the Fedora images be defined in the same
source language as the Arch Linux and Ubuntu images, and certainly the
RHEL images.  However, ultimately, I see it only as a convenience. 
Folks can take the Fedora images as a reference and tweak them to fit
their own distribution, or copy paste the files across Git
repositories, etc..  If we lose that, then I would learn to live with
it.  :)

If we do move away from Container/Dockerfiles, then my main question
would be whether it'll still be easy to locally build the images.  Will
we need something a lot more elaborate than the simplicity of a 'podman
build'?

Toolbx has two parts.  A somewhat distro-agnostic toolbox(1) binary,
and an OCI image for a distribution.  Both work together to provide an
interactive CLI environment, and hence both should be tested together.

If a contributor attempts to change one or both of those two, then it
should be easy for them to verify whether both would continue to work
together.  Currently, they can do that by doing a local 'podman build'
on the image definitions in toolbox.git and try to use them with their
modified toolbox(1) binary.  I want to formalize this by making the
upstream CI run against both:
  (a) images that are currently published on the OCI registries and 
  (b) images that are built on-the-fly from the sources in toolbox.git

... so that it will validate any changes to the image sources in
toolbox.git, and alert us to any changes in the underlying distribution
(eg., packages, dependencies, base images, etc.) that could break
future image rebuilds.

Currently, the upstream CI runs only against (a).

If we lose the simplicity of a 'podman build' then we lose the testing.
That's my main worry.

If the new set-up offers a similarly simple way of building the images
from source, then I am happy.

>  but maybe that's bearable if we can substantially improve the
>  workflow? 
The phrase "we can substantially improve the workflow" makes me
curious.  Any details or pointers?

>  With some refactoring of the kickstarts for Fedora containers, we
>  could also remove some of the hackiness in the Dockerfile where it is
>  undoing choices in the base image. 
*nod*

>  When I ask this question, I do have a hidden agenda - I'm
> thinking
>  that it might be possible to move Flatpak generation to a Koji plugin
>  that runs directly on the builders, but that would leave the toolbox
>  as the sole user of the OSBS cluster, which seems dubiously
>  sustainable. So maybe we can take Toolbox off of OSBS as well and
>  retire OSBS? 
Yes, I saw you mention this elsewhere.  I totally understand and agree
with your motivation.  Trying to consolidate the infrastructure does
sound like a good idea.

>  Now, if we could instead move to OSBS 2 and have that actively
>  maintained and across architecture, that could be better, but I don't
>  really see where the resources are going to come from to do that.  
*nod*

Cheers,
Rishi

[Owen Taylor]

On Tue, May 30, 2023 at 9:47 AM Debarshi Ray <rishi.is(a)lostca.se&gt; wrote:

>  Hey Owen,
>
>  On Mon, 2023-05-29 at 12:39 -0400, Owen Taylor wrote:
>  > On Mon, May 29, 2023 at 8:16 AM Debarshi Ray via devel
>  > <devel(a)lists.fedoraproject.org&gt; wrote:
>  > >
>  > > My main concern, which I had brought up in the Release Engineering
>  > > tickets before [1,2] is whether the fedora-toolbox images would
>  > > continue to be defined as a Docker/Containerfile.  I am asking
>  > > because the fedora base images are defined as kickstart files [3].
>  > >
>  > > There's some value in continuing to use a Docker/Containerfile
>  > > because it's widely known and leads to a common workflow.  It is
>  > > used for the official Toolbx images for other distributions like
>  > > Arch Linux, RHEL and Ubuntu, and a growing list of third-party
>  > > Toolbx images [4].  A Docker/Containerfile is easy to use with
>  > > 'podman build' as part of the upstream CI, which makes it easier to
>  > > test all the different parts.
>  >
>  > Unfortunately, if we switched to using ImageFactory or pretty much
>  > anything other than OSBS, we'd no longer be able to define the Fedora
>  > toolbox image as a Dockerfile. It certainly is a disadvantage that
>  > it's no longer connected to the way that the other toolbox images are
>  > defined,
>
>  It's an advantage to have the Fedora images be defined in the same
>  source language as the Arch Linux and Ubuntu images, and certainly the
>  RHEL images.  However, ultimately, I see it only as a convenience.
>  Folks can take the Fedora images as a reference and tweak them to fit
>  their own distribution, or copy paste the files across Git
>  repositories, etc..  If we lose that, then I would learn to live with
>  it.  :)
>  
I guess in this scenario, the Fedora images are no longer the reference for
creating things for other distributions and something else (RHEL, Ubuntu,
...) would have to take over that role. For "tweaking", I'd think we'd
definitely want to promote "FROM fedora-toolbox:latest".


>  If we do move away from Container/Dockerfiles, then my main question
>  would be whether it'll still be easy to locally build the images.  Will
>  we need something a lot more elaborate than the simplicity of a 'podman
>  build'?
>  
I haven't actually tried building the Fedora container images locally, but
it it looks relatively documented and straightforward, and someone could
write a shell script to automate it pretty easily. But it is still going to
require tools (ImageFactory) that people don't have installed and probably
take a while to run. It seems like more of a "want to contribute to the the
Fedora toolbox image" thing than a "would do it to run tests when
contributing to Toolbx" thing.

Toolbx has two parts.  A somewhat distro-agnostic toolbox(1) binary,
>  and an OCI image for a distribution.  Both work together to provide
> an
>  interactive CLI environment, and hence both should be tested together.
>
>  If a contributor attempts to change one or both of those two, then it
>  should be easy for them to verify whether both would continue to work
>  together.  Currently, they can do that by doing a local 'podman build'
>  on the image definitions in toolbox.git and try to use them with their
>  modified toolbox(1) binary.  I want to formalize this by making the
>  upstream CI run against both:
>    (a) images that are currently published on the OCI registries and
>    (b) images that are built on-the-fly from the sources in toolbox.git
>  
In this scenario the Fedora Toolbx image definition would live in
fedora-kickstarts, so it doesn't really make sense for me that Toolbox CI
would *recreate it*, rather than just using the version from the repository.

Ideally, we'd run the Toolbox tests against the newly built Toolbx image as
part of QE for the Fedora compose.


>  ... so that it will validate any changes to the image sources in
>  toolbox.git, and alert us to any changes in the underlying distribution
>  (eg., packages, dependencies, base images, etc.) that could break
>  future image rebuilds.
>
>  Currently, the upstream CI runs only against (a).
>
>  If we lose the simplicity of a 'podman build' then we lose the testing.
>  That's my main worry.
>
>  If the new set-up offers a similarly simple way of building the images
>  from source, then I am happy.
>
>  > but maybe that's bearable if we can substantially improve the
>  > workflow?
>
>  The phrase "we can substantially improve the workflow" makes me
>  curious.  Any details or pointers?
>  
What I mean by this was simply the obvious - that the Toolbx image is built
as part of the compose, and can be tested as part of the compose
validation. That nobody is sitting there manually typing 'fedpkg
container-build' and creating Bodhi updates. (*)

- Owen

(*) I'm not saying that this level of integration is impossible with OSBS -
in fact there is some level of Pungi support for OSBS container builds
already. Just that it's basically *already* there in Fedora for the base
images.

[Clement Verna]

I think it's worth to mention that many layered container images are
published directly on the Fedora's Quay.io organisation (
https://quay.io/organization/fedora).
The builds are not happening in OSBS or the Fedora infrastructure but IIRC
it's a GitHub Action that does the build and pushes the container image.
There could be an option to leverage Quay.io build capabilities too.

On Tue, 30 May 2023, 17:02 Owen Taylor, <otaylor(a)redhat.com&gt; wrote:

>  On Tue, May 30, 2023 at 9:47 AM Debarshi Ray <rishi.is(a)lostca.se&gt; wrote:
>
> > Hey Owen,
> >
> > On Mon, 2023-05-29 at 12:39 -0400, Owen Taylor wrote:
> > > On Mon, May 29, 2023 at 8:16 AM Debarshi Ray via devel
> > > <devel(a)lists.fedoraproject.org&gt; wrote:
> > > >
> > > > My main concern, which I had brought up in the Release Engineering
> > > > tickets before [1,2] is whether the fedora-toolbox images would
> > > > continue to be defined as a Docker/Containerfile.  I am asking
> > > > because the fedora base images are defined as kickstart files [3].
> > > >
> > > > There's some value in continuing to use a Docker/Containerfile
> > > > because it's widely known and leads to a common workflow.  It is
> > > > used for the official Toolbx images for other distributions like
> > > > Arch Linux, RHEL and Ubuntu, and a growing list of third-party
> > > > Toolbx images [4].  A Docker/Containerfile is easy to use with
> > > > 'podman build' as part of the upstream CI, which makes it easier
> to
> > > > test all the different parts.
> > >
> > > Unfortunately, if we switched to using ImageFactory or pretty much
> > > anything other than OSBS, we'd no longer be able to define the Fedora
> > > toolbox image as a Dockerfile. It certainly is a disadvantage that
> > > it's no longer connected to the way that the other toolbox images are
> > > defined,
> >
> > It's an advantage to have the Fedora images be defined in the same
> > source language as the Arch Linux and Ubuntu images, and certainly the
> > RHEL images.  However, ultimately, I see it only as a convenience.
> > Folks can take the Fedora images as a reference and tweak them to fit
> > their own distribution, or copy paste the files across Git
> > repositories, etc..  If we lose that, then I would learn to live with
> > it.  :)
> >
>
>  I guess in this scenario, the Fedora images are no longer the reference
>  for creating things for other distributions and something else (RHEL,
>  Ubuntu, ...) would have to take over that role. For "tweaking", I'd think
>  we'd definitely want to promote "FROM fedora-toolbox:latest".
>
>
> > If we do move away from Container/Dockerfiles, then my main question
> > would be whether it'll still be easy to locally build the images.  Will
> > we need something a lot more elaborate than the simplicity of a 'podman
> > build'?
> >
>
>  I haven't actually tried building the Fedora container images locally,
>  but  it it looks relatively documented and straightforward, and someone
>  could write a shell script to automate it pretty easily. But it is still
>  going to require tools (ImageFactory) that people don't have installed and
>  probably take a while to run. It seems like more of a "want to contribute
>  to the the Fedora toolbox image" thing than a "would do it to run tests
>  when contributing to Toolbx" thing.
>
>  Toolbx has two parts.  A somewhat distro-agnostic toolbox(1) binary,
> > and an OCI image for a distribution.  Both work together to provide an
> > interactive CLI environment, and hence both should be tested together.
> >
> > If a contributor attempts to change one or both of those two, then it
> > should be easy for them to verify whether both would continue to work
> > together.  Currently, they can do that by doing a local 'podman build'
> > on the image definitions in toolbox.git and try to use them with their
> > modified toolbox(1) binary.  I want to formalize this by making the
> > upstream CI run against both:
> >   (a) images that are currently published on the OCI registries and
> >   (b) images that are built on-the-fly from the sources in toolbox.git
> >
>
>  In this scenario the Fedora Toolbx image definition would live in
>  fedora-kickstarts, so it doesn't really make sense for me that Toolbox CI
>  would *recreate it*, rather than just using the version from the repository.
>
>  Ideally, we'd run the Toolbox tests against the newly built Toolbx image
>  as part of QE for the Fedora compose.
>
>
> > ... so that it will validate any changes to the image sources in
> > toolbox.git, and alert us to any changes in the underlying distribution
> > (eg., packages, dependencies, base images, etc.) that could break
> > future image rebuilds.
> >
> > Currently, the upstream CI runs only against (a).
> >
> > If we lose the simplicity of a 'podman build' then we lose the testing.
> > That's my main worry.
> >
> > If the new set-up offers a similarly simple way of building the images
> > from source, then I am happy.
> >
> > > but maybe that's bearable if we can substantially improve the
> > > workflow?
> >
> > The phrase "we can substantially improve the workflow" makes me
> > curious.  Any details or pointers?
> >
>
>  What I mean by this was simply the obvious - that the Toolbx image is
>  built as part of the compose, and can be tested as part of the compose
>  validation. That nobody is sitting there manually typing 'fedpkg
>  container-build' and creating Bodhi updates. (*)
>
>  - Owen
>
>  (*) I'm not saying that this level of integration is impossible with OSBS
>  - in fact there is some level of Pungi support for OSBS container builds
>  already. Just that it's basically *already* there in Fedora for the base
>  images.
>
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>

[Debarshi Ray]

Hey,

I wanted to wrap up this sub-thread on-list, after Owen and I chatted
about it off-list.

I am fine with having the fedora-toolbox OCI images being defined as
kickstart files in the Fedora infrastructure and built by ImageFactory
and published as another base image, just like the fedora base image
currently is.

This means that the downstream sources of the images in Fedora that are
actually used to make them available to users will differ from the
upstream sources that are used for testing and CI and are currently
represented as Containerfiles.  If we can't use ImageFactory for
upstream testing and CI, we will have to carefully keep them
synchronized whenever there are changes.  If this is the price to pay
for making the fedora-toolbox images release-blocking deliverables,
then so be it.  :)

That said, my understanding is that Owen is still not settled on all
the tooling changes for building Fedora Flatpaks.  So, we may or may
not end up using ImageFactory for the fedora-toolbox images in the end.

Cheers,
Rishi

[Josh Boyer]

On Mon, Jun 5, 2023 at 6:28 AM Debarshi Ray via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  Hey,
>
>  I wanted to wrap up this sub-thread on-list, after Owen and I chatted
>  about it off-list.
>
>  I am fine with having the fedora-toolbox OCI images being defined as
>  kickstart files in the Fedora infrastructure and built by ImageFactory
>  and published as another base image, just like the fedora base image
>  currently is. 
Just an FYI, but we're trying to move away from ImageFactory in the
general sense.  I'm not sure if/how that will manifest itself in
Fedora, but ImageFactory is pretty terrible for debugging things when
it goes wrong.  The options we're looking at elsewhere are Image
Builder, OSBS2, and RHTAP.

Honestly, for base container images the simplest solution is just
buildah but nobody has put a lot of time into making that workable in
the ways that many projects would view as "official".

josh

> This means that the downstream sources of the images in Fedora that are
> actually used to make them available to users will differ from the
> upstream sources that are used for testing and CI and are currently
> represented as Containerfiles.  If we can't use ImageFactory for
> upstream testing and CI, we will have to carefully keep them
> synchronized whenever there are changes.  If this is the price to pay
> for making the fedora-toolbox images release-blocking deliverables,
> then so be it.  :)
>
> That said, my understanding is that Owen is still not settled on all
> the tooling changes for building Fedora Flatpaks.  So, we may or may
> not end up using ImageFactory for the fedora-toolbox images in the end.
>
> Cheers,
> Rishi
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Owen Taylor]

On Mon, Jun 5, 2023 at 8:35 AM Josh Boyer <jwboyer(a)fedoraproject.org&gt; wrote:

>  On Mon, Jun 5, 2023 at 6:28 AM Debarshi Ray via devel
>  <devel(a)lists.fedoraproject.org&gt; wrote:
>
>  I wanted to wrap up this sub-thread on-list, after Owen and I chatted
>  > about it off-list.
>  >
>  > I am fine with having the fedora-toolbox OCI images being defined as
>  > kickstart files in the Fedora infrastructure and built by ImageFactory
>  > and published as another base image, just like the fedora base image
>  > currently is.
>
>  Just an FYI, but we're trying to move away from ImageFactory in the
>  general sense.  I'm not sure if/how that will manifest itself in
>  Fedora, but ImageFactory is pretty terrible for debugging things when
>  it goes wrong.  The options we're looking at elsewhere are Image
>  Builder, OSBS2, and RHTAP.
>  
If we want to move Fedora images to ImageBuilder, then the right thing to
do here is still to merge the toolbox image into fedora-kickstarts and then
work on migrating that, rather than have it be a one-off outlier. (*)

- Owen

(*) Yes, I know Fedora-IOT uses ImageBuilder. but in the context of the
main Fedora compose.

[Clement Verna]

On Mon, 8 May 2023 at 22:11, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:

>  On Mon, May 08, 2023 at 07:57:30PM +0000, Zbigniew
> Jędrzejewski-Szmek
>  wrote:
>  >
>  > I think we need some clarity wrt. to the dependency order here.
>  > Let's say we:
>  > > In order to do this at branch point, we will need to move building this
>  > > image into the compose process and mark it blocking.
>  > OK, so we build things, but then we need to publish to registry.fp.o,
>  > which is asynchrounous (?). When we test the newly built ISOs, do
>
>  No, it happens at the end of the compose (if no blocking deliverables
>  failed causing the compose to fail)
>  
If we do this, we should also make the container base images release
blocking since AFAIK they are currently not release blocking.


>  > we test them also with the latest image that we get from
> registry.fp.o?
>  > And if we find a bug anywhere in this pipeline, we respin everything?
>
>  Good question. I'll note that currently we do not do any specific
>  testing after branch point. We freeze things to get a compose to
>  complete, but then we move on. It's not like Beta or Final.
>
>  > > I'd like to note that making this blocking doesn't waive any kind of
>  > > magic wand that makes our infrastructure more reliable. ;)
>  > > The container build pipeline is a long collection of fragile things.
>  > > It may well result in us slipping more based on things not working. ;(
>  >
>  > Hmm, quoting from https://pagure.io/releng/issue/11092:
>  > >> Also the aarch64 cluster is running on Fedora 33 boxes, so we
>  > >> should probably try to do a full redeploy :-(
>  > > We can't upgrade it from f33 because docker is no longer in f34+ and
>  > > openshift origin / 3.11 doesn't support any newer either.
>  >
>  > Is this still true? I don't think we want to make the Fedora release
>  > process contingent on something that requires F33.
>
>  yes, it's still true. Note thats the aarch64 osbs cluster.
>  The x86_64 one is rhel7.
>
>  So, perhaps it would make sense to only make the x86_64 one blocking? 

>  kevin
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>

[Debarshi Ray]

Hey Clement,

On Tue, 2023-05-09 at 09:45 +0200, Clement Verna wrote:
>  
>  
>  On Mon, 8 May 2023 at 22:11, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  > On Mon, May 08, 2023 at 07:57:30PM +0000, Zbigniew Jędrzejewski-
>  > Szmek wrote:
>  > > 
>  > > I think we need some clarity wrt. to the dependency order here.
>  > > Let's say we:
>  > > > In order to do this at branch point, we will need to move
>  > > > building this
>  > > > image into the compose process and mark it blocking.
>  > > OK, so we build things, but then we need to publish to
>  > > registry.fp.o,
>  > > which is asynchrounous (?). When we test the newly built ISOs, do
>  > 
>  > No, it happens at the end of the compose (if no blocking
>  > deliverables failed causing the compose to fail)
>  
>  If we do this, we should also make the container base images release
>  blocking since AFAIK they are currently not release blocking.  
Yes, that's a good point.

The OCI base images aren't currently listed as release-blocking
deliverables:
https://docs.fedoraproject.org/en-US/releases/f39/blocking/

... but if this Change goes through, then they will implicitly become
release-blocking deliverables because the fedora-toolbox images are
based on them.

Should we explicitly mention this in the Change?  Or, something else? 
Or, is it fine as it is?

Thanks,
Rishi

[Clement Verna]

On Tue, 9 May 2023, 15:42 Debarshi Ray via devel, <
devel(a)lists.fedoraproject.org&gt; wrote:

>  Hey Clement,
>
>  On Tue, 2023-05-09 at 09:45 +0200, Clement Verna wrote:
>  >
>  >
>  > On Mon, 8 May 2023 at 22:11, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  > > On Mon, May 08, 2023 at 07:57:30PM +0000, Zbigniew Jędrzejewski-
>  > > Szmek wrote:
>  > > >
>  > > > I think we need some clarity wrt. to the dependency order here.
>  > > > Let's say we:
>  > > > > In order to do this at branch point, we will need to move
>  > > > > building this
>  > > > > image into the compose process and mark it blocking.
>  > > > OK, so we build things, but then we need to publish to
>  > > > registry.fp.o,
>  > > > which is asynchrounous (?). When we test the newly built ISOs, do
>  > >
>  > > No, it happens at the end of the compose (if no blocking
>  > > deliverables failed causing the compose to fail)
>  >
>  > If we do this, we should also make the container base images release
>  > blocking since AFAIK they are currently not release blocking.
>
>  Yes, that's a good point.
>
>  The OCI base images aren't currently listed as release-blocking
>  deliverables:
>  https://docs.fedoraproject.org/en-US/releases/f39/blocking/
>
>  ... but if this Change goes through, then they will implicitly become
>  release-blocking deliverables because the fedora-toolbox images are
>  based on them.
>
>  Should we explicitly mention this in the Change?  Or, something else?
>  Or, is it fine as it is?
>  
I think mentioning explicitly in this change is good enough :)

>  Thanks,
>  Rishi
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>

[Debarshi Ray]

On Wed, 2023-05-10 at 21:35 +0200, Clement Verna wrote:
>  
>  On Tue, 9 May 2023, 15:42 Debarshi Ray via devel,
>  <devel(a)lists.fedoraproject.org&gt; wrote:
>  > 
>  > On Tue, 2023-05-09 at 09:45 +0200, Clement Verna wrote:
>  > > 
>  > > If we do this, we should also make the container base images
>  > > release blocking since AFAIK they are currently not release
>  > > blocking. 
>  > 
>  > Yes, that's a good point.
>  > 
>  > The OCI base images aren't currently listed as release-blocking
>  > deliverables:
>  > https://docs.fedoraproject.org/en-US/releases/f39/blocking/
>  > 
>  > ... but if this Change goes through, then they will implicitly
>  > become release-blocking deliverables because the fedora-toolbox
>  > images are based on them.
>  > 
>  > Should we explicitly mention this in the Change?  Or, something
>  > else?  Or, is it fine as it is?
>  
>  I think mentioning explicitly in this change is good enough :)
>   
Okay!  I add a note to the Detailed Description section of the Change.

[Peter Robinson]

On Mon, May 8, 2023 at 9:11 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  On Mon, May 08, 2023 at 07:57:30PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  >
>  > I think we need some clarity wrt. to the dependency order here.
>  > Let's say we:
>  > > In order to do this at branch point, we will need to move building this
>  > > image into the compose process and mark it blocking.
>  > OK, so we build things, but then we need to publish to registry.fp.o,
>  > which is asynchrounous (?). When we test the newly built ISOs, do
>
>  No, it happens at the end of the compose (if no blocking deliverables
>  failed causing the compose to fail)
>
>  > we test them also with the latest image that we get from registry.fp.o?
>  > And if we find a bug anywhere in this pipeline, we respin everything?
>
>  Good question. I'll note that currently we do not do any specific
>  testing after branch point. We freeze things to get a compose to
>  complete, but then we move on. It's not like Beta or Final.
>
>  > > I'd like to note that making this blocking doesn't waive any kind of
>  > > magic wand that makes our infrastructure more reliable. ;)
>  > > The container build pipeline is a long collection of fragile things.
>  > > It may well result in us slipping more based on things not working. ;(
>  >
>  > Hmm, quoting from https://pagure.io/releng/issue/11092:
>  > >> Also the aarch64 cluster is running on Fedora 33 boxes, so we
>  > >> should probably try to do a full redeploy :-(
>  > > We can't upgrade it from f33 because docker is no longer in f34+ and
>  > > openshift origin / 3.11 doesn't support any newer either.
>  >
>  > Is this still true? I don't think we want to make the Fedora release
>  > process contingent on something that requires F33.
>
>  yes, it's still true. Note thats the aarch64 osbs cluster.
>  The x86_64 one is rhel7.
>
>  So, perhaps it would make sense to only make the x86_64 one blocking? 
Or possibly build it with osbuild, the infra there is pretty stable
now for IoT, it's not perfect, but it does have x86_64/aarch64.

[Neal Gompa]

On Tue, May 9, 2023 at 6:19 AM Peter Robinson <pbrobinson(a)gmail.com&gt; wrote:
>  On Mon, May 8, 2023 at 9:11 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  >
>  > On Mon, May 08, 2023 at 07:57:30PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > >
>  > > I think we need some clarity wrt. to the dependency order here.
>  > > Let's say we:
>  > > > In order to do this at branch point, we will need to move building this
>  > > > image into the compose process and mark it blocking.
>  > > OK, so we build things, but then we need to publish to registry.fp.o,
>  > > which is asynchrounous (?). When we test the newly built ISOs, do
>  >
>  > No, it happens at the end of the compose (if no blocking deliverables
>  > failed causing the compose to fail)
>  >
>  > > we test them also with the latest image that we get from registry.fp.o?
>  > > And if we find a bug anywhere in this pipeline, we respin everything?
>  >
>  > Good question. I'll note that currently we do not do any specific
>  > testing after branch point. We freeze things to get a compose to
>  > complete, but then we move on. It's not like Beta or Final.
>  >
>  > > > I'd like to note that making this blocking doesn't waive any kind
> of
>  > > > magic wand that makes our infrastructure more reliable. ;)
>  > > > The container build pipeline is a long collection of fragile things.
>  > > > It may well result in us slipping more based on things not working. ;(
>  > >
>  > > Hmm, quoting from https://pagure.io/releng/issue/11092:
>  > > >> Also the aarch64 cluster is running on Fedora 33 boxes, so we
>  > > >> should probably try to do a full redeploy :-(
>  > > > We can't upgrade it from f33 because docker is no longer in f34+ and
>  > > > openshift origin / 3.11 doesn't support any newer either.
>  > >
>  > > Is this still true? I don't think we want to make the Fedora release
>  > > process contingent on something that requires F33.
>  >
>  > yes, it's still true. Note thats the aarch64 osbs cluster.
>  > The x86_64 one is rhel7.
>  >
>  > So, perhaps it would make sense to only make the x86_64 one blocking?
>
>  Or possibly build it with osbuild, the infra there is pretty stable
>  now for IoT, it's not perfect, but it does have x86_64/aarch64. 
We have multiple options that aren't the OSBS stuff:

* kiwi (there's a koji plugin for it)
* osbuild
* lorax

I'd be happy to help with kiwi (as one of the upstream developers of it).

The Fedora Asahi, CentOS Hyperscale, and CentOS Alternative Image
teams have been using kiwi to build images for a while now. :)




--
真実はいつも一つ！/ Always, there's only one truth!

[Debarshi Ray]

Hey Zbigniew,

On Mon, 2023-05-08 at 19:57 +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  On Mon, May 08, 2023 at 09:24:08AM -0700, Kevin Fenzi wrote:
>  > I'm broadly in favor here, some comments in line...
>  > 
>  > ...snip...
>  > > First, we want to ensure that there are up to date
>  > > [https://src.fedoraproject.org/container/fedora-toolbox
>  > > fedora-toolbox] OCI images published on
>  > > [https://registry.fedoraproject.org/ registry.fedoraproject.org]
>  > > as
>  > > release-blocking deliverables at critical points in the
>  > > development
>  > > schedule, just like the installation ISOs for the Editions from
>  > > [https://download.fedoraproject.org/pub/fedora/linux/releases/
>  > > download.fedoraproject.org].  This must at least happen when an
>  > > upcoming Fedora release is branched from Rawhide, and for the
>  > > Beta and
>  > > Final release candidates.  If possible, they should be updated
>  > > more
>  > > frequently as part of the nightly composes.  We do not expect
>  > > this to
>  > > happen after a Fedora release has gone GA.
>  
>  I think we need some clarity wrt. to the dependency order here.
>  Let's say we:
>  > In order to do this at branch point, we will need to move building
>  > this
>  > image into the compose process and mark it blocking.
>  OK, so we build things, but then we need to publish to registry.fp.o,
>  which is asynchrounous (?). When we test the newly built ISOs, do
>  we test them also with the latest image that we get from
>  registry.fp.o?
>  And if we find a bug anywhere in this pipeline, we respin everything? 
That's our expectation, yes.

When we apply the test criteria to a compose, if the freshly composed
fedora-toolbox OCI images don't meet the test criteria on hosts running
the freshly composed Fedora ISOs, then we track down the bugs, get them
fixed, re-compose, and try again.  Currently, this will happen for the
Beta and Final release candidates.

Cheers,
Rishi

[Zbigniew Jędrzejewski-Szmek]

On Tue, May 09, 2023 at 03:46:21PM +0200, Debarshi Ray via devel wrote:
>  Hey Zbigniew,
>  
>  On Mon, 2023-05-08 at 19:57 +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > On Mon, May 08, 2023 at 09:24:08AM -0700, Kevin Fenzi wrote:
>  > > I'm broadly in favor here, some comments in line...
>  > > 
>  > > ...snip...
>  > > > First, we want to ensure that there are up to date
>  > > > [https://src.fedoraproject.org/container/fedora-toolbox
>  > > > fedora-toolbox] OCI images published on
>  > > > [https://registry.fedoraproject.org/ registry.fedoraproject.org]
>  > > > as
>  > > > release-blocking deliverables at critical points in the
>  > > > development
>  > > > schedule, just like the installation ISOs for the Editions from
>  > > > [https://download.fedoraproject.org/pub/fedora/linux/releases/
>  > > > download.fedoraproject.org].  This must at least happen when an
>  > > > upcoming Fedora release is branched from Rawhide, and for the
>  > > > Beta and
>  > > > Final release candidates.  If possible, they should be updated
>  > > > more
>  > > > frequently as part of the nightly composes.  We do not expect
>  > > > this to
>  > > > happen after a Fedora release has gone GA.
>  > 
>  > I think we need some clarity wrt. to the dependency order here.
>  > Let's say we:
>  > > In order to do this at branch point, we will need to move building
>  > > this
>  > > image into the compose process and mark it blocking.
>  > OK, so we build things, but then we need to publish to registry.fp.o,
>  > which is asynchrounous (?). When we test the newly built ISOs, do
>  > we test them also with the latest image that we get from
>  > registry.fp.o?
>  > And if we find a bug anywhere in this pipeline, we respin everything?
>  
>  That's our expectation, yes.
>  
>  When we apply the test criteria to a compose, if the freshly composed
>  fedora-toolbox OCI images don't meet the test criteria on hosts running
>  the freshly composed Fedora ISOs, then we track down the bugs, get them
>  fixed, re-compose, and try again.  Currently, this will happen for the
>  Beta and Final release candidates. 
Thanks. Makes sense.

Zbyszek

[Debarshi Ray]

Hey Kevin,

On Mon, 2023-05-08 at 09:24 -0700, Kevin Fenzi wrote:
>  I'm broadly in favor here, some comments in line...
>  
>  ...snip...
>  > It will be beneficial to consider the
>  > [https://src.fedoraproject.org/container/fedora-toolbox
>  > fedora-toolbox] images as release-blocking deliverables because
>  > Fedora's [https://opencontainers.org/ OCI] infrastructure is often
>  > broken.  Here are [https://pagure.io/releng/issue/11092 two]
>  > [https://pagure.io/releng/issue/11367 recent] examples of
>  > <code>fedpkg
>  > container-build</code> not working.  In the second case, it was
>  > preventing the images from being rebuilt to pull in an
>  > [https://bugzilla.redhat.com/show_bug.cgi?id=2170878 important]
>  > bug-fix.  The broken infrastructure prevents regular Fedora
>  > contributors from jumping in to rebuild and publish the images at
>  > critical points in the development schedule.  Making them
>  > release-blocking deliverables would attract greater attention and
>  > scrutiny from release engineering and ensure that a Fedora
>  > development
>  > cycle does not proceed with broken or outdated or missing
>  > <code>fedora-toolbox</code> images.
>  
>  I'd like to note that making this blocking doesn't waive any kind of
>  magic wand that makes our infrastructure more reliable. ;) 
>  The container build pipeline is a long collection of fragile things. 
>  It may well result in us slipping more based on things not working.
>  ;(  
Yes, I understand.

Currently, what happens is that someone (very often Jens) jumps in to
update the fedora-toolbox images from time to time, sometimes because
users started to complain about missing images after branching or some
other problem, then it turns out that Fedora's OCI infrastructure is
broken, then it leads to a pagure.io/releng issue, and then someone
(very often you, Kevin) jumps in to frantically debug and get things
going again.

I was hoping that by making the fedora-toolbox images release-blocking
deliverables, we will avoid some of the ad-hoc frantic last-minute
fire-fighting and get more eye balls on this part of the Fedora
infrastructure.  Most importantly, Fedora users would get a more robust
and smoother user experience because they won't have to scream and yell
on bug trackers to get new images pushed out the door.

Also, it's not really possible to test the toolbox(1) binary unless the
corresponding fedora-toolbox images are updated and published reliably.

In practical terms, if this means that the Toolbx developers need to
get involved to help out with Fedora's OCI infrastructure, then so be
it.  :)

Cheers,
Rishi