F39 Change Proposal: Flatpaks without Modules (System-Wide Change)

[Aoife Moloney]

https://fedoraproject.org/wiki/Changes/FlatpaksWithoutModules

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.

== Owner ==
* Name: [[User:Otaylor|Owen Taylor]]
* Email: otaylor(a)redhat.com


== Detailed Description ==
There are two types of Flatpak containers -
runtimes - which contain unmodified Fedora packages,
and applications - which contain Fedora packages rebuilt to relocate
them from /usr to /app.
Currently, the rebuild process is done using modules -
packages are either rebuilt in a shared flatpak-common module, or in a
module specific to the application.
This change replaces this with a separate build target that all
rebuilds are done in.

=== Rebuilding RPMs ===
For each Fedora release, we'll have two additional targets

 f39-flatpak-runtime (inherits f39), tags: f39-flatpak-runtime-build,
f39-flatpak-runtime
 f39-app (inherits f39-flatpak-runtime), tags: f39-app-build, f39-app

The f39-flatpak-runtime target has a limited use -
it's used to build packages that go into the Flatpak runtimes
(standard and KDE).
This would include flatpak-runtime-config, flatpak-rpm-macros,
but also rebuilds of standard packages if needed to prune dependencies:
currently this is done for gstreamer-plugins-good.

The f39-app target is the main target that is used for prefix=/app rebuilds.
flatpak-rpm-macros is part of the build group for this target,
resulting in builds that target /app.
As currently, spec files can be conditionalized with '%if 0%{?flatpak}'

Builds in f39-flatpak-runtime and f39-app have the dist tags
.f39runtime and .f39app respectively.
This is necessary so that we can automatically rebuild a package
without having to bump the release in the specfile.
(Koji requires everything to have a unique NVR.)
It also allows humans and tools to easily distinguish these specialized builds.

Any Fedora packager can rebuild a package into f39-app using 'fepdkg
build --target=f39-app'.
There will also be a new convenience subcommands of fedpkg 'fedpkg
build-flatpak-rpms',
'fedpkg build-flatpak-rpms-local' that can be used to rebuild all
missing bundled dependencies of a Flatpak,
in Koji or locally.

Once a package exists in f39-app or f38-app, then
[[https://gitlab.com/redhat/centos-stream/ci-cd/distrosync/distrobuildsync
distrobuildsync]] will be used to do a build into f39-app each time a
standard build completes.

=== Container Versioning ===

Currently, the version is based on the module version:

  firefox:stable:3820230427145713:b1edb643 =>
firefox-stable-3820230427145713.b1edb643

Where the module version is created from platform versions and git
commit timestamps
for the module repository.
We could reproduce something very much like this based on the runtime branch and
git commit timestamp, but since application Flatpaks have an
identifiable "Main RPM"
(unlike the general case of modules and containers), a better scheme is:

  firefox-112.0.2-1.fc38 => firefox-flatpak-112.0.2-1
  name: main RPM name + -flatpak
  version: main RPM version
  release: automatically incremented based on existing Koji builds

Because upgrades are done on Flatpak ID (org.mozilla.Firefox),
there is no compatibility impact of switching Flatpaks from 'firefox'
to 'firefox-flatpak'.

There is considerable implementation complexity within OSBS to implement this,
because the N/V/R need to be written into generated Dockerfile as
labels ''before'' building it,
but it should be manageable. If not, we'll need to switch to a different scheme.
(Running dnf with <code>--best</code> when building the container will
help make this reliable
and would be a general improvement.)

=== Updates ===

The current plan is that builds of RPMs will be tagged directly into f39-app and
containers will be composed from the contents of  f39-app with no
involvement of Bodhi.
Bodhi is only involved in releasing the container.

We can add CI checks on container updates to compare the rebuilt
packages in the container
to the latest released versions for Fedora, though it's not always
easy to distinguish:

* Newer because of a RPM change that hasn't passed testing yet
* vs. Newer because a spec file was needed to get the package to
rebuild with prefix=/app

Since Fedora does not embargo builds,
there's no potential information leak if a change to dist-git is
released via container before it is released by RPM.

=== Disadvantages and issues===
In moving to this system, we lose the ability to take advantage of the
capabilities of modularity -
to use a different stream branch of a library for an application than
the one shipped in F39, say.
There are essentially no examples of this being used in all the
Flatpaks created for Fedora over the last 5 years,
probably because modularity has gained little traction within Fedora
more broadly.

Some Flatpak modules strip down bundled packages by setting macros in
the buildopts/rpms/macros section of the
modulemd file. For example, flatpak-common has:

  buildopts:
    rpms:
      macros: |
        %_without_mingw 1
        %_without_tkinter 1

This is no longer possible, so these changes need to be fed back into
spec files with
'%if 0%{?flatpak}' style conditionalization.

Additionally a couple of packages (evolution-data-services and
tracker-miners) are set up so they can be
built with an application-specific D-Bus prefix. Evolution has:

  buildopts:
    rpms:
      macros: |
        %_eds_dbus_services_prefix org.gnome.Evolution

This will need to be redone so that evolution-data-services doesn't
need recompilation
and the prefixing can be done by changing configuration files at
container build time.

== Feedback ==


== Benefit to Fedora ==
The experience for Fedora developers who want to create Flatpaks of
their applications will be improved:
* They no longer need to understand modularity and modulemd files -
instead of a modulemd file *and* a container.yaml, they simply create
a container.yaml.
* Rebuilds are automatically shared between Flatpaks, without having
to get them added to the flatpak-common module.
* Because the local build experience is no longer using the poorly
maintained local-build mode of MBS, it can be more efficient and
easier to understand.

This should result in more Fedora Flatpaks and more up-to-date Fedora Flatpaks.

We also greatly reduce the usage of Module Build Service within Fedora.
This is important because upstream maintenance is currently reduced,
and it's not clear that we'll be able to sustain a deployment indefinitely.

== Scope ==

* Proposal owners:
** Update flatpak-module-tools to be able to work without modularity
** Create changes to fedpkg to add 'rebuild-flatpak-packages'
(implementation likely within flatpak-module-tools)
** Update atomic-reactor to be able to build Flatpaks without modularity

* Other developers:
** Flatpaks will need to be migrated to the new system by Flatpak
packagers. While we could make the new system opt-in for F39, it's
probably easiest to just require all Flatpaks to be moved over. In
many cases, this should consist of just a few minor changes to
container.yaml.
** Changes to fedpkg and atomic-reactor need to be reviewed, and for
atomic-reactor deployed. '''Note''' Fedora is on OSBS 1, which is
not
a current release. Backporting changes to OSBS 1 seems more realistic
than upgrading Fedora to OSBS 2 at this point.

* Release engineering: [https://pagure.io/releng/issue/11415 #Releng
issue number]
** Release engineering will need to manually create the tags/targets
for F39 and update the tag maintenance scripts to do this for future
releases.

* Policies and guidelines:
https://docs.fedoraproject.org/en-US/flatpak/ will need to be updated
once we have an initial implementation. No impact to packaging
guidelines is expected.

* Trademark approval: N/A (not needed for this Change)

* Alignment with Community Initiatives: No


== Upgrade/compatibility impact ==
The Flatpaks built will look exactly the same as before from a user perspective.

== How To Test ==

Testing here will be done by selecting a few Flatpaks and trying to
rebuild them using the new system,
first locally and then on infrastructure.

The final validation will be when we mass rebuild Flatpaks for F39.

== User Experience ==
This should be largely invisible to users.
Hopefully, by simplifying the developer experience around creating Flatpaks,
more Fedora Flatpaks will be available and they will be kept more up-to-date.

== Dependencies ==
This is unrelated to other Fedora changes or to changes within packages.

== Contingency Plan ==

* Contingency mechanism: The current infrastructure for module-based
builds will be maintained, so that if we can't get this done for F39,
we'll build Flatpaks as modules for F39.
* Contingency deadline: By 2023-08-01 we should have runtimes built
and a number of applications or we'll plan on using modules for F39
Flatpaks.
* Blocks release? No

== Documentation ==
At the moment, this change proposal is standalone documentation of the
change. As noted above, https://docs.fedoraproject.org/en-US/flatpak/
will need to be updated once we have an initial implementation. If you
are interested in getting involved in planning and implementing this
change, please contact the
[https://fedoraproject.org/wiki/SIGs/Flatpak Flatpak SIG] matrix room.

== Release Notes ==




-- 
Aoife Moloney

Product Owner

Community Platform Engineering Team

Red Hat EMEA

Communications House

Cork Road

Waterford

[Milan Crha]

On Wed, 2023-05-10 at 11:54 +0100, Aoife Moloney wrote:
>  Additionally a couple of packages (evolution-data-services and
>  tracker-miners) are set up so they can be
>  built with an application-specific D-Bus prefix. Evolution has:
>  
>    buildopts:
>      rpms:
>        macros: |
>          %_eds_dbus_services_prefix org.gnome.Evolution
>  
>  This will need to be redone so that evolution-data-services doesn't
>  need recompilation
>  and the prefixing can be done by changing configuration files at
>  container build time. 
	Hi,
I cannot speak for the tracker-miners, but in case of the Evolution, it
runs in its own sandbox, separated from the host system, with bundled
evolution-data-server (eds) services, thus when the user installs the
Flatpak version, he/she gets the expected Evolution and eds versions,
independent of what the host system has installed. Advantage: the user
gets all fixes, not only client-side (Evolution) fixes. It's important,
from my point of view.

>  In many cases, this should consist of just a few minor changes to
>  container.yaml. 
Do you mean the `evolution.yaml` is gone with this change and the
dependencies are taken directly from the .spec file? The eds dependency
is a problem here, as you noted, not talking that not everything from
the .spec file requires a rebuild for the flatpak (most dependencies
are part of the runtime).

	Bye,
	Milan

[Owen Taylor]

On Wed, May 10, 2023 at 8:06 AM Milan Crha <mcrha(a)redhat.com&gt; wrote:

>  On Wed, 2023-05-10 at 11:54 +0100, Aoife Moloney wrote:
>  > Additionally a couple of packages (evolution-data-services and
>  > tracker-miners) are set up so they can be
>  > built with an application-specific D-Bus prefix. Evolution has:
>  >
>  >   buildopts:
>  >     rpms:
>  >       macros: |
>  >         %_eds_dbus_services_prefix org.gnome.Evolution
>  >
>  > This will need to be redone so that evolution-data-services doesn't
>  > need recompilation
>  > and the prefixing can be done by changing configuration files at
>  > container build time.
>
>          Hi,
>  I cannot speak for the tracker-miners, but in case of the Evolution, it
>  runs in its own sandbox, separated from the host system, with bundled
>  evolution-data-server (eds) services, thus when the user installs the
>  Flatpak version, he/she gets the expected Evolution and eds versions,
>  independent of what the host system has installed. Advantage: the user
>  gets all fixes, not only client-side (Evolution) fixes. It's important,
>  from my point of view.
>  
We'll still have an evolution-data-server that runs inside the sandbox and
has prefixed names, it will just have to be done a different way. My best
current idea is that for the (single) Flatpak build, the
evolution-data-services would have

-DDBUS_SERVICES_PREFIX=com.example.FlatpakApp

Then we'd change things (preferably upstream) so instead of building
DBUS_SERVICES_PREFIX into the source code,


com.example.FlatpakApp.org.gnome.evolution.dataserver.AddressBook10.service

would have:

 Exec=/app/libexec/evolution-addressbook-factory
--dbus-services-prefix=com.example.FlatpakApp

Then container.yaml would have something like:

 dbus-service-substitute=com.example.FlatpakApp:org.gnome.evolution

And that would do that substitution in the names and contents of any
service files when building the container. Does that sound workable? Are
there better ways we could do it?

>  In many cases, this should consist of just a few minor changes to
>  > container.yaml.
>
>  Do you mean the `evolution.yaml` is gone with this change and the
>  dependencies are taken directly from the .spec file? The eds dependency
>  is a problem here, as you noted, not talking that not everything from
>  the .spec file requires a rebuild for the flatpak (most dependencies
>  are part of the runtime).
>  
Yeah, evolution.yaml is gone with the change. Any dependency that is not
part of the runtime will have to have been rebuilt in the f39-app target,
or the container build will fail. Then we'll have some convenience at the
fedpkg level to make that not too painful. ('fedpkg flatpak-build' will
check that things seem ok before starting the build, and there will be
'fedpkg flatpak-build-rpms --all-missing' to rebuild the missing
dependencies.)

- Owen

[Milan Crha]

On Wed, 2023-05-10 at 09:30 -0400, Owen Taylor wrote:
>  Does that sound workable? Are there better ways we could do it?
	Hi,
if I recall correctly, using the custom D-Bus prefix is there to match
application's D-Bus prefix defined for the flatpak, thus:
a) the services run independently from the system, inside the sandbox;
b) they can be started without any additional effort on the Flatpak
   configuration side (because the services share the app's D-Bus
   prefix).

I briefly grepped the evolution-data-server sources and it seems that
most of the places in the .c files can be changed in runtime, but there
are places where the things can break, like the `evolution-data-
server.pc` file or the D-Bus .service files, which both reference the
D-Bus name from the compile time and those .service files are also
named by the service (otherwise there's a runtime warning in the
journal about the file not matching the service name). Those might not
be show stoppers, I guess.

By the way, when people install Evolution flatpak, they have
preinstalled evolution-ews inside it. How will they install it after
this change?

	Bye,
	Milan

[Owen Taylor]

On Mon, May 15, 2023 at 1:51 AM Milan Crha <mcrha(a)redhat.com&gt; wrote:

>  On Wed, 2023-05-10 at 09:30 -0400, Owen Taylor wrote:
>  > Does that sound workable? Are there better ways we could do it?
>
>          Hi,
>  if I recall correctly, using the custom D-Bus prefix is there to match
>  application's D-Bus prefix defined for the flatpak, thus:
>  a) the services run independently from the system, inside the sandbox;
>  b) they can be started without any additional effort on the Flatpak
>     configuration side (because the services share the app's D-Bus
>     prefix).
>
>  I briefly grepped the evolution-data-server sources and it seems that
>  most of the places in the .c files can be changed in runtime, but there
>  are places where the things can break, like the `evolution-data-
>  server.pc` file or the D-Bus .service files, which both reference the
>  D-Bus name from the compile time and those .service files are also
>  named by the service (otherwise there's a runtime warning in the
>  journal about the file not matching the service name). Those might not
>  be show stoppers, I guess.
>  
The idea is that we'd substitute the name and contents of the service files
at container creation time. The .pc file is tricker, since it is build
time. Those variables aren't used much.
I found two examples:
 - libfolks uses it when setting up a private bus for test cases -
shouldn't matter
 - You use it in to fix bus names in flatpak-evolution-wrapper.sh - those
could just be hardcoded since the bus name will be always the same for the
Evolution Flatpak

By the way, when people install Evolution flatpak, they have
>  preinstalled evolution-ews inside it. How will they install it after
>  this change?
>  
The idea is that the container.yaml can list any number of packages to be
installed in the Flatpak. The first one is the "main" one, but that only
means that the name/version of the Flatpak come from it by default. Adding
evolution-ews as an additional package will work fine.

- Owen

[Milan Crha]

	Hi,

On Wed, 2023-05-17 at 12:23 -0400, Owen Taylor wrote:
>  You use it in to fix bus names in flatpak-evolution-wrapper.sh -
>  those could just be hardcoded since the bus name will be always the
>  same for the Evolution Flatpak 
It will work unless the version of the D-Bus service is bumped for
whatever reason. That's the only reason why the variables are part of
the .pc file, to have correct D-Bus service name provided, not hard-
coded.

In any case, I understand these are not obstacles, not the big one, if
the runtime change for the D-Bus service names will be possible. Feel
free to file an upstream bug against the evolution-data-server, thus
it's tracked (and not forgotten).

	Thanks and bye,
	Milan

[Zbigniew Jędrzejewski-Szmek]

On Wed, May 10, 2023 at 11:54:39AM +0100, Aoife Moloney wrote:
>  https://fedoraproject.org/wiki/Changes/FlatpaksWithoutModules 
>  For each Fedora release, we'll have two additional targets
>  
>   f39-flatpak-runtime (inherits f39), tags: f39-flatpak-runtime-build,
>  f39-flatpak-runtime
>   f39-app (inherits f39-flatpak-runtime), tags: f39-app-build, f39-app
>  
>  The f39-flatpak-runtime target has a limited use -
>  it's used to build packages that go into the Flatpak runtimes
>  (standard and KDE).
>  This would include flatpak-runtime-config, flatpak-rpm-macros,
>  but also rebuilds of standard packages if needed to prune dependencies:
>  currently this is done for gstreamer-plugins-good.
>  
>  The f39-app target is the main target that is used for prefix=/app rebuilds.
>  flatpak-rpm-macros is part of the build group for this target,
>  resulting in builds that target /app.
>  As currently, spec files can be conditionalized with '%if 0%{?flatpak}'
>  
>  Builds in f39-flatpak-runtime and f39-app have the dist tags
>  .f39runtime and .f39app respectively.
>  This is necessary so that we can automatically rebuild a package
>  without having to bump the release in the specfile.
>  (Koji requires everything to have a unique NVR.)
>  It also allows humans and tools to easily distinguish these specialized builds. 
I love how simple this is. Mostly just using koji as it's supposed to be used.

Zbyszek

[Florian Weimer]

* Aoife Moloney:

>  There are two types of Flatpak containers -
>  runtimes - which contain unmodified Fedora packages,
>  and applications - which contain Fedora packages rebuilt to relocate
>  them from /usr to /app. 
Is this relocation still required?

Thanks,
Florian

[Owen Taylor]

On Wed, May 10, 2023 at 9:48 AM Florian Weimer <fweimer(a)redhat.com&gt; wrote:

>  * Aoife Moloney:
>
>  > There are two types of Flatpak containers -
>  > runtimes - which contain unmodified Fedora packages,
>  > and applications - which contain Fedora packages rebuilt to relocate
>  > them from /usr to /app.
>
>  Is this relocation still required?
>  
Yes - the /app vs. /usr split is pretty fundamental to the operation of
Flatpak, and this Change doesn't propose any changes to how Flatpak works,
just how Flatpaks are built in Fedora.

- Owen

[Michael J Gruber]

Makes a lot of sense and improves the information value of the version greatly.

I'm just wondering whether the name suffix is the right place to put
"flatpak" as if it were a subpackage - can we use the dist tag instead, say
`fi39` for "fedora immutable based on fedora 39" or something like that? Or
`fp39` for `fedora flatpak ...`?

[Owen Taylor]

>  I'm just wondering whether the name suffix is the right place to
> put 
"flatpak" as if it were a subpackage - can we use the dist tag
instead, say
`fi39` for "fedora immutable based on fedora 39" or something like that? Or
`fp39` for `fedora flatpak ...`?

The main difference is what happens when you type firefox into koji - do
you see

  firefox-112.0.2-1.fc38     [the RPM]
  firefox-112.0.2-1.flatpak     [the Flatpak]

Or do you just see the RPM build  and you have to type firefox-flatpak to
see the container build? Which one is clearer? Which one is most useful? I
don't honestly have a big opinion here. I'll bring this and some other
naming issues up in the Fedora Flatpaks SIG next Monday.

- Owen


On Thu, May 11, 2023 at 7:44 AM Michael J Gruber <mjg(a)fedoraproject.org&gt;
wrote:

> Makes a lot of sense and improves the information value of the version
> greatly.
>
>  I'm just wondering whether the name suffix is the right place to
> put 
> "flatpak" as if it were a subpackage - can we use the dist tag
instead, say
> `fi39` for "fedora immutable based on fedora 39" or something like that?
Or
> `fp39` for `fedora flatpak ...`?
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue
>

[Florian Weimer]

* Owen Taylor:

>  On Wed, May 10, 2023 at 9:48 AM Florian Weimer
> <fweimer(a)redhat.com&gt; wrote:
>
>   * Aoife Moloney:
>
>   > There are two types of Flatpak containers -
>   > runtimes - which contain unmodified Fedora packages,
>   > and applications - which contain Fedora packages rebuilt to relocate
>   > them from /usr to /app.
>
>   Is this relocation still required?
>
>  Yes - the /app vs. /usr split is pretty fundamental to the operation
>  of Flatpak, and this Change doesn't propose any changes to how Flatpak
>  works, just how Flatpaks are built in Fedora. 
Do you have a pointer where I can read up on the current approach?

I'm concerned that this is just like Software Collections with a
different path, which I believe where previously banned in Fedora (and
are generally a huge hassle on the releng side, I think).

Thanks,
Florian

[Owen Taylor]

On Fri, May 12, 2023 at 1:34 PM Florian Weimer <fweimer(a)redhat.com&gt; wrote:

>  * Owen Taylor:
>
>  > On Wed, May 10, 2023 at 9:48 AM Florian Weimer <fweimer(a)redhat.com&gt;
>  wrote:
>  >
>  >  * Aoife Moloney:
>  >
>  >  > There are two types of Flatpak containers -
>  >  > runtimes - which contain unmodified Fedora packages,
>  >  > and applications - which contain Fedora packages rebuilt to relocate
>  >  > them from /usr to /app.
>  >
>  >  Is this relocation still required?
>  >
>  > Yes - the /app vs. /usr split is pretty fundamental to the operation
>  > of Flatpak, and this Change doesn't propose any changes to how Flatpak
>  > works, just how Flatpaks are built in Fedora.
>
>  Do you have a pointer where I can read up on the current approach?
>  
Some references that I found quickly:

https://docs.fedoraproject.org/en-US/flatpak/concepts/
https://docs.flatpak.org/en/latest/basic-concepts.html

But neither goes into a lot of detail to motivation. I'm certainly willing
to describe things in more detail or look around further if you still have
questions.


>  I'm concerned that this is just like Software Collections with a
>  different path, which I believe where previously banned in Fedora (and
>  are generally a huge hassle on the releng side, I think). 

The big distinction here is that there is no idea that these RPMs will be
installed directly on a Fedora system. They are used only when building
Flatpak containers. So I don't think there is much similarity to SCL's
other than "built with a different prefix".

- Owen

[Florian Weimer]

* Owen Taylor:

>  On Fri, May 12, 2023 at 1:34 PM Florian Weimer
> <fweimer(a)redhat.com&gt; wrote:
>
>   * Owen Taylor:
>
>   > On Wed, May 10, 2023 at 9:48 AM Florian Weimer <fweimer(a)redhat.com&gt; wrote:
>   >
>   >  * Aoife Moloney:
>   >
>   >  > There are two types of Flatpak containers -
>   >  > runtimes - which contain unmodified Fedora packages,
>   >  > and applications - which contain Fedora packages rebuilt to relocate
>   >  > them from /usr to /app.
>   >
>   >  Is this relocation still required?
>   >
>   > Yes - the /app vs. /usr split is pretty fundamental to the operation
>   > of Flatpak, and this Change doesn't propose any changes to how Flatpak
>   > works, just how Flatpaks are built in Fedora.
>
>   Do you have a pointer where I can read up on the current approach?
>
>  Some references that I found quickly:
>
>  https://docs.fedoraproject.org/en-US/flatpak/concepts/
>  https://docs.flatpak.org/en/latest/basic-concepts.html
>
>  But neither goes into a lot of detail to motivation. I'm certainly
>  willing to describe things in more detail or look around further if
>  you still have questions. 
As far as I can see, it doesn't explain why we need rebuilds.

Toolbx seems to be able achieve something similar without rebuilds.

Thanks,
Florian

[Owen Taylor]

On Thu, May 25, 2023 at 4:02 AM Florian Weimer <fweimer(a)redhat.com&gt; wrote:

>  * Owen Taylor:
>
>  > On Fri, May 12, 2023 at 1:34 PM Florian Weimer <fweimer(a)redhat.com&gt;
>  wrote:
>
>  >  Do you have a pointer where I can read up on the current approach?
>  >
>  > Some references that I found quickly:
>  >
>  > https://docs.fedoraproject.org/en-US/flatpak/concepts/
>  > https://docs.flatpak.org/en/latest/basic-concepts.html
>  >
>  > But neither goes into a lot of detail to motivation. I'm certainly
>  > willing to describe things in more detail or look around further if
>  > you still have questions.
>
>  As far as I can see, it doesn't explain why we need rebuilds.
>
>  Toolbx seems to be able achieve something similar without rebuilds. 

 In the docker/oci ecosystem, layers are explicitly dependent on the exact
underlying layers. If I modify a base image to upgrade one library, I have
to rebuild images that depend on it. Similarly, once you've created a
Toolbx container based on a  verison of the image, you can't replace the
image without recreating the container and losing local modifications.

If Flatpak was based on docker/oci, then every time a single library in the
runtime was modified, all 2000+ apps on Flathub would need to be rebuilt
and the upgraded versions downloaded by every single user. In the ideal
case, the upgraded application layers delta-compress well, but we're still
talking about a huge amount of wasted resources. It works better for
server-side containers because of datacenter-level networking and
resources, and because server-side container images are often simpler and
are based on simpler base images.

So, we need to preserve the ability to separately upgrade the application
and the runtime. This is currently achieved in Flatpak by the "low-tech"
way that the runtime is in /usr and the application in /app and we mount
those together into final namespace. If we want to achieve the same thing
with everything in /usr, then we have a series of technical challenges to
overcome. Among others:

 * We need a way to merge filesystems (fuse-overlayfs, composefs, huge
quantities of bind mounts, hardlink trees...)
 * We need to rebuild files that can contain data from both parts
(/etc/ld.so.conf, /usr/lib/fontconfig/cache, etc.)
 * We have to handle the case where the application layer needs to bundle a
newer/different version of a library than the runtme

These aren't impossible to solve, but to come up with solutions, implement
them in the Flatpak codebase, validate them with real applications, update
all the build infrastructure is a large (year or years scale) project.
Beyond that, you have to convince upstream Flatpak that this is worthwhile
- to add a whole second mode of operation for Flatpak when the current mode
of putting application code in /app largely works fine in the "normal" case
where the application is rebuilt from source code.

So, to recap, Flatpak separates out /usr and /app because that allows the
runtime and application to be separately upgraded. Modifying Flatpak to
solve that problem in a different way would be a technically difficult and
long process that, even if we had the resources to do it, would not solve
immediate problems with our dependency on modularity.

- Owen

[Petr Pisar]

V Thu, May 25, 2023 at 06:04:51AM -0400, Owen Taylor napsal(a):
>   * We need to rebuild files that can contain data from both parts
>  (/etc/ld.so.conf, /usr/lib/fontconfig/cache, etc.) 
How does splitting files from /usr to /app helps with this item?  Or is it
that application flatpaks are not supposed to deliver their own libraries and
fonts and if they do, they have to vendor them in a specific way (e.g.
storing a location of the application libraries into application executables
with RPATH)? 

-- Petr

[Owen Taylor]

On Thu, May 25, 2023 at 6:57 AM Petr Pisar <ppisar(a)redhat.com&gt; wrote:

>  V Thu, May 25, 2023 at 06:04:51AM -0400, Owen Taylor napsal(a):
>  >  * We need to rebuild files that can contain data from both parts
>  > (/etc/ld.so.conf, /usr/lib/fontconfig/cache, etc.)
>
>  How does splitting files from /usr to /app helps with this item?  Or is it
>  that application flatpaks are not supposed to deliver their own libraries
>  and
>  fonts and if they do, they have to vendor them in a specific way (e.g.
>  storing a location of the application libraries into application
>  executables
>  with RPATH)?
>  
Generally, if the runtime includes a library (like fontconfig) where it's
useful for an application to drop data into /app, the runtime is expected
to set the necessary configuration files, environment variables, or
whatever so that they are found:

For example, fontconfig is handled by:


https://src.fedoraproject.org/rpms/flatpak-runtime-config/blob/f38/f/50-f...

which is installed in all Fedora runtime images. Setting LD_LIBRARY_PATH is
handled by the Flatpak core.

- Owen

[Matthew Miller]

On Fri, May 12, 2023 at 07:33:56PM +0200, Florian Weimer wrote:
>  I'm concerned that this is just like Software Collections with a
>  different path, which I believe where previously banned in Fedora (and
>  are generally a huge hassle on the releng side, I think). 
The main sticking point with Software Collections was the complixity it
introduced into spec files -- lots of special macros that had to be used
just-so.

(And of course there were a lot of disagreements about naming things.)



-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Dan Čermák]

Hi,

Aoife Moloney <amoloney(a)redhat.com&gt; writes:

>  https://fedoraproject.org/wiki/Changes/FlatpaksWithoutModules
>  
snip
>  There is considerable implementation complexity within OSBS to implement this,
>  because the N/V/R need to be written into generated Dockerfile as
>  labels ''before'' building it,
>  but it should be manageable. If not, we'll need to switch to a different scheme.
>  (Running dnf with <code>--best</code> when building the container will
>  help make this reliable
>  and would be a general improvement.) 
Just out of interest, has this complicated part been already implemented
or is it still on the todo list? Would using a different container build
system reduce the complexity here?


Cheers,

Dan

[Owen Taylor]

On Thu, May 11, 2023 at 2:31 PM Dan Čermák <dan.cermak(a)cgc-instruments.com&gt;
wrote:

>  Hi,
>
>  Aoife Moloney <amoloney(a)redhat.com&gt; writes:
>
>  > https://fedoraproject.org/wiki/Changes/FlatpaksWithoutModules
>  >
>  snip
>  >
>  > There is considerable implementation complexity within OSBS to implement
>  this,
>  > because the N/V/R need to be written into generated Dockerfile as
>  > labels ''before'' building it,
>  > but it should be manageable. If not, we'll need to switch to a different
>  scheme.
>  > (Running dnf with <code>--best</code> when building the container will
>  > help make this reliable
>  > and would be a general improvement.)
>
>  Just out of interest, has this complicated part been already implemented
>  or is it still on the todo list? Would using a different container build
>  system reduce the complexity here?
>  
I have a fairly strong sense of what needs to be done, and have written
similar code before, but haven't actually written the patches for the
atomic-reactor component of OSBS.

Honestly, the "complexity" here is dwarfed both by the complexity of
maintaining OSBS in Fedora *and* by the complexity of switching to a
different container build system. I don't think it factors in to whether
Fedora switches and on what timescale.

- Owen

[Mattia Verga]

Il 10/05/23 12:54, Aoife Moloney ha scritto:
>  https://fedoraproject.org/wiki/Changes/FlatpaksWithoutModules
>
>  
I've never tried to make a flatpak because I was scared by the need of 
firstly build modules and I'm really happy to see this change moving on.

However, there's something I can't understand: AFAIK, a flatpak is 
platform independent, so a flatpak built on F39 can be installed on any 
Fedora version or even on other Linux distributions... right?
So, why having all those "Fedora Containers" releases in Bodhi which 
follow Fedora branches? Isn't just one Fedora Containers release enough? 
What happens if one builds the same flatpak on multiple Fedora 
Containers releases?

Mattia

[Owen Taylor]

On Fri, May 12, 2023, 1:03 PM Mattia Verga via devel <
devel(a)lists.fedoraproject.org&gt; wrote:

>  Il 10/05/23 12:54, Aoife Moloney ha scritto:
>  > https://fedoraproject.org/wiki/Changes/FlatpaksWithoutModules
>  >
>  >
>  I've never tried to make a flatpak because I was scared by the need of
>  firstly build modules and I'm really happy to see this change moving on.
>
>  However, there's something I can't understand: AFAIK, a flatpak is
>  platform independent, so a flatpak built on F39 can be installed on any
>  Fedora version or even on other Linux distributions... right?
>  So, why having all those "Fedora Containers" releases in Bodhi which
>  follow Fedora branches? Isn't just one Fedora Containers release enough?
>  What happens if one builds the same flatpak on multiple Fedora
>  Containers releases?
>  infrastructure/new_issue
>  <https://pagure.io/fedora-infrastructure/new_issue> 

The "F38 Flatpaks" release in Bodhi represents Flatpaks built with the F38
package set against the F8 runtime. But, yes, as you say we handle Flatpaks
as a single stream. Once we release Firefox into "F39 Flatpaks", everybody
on all releases gets that and we never do an update in "F38 Flatpaks" again.

If updates *do* get pushed on multiple releases, last pushed wins. Might be
useful if we found that we pushed something to early or broken - but isn't
normal.

But what if we had a single release instead?

On a technical level, we rely on separate releases because Bodhi is using
that to know what koji tag to pull from. So to merge them, we'd probably
need a single dest tag in Koji as well. But would it be more convenient and
less confusing for packagers and users? Would there be any performance or
UI problems from having a release in Bodhi that extends indefinitely?

In any case, a change to how we handle Flatpak releases in Koji could be
done separately from this change proposal :-)

- Owen

[Mattia Verga]

Il 13/05/23 03:04, Owen Taylor ha scritto:

>  On Fri, May 12, 2023, 1:03 PM Mattia Verga via devel
> <devel(a)lists.fedoraproject.org&gt; wrote:
>
> > Il 10/05/23 12:54, Aoife Moloney ha scritto:
> >> https://fedoraproject.org/wiki/Changes/FlatpaksWithoutModules
> >>
> >>
> > I've never tried to make a flatpak because I was scared by the need of
> > firstly build modules and I'm really happy to see this change moving on.
> >
> > However, there's something I can't understand: AFAIK, a flatpak is
> > platform independent, so a flatpak built on F39 can be installed on any
> > Fedora version or even on other Linux distributions... right?
> > So, why having all those "Fedora Containers" releases in Bodhi which
> > follow Fedora branches? Isn't just one Fedora Containers release enough?
> > What happens if one builds the same flatpak on multiple Fedora
> > Containers releases?
> > [infrastructure/new_issue](https://pagure.io/fedora-infrastructure/new_issue)
>
>  The "F38 Flatpaks" release in Bodhi represents Flatpaks built with the F38
> package set against the F8 runtime. But, yes, as you say we handle Flatpaks as a single
> stream. Once we release Firefox into "F39 Flatpaks", everybody on all releases
> gets that and we never do an update in "F38 Flatpaks" again.
>
>  If updates *do* get pushed on multiple releases, last pushed wins. Might be useful if we
> found that we pushed something to early or broken - but isn't normal. 
I was wondering if that was due to have the ability of building the flatpak against a
specific runtime. I mean, as I understand how a flatpak works, a flatpak created from a
koji build for f39 will be built against a specific runtime (say, KDE available in f39
stable tags), but when we build the same koji build for f40 we will be using a different
runtime which may not be compatible... so still having a Fedora Containers 39 release will
make the maintainer able to continue using the f39 runtime... I know I'm quite
confused, aren't I?

If I search for fedora-toolbox updates in Bodhi, I now see there are multiple updates
released alongside each others at the same time.

>  But what if we had a single release instead?
>
>  On a technical level, we rely on separate releases because Bodhi is using that to know
> what koji tag to pull from. So to merge them, we'd probably need a single dest tag in
> Koji as well. But would it be more convenient and less confusing for packagers and users?
> Would there be any performance or UI problems from having a release in Bodhi that extends
> indefinitely? 
Would be simpler to have a single Koji dest tag which doesn't change alongside Fedora
branches, so that packagers only have to remember that one? (provided that we don't
want the ability to build multiple updates as discussed above)

From a Bodhi POV, I don't think there would be any problem in extending a release
indefinitely... we currently have Fedora ELN running since 2020 and with something like
80K updates and nothing as blown up (yet...).

>  In any case, a change to how we handle Flatpak releases in Koji could
> be done separately from this change proposal :-)
>
>  - Owen

[Owen Taylor]

On Sat, May 13, 2023, 4:24 AM Mattia Verga via devel <
devel(a)lists.fedoraproject.org&gt; wrote:

>  Il 13/05/23 03:04, Owen Tay rt I look
>
>  On Fri, May 12, 2023, 1:03 PM Mattia Verga via devel <
>  devel(a)lists.fedoraproject.org&gt; wrote:
>
> > Il 10/05/23 12:54, Aoife Moloney ha scritto:
> > M kk kk kk kk o kk kk kk I I'll k>
> > https://fedoraproject.org/wiki/Changes/FlatpaksWithoutModules
> > >
> > >
> > I've never tried to make a flatpak because I was scared by the need of
> > firstly build modules and I'm really happy to see this change moving on.
> >
> > However, there's something I can't understand: AFAIK, a flatpak is
> > platform independent, so a flatpak built on F39 can be installed on any
> > Fedora version or even on other Linux distributions... right?
> > So, why having all those "Fedora Containers" releases in Bodhi which
> > follow Fedora branches? Isn't just one Fedora Containers release enough?
> > What happens if one builds the same flatpak on multiple Fedora
> > Containers releases?
> > infrastructure/new_issue
> > <https://pagure.io/fedora-infrastructure/new_issue>
>
>
>  The "F38 Flatpaks" release in Bodhi represents Flatpaks built with the F38
>  package set against the F8 runtime. But, yes, as you say we handle Flatpaks
>  as a single stream. Once we release Firefox into "F39 Flatpaks", everybody
>  on all releases gets that and we never do an update in "F38 Flatpaks" again.
>
>  If updates *do* get pushed on multiple releases, last pushed wins. Might
>  be useful if we found that we pushed something to early or broken - but
>  isn't normal.
>
>  I was wondering if that was due to have the ability of building the
>  flatpak against a specific runtime. I mean, as I understand how a flatpak
>  works, a flatpak created from a koji build for f39 will be built against a
>  specific runtime (say, KDE available in f39 stable tags), but when we build
>  the same koji build for f40 we will be using a different runtime which may
>  not be compatible... so still having a Fedora Containers 39 release will
>  make the maintainer able to continue using the f39 runtime... I know I'm
>  quite confused, aren't I?
>
>  If I search for fedora-toolbox updates in Bodhi, I now see there are
>  multiple updates released alongside each others at the same time
>  
I think some of the confusion comes from the fact that fedora-toolbox is
not a Flatpak at all and we have different bodhi releases and policies for
Flatpaks and other containers. We naturally do want parallel versions of
the toolbox container - you should be able to create a F37 Toolbox or a F38
Toolbox or ...

>  But what if we had a single release instead?
>
>  On a technical level, we rely on separate releases because Bodhi is using
>  that to know what koji tag to pull from. So to merge them, we'd probably
>  need a single dest tag in Koji as well. But would it be more convenient and
>  less confusing for packagers and users? Would there be any performance or
>  UI problems from having a release in Bodhi that extends indefinitely?
>
>  Would be simpler to have a single Koji dest tag which doesn't change
>  alongside Fedora branches, so that packagers only have to remember that
>  one? (provided that we don't want the ability to build multiple updates as
>  discussed above)
>  
fedpkg automatically determined the target based on the runtime specified
in container.yaml. We'd need separate targets in any case because the build
tag where the packages installed in the container come from needs to be
different based on the runtime version / fedora release. But the different
targets could potentially share a single dest tag.

One thing that would need to change in this model would be the name of the
runtime container in koji/bodhi. We'd probably need to have it be
flatpak-runtime-f38 or something like that, or it would be confusing to be
doing parallel updates into a single dest tag and bodhi release.

(The runtime needs security updates as long as we have any applications
still using it.)

>  From a Bodhi POV, I don't think there would be any problem in
> extending a
>  release indefinitely... we currently have Fedora ELN running since 2020 and
>  with something like 80K updates and nothing as blown up (yet...)
>  
Well, that answers that question :-)

I've created:
https://gitlab.com/fedora/sigs/flatpak/fedora-flatpaks/-/issues/15 to track
this idea, and will bring it up in the next Fedora Flatpaks SIG meeting.

Owen

[Matthew Miller]

On Fri, May 12, 2023 at 09:04:40PM -0400, Owen Taylor wrote:
>  > So, why having all those "Fedora Containers" releases
> in Bodhi which
>  > follow Fedora branches? Isn't just one Fedora Containers release enough?
[...]
>  The "F38 Flatpaks" release in Bodhi represents Flatpaks
> built with the F38
>  package set against the F8 runtime. But, yes, as you say we handle Flatpaks
>  as a single stream. Once we release Firefox into "F39 Flatpaks", everybody
>  on all releases gets that and we never do an update in "F38 Flatpaks" again.
>  
>  If updates *do* get pushed on multiple releases, last pushed wins. Might be
>  useful if we found that we pushed something to early or broken - but isn't
>  normal.
>  
>  But what if we had a single release instead? 
A few years ago, a design team contributor was working on an animated video
extolling our painless upgrade process. But, in the midst of that, they
updated to a new Fedora release, and discovered that Inkscape had dropped an
obscure export format which they happened to rely on.

I'd love it if we could produce both "fast" and "slow" streams.
Ideally
that's orthogonal to Fedora runtime release, but practically speaking that's
what we have now. Maybe in the future we could build a "slow" stream as part
of EPEL on a UBI or CentOS Stream runtime?



-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Petr Pisar]

V Wed, May 10, 2023 at 11:54:39AM +0100, Aoife Moloney napsal(a):
>  https://fedoraproject.org/wiki/Changes/FlatpaksWithoutModules
>   
[...]
>  === Rebuilding RPMs ===
>  For each Fedora release, we'll have two additional targets
>  
>   f39-flatpak-runtime (inherits f39), tags: f39-flatpak-runtime-build,
>  f39-flatpak-runtime
>   f39-app (inherits f39-flatpak-runtime), tags: f39-app-build, f39-app
>  
>  The f39-flatpak-runtime target has a limited use -
>  it's used to build packages that go into the Flatpak runtimes
>  (standard and KDE).
>  This would include flatpak-runtime-config, flatpak-rpm-macros,
>  but also rebuilds of standard packages if needed to prune dependencies:
>  currently this is done for gstreamer-plugins-good.
>  
>  The f39-app target is the main target that is used for prefix=/app rebuilds.
>  flatpak-rpm-macros is part of the build group for this target,
>  resulting in builds that target /app.
>  As currently, spec files can be conditionalized with '%if 0%{?flatpak}'
>   
[...]
>  Any Fedora packager can rebuild a package into f39-app using
> 'fepdkg
>  build --target=f39-app'.
>   
[...]
>  Once a package exists in f39-app or f38-app, then
>  [[https://gitlab.com/redhat/centos-stream/ci-cd/distrosync/distrobuildsync
>  distrobuildsync]] will be used to do a build into f39-app each time a
>  standard build completes.
>   
Do I understand correctly that flapaks containing the same package actually
reuses the same package build? E.g. if a flatpak for Firefox requires a nss
package and a flatpak for Thunderbird requires the same nss package, then nss
is built into f39-app only once and both Firefox and Thunderbird flapaks are
created from the same nss package build?

Because if there were a need for rebuilding the same package for different
flapaks differently, using a single Koji target would not not work. A Koji tag
only presents the latest build of a package.

-- Petr

[Owen Taylor]

On Wed, May 17, 2023 at 4:42 AM Petr Pisar <ppisar(a)redhat.com&gt; wrote:

>  [...]
>  > Any Fedora packager can rebuild a package into f39-app using 'fepdkg
>  > build --target=f39-app'.
>  >
>  [...]
>  > Once a package exists in f39-app or f38-app, then
>  > [[
>  https://gitlab.com/redhat/centos-stream/ci-cd/distrosync/distrobuildsync
>  > distrobuildsync]] will be used to do a build into f39-app each time a
>  > standard build completes.
>  >
>  Do I understand correctly that flapaks containing the same package actually
>  reuses the same package build? E.g. if a flatpak for Firefox requires a nss
>  package and a flatpak for Thunderbird requires the same nss package, then
>  nss
>  is built into f39-app only once and both Firefox and Thunderbird flapaks
>  are
>  created from the same nss package build?
>  
Yes, that's the idea. While we had the theoretical ability to build
libraries differently when bundling into different Flatpaks, in the vast
majority of cases, we did not use that capability.


>  Because if there were a need for rebuilding the same package for
> different
>  flapaks differently, using a single Koji target would not not work. A Koji
>  tag
>  only presents the latest build of a package.
>  
Yep. The side tag mechanism could possibly be used if a situation arose
where we *did* need different builds, but I don't see that need.

- Owen

[Aoife Moloney]

This proposal has now been submitted to FESCo for voting
https://pagure.io/fesco/issue/3005
_______________________________________________
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue