F39 proposal: Replace DNF with DNF5 (System-Wide Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/ReplaceDnfWithDnf5

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.

== Summary ==
Make DNF5 the new default packaging tool. The change will replace DNF,
LIBDNF, and DNF-AUTOMATIC with the new DNF5 and new Libdnf5 library.
It is a second step after
https://fedoraproject.org/wiki/Changes/MajorUpgradeOfMicrodnf.

== Owner ==
* Name: [[User:jmracek| Jaroslav Mracek]]
* Email: jmracek(a)redhat.com


== Detailed Description ==
The new DNF5 will provide a significant improvement in user
experiences and performance. The replacement is the second step in
upgrade of Fedora Software Management stack. Without the change there
will be multiple software management tool (DNF5, old Microdnf,
PackageKit, and DNF) based on different libraries (libdnf, libdnf5),
providing a different behavior, and not sharing a history. We can also
expect that DNF will have only limited support from upstream. The DNF5
development was announced on
[https://lists.fedoraproject.org/archives/list/[e-mail].o...
Fedora-Devel] list in 2020.

=== New DNF5 Features ===

* Fully featured package manager without requirement of Python
** Smaller system
** Faster
** Replace DNF and Microdnf

* Unified behavior of in the software management stack
** New Libdnf5 plugins (C++, Python) will be applicable to DNF5, Dnf5Daemon
*** DNF4 plugins were not applicable for PackageKit and Microdnf (e.g.
versionlock, subscription-manager), therefore PackageKit behaves
differently in comparison to DNF
** Shared configurations
*** In DNF4 not all configuration is honored by PackageKit and Microdnf
** DNF/YUM was developed for decades with impact of multiple styles
and naming conventions (options, configuration, options, commands)

* New Daemon
** The new daemon can provide an alternative to PackageKit for RPMs
(only one backend of PackageKit) if it will be integrated into Desktop
** Support of Modularity and Comps group
* Performance improvement
** Loading of repositories
** Advisory operations
** RPM queries
*** Name filters with a case-insensitive search (the `repoquery` command)
** Smart sharing of metadata between dnf5 and daemon
*** Reduce disk and downloads requirements
*** Currently, DNF, Microdnf, and PackageKit use their own cache
*** Optional, may be not available for Fedora 39

* Decrease of a maintenance cost in the long term
** Shared plugins
** Removal of functional duplicates

* Fully integrated Modularity in LIBDNF5 workflows
** The Modularity is supported in DNF and LIBDNF but it is not fully
integrated. Integration was not possible due to limitation of
compatibility with other tools (PackageKit)
** Fully integrated Modularity required changes in the library workflow

=== Major codebase improvements ===

*Reports in structure
** DNF reports a lot of important information only in logs

* Removal of duplicated implementation
** LIBDNF evolved from LIBHIF (PackageKit library) and HAWKEY (DNF
library). The integration was never finished, therefore LIBDNF still
contains duplicated functionality.
** decrease of the code maintenance cost in future

* Unify Python bindings
** Formal Libdnf provides two types of Python bindings
*** CPython (hawkey)
*** SWIG (libdnf)
** Maintaining and communication between both bindings requires a lot
of resources
** Binding unification was not possible without breaking API compatibility

* SWIG bindings
** With SWIG we can generate additional bindings without spending huge resources
** Code in particular languages will be very similar to each other

* Separation of system state from history DB and `/etc/dnf/module.d`
** In dnf-4 the list of userinstalled packages and list of installed
groups along with the lists of packages installed from them is
computed as an aggregation of transaction history. In dnf5 it will be
stored separately, having multiple benefits, among them that the
history database will serve for informational purposes only and will
not define the state of the system (it gets corrupted occasionally
etc.).
** Data stored in `/etc/dnf/module.d` were not supposed to be user
modifiable and their format is not sufficient (missing information
about installed packages with installed profiles)
*** Content of `/etc/dnf/module.d` will be moved into the System State

== Feedback ==


== Benefit to Fedora ==


== Scope ==
* Proposal owners:

DNF5 is still in the development and some of the features or options
are not yet available. We still have to finish the implementation of
Modularity, storing internal data related to History and System State,
and also documentation and man pages. DNF5 can be tested from
repository with upstream nightly builds -
https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/dnf5-unstable/.
The project's github repository is here -
https://github.com/rpm-software-management/dnf5/

* Other developers:
* Release engineering: [https://pagure.io/releng/issues #Releng issue number]

* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives:


== Upgrade/compatibility impact ==
The new DNF5 will obsolete `dnf`, `yum`, `dnf-automatic`, `yum-utils`,
and DNF plugins (core and extras). python3-dnf and LIBDNF (`libdnf`,
`python3-hawkey`) will be obsoleted by `fedora-obsolete-packages`.


=== Compatibility ===
The new DNF5 will provide a symlink to `/usr/bin/dnf` therefore users
will see the replacement as an upgrade of DNF with limited but
documented syntax changes. The DNF5 will provide some compatible
aliases of commands and options to improve adoption of the DNF5.

== How To Test ==

Install `dnf5` package from
https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/dnf5-unstable/


== User Experience ==

* Improved progress bars
* Improved transaction table
* Transaction progress reports including scriptlets reports
* Support of local rpm for transaction operation
* Great bash completion (better then DNF has)
* New commands and options that are only available with `DNF`


== Dependencies ==
There is a long list of dependent packages

=== dnf ===

 auter
 calamares
 copr-builder
 cpanspec
 dnf-plugin-diff
 dnfdragora
 etckeeper-dnf
 fedora-review
 fedora-upgrade
 kiwi-systemdeps-core
 libdnf-plugin-subscription-manager
 lpf
 mock
 osbuild
 perl-CPAN-Plugin-Sysdeps
 policycoreutils-devel
 rbm
 subscription-manager
 supermin
 system-config-language

=== python3-dnf ===

 anaconda-core
 dnf-plugin-ovl
 dnfdaemon
 fedora-easy-karma
 fedora-review
 lorax
 mock-core-configs
 module-build-service
 modulemd-tools
 needrestart
 pungi
 python3-bodhi-client
 python3-dnf-plugin-cow
 python3-dnf-plugin-flunk_dependent_remove
 python3-imgcreate
 python3-libreport
 retrace-server
 system-config-language

=== libdnf ===

 PackageKit
 copr-builder
 gnome-software-rpm-ostree
 libdnf-plugin-subscription-manager
 libdnf-plugin-swidtags
 libdnf-plugin-txnupd

=== python3-hawkey ===

 mock-core-configs
 modulemd-tools
 python3-rpmdeplint
 retrace-server


== Contingency Plan ==
* Contingency mechanism: (What to do?  Who will do it?)
* Contingency deadline:
* Blocks release?


== Documentation ==
none

== Release Notes ==



-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

[Vitaly Zaitsev]

On 06/09/2022 20:28, Ben Cotton wrote:
>  The new DNF5 will provide a symlink to `/usr/bin/dnf` therefore
> users
>  will see the replacement as an upgrade of DNF with limited but
>  documented syntax changes. The DNF5 will provide some compatible
>  aliases of commands and options to improve adoption of the DNF5. 
Why not just update existing dnf packages to version 5?

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Jaroslav Mracek]

DNF5 is completely a different component. It does not depend like microdnf on Python. DNF
plugins are not compatible with DNF5. There will be changes in commands, options, outputs
and so on therefore selling it as an update will be quite confusing.

[Milan Crha]

On Thu, 2022-09-08 at 12:20 +0200, Jaroslav Mracek wrote:
>  DNF5 is completely a different component. It does not depend like
>  microdnf on Python. DNF plugins are not compatible with DNF5. There
>  will be changes in commands, options, outputs and so on therefore
>  selling it as an update will be quite confusing. 
	Hi,
the current COPR builds for early testers conflict on the libdnf5-devel
package:

  - package libdnf5-devel-5.0.0-20220826011404.2.g05e0c2c0.fc37.x86_64
conflicts with libdnf-devel < 5 provided by libdnf-devel-0.68.0-
1.fc37.x86_64

Is that going to be solved anyhow, or it's expected? The
library/executable files can be installed in parallel, it seems, or at
least those I tried, but having the devel files for both is not
possible at the moment. It would be useful to not conflict the files,
thus one can build on one machine for both versions, not replace the
devel files constantly, if the two are supposed to live together for
some time.
	Bye,
	Milan

[Nico Kadel-Garcia]

On Thu, Sep 8, 2022 at 8:27 AM Milan Crha <mcrha(a)redhat.com&gt; wrote:
>  On Thu, 2022-09-08 at 12:20 +0200, Jaroslav Mracek wrote:
>  > DNF5 is completely a different component. It does not depend like
>  > microdnf on Python. DNF plugins are not compatible with DNF5. There
>  > will be changes in commands, options, outputs and so on therefore
>  > selling it as an update will be quite confusing.
>
>          Hi,
>  the current COPR builds for early testers conflict on the libdnf5-devel
>  package:
>
>    - package libdnf5-devel-5.0.0-20220826011404.2.g05e0c2c0.fc37.x86_64
>  conflicts with libdnf-devel < 5 provided by libdnf-devel-0.68.0-
>  1.fc37.x86_64
>
>  Is that going to be solved anyhow, or it's expected? The
>  library/executable files can be installed in parallel, it seems, or at
>  least those I tried, but having the devel files for both is not
>  possible at the moment. It would be useful to not conflict the files,
>  thus one can build on one machine for both versions, not replace the
>  devel files constantly, if the two are supposed to live together for
>  some time.
>          Bye,
>          Milan 
And I'd like a pony. More seriously, installing parallel versions of
the same development tools overn conflict due to files in
/usr/include/ unless the original others thought to distribute the
software in major version specific subdirectories. Not many utilities
do that.

[Milan Crha]

On Thu, 2022-09-08 at 08:39 -0400, Nico Kadel-Garcia wrote:
>  installing parallel versions of the same development tools 
	Hi,
Jaroslav said they are not the same, hence I asked.
	Bye,
	Milan

[Adam Williamson]

On Tue, 2022-09-06 at 14:28 -0400, Ben Cotton wrote:
>  
>  == Scope ==
>  * Proposal owners:
>  
>  DNF5 is still in the development and some of the features or options
>  are not yet available. We still have to finish the implementation of
>  Modularity, storing internal data related to History and System State,
>  and also documentation and man pages. DNF5 can be tested from
>  repository with upstream nightly builds -
>  https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/dnf5-unstable/.
>  The project's github repository is here -
>  https://github.com/rpm-software-management/dnf5/
>  
>  * Other developers:
>  * Release engineering: [https://pagure.io/releng/issues #Releng issue number] 
[snip]

>  == Dependencies ==
>  There is a long list of dependent packages
>  
>  === dnf ===
>  
>   auter
>   calamares
>   copr-builder
>   cpanspec
>   dnf-plugin-diff
>   dnfdragora
>   etckeeper-dnf
>   fedora-review
>   fedora-upgrade
>   kiwi-systemdeps-core
>   libdnf-plugin-subscription-manager
>   lpf
>   mock
>   osbuild
>   perl-CPAN-Plugin-Sysdeps
>   policycoreutils-devel
>   rbm
>   subscription-manager
>   supermin
>   system-config-language
>  
>  === python3-dnf ===
>  
>   anaconda-core
>   dnf-plugin-ovl
>   dnfdaemon
>   fedora-easy-karma
>   fedora-review
>   lorax
>   mock-core-configs
>   module-build-service
>   modulemd-tools
>   needrestart
>   pungi
>   python3-bodhi-client
>   python3-dnf-plugin-cow
>   python3-dnf-plugin-flunk_dependent_remove
>   python3-imgcreate
>   python3-libreport
>   retrace-server
>   system-config-language
>  
>  === libdnf ===
>  
>   PackageKit
>   copr-builder
>   gnome-software-rpm-ostree
>   libdnf-plugin-subscription-manager
>   libdnf-plugin-swidtags
>   libdnf-plugin-txnupd
>  
>  === python3-hawkey ===
>  
>   mock-core-configs
>   modulemd-tools
>   python3-rpmdeplint
>   retrace-server 
So, I feel like it's an issue that we have this huge list of
dependencies of the current implementation, but up there under "Other
developers:" in the Scope section, there is nothing.

The list of dependencies implies a fairly considerable amount of work
is going to be necessary on the part of "other developers" to adapt all
of these dependencies to dnf5. A lot of those dependencies are very
important - they are core components of how we work on, build, and use
Fedora.

I think this Change needs explicit buy-in from the maintainers of
dependent packages, and a clear plan and timeline for how those
packages will be ported to dnf5 such that the Change can land smoothly
in F39 timeframe.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Jaroslav Mracek]

It is true, we need a cooperation with maintainers of many components, but not everything
on the list is critical for transition or proposal itself. First of all we are not going
to remove old DNF from the distribution. Then only DNF and DNF5 directly conflict
therefore there is some room for an extension of transition period, but it depends on
particular component. Components that installs content on the system has higher priority
for transition then other use-cases with DNF.

[Maxwell G]

On Thursday, September 8, 2022 Jaroslav Mracek wrote:
>  First of all we are not going to remove old DNF from the
> distribution. 
Isn't that what

>  The new DNF5 will obsolete `dnf`, `yum`, `dnf-automatic`,
> `yum-utils`,
>  and DNF plugins (core and extras). python3-dnf and LIBDNF (`libdnf`,
>  `python3-hawkey`) will be obsoleted by `fedora-obsolete-packages`. 
says?

-- 
Maxwell G (@gotmax23)
Pronouns: He/Him/His

[Maxwell G]

Sep 8, 2022 8:45:19 AM Maxwell G via devel 
<devel(a)lists.fedoraproject.org&gt;:

>  On Thursday, September 8, 2022 Jaroslav Mracek wrote:
> > First of all we are not going to remove old DNF from the distribution.
>
>  Isn't that what
>
> > The new DNF5 will obsolete `dnf`, `yum`, `dnf-automatic`, `yum-utils`,
> > and DNF plugins (core and extras). python3-dnf and LIBDNF (`libdnf`,
> > `python3-hawkey`) will be obsoleted by `fedora-obsolete-packages`.
>
>  says?
>  
Sorry, I was unclear. I meant to say that those two statements appear 
contradictory.
--
Best,

Maxwell G (@gotmax23)
Pronouns: He/Him/His

[Richard W.M. Jones]

On Tue, Sep 06, 2022 at 02:28:41PM -0400, Ben Cotton wrote:
>   supermin 
Supermin requires only that:

  dnf download --destdir=<DIR> [list of pkgs] [-c configfile]

works *as non-root* (or some equivalent of that command as non-root)
to download the RPMs.  Does dnf5 support that?  Note it's especially
important that it does not require special privileges.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Jaroslav Mracek]

Such a user-case should be covered by DNF5 compatibility with DNF and maybe there will be
not a requirement for any change in your code. There are changes in DNF5 but only when it
provides a benefit and resolve some issues.

Jaroslav

[Josh Boyer]

On Tue, Sep 6, 2022 at 2:29 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  https://fedoraproject.org/wiki/Changes/ReplaceDnfWithDnf5
>
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>
>  == Summary ==
>  Make DNF5 the new default packaging tool. The change will replace DNF,
>  LIBDNF, and DNF-AUTOMATIC with the new DNF5 and new Libdnf5 library.
>  It is a second step after
>  https://fedoraproject.org/wiki/Changes/MajorUpgradeOfMicrodnf.
>
>  == Owner ==
>  * Name: [[User:jmracek| Jaroslav Mracek]]
>  * Email: jmracek(a)redhat.com
>
>
>  == Detailed Description ==
>  The new DNF5 will provide a significant improvement in user
>  experiences and performance. The replacement is the second step in
>  upgrade of Fedora Software Management stack. Without the change there
>  will be multiple software management tool (DNF5, old Microdnf,
>  PackageKit, and DNF) based on different libraries (libdnf, libdnf5),
>  providing a different behavior, and not sharing a history. We can also
>  expect that DNF will have only limited support from upstream. The DNF5
>  development was announced on
>
> [https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  Fedora-Devel] list in 2020.
>
>  === New DNF5 Features ===
>
>  * Fully featured package manager without requirement of Python
>  ** Smaller system
>  ** Faster
>  ** Replace DNF and Microdnf
>
>  * Unified behavior of in the software management stack
>  ** New Libdnf5 plugins (C++, Python) will be applicable to DNF5, Dnf5Daemon
>  *** DNF4 plugins were not applicable for PackageKit and Microdnf (e.g.
>  versionlock, subscription-manager), therefore PackageKit behaves
>  differently in comparison to DNF 
Is there any analysis on how many yum3/dnf4 plugins exist outside of
the core set that the DNF team maintains?  I'm curious how much of a
porting effort is required to move from yum3/dnf4 for plugin authors.

Will there be documentation and best practices on migration for plugin
maintainers?

>  ** Shared configurations
>  *** In DNF4 not all configuration is honored by PackageKit and Microdnf
>  ** DNF/YUM was developed for decades with impact of multiple styles
>  and naming conventions (options, configuration, options, commands)
>
>  * New Daemon
>  ** The new daemon can provide an alternative to PackageKit for RPMs
>  (only one backend of PackageKit) if it will be integrated into Desktop 
Is this daemon optional depending on installation type, or will it be
running on all installations?  I am assuming it is optional and
centered mostly around whatever currently needs PackageKit.

>  ** Support of Modularity and Comps group
>  * Performance improvement
>  ** Loading of repositories
>  ** Advisory operations
>  ** RPM queries
>  *** Name filters with a case-insensitive search (the `repoquery` command)
>  ** Smart sharing of metadata between dnf5 and daemon
>  *** Reduce disk and downloads requirements
>  *** Currently, DNF, Microdnf, and PackageKit use their own cache
>  *** Optional, may be not available for Fedora 39
>
>  * Decrease of a maintenance cost in the long term
>  ** Shared plugins
>  ** Removal of functional duplicates
>
>  * Fully integrated Modularity in LIBDNF5 workflows
>  ** The Modularity is supported in DNF and LIBDNF but it is not fully
>  integrated. Integration was not possible due to limitation of
>  compatibility with other tools (PackageKit)
>  ** Fully integrated Modularity required changes in the library workflow
>
>  === Major codebase improvements ===
>
>  *Reports in structure
>  ** DNF reports a lot of important information only in logs
>
>  * Removal of duplicated implementation
>  ** LIBDNF evolved from LIBHIF (PackageKit library) and HAWKEY (DNF
>  library). The integration was never finished, therefore LIBDNF still
>  contains duplicated functionality.
>  ** decrease of the code maintenance cost in future 
Do we have some statistics on the consolidated installation size of
DNF5 vs. dnf4?

Any performance comparisons?

josh

> * Unify Python bindings
> ** Formal Libdnf provides two types of Python bindings
> *** CPython (hawkey)
> *** SWIG (libdnf)
> ** Maintaining and communication between both bindings requires a lot
> of resources
> ** Binding unification was not possible without breaking API compatibility
>
> * SWIG bindings
> ** With SWIG we can generate additional bindings without spending huge resources
> ** Code in particular languages will be very similar to each other
>
> * Separation of system state from history DB and `/etc/dnf/module.d`
> ** In dnf-4 the list of userinstalled packages and list of installed
> groups along with the lists of packages installed from them is
> computed as an aggregation of transaction history. In dnf5 it will be
> stored separately, having multiple benefits, among them that the
> history database will serve for informational purposes only and will
> not define the state of the system (it gets corrupted occasionally
> etc.).
> ** Data stored in `/etc/dnf/module.d` were not supposed to be user
> modifiable and their format is not sufficient (missing information
> about installed packages with installed profiles)
> *** Content of `/etc/dnf/module.d` will be moved into the System State
>
> == Feedback ==
>
>
> == Benefit to Fedora ==
>
>
> == Scope ==
> * Proposal owners:
>
> DNF5 is still in the development and some of the features or options
> are not yet available. We still have to finish the implementation of
> Modularity, storing internal data related to History and System State,
> and also documentation and man pages. DNF5 can be tested from
> repository with upstream nightly builds -
> https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/dnf5-unstable/.
> The project's github repository is here -
> https://github.com/rpm-software-management/dnf5/
>
> * Other developers:
> * Release engineering: [https://pagure.io/releng/issues #Releng issue number]
>
> * Policies and guidelines: N/A (not needed for this Change)
> * Trademark approval: N/A (not needed for this Change)
> * Alignment with Objectives:
>
>
> == Upgrade/compatibility impact ==
> The new DNF5 will obsolete `dnf`, `yum`, `dnf-automatic`, `yum-utils`,
> and DNF plugins (core and extras). python3-dnf and LIBDNF (`libdnf`,
> `python3-hawkey`) will be obsoleted by `fedora-obsolete-packages`.
>
>
> === Compatibility ===
> The new DNF5 will provide a symlink to `/usr/bin/dnf` therefore users
> will see the replacement as an upgrade of DNF with limited but
> documented syntax changes. The DNF5 will provide some compatible
> aliases of commands and options to improve adoption of the DNF5.
>
> == How To Test ==
>
> Install `dnf5` package from
> https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/dnf5-unstable/
>
>
> == User Experience ==
>
> * Improved progress bars
> * Improved transaction table
> * Transaction progress reports including scriptlets reports
> * Support of local rpm for transaction operation
> * Great bash completion (better then DNF has)
> * New commands and options that are only available with `DNF`
>
>
> == Dependencies ==
> There is a long list of dependent packages
>
> === dnf ===
>
>  auter
>  calamares
>  copr-builder
>  cpanspec
>  dnf-plugin-diff
>  dnfdragora
>  etckeeper-dnf
>  fedora-review
>  fedora-upgrade
>  kiwi-systemdeps-core
>  libdnf-plugin-subscription-manager
>  lpf
>  mock
>  osbuild
>  perl-CPAN-Plugin-Sysdeps
>  policycoreutils-devel
>  rbm
>  subscription-manager
>  supermin
>  system-config-language
>
> === python3-dnf ===
>
>  anaconda-core
>  dnf-plugin-ovl
>  dnfdaemon
>  fedora-easy-karma
>  fedora-review
>  lorax
>  mock-core-configs
>  module-build-service
>  modulemd-tools
>  needrestart
>  pungi
>  python3-bodhi-client
>  python3-dnf-plugin-cow
>  python3-dnf-plugin-flunk_dependent_remove
>  python3-imgcreate
>  python3-libreport
>  retrace-server
>  system-config-language
>
> === libdnf ===
>
>  PackageKit
>  copr-builder
>  gnome-software-rpm-ostree
>  libdnf-plugin-subscription-manager
>  libdnf-plugin-swidtags
>  libdnf-plugin-txnupd
>
> === python3-hawkey ===
>
>  mock-core-configs
>  modulemd-tools
>  python3-rpmdeplint
>  retrace-server
>
>
> == Contingency Plan ==
> * Contingency mechanism: (What to do?  Who will do it?)
> * Contingency deadline:
> * Blocks release?
>
>
> == Documentation ==
> none
>
> == Release Notes ==
>
>
>
> --
> Ben Cotton
> He / Him / His
> Fedora Program Manager
> Red Hat
> TZ=America/Indiana/Indianapolis
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Neal Gompa]

On Wed, Sep 7, 2022 at 3:17 PM Josh Boyer <jwboyer(a)fedoraproject.org&gt; wrote:
>  On Tue, Sep 6, 2022 at 2:29 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  >
>  > https://fedoraproject.org/wiki/Changes/ReplaceDnfWithDnf5
>  >
>  > This document represents a proposed Change. As part of the Changes
>  > process, proposals are publicly announced in order to receive
>  > community feedback. This proposal will only be implemented if approved
>  > by the Fedora Engineering Steering Committee.
>  >
>  > == Summary ==
>  > Make DNF5 the new default packaging tool. The change will replace DNF,
>  > LIBDNF, and DNF-AUTOMATIC with the new DNF5 and new Libdnf5 library.
>  > It is a second step after
>  > https://fedoraproject.org/wiki/Changes/MajorUpgradeOfMicrodnf.
>  >
>  > == Owner ==
>  > * Name: [[User:jmracek| Jaroslav Mracek]]
>  > * Email: jmracek(a)redhat.com
>  >
>  >
>  > == Detailed Description ==
>  > The new DNF5 will provide a significant improvement in user
>  > experiences and performance. The replacement is the second step in
>  > upgrade of Fedora Software Management stack. Without the change there
>  > will be multiple software management tool (DNF5, old Microdnf,
>  > PackageKit, and DNF) based on different libraries (libdnf, libdnf5),
>  > providing a different behavior, and not sharing a history. We can also
>  > expect that DNF will have only limited support from upstream. The DNF5
>  > development was announced on
>  >
> [https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  > Fedora-Devel] list in 2020.
>  >
>  > === New DNF5 Features ===
>  >
>  > * Fully featured package manager without requirement of Python
>  > ** Smaller system
>  > ** Faster
>  > ** Replace DNF and Microdnf
>  >
>  > * Unified behavior of in the software management stack
>  > ** New Libdnf5 plugins (C++, Python) will be applicable to DNF5, Dnf5Daemon
>  > *** DNF4 plugins were not applicable for PackageKit and Microdnf (e.g.
>  > versionlock, subscription-manager), therefore PackageKit behaves
>  > differently in comparison to DNF
>
>  Is there any analysis on how many yum3/dnf4 plugins exist outside of
>  the core set that the DNF team maintains?  I'm curious how much of a
>  porting effort is required to move from yum3/dnf4 for plugin authors.
>
>  Will there be documentation and best practices on migration for plugin
>  maintainers?
>  
As a plugin author, I'd like something like this too...

>  > ** Shared configurations
>  > *** In DNF4 not all configuration is honored by PackageKit and Microdnf
>  > ** DNF/YUM was developed for decades with impact of multiple styles
>  > and naming conventions (options, configuration, options, commands)
>  >
>  > * New Daemon
>  > ** The new daemon can provide an alternative to PackageKit for RPMs
>  > (only one backend of PackageKit) if it will be integrated into Desktop
>
>  Is this daemon optional depending on installation type, or will it be
>  running on all installations?  I am assuming it is optional and
>  centered mostly around whatever currently needs PackageKit.
>  
Porting PackageKit mostly requires some API documentation and examples
for porting the existing DNF backend code to the new one. Some
assistance from the DNF team might be needed too.

We can ship a DNF5 backend alongside the DNF4 one in the upstream
sources and switch backends in Fedora as needed.




--
真実はいつも一つ！/ Always, there's only one truth!

[Jaroslav Mracek]

>  On Wed, Sep 7, 2022 at 3:17 PM Josh Boyer
> <jwboyer(a)fedoraproject.org&gt; wrote:
>  
>  As a plugin author, I'd like something like this too...
>  
>  
>  Porting PackageKit mostly requires some API documentation and examples
>  for porting the existing DNF backend code to the new one. Some
>  assistance from the DNF team might be needed too. 
We work on improving documentation. There ate already tutorials. 
Tutorials: https://libdnf.readthedocs.io/en/dnf-5-devel/tutorial/index.html - Recently we
rename the project to DNF5 therefore updated documentation is not yet available.

Jaroslav

> 
> We can ship a DNF5 backend alongside the DNF4 one in the upstream
> sources and switch backends in Fedora as needed.
> 
> 
> 
> 
> --
> 真実はいつも一つ！/ Always, there's only one truth!

[Jaroslav Mracek]

>  On Tue, Sep 6, 2022 at 2:29 PM Ben Cotton
> <bcotton(a)redhat.com&gt; wrote:
>  
>  Is there any analysis on how many yum3/dnf4 plugins exist outside of
>  the core set that the DNF team maintains?  I'm curious how much of a
>  porting effort is required to move from yum3/dnf4 for plugin authors. 
I think as an analyses can be count the list of dependencies mentioned in Dependencis
section. Specially I focused there packages not maintained by DNF team. Plugins that are
not in Fedora are difficult to track. The change announcement is also the way how to get
information about external dependencies.

>  
>  Will there be documentation and best practices on migration for plugin
>  maintainers? 
Yes, we will provide documentation and an example plugin. But we also are willing to help
with the transition.

>  
>  
>  Is this daemon optional depending on installation type, or will it be
>  running on all installations?  I am assuming it is optional and
>  centered mostly around whatever currently needs PackageKit. 
DNF5 runs without a daemon. There are user-cases where daemon cannot be used. 

>  
>  
>  Do we have some statistics on the consolidated installation size of
>  DNF5 vs. dnf4? 
Yes, 
DNF - download size 48 MB, install size 166 MB, 125 Packages
Microdnf - download size 34 MB, install size 115 MB, 101 Packages
Dnf5 with libmodulemd - download size 33 MB, install size 114 MB, 100 Packages

>  
>  Any performance comparisons? 
Yes, but take them as preliminary:
dnf repoquery $(rpm -qa) - 4.06s
dnf5 repoquery $(rpm -qa) - 1.42s
dnf upgrade $(rpm -qa) --assumeno - 15.77s
dnf5 upgrade $(rpm -qa) --assumeno - 2.57s

Jaroslav

> 
> josh
>

[Marius Schwarz]

Am 06.09.22 um 20:28 schrieb Ben Cotton:
>  * Removal of duplicated implementation
>  ** LIBDNF evolved from LIBHIF (PackageKit library) and HAWKEY (DNF
>  library). The integration was never finished, therefore LIBDNF still
>  contains duplicated functionality.
>  ** decrease of the code maintenance cost in future
>
>  * Unify Python bindings 
If it's still written in python, it will still be slow on devices like 
Pinephones. I was under the impression, that microdnf + libdnf was 
developed to counter this slowness?

best regards,
Marius Schwarz

[Fabio Alessandro Locati]

On Thu, Sep 8, 2022, at 10:13, Marius Schwarz wrote:
>  Am 06.09.22 um 20:28 schrieb Ben Cotton:
> > * Unify Python bindings
>
>  If it's still written in python, it will still be slow on devices like 
>  Pinephones. I was under the impression, that microdnf + libdnf was 
>  developed to counter this slowness? 
My understanding is that dnf is written 100% in Python [0], while libdnf is written in C
[1], dnf5 is written in C++ [2]. Though, to guarantee compatibility, dnf5 will have Python
bindings  to allow Python software to easily use dnf5 (as well as bindings for other
languages such as Perl and Ruby).

[0] https://github.com/rpm-software-management/dnf
[1] https://github.com/rpm-software-management/microdnf
[2] https://github.com/rpm-software-management/dnf5
-- 
Fabio Alessandro "Fale" Locati
fale.io

[Jaroslav Mracek]

Correct, DNF5 will provide a library - LIBDNF5 that will provide C++ API plus bindings to
various languages including Python.

Jaroslav

[Tommy Nguyen]

On Thu, 2022-09-08 at 10:13 +0200, Marius Schwarz wrote:
>  If it's still written in python, it will still be slow on
> devices
>  like 
>  Pinephones. I was under the impression, that microdnf + libdnf was 
>  developed to counter this slowness?
>  
>  best regards,
>  Marius Schwarz 
Though I don't know how it'll perform on an embedded device, I tested
it in toolbox and it's MUCH faster than dnf 4.

[Jaroslav Mracek]

Happy to hear that.

Jaroslav

[Jaroslav Mracek]

>  Am 06.09.22 um 20:28 schrieb Ben Cotton:
>  
>  If it's still written in python, it will still be slow on devices like 
>  Pinephones. I was under the impression, that microdnf + libdnf was 
>  developed to counter this slowness?
>  
>  best regards,
>  Marius Schwarz 
No, DNF5 is not written in Python. Performance improvements are not based on dropping
Python, but by better logic in code.

Jaroslav

[Otto Liljalaakso]

Ben Cotton kirjoitti 6.9.2022 klo 21.28:
>  == Summary ==
>  Make DNF5 the new default packaging tool. The change will replace DNF,
>  LIBDNF, and DNF-AUTOMATIC with the new DNF5 and new Libdnf5 library. 
>  == Upgrade/compatibility impact ==
>  The new DNF5 will obsolete `dnf`, `yum`, `dnf-automatic`, `yum-utils`,
>  and DNF plugins (core and extras).d by `fedora-obsolete-packages`. 
These two are the only mentions of dnf-automatic in this change. What 
does "replace" mean here? How does DNF5 cover dnf-automatic's use case?

[Jaroslav Mracek]

>  Ben Cotton kirjoitti 6.9.2022 klo 21.28:
>  
>  
>  These two are the only mentions of dnf-automatic in this change. What 
>  does "replace" mean here? How does DNF5 cover dnf-automatic's use case?
This is a good question. We will replace the functionality, but it is still on our TODO
list, therefore I cannot provide details right now.
Jaroslav Mracek

[Tommy Nguyen]

For those who are still not convinced, here is a comparison:

$ toolbox create -d fedora -r 37 && toolbox enter
$ sudo time dnf upgrade -y
26.79user 3.46system 0:49.09elapsed 61%CPU (0avgtext+0avgdata
489304maxresident)k
47400inputs+1243320outputs (266major+377843minor)pagefaults 0swaps

$ toolbox create -d fedora -r 37 && toolbox enter
$ sudo dnf copr enable rpmsoftwaremanagement/dnf5-unstable 
$ sudo dnf install dnf5 -y
$ sudo time dnf5 upgrade -y
11.55user 3.40system 0:33.98elapsed 44%CPU (0avgtext+0avgdata
199056maxresident)k
72inputs+1203072outputs (280major+191432minor)pagefaults 0swaps

DNF5 is ridiculously fast. The new text output using the C++ fmt
library is also a bonus.

[Kevin Kofler]

Tommy Nguyen wrote:
>  DNF5 is ridiculously fast. 
It is faster, but "ridiculously"? In the metric that matters (elapsed 
wallclock time), your benchmark shows the update taking 30% less time.

That said, there are other features of DNF5 (no more Python, shared cache 
between PackageKit and CLI) that are IMHO even more interesting than the raw 
speed gain.

        Kevin Kofler

[Tommy Nguyen]

On Wed, 2022-09-21 at 02:53 +0200, Kevin Kofler via devel wrote:
>  Tommy Nguyen wrote:
>  > DNF5 is ridiculously fast.
>  
>  It is faster, but "ridiculously"? In the metric that matters (elapsed
>  wallclock time), your benchmark shows the update taking 30% less
>  time.
>  
>  That said, there are other features of DNF5 (no more Python, shared
>  cache 
>  between PackageKit and CLI) that are IMHO even more interesting than
>  the raw 
>  speed gain. 
Would you have preferred if I used a different superlative? I still
consider 30% less to be significant, especially because DNF5 processes
metadata in parallel which makes things seem faster, whether perceived
or real. And yes, the other improvements are good as well, but end
users focus on speed. See: constant discussions about how to make DNF4
faster (including enabling fastestmirror which can make things slower)
or perception that it is slower because it processes metadata at a
different stage than apt for example.

[Sandro]

On 21-09-2022 03:11, Tommy Nguyen wrote:
>  On Wed, 2022-09-21 at 02:53 +0200, Kevin Kofler via devel wrote:
> > Tommy Nguyen wrote:
> >> DNF5 is ridiculously fast.
> > 
> > It is faster, but "ridiculously"? In the metric that matters
> > (elapsed wallclock time), your benchmark shows the update taking
> > 30% less time.
> > 
> > That said, there are other features of DNF5 (no more Python,
> > shared cache between PackageKit and CLI) that are IMHO even more
> > interesting than the raw speed gain.
>  
>  Would you have preferred if I used a different superlative? I still 
>  consider 30% less to be significant, especially because DNF5
>  processes metadata in parallel which makes things seem faster,
>  whether perceived or real. And yes, the other improvements are good
>  as well, but end users focus on speed. See: constant discussions
>  about how to make DNF4 faster (including enabling fastestmirror which
>  can make things slower) or perception that it is slower because it
>  processes metadata at a different stage than apt for example. 
As of recent, memory usage might be of a larger concern. Of which your
data shows a very welcome 40% reduction. I'll gladly take the speed
improvement as a bonus.

Having followed the discussion, the challenging part will be the
adaptation of all dependent packages to DNF5. I'll leave it to others to
coin terms and attributes for the new DNF5.

-- Sandro

[Sandro]

On 21-09-2022 09:33, Sandro wrote:
>  As of recent, memory usage might be of a larger concern. Of which
> your
>  data shows a very welcome 40% reduction. I'll gladly take the speed
>  improvement as a bonus. 
I wrote that before morning coffee. It's a 60% reduction. DNF5 is using 
only 40% of what DNF uses in Tommy's test case.

[Marius Schwarz]

Am 20.09.22 um 22:53 schrieb Tommy Nguyen:
>  DNF5 is ridiculously fast. The new text output using the C++ fmt
>  library is also a bonus. 
Yes, it's fast, which is a great improvement \o/ The new output format 
is ... debateable. It has advantages with "screen -x"-sessions with 
different resolutions,
but hardcore dnf fans will need to adapt to it. The old one was a bit 
simplier and had a more eased, not to say relaxed, flair ;)

best regards,
Marius Schwarz

DNF5 wants to replace regular rpms with modules

[Otto Liljalaakso]

Ben Cotton kirjoitti 6.9.2022 klo 21.28:
>  https://fedoraproject.org/wiki/Changes/ReplaceDnfWithDnf5
>  
>  == Summary ==
>  Make DNF5 the new default packaging tool. The change will replace DNF,
>  LIBDNF, and DNF-AUTOMATIC with the new DNF5 and new Libdnf5 library.
>  It is a second step after
>  https://fedoraproject.org/wiki/Changes/MajorUpgradeOfMicrodnf. 
>  == How To Test ==
>  
>  Install `dnf5` package from
>  https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/dnf5-unstable/ 
I did this, and tried using dnf5 instead of dnf. It look like dnf5 wants 
to switch a lot of packages I have to modules. It is my understanding 
that in Fedora, nothing should be installed from modules unless 
explictly requested. What is going on here?

Using postgresql as an example, I have it installed as regular rpm, not 
module:

$ LANG=C.UTF-8 dnf -C info --installed postgresql
Installed Packages
Name         : postgresql
Version      : 14.3
Release      : 8.fc37
Architecture : x86_64
Size         : 6.0 M
Source       : postgresql-14.3-8.fc37.src.rpm
Repository   : @System
 From repo    : fedora

But dnf5 wants to use a module instead:

$ LANG=C.UTF-8 sudo dnf5 upgrade --assumeno postgresql
Updating and loading repositories:
Repositories loaded.
Package                  Arch   Version              Repository
Upgrading:
  postgresql              x86_64 15beta2-1.module_f37 fedora-mod
   replacing postgresql   x86_64 14.3-8.fc37          <unknown>
  postgresql-private-libs x86_64 15beta2-1.module_f37 fedora-mod
   replacing postgresql-private-libs
                          x86_64 14.3-8.fc37          <unknown>
  postgresql-server       x86_64 15beta2-1.module_f37 fedora-mod
    replacing postgresql-server
                          x86_64 14.3-8.fc37          <unknown>
Installing dependencies:
  libicu69                x86_64 69.1-1.fc37          fedora

Transaction Summary:
  Installing:        1 packages
  Upgrading:         3 packages
  Replacing:         3 packages

Operation aborted by the user.

DNF5 wants to replace regular rpms with modules

[Petr Pisar]

As far as I know DNF5 has not yet fully support modules. Probably the
build of DNF5 you use still does not perform modular filtering. I
believe that should change before adding DNF5 into Fedora. As a
workaround, if you do not use modules, simply disable the modular
repositories.

On Thu, Sep 22, 2022 at 3:59 PM Otto Liljalaakso
<otto.liljalaakso(a)iki.fi&gt; wrote:
>  Ben Cotton kirjoitti 6.9.2022 klo 21.28:
>  > https://fedoraproject.org/wiki/Changes/ReplaceDnfWithDnf5
>  >
>  > == Summary ==
>  > Make DNF5 the new default packaging tool. The change will replace DNF,
>  > LIBDNF, and DNF-AUTOMATIC with the new DNF5 and new Libdnf5 library.
>  > It is a second step after
>  > https://fedoraproject.org/wiki/Changes/MajorUpgradeOfMicrodnf.
>
>  > == How To Test ==
>  >
>  > Install `dnf5` package from
>  > https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/dnf5-unstable/
>
>  I did this, and tried using dnf5 instead of dnf. It look like dnf5 wants
>  to switch a lot of packages I have to modules. It is my understanding
>  that in Fedora, nothing should be installed from modules unless
>  explictly requested. What is going on here?
>
>  Using postgresql as an example, I have it installed as regular rpm, not
>  module:
>
>  $ LANG=C.UTF-8 dnf -C info --installed postgresql
>  Installed Packages
>  Name         : postgresql
>  Version      : 14.3
>  Release      : 8.fc37
>  Architecture : x86_64
>  Size         : 6.0 M
>  Source       : postgresql-14.3-8.fc37.src.rpm
>  Repository   : @System
>   From repo    : fedora
>
>  But dnf5 wants to use a module instead:
>
>  $ LANG=C.UTF-8 sudo dnf5 upgrade --assumeno postgresql
>  Updating and loading repositories:
>  Repositories loaded.
>  Package                  Arch   Version              Repository
>  Upgrading:
>    postgresql              x86_64 15beta2-1.module_f37 fedora-mod
>     replacing postgresql   x86_64 14.3-8.fc37          <unknown>
>    postgresql-private-libs x86_64 15beta2-1.module_f37 fedora-mod
>     replacing postgresql-private-libs
>                            x86_64 14.3-8.fc37          <unknown>
>    postgresql-server       x86_64 15beta2-1.module_f37 fedora-mod
>      replacing postgresql-server
>                            x86_64 14.3-8.fc37          <unknown>
>  Installing dependencies:
>    libicu69                x86_64 69.1-1.fc37          fedora
>
>  Transaction Summary:
>    Installing:        1 packages
>    Upgrading:         3 packages
>    Replacing:         3 packages
>
>  Operation aborted by the user.
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

DNF5 wants to replace regular rpms with modules

[Jaroslav Mracek]

Correct, the modular filtering is not yet implemented and this is the last blocker for
rawhide release.

Jaroslav Mracek

DNF5 wants to replace regular rpms with modules

[Rahul Sundaram]

Hi

On Fri, Sep 23, 2022 at 9:50 AM Jaroslav Mracek wrote:

>  Correct, the modular filtering is not yet implemented and this is the
> last
>  blocker for rawhide release.
>  
Quick notes from what I am seeing with limiting testing using the copr repo

* Performance is much better
* Search seems busted, no output at all
* "update" seems to have been dropped (and
/usr/lib/dnf5/aliases.d/compatibility.conf didn't appear to be user
configurable). update should really be kept as a compatibility alias
* Install is overtly verbose

Rahul

[Maxwell G]

On Tue Sep 6, 2022, Ben Cotton wrote:
>  == Upgrade/compatibility impact ==
>  The new DNF5 will obsolete `dnf`, `yum`, `dnf-automatic`, `yum-utils`,
>  and DNF plugins (core and extras). python3-dnf and LIBDNF (`libdnf`,
>  `python3-hawkey`) will be obsoleted by `fedora-obsolete-packages`. 
I am worried about removing python3-dnf this early. As far as I can
tell, the new Python libdnf bindings are very much not a drop in
replacement. There are a fair amount of tools and scripts that depend on
python3-dnf. The change proposal listed some of them, but Ansible's dnf
module is notably missing. Personal user scripts (such as the one I use
for Go CVE rebuilds) and some of releng's scripts will also be broken. I
don't think we should proceed with removing python3-dnf until the
majority of the important dependent software is ported. (To be clear, I
don't consider my personal scripts to be important dependent software
:).

In general, I think the "If DNF5 will be not ready to replace DNF"
contingent needs to have definite criteria and the compatibility section
needs to be expanded. Of course FESCo can do as they see fit, but I
personally have qualms with approving a major change like this in
advanced of an actual stable release and major testing, especially
without a clear definition of when DNF5 is considered ready to replace
DNF4.

--
Best,

Maxwell G (@gotmax23)
Pronouns: He/Him/His