F38 Change: Major upgrade of Microdnf (Self-Contained Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/MajorUpgradeOfMicrodnf

== Summary ==
A major upgrade of Microdnf is the first step in the evolution of
package management in Fedora. The new microdnf has ambitions to
provide all major features of DNF without losing its minimal
footprint.

== Owner ==
* Name: [[User:jmracek| Jaroslav Mracek]]
* Email: [e-mail]

== Detailed Description ==
The new major Microdnf will provide huge improvements and in some
cases better behavior then DNF. In the future, the new Microdnf will
replace DNF. The new Microdnf will be accompanied by a new library
(`libdnf5`) and a new DNF Daemon.

=== MICRODNF features ===
* Improved user experience
** Improved progress bars
** Improved transaction table
** Transaction progress reports including scriptlets reports
** Support of local rpm for transaction operation
** Great bash completion (better then DNF has)

=== LIBDNF5 features ===

* Fully integrated Modularity in LIBDNF workflows
** Modularity is currently supported in DNF and LIBDNF, but it is not
fully integrated due to limitations in compatibility with other tools
(PackageKit)
** Fully integrated Modularity requires changes in library workflow
* Unified user interface
** DNF/YUM was developed for decades with the impact of multiple
styles and naming conventions (options, configuration options,
commands)
* Plugins
** DNF plugins are not applicable for PackageKit and Microdnf (e.g.
versionlock, subscription-manager), therefore PackageKit behaves
differently to DNF
* New plugins (C++, Python) will be available for all users
** Unified behavior
** Removal of functional duplicates
*** Decrease maintenance cost
* Shared configurations
** In DNF4 the configuration is only partially honored by PackageKit
and Microdnf
* New Daemon
** The new daemon can provide an alternative to PackageKit for RPMs
(only one backend of PackageKit) if it will be integrated into the
Desktop
* Additional improvements
** Reports in structure (API)
*** DNF reports a lot of important information only in logs
* Shared cache and improved cache handling (optional, not available in
Fedora 38)
** Microdnf, DNF4, and PackageKit use cached repositories on a
different location with different cache structure
* Performance improvement
** Loading of repositories
** Advisory operation
** RPM query
*** Name filters with a case-insensitive search
** Smart sharing of metadata between dnf, microdnf, daemon (optional,
not available in Fedora 38)
*** Reduce disk and downloads requirements
* Relocation of internal databases into `/usr`
** Make rollback more easy

=== Downsides ===
* Relocation of internal databases and different structure of internal databases
** The transaction performed by the new MICRODNF will be not visible by DNF
** The transaction performed by DNF or PackageKit will be not visible
by the new MICRODNF
** Packages installed by another packager will be handled as userinstalled
*** Consequence => The removal of a package will not trigger removal
of unused dependencies
* Compatibility
** To improve user experience and to unify dnf/microdnf behavior we
were unable to keep 100% compatibility with formal Microdnf in
command-line and in behavior

== Benefit to Fedora ==
The new MICRODNF significantly improves the user experience and in the
future it will provide all important features of DNF. It will also
keep all advantages of the original MICRODNF, like minimal size
required by containers.
The presence of MICRODNF, LIBDNF5, and DNFDAEMON in the distribution
will also allow for a smooth transition of components using `dnf`,
`python3-dnf`, `python3-hawkey`, `libdnf`, `dnfdragora`, and
`python3-dnfdaemon` to a new library.

* Improved progress bars
* Improved transaction table
* Transaction progress reports including scriptlets reports
* Support of local rpm for transaction operation
* Great bash completion (better then DNF has)
* `builddep` command without Python on the system

== Scope ==
* Proposal owners:
The new Microdnf is still in the development and some of the features
or options are not yet available. We still have to finish the
implementation of Modularity, storing internal data related to History
and System State, and also documentation and man pages. The new
Microdnf can be tested from repository with upstream nightly builds -
https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/dnf5-unstable/.
The project's github repository is here -
https://github.com/rpm-software-management/libdnf/tree/dnf-5-devel

* Other developers:

* Release engineering:
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives:

== Upgrade/compatibility impact ==

== How To Test ==

== User Experience ==
* Improved progress bars
* Improved transaction table
* Transaction progress reports including scriptlets reports
* Support of local rpm for transaction operation
* Great bash completion (better then DNF has)
* New commands and options that are only available with `DNF`

== Dependencies ==

== Contingency Plan ==
* Contingency mechanism: (What to do?  Who will do it?) N/A (not a
System Wide Change)
* Contingency deadline: N/A (not a System Wide Change)
* Blocks release? N/A (not a System Wide Change), Yes/No

== Documentation ==
N/A (not a System Wide Change)

== Release Notes ==

[Vít Ondruch]

Dne 12. 04. 22 v 22:54 Ben Cotton napsal(a):
> https://fedoraproject.org/wiki/Changes/MajorUpgradeOfMicrodnf
>
> == Summary ==
> A major upgrade of Microdnf is the first step in the evolution of
> package management in Fedora. The new microdnf has ambitions to
> provide all major features of DNF without losing its minimal
> footprint.
>
> == Owner ==
> Name: [[User:jmracek| Jaroslav Mracek]]
> Email: [e-mail]
>
> == Detailed Description ==
> The new major Microdnf will provide huge improvements and in some
> cases better behavior then DNF. In the future, the new Microdnf will
> replace DNF. The new Microdnf will be accompanied by a new library
> (`libdnf5`) and a new DNF Daemon.
>
> === MICRODNF features ===
> Improved user experience
>
> ** Improved progress bars
> ** Improved transaction table
> ** Transaction progress reports including scriptlets reports
> ** Support of local rpm for transaction operation
> ** Great bash completion (better then DNF has)
>
> === LIBDNF5 features ===
> Fully integrated Modularity in LIBDNF workflows
>
> ** Modularity is currently supported in DNF and LIBDNF, but it is not
> fully integrated due to limitations in compatibility with other tools
> (PackageKit)
> ** Fully integrated Modularity requires changes in library workflow
> Unified user interface
>
> ** DNF/YUM was developed for decades with the impact of multiple
> styles and naming conventions (options, configuration options,
> commands)
> Plugins
>
> ** DNF plugins are not applicable for PackageKit and Microdnf (e.g.
> versionlock, subscription-manager), therefore PackageKit behaves
> differently to DNF
> New plugins (C++, Python) will be available for all users
>
> ** Unified behavior
> ** Removal of functional duplicates
> *** Decrease maintenance cost
> Shared configurations
>
> ** In DNF4 the configuration is only partially honored by PackageKit
> and Microdnf
> New Daemon
>
> ** The new daemon can provide an alternative to PackageKit for RPMs
> (only one backend of PackageKit) if it will be integrated into the
> Desktop
> Additional improvements
>
> ** Reports in structure (API)
> *** DNF reports a lot of important information only in logs
> Shared cache and improved cache handling (optional, not available in
>
> Fedora 38)
> ** Microdnf, DNF4, and PackageKit use cached repositories on a
> different location with different cache structure
> Performance improvement
>
> ** Loading of repositories
> ** Advisory operation
> ** RPM query
> *** Name filters with a case-insensitive search
> ** Smart sharing of metadata between dnf, microdnf, daemon (optional,
> not available in Fedora 38)
> *** Reduce disk and downloads requirements
> Relocation of internal databases into `/usr`
>
> ** Make rollback more easy
>
> === Downsides ===
> Relocation of internal databases and different structure of internal databases
>
> ** The transaction performed by the new MICRODNF will be not visible by DNF
> ** The transaction performed by DNF or PackageKit will be not visible
> by the new MICRODNF

Can you please elaborate what is the migration path? What precisely will 
be using the MICRODNF/libdnf5 in F38. Will the DNF db be migrated? When 
the DNF will be dropped or the switch from DNF to MICRODNF will happen?

Vít
> ** Packages installed by another packager will be handled as userinstalled
> *** Consequence => The removal of a package will not trigger removal
> of unused dependencies
> Compatibility
>
> ** To improve user experience and to unify dnf/microdnf behavior we
> were unable to keep 100% compatibility with formal Microdnf in
> command-line and in behavior
>
> == Benefit to Fedora ==
> The new MICRODNF significantly improves the user experience and in the
> future it will provide all important features of DNF. It will also
> keep all advantages of the original MICRODNF, like minimal size
> required by containers.
> The presence of MICRODNF, LIBDNF5, and DNFDAEMON in the distribution
> will also allow for a smooth transition of components using `dnf`,
> `python3-dnf`, `python3-hawkey`, `libdnf`, `dnfdragora`, and
> `python3-dnfdaemon` to a new library.
> Improved progress bars
> Improved transaction table
> Transaction progress reports including scriptlets reports
> Support of local rpm for transaction operation
> Great bash completion (better then DNF has)
> `builddep` command without Python on the system
>
> == Scope ==
> Proposal owners:
>
> The new Microdnf is still in the development and some of the features
> or options are not yet available. We still have to finish the
> implementation of Modularity, storing internal data related to History
> and System State, and also documentation and man pages. The new
> Microdnf can be tested from repository with upstream nightly builds -
> https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/dnf5-unstable/.
> The project's github repository is here -
> https://github.com/rpm-software-management/libdnf/tree/dnf-5-devel
>
> Other developers:
>
> Release engineering:
>
> Policies and guidelines: N/A (not needed for this Change)
>
> Trademark approval: N/A (not needed for this Change)
>
> Alignment with Objectives:
>
> == Upgrade/compatibility impact ==
>
> == How To Test ==
>
> == User Experience ==
> Improved progress bars
> Improved transaction table
> Transaction progress reports including scriptlets reports
> Support of local rpm for transaction operation
> Great bash completion (better then DNF has)
> New commands and options that are only available with `DNF`
>
> == Dependencies ==
>
> == Contingency Plan ==
> Contingency mechanism: (What to do?  Who will do it?) N/A (not a
>
> System Wide Change)
> Contingency deadline: N/A (not a System Wide Change)
> Blocks release? N/A (not a System Wide Change), Yes/No
>
> == Documentation ==
> N/A (not a System Wide Change)
>
> == Release Notes ==

[Jaroslav Mracek]

On Wed, Apr 13, 2022 at 10:15 AM Vít Ondruch  wrote:
> Dne 12. 04. 22 v 22:54 Ben Cotton napsal(a):
> > https://fedoraproject.org/wiki/Changes/MajorUpgradeOfMicrodnf
> >
> > == Summary ==
> > A major upgrade of Microdnf is the first step in the evolution of
> > package management in Fedora. The new microdnf has ambitions to
> > provide all major features of DNF without losing its minimal
> > footprint.
> >
> > == Owner ==
> > Name: [[User:jmracek| Jaroslav Mracek]]
> > Email: [e-mail]
> >
> > == Detailed Description ==
> > The new major Microdnf will provide huge improvements and in some
> > cases better behavior then DNF. In the future, the new Microdnf will
> > replace DNF. The new Microdnf will be accompanied by a new library
> > (`libdnf5`) and a new DNF Daemon.
> >
> > === MICRODNF features ===
> > Improved user experience
> >
> > ** Improved progress bars
> > ** Improved transaction table
> > ** Transaction progress reports including scriptlets reports
> > ** Support of local rpm for transaction operation
> > ** Great bash completion (better then DNF has)
> >
> > === LIBDNF5 features ===
> > Fully integrated Modularity in LIBDNF workflows
> >
> > ** Modularity is currently supported in DNF and LIBDNF, but it is not
> > fully integrated due to limitations in compatibility with other tools
> > (PackageKit)
> > ** Fully integrated Modularity requires changes in library workflow
> > Unified user interface
> >
> > ** DNF/YUM was developed for decades with the impact of multiple
> > styles and naming conventions (options, configuration options,
> > commands)
> > Plugins
> >
> > ** DNF plugins are not applicable for PackageKit and Microdnf (e.g.
> > versionlock, subscription-manager), therefore PackageKit behaves
> > differently to DNF
> > New plugins (C++, Python) will be available for all users
> >
> > ** Unified behavior
> > ** Removal of functional duplicates
> > *** Decrease maintenance cost
> > Shared configurations
> >
> > ** In DNF4 the configuration is only partially honored by PackageKit
> > and Microdnf
> > New Daemon
> >
> > ** The new daemon can provide an alternative to PackageKit for RPMs
> > (only one backend of PackageKit) if it will be integrated into the
> > Desktop
> > Additional improvements
> >
> > ** Reports in structure (API)
> > *** DNF reports a lot of important information only in logs
> > Shared cache and improved cache handling (optional, not available in
> >
> > Fedora 38)
> > ** Microdnf, DNF4, and PackageKit use cached repositories on a
> > different location with different cache structure
> > Performance improvement
> >
> > ** Loading of repositories
> > ** Advisory operation
> > ** RPM query
> > *** Name filters with a case-insensitive search
> > ** Smart sharing of metadata between dnf, microdnf, daemon (optional,
> > not available in Fedora 38)
> > *** Reduce disk and downloads requirements
> > Relocation of internal databases into `/usr`
> >
> > ** Make rollback more easy
> >
> > === Downsides ===
> > Relocation of internal databases and different structure of internal
>
> databases
> > ** The transaction performed by the new MICRODNF will be not visible by
>
> DNF
> > ** The transaction performed by DNF or PackageKit will be not visible
> > by the new MICRODNF

Thank you very much for very good questions.
> Can you please elaborate what is the migration path?

The new Microdnf will obsolete the old microdnf or new microdnf will have
a higher version.

What precisely will
> be using the MICRODNF/libdnf5 in F38.

The new Microdnf will use a new library `libdnf5` that will be parallel
installable with `dnf` and `libdnf`
> Will the DNF db be migrated?

Right now I do not have the answer for that.
> When
> the DNF will be dropped or the switch from DNF to MICRODNF will happen?

I am going to create a System Wide Change for Fedora 39 to replace DNF with
the new Microdnf.

Jaroslav

Vít
> > ** Packages installed by another packager will be handled as
>
> userinstalled
> > *** Consequence => The removal of a package will not trigger removal
> > of unused dependencies
> > Compatibility
> >
> > ** To improve user experience and to unify dnf/microdnf behavior we
> > were unable to keep 100% compatibility with formal Microdnf in
> > command-line and in behavior
> >
> > == Benefit to Fedora ==
> > The new MICRODNF significantly improves the user experience and in the
> > future it will provide all important features of DNF. It will also
> > keep all advantages of the original MICRODNF, like minimal size
> > required by containers.
> > The presence of MICRODNF, LIBDNF5, and DNFDAEMON in the distribution
> > will also allow for a smooth transition of components using `dnf`,
> > `python3-dnf`, `python3-hawkey`, `libdnf`, `dnfdragora`, and
> > `python3-dnfdaemon` to a new library.
> > Improved progress bars
> > Improved transaction table
> > Transaction progress reports including scriptlets reports
> > Support of local rpm for transaction operation
> > Great bash completion (better then DNF has)
> > `builddep` command without Python on the system
> >
> > == Scope ==
> > Proposal owners:
> >
> > The new Microdnf is still in the development and some of the features
> > or options are not yet available. We still have to finish the
> > implementation of Modularity, storing internal data related to History
> > and System State, and also documentation and man pages. The new
> > Microdnf can be tested from repository with upstream nightly builds -
>
> https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/dnf5-unstable/
> .
> > The project's github repository is here -
> > https://github.com/rpm-software-management/libdnf/tree/dnf-5-devel
> >
> > Other developers:
> >
> > Release engineering:
> >
> > Policies and guidelines: N/A (not needed for this Change)
> >
> > Trademark approval: N/A (not needed for this Change)
> >
> > Alignment with Objectives:
> >
> > == Upgrade/compatibility impact ==
> >
> > == How To Test ==
> >
> > == User Experience ==
> > Improved progress bars
> > Improved transaction table
> > Transaction progress reports including scriptlets reports
> > Support of local rpm for transaction operation
> > Great bash completion (better then DNF has)
> > New commands and options that are only available with `DNF`
> >
> > == Dependencies ==
> >
> > == Contingency Plan ==
> > Contingency mechanism: (What to do?  Who will do it?) N/A (not a
> >
> > System Wide Change)
> > Contingency deadline: N/A (not a System Wide Change)
> > Blocks release? N/A (not a System Wide Change), Yes/No
> >
> > == Documentation ==
> > N/A (not a System Wide Change)
> >
> > == Release Notes ==
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Zbigniew Jędrzejewski-Szmek]

> https://fedoraproject.org/wiki/Changes/MajorUpgradeOfMicrodnf

Overall, the new architecture is a huge improvement and should fix many
of the long-standing issues.

Would it be possible to extend the How To Test section with installation
and commandline-use instructions? Is the stuff planned in the copr repo already
usable for early adopters?

On Wed, Apr 13, 2022 at 01:34:10PM +0200, Jaroslav Mracek wrote:
> On Wed, Apr 13, 2022 at 10:15 AM Vít Ondruch  wrote:
> > What precisely will be using the MICRODNF/libdnf5 in F38.
>
> The new Microdnf will use a new library `libdnf5` that will be parallel
> installable with `dnf` and `libdnf`
> > Will the DNF db be migrated?
>
> Right now I do not have the answer for that.

It would be nice to have *some* migration plan, even if the details
are left unspecified for now.

Obviously the best answer would be if the dnf history db is migrated.
But if it is only partially migrated, this would be acceptable too.
I.e. we could have a "flag day" where the user install dnf5 and uninstalls
dnf4 and this step is irrersible, and dnf5 cannot undo or report on
previous dnf transactions. The most important part seems to be the
coloring of packages for installation reason, and this should be migrateable.
If not, I think we should discuss what can be done.
> > When
> > the DNF will be dropped or the switch from DNF to MICRODNF will happen?
>
> I am going to create a System Wide Change for Fedora 39 to replace DNF with
> the new Microdnf.

Looking forward to that ;)

Zbyszek

[Mattia Verga]

Il 12/04/22 22:54, Ben Cotton ha scritto:
> https://fedoraproject.org/wiki/Changes/MajorUpgradeOfMicrodnf
>
> == Summary ==
> A major upgrade of Microdnf is the first step in the evolution of
> package management in Fedora. The new microdnf has ambitions to
> provide all major features of DNF without losing its minimal
> footprint.
>
> == Owner ==
> Name: [[User:jmracek| Jaroslav Mracek]]
> Email: [e-mail]
>
> == Detailed Description ==
> The new major Microdnf will provide huge improvements and in some
> cases better behavior then DNF. In the future, the new Microdnf will
> replace DNF. The new Microdnf will be accompanied by a new library
> (`libdnf5`) and a new DNF Daemon.

Gosh, this is all so confusing... so we currently have PackageKit, DNF
and microdnf on top of libdnf and we're going to have Microdnf (which is
not the same as microdnf) on top of libdnf5 (which is not the same as
libdnf)...? \o/

Mattia

[Neal Gompa]

On Wed, Apr 13, 2022 at 10:07 AM Mattia Verga via devel
 wrote:
> Il 12/04/22 22:54, Ben Cotton ha scritto:
> > https://fedoraproject.org/wiki/Changes/MajorUpgradeOfMicrodnf
> >
> > == Summary ==
> > A major upgrade of Microdnf is the first step in the evolution of
> > package management in Fedora. The new microdnf has ambitions to
> > provide all major features of DNF without losing its minimal
> > footprint.
> >
> > == Owner ==
> > Name: [[User:jmracek| Jaroslav Mracek]]
> > Email: [e-mail]
> >
> > == Detailed Description ==
> > The new major Microdnf will provide huge improvements and in some
> > cases better behavior then DNF. In the future, the new Microdnf will
> > replace DNF. The new Microdnf will be accompanied by a new library
> > (`libdnf5`) and a new DNF Daemon.
>
> Gosh, this is all so confusing... so we currently have PackageKit, DNF
> and microdnf on top of libdnf and we're going to have Microdnf (which is
> not the same as microdnf) on top of libdnf5 (which is not the same as
> libdnf)...? \o/

In the future, the plan is to consolidate everything into libdnf5
(dnf, microdnf, dnfdaemon, packagekit, etc.).

[Kevin Kofler]

Neal Gompa wrote:
> dnf, microdnf

Doesn't the change page say the Python DNF will go away?

I would welcome a pure C/C++ base system, with no language interpreters 
beyond bash.

Kevin Kofler

[Jaroslav Mracek]

On Wed, Apr 13, 2022 at 4:30 PM Kevin Kofler via devel <
[e-mail]> wrote:
> Neal Gompa wrote:
> > dnf, microdnf
>
> Doesn't the change page say the Python DNF will go away?

This question will be addressed in a separate change proposal. Fedora 39
can be taken as a primary target.

Jaroslav
> I would welcome a pure C/C++ base system, with no language interpreters
> beyond bash.
>     Kevin Kofler
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Demi Marie Obenour]

On 4/12/22 16:54, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/MajorUpgradeOfMicrodnf
>
> == Summary ==
> A major upgrade of Microdnf is the first step in the evolution of
> package management in Fedora. The new microdnf has ambitions to
> provide all major features of DNF without losing its minimal
> footprint.

Will this come with better support for repo_gpgcheck=1?

[Gordon Messmer]

On 4/12/22 13:54, Ben Cotton wrote:
> == Detailed Description ==
> The new major Microdnf will provide huge improvements and in some
> cases better behavior then DNF. In the future, the new Microdnf will
> replace DNF. The new Microdnf will be accompanied by a new library
> (`libdnf5`) and a new DNF Daemon.

I've gotta ask... How much memory does the new dnf daemon take while idle?

I know this comes up time to time... As it is, PackageKitd and 
gnome-software both, individually, take ~ 450MB of RAM without any user 
interaction (other than logging in to a desktop) on a F36 system I 
updated for general testing.  This situation isn't new; on a quick 
search, I see reports of this problem on F25 in the top results.

https://www.google.com/search?client=firefox-b-1-d&q=fedora+packagekitd+...

Using close to a gig of RAM for a couple of services that do nothing 
99.99% of the time isn't great, and I'm a little worried that another 
daemon is going to make things worse, especially until microdnf replaces 
dnf sometime "in the future."

[Kevin Kofler]

Gordon Messmer wrote:
> I've gotta ask... How much memory does the new dnf daemon take while idle?

As I understand it, the new DNF daemon would mostly only replace/upgrade the 
already existing dnfdaemon, for users of tools like Dnfdragora. It would not 
be required otherwise, or would it?

Kevin Kofler

[Neal Gompa]

On Thu, Apr 14, 2022 at 7:40 AM Kevin Kofler via devel
 wrote:
> Gordon Messmer wrote:
> > I've gotta ask... How much memory does the new dnf daemon take while idle?
>
> As I understand it, the new DNF daemon would mostly only replace/upgrade the
> already existing dnfdaemon, for users of tools like Dnfdragora. It would not
> be required otherwise, or would it?

Yes. Applications need to directly use the dnfdaemon API to use the dnfdaemon.

[Chris Snyder]

For dnf plugin writers, what will the migration path look like to switch
over to be compatible with microdnf?

On Thu, Apr 14, 2022 at 8:05 AM Neal Gompa  wrote:
> On Thu, Apr 14, 2022 at 7:40 AM Kevin Kofler via devel
>  wrote:
> > Gordon Messmer wrote:
> > > I've gotta ask... How much memory does the new dnf daemon take while
>
> idle?
> > As I understand it, the new DNF daemon would mostly only replace/upgrade
>
> the
> > already existing dnfdaemon, for users of tools like Dnfdragora. It would
>
> not
> > be required otherwise, or would it?
>
> Yes. Applications need to directly use the dnfdaemon API to use the
> dnfdaemon.
>
> --
> 真実はいつも一つ！/ Always, there's only one truth!
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Jaroslav Mracek]

On Thu, Apr 14, 2022 at 8:48 PM Chris Snyder  wrote:
> For dnf plugin writers, what will the migration path look like to switch
> over to be compatible with microdnf?

The new Plugins for LIBDNF5 or Microdnf will be required to use the new API
therefore it will be required to rewrite them. The DNF team will help with
the plugin transition to DNF5.

Jaroslav
> On Thu, Apr 14, 2022 at 8:05 AM Neal Gompa  wrote:
> > On Thu, Apr 14, 2022 at 7:40 AM Kevin Kofler via devel
> >  wrote:
> > > Gordon Messmer wrote:
> > > > I've gotta ask... How much memory does the new dnf daemon take while
> >
> > idle?
> > > As I understand it, the new DNF daemon would mostly only
> >
> > replace/upgrade the
> > > already existing dnfdaemon, for users of tools like Dnfdragora. It
> >
> > would not
> > > be required otherwise, or would it?
> >
> > Yes. Applications need to directly use the dnfdaemon API to use the
> > dnfdaemon.
> >
> > --
> > 真実はいつも一つ！/ Always, there's only one truth!
> > _______________________________________________
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedoraproject.org/archives/list/[e-mail]
> > Do not reply to spam on the list, report it:
> > https://pagure.io/fedora-infrastructure
>
> --
>
> Christopher Snyder, RHCSA
>
> he, him, his
>
> Associate Manager, Software Engineering
>
> Red Hat https://www.redhat.com
>
> IRC: csnyder
> https://www.redhat.com
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Jaroslav Mracek]

On Thu, Apr 14, 2022 at 8:30 AM Gordon Messmer 
wrote:
> On 4/12/22 13:54, Ben Cotton wrote:
> > == Detailed Description ==
> > The new major Microdnf will provide huge improvements and in some
> > cases better behavior then DNF. In the future, the new Microdnf will
> > replace DNF. The new Microdnf will be accompanied by a new library
> > (`libdnf5`) and a new DNF Daemon.
>
> I've gotta ask... How much memory does the new dnf daemon take while idle?

We do not have any measurements right now. Please feel free to test it. We
have a repository with DNF5/Microdnf nightly builds -
https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/dnf5-unstable/

Jaroslav
> I know this comes up time to time... As it is, PackageKitd and
> gnome-software both, individually, take ~ 450MB of RAM without any user
> interaction (other than logging in to a desktop) on a F36 system I
> updated for general testing.  This situation isn't new; on a quick
> search, I see reports of this problem on F25 in the top results.
>
> https://www.google.com/search?client=firefox-b-1-d&q=fedora+packagekitd+...
>
> Using close to a gig of RAM for a couple of services that do nothing
> 99.99% of the time isn't great, and I'm a little worried that another
> daemon is going to make things worse, especially until microdnf replaces
> dnf sometime "in the future."
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Miroslav Suchý]

Dne 20. 04. 22 v 8:55 Jaroslav Mracek napsal(a):
> I've gotta ask... How much memory does the new dnf daemon take while idle?
>
>
> We do not have any measurements right now. Please feel free to test it. We have a repository with DNF5/Microdnf 
> nightly builds - https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/dnf5-unstable/ 
> https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/dnf5-unstable/

My measurements (as reported by systemctl status):

After installation:
     Memory: 1.6M
        CPU: 23ms

After upgrade of few packages:
     Memory: 917.6M
        CPU: 1min 44.767

This is a lot. To dive deep. `top` reports VIRT: 797232, RES: 186596

To be fair, I have huge repository enabled. `dnf repoquery |wc -l` says 94 803.

Other feedback:

# systemctl enable dnfdaemon-server.service
The unit files have no installation config (WantedBy=, RequiredBy=, Also=,
Alias= settings in the [Install] section, and DefaultInstance= for template
units). This means they are not meant to be enabled using systemctl.

Is this expected? `start` works well. I am curious how we should enable it persistently.

BTW when I was looking for documentation, it was clumsy to find that documentation for DNF5 is in libdnf repository (and 
specific branch).

The fact that https://rpm-software-management.github.io/ has not been updated for years does not help either.

Miroslav

[Zbigniew Jędrzejewski-Szmek]

On Thu, Apr 21, 2022 at 12:45:48PM +0200, Miroslav Suchý wrote:
> Dne 20. 04. 22 v 8:55 Jaroslav Mracek napsal(a):
> > I've gotta ask... How much memory does the new dnf daemon take while idle?
> After installation:
>     Memory: 1.6M
>        CPU: 23ms
>
> After upgrade of few packages:
>     Memory: 917.6M
>        CPU: 1min 44.767
>
> This is a lot. To dive deep. `top` reports VIRT: 797232, RES: 186596

182 MB RSS is a lot, but should be acceptable on today's systems, as long
as it's transient.

Ideally:
- dnf-daemon would be dbus-activated and exit-on-idle after a suitable timeout
  (the timeout can be something like 15 minutes, so that it's not
   needlessly respawned if somebody is doing some interactive work. It just
   shouldn't stay around "forever".)
- If it is spawned also by things like command-not-found, it would
  need to start quickly. Possibly some caching of the repo state would need to be done.
  (Though maybe not. In a container where I'm testing this, i.e. with
   just the basic repos and a few packages, microdnf loads the metadata in <1s.
   If it stays <1s, this should be good enough.)

Zbyszek

[Colin Walters]

On Thu, Apr 21, 2022, at 7:19 AM, Zbigniew Jędrzejewski-Szmek wrote:
> dnf-daemon would be dbus-activated and exit-on-idle after a suitable
>
> timeout

This is how rpm-ostree has worked for about 5 years now: https://github.com/coreos/rpm-ostree/pull/606

(Lots of useful references in that thread - doing exit-on-idle in a *race free* way with DBus is unfortunately very tricky.  I am still thinking about switching the default client mode to direct systemd socket activation.  As of recently, the client also directly invokes `systemctl start` (if privileged))

The problem isn't really inherent to a daemon, the problem is the *gigantic* size of the repodata.

Also on the background in my todo list is to move all the RPM stuff into a forked subprocess from the daemon that itself is only run on demand - that would mean an idle daemon still has low memory usage.

I haven't dug into the libdnf5 code, but today it actually embeds the daemon code in the libdnf repository - I hope we can compile that out or split it out, because rpm-ostree already has an working DBus API.  Maybe in some cases we can try to expose some of the same API entrypoints, but on the other hand the entire rpm-ostree design is oriented by default around offline-by-default transactional updates, which is going to look quite different from an API perspective.

[Neal Gompa]

On Thu, Apr 21, 2022 at 10:01 AM Colin Walters  wrote:
> On Thu, Apr 21, 2022, at 7:19 AM, Zbigniew Jędrzejewski-Szmek wrote:
> > dnf-daemon would be dbus-activated and exit-on-idle after a suitable
> >
> > timeout
>
> This is how rpm-ostree has worked for about 5 years now: https://github.com/coreos/rpm-ostree/pull/606
>
> (Lots of useful references in that thread - doing exit-on-idle in a *race free* way with DBus is unfortunately very tricky.  I am still thinking about switching the default client mode to direct systemd socket activation.  As of recently, the client also directly invokes `systemctl start` (if privileged))
>
> The problem isn't really inherent to a daemon, the problem is the *gigantic* size of the repodata.
>
> Also on the background in my todo list is to move all the RPM stuff into a forked subprocess from the daemon that itself is only run on demand - that would mean an idle daemon still has low memory usage.
>
> I haven't dug into the libdnf5 code, but today it actually embeds the daemon code in the libdnf repository - I hope we can compile that out or split it out, because rpm-ostree already has an working DBus API.  Maybe in some cases we can try to expose some of the same API entrypoints, but on the other hand the entire rpm-ostree design is oriented by default around offline-by-default transactional updates, which is going to look quite different from an API perspective.

My hope is that the repository gets split out once the libdnf5 API is
finalized. Maintaining it all in one repo is going to be painful for
long-term maintenance.

[Neal Gompa]

On Thu, Apr 21, 2022 at 6:56 AM Miroslav Suchý  wrote:
> Dne 20. 04. 22 v 8:55 Jaroslav Mracek napsal(a):
> > I've gotta ask... How much memory does the new dnf daemon take while idle?
>
> We do not have any measurements right now. Please feel free to test it. We have a repository with DNF5/Microdnf nightly builds - https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/dnf5-unstable/
>
> My measurements (as reported by systemctl status):
>
> After installation:
>     Memory: 1.6M
>        CPU: 23ms
>
> After upgrade of few packages:
>     Memory: 917.6M
>        CPU: 1min 44.767
>
> This is a lot. To dive deep. `top` reports VIRT: 797232, RES: 186596
>
> To be fair, I have huge repository enabled. `dnf repoquery |wc -l` says 94 803.
>
> Other feedback:
>
> # systemctl enable dnfdaemon-server.service
> The unit files have no installation config (WantedBy=, RequiredBy=, Also=,
> Alias= settings in the [Install] section, and DefaultInstance= for template
> units). This means they are not meant to be enabled using systemctl.
>
> Is this expected? `start` works well. I am curious how we should enable it persistently.

Like the current dnfdaemon from ManaTools, it is dbus-activated. You
need something that uses its D-Bus APIs to have it start the service.

[John Reiser]

On 4/13/22 23:28, Gordon Messmer wrote:
> I've gotta ask... How much memory does the new dnf daemon take while idle?
>
> I know this comes up time to time... As it is, PackageKitd and gnome-software both, individually, take ~ 450MB of RAM without any user interaction (other than logging in to a desktop) on a F36 system I updated for general testing.

Assuming that there is minimal "bloat" in app data structures, then
the root cause of such high usage of address space is python memory management.
By default, python does not collect and re-use memory "garbage" (previous
allocations that no longer are accessible) until the OS denies a request to
allocate a new page.  The typical result is a long time before the first collection,
because Linux will allocate new pages until /proc/meminfo.CommitLimit is reached
(and/or swap space is exhausted, which invokes OutOfMemory process killer.)

In particular, a python daemon which starts and reaches a (near-)steady state
soon after system boot might not ever collect garbage unless the system
runs for weeks or months before a re-boot.

In order to reduce the memory footprint then the python strategy
must be changed, and/or the app must pay attention (explicitly delete
and collect garbage.)  In the particular case of rpm package management
on a system with 64-bit pointers, then the app also should use arrays
instead of linked structures.  There is no case of more than (1<<24)
packages, and nearly all invocations manage fewer than (1<<16) packages.
Thus a 2-byte or 3-byte ordinal (packed) can be used instead of an 8-byte pointer.
This may require non-trivial code changes.  For instance, the stack
cannot be used as the residence for a temporary package structure.
Also, using text compression might reduce significantly the RAM
that is required to store character strings.  Even without compression,
in many apps character strings tend to persist forever without
modification, so just append them into chunks of (1<<20) bytes,
with a separate array of (1<<12) pointers to chunks.  Then such a string
can be identified by a 32-bit ID instead of a 64-bit pointer,
which saves 4 bytes for the pointer plus at least 8 bytes of
accounting overhead used by the default malloc().

[Zbigniew Jędrzejewski-Szmek]

> Compatibility
>
> ** To improve user experience and to unify dnf/microdnf behavior we
> were unable to keep 100% compatibility with formal Microdnf in
> command-line and in behavior

Can you comment more on this part? yum/dnf command-line and behaviour
compatiblity made adoption fairly easy. (I know it wasn't 100%, but the
major parts were compatible.)

I see that e.g. microdnf5 doesn't have 'list': is this intentional?
Lack of 'list' would break many basic dnf uses…

Also, what is the plan for normal command-line use: microdnf5 or the dnfdaemon
client?

Zbyszek

[Fabio Valentini]

On Thu, Apr 21, 2022 at 11:01 AM Zbigniew Jędrzejewski-Szmek
 wrote:
> > Compatibility
> >
> > ** To improve user experience and to unify dnf/microdnf behavior we
> > were unable to keep 100% compatibility with formal Microdnf in
> > command-line and in behavior
>
> Can you comment more on this part? yum/dnf command-line and behaviour
> compatiblity made adoption fairly easy. (I know it wasn't 100%, but the
> major parts were compatible.)
>
> I see that e.g. microdnf5 doesn't have 'list': is this intentional?
> Lack of 'list' would break many basic dnf uses…
>
> Also, what is the plan for normal command-line use: microdnf5 or the dnfdaemon
> client?

Yeah, this is something that I'm also worried about in the long-term.
We've had dnf for so long now that there's tons of guides and docs
that reference dnf command line stuff.

And I don't even want to guess how many scripts there are that call
dnf on the command line that would be broken if it would be replaced
by something that's not a drop-in replacement ... most importantly,
what will happen to stuff from dnf-utils, like repoquery, and
repoclosure commands?
repoquery is an essential tool for package maintainers, and from my
experience, I'm often using it from a scripted context. If that would
no longer work or require reimplementing or adapting all my scripts
that call dnf commands, that's a pretty big deal :(

Fabio