F38 proposal: Rpmautospec by Default (System-Wide Change proposal)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/Rpmautospec_by_Default
This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.
== Summary ==
Rpmautospec (`%autorelease` and `%autochangelog`) is recommended as
the default approach.
Packaging Guidelines and other documentation are adjusted to describe
this approach first.
Various tools that provide spec file templates are adjusted.
== Owner ==
* Name: [[User:Nphilipp| Nils Philippsen]], [[User:Zbyszek| Zbigniew
Jędrzejewski-Szmek]]
* Email: nphilipp - at - redhat.com, zbyszek - at - in.waw.pl
== Detailed Description ==
{{admon/note|Brief reminder about rpmautospec|
The spec file contains:
<pre>
Version: 1.2.3
Release: %autorelease
...
%changelog
%autochangelog
</pre>
Rpmautospec uses git history. Whenever the package is built
(`.src.rpm` is generated), rpmautospec tooling will replace the
`%autorelease` macro with the number of commits since the last commit
that changed the `Version` field, and the `%autochangelog` macro with
a text generated from `git log`.
For details see the
[https://docs.pagure.org/fedora-infra.rpmautospec/principle.html
docs].
}}
Rpmautospec has been deployed in Fedora since F35
([[Changes/rpmautospec]]), and 3423/23045 packages use it (15%).
But it is still a "second-class citizen": most documentation doesn't
mention it, and many packagers know that it exists but don't use it in
their packages. We think that it's reasonable to switch to
`%autorelease`+`%autochangelog` for almost all packages and that
Packaging Guidelines and various packaging howtos should recommend
that approach to packagers. The "traditional" approach of
manually-managed `Release` and `%changelog` will remain valid and will
be documented as a fallback.
This change is targeted at Fedora 38, but it will actually apply to
all releases. The goal is to update the Packaging Guidelines and other
prominent documentation and tools now, and other docs and tools
possibly at a later time. Changing packages is out of scope.
It is worth mentioning that `rust2rpm` uses
`%autorelease`+`%autochangelog` since a few releases, so most rust
packages have switched. (Generally, rust spec files are recreated
using the generator for each new version, so the switch would happen
whenever a new version is packaged unless the packager opts out.)
== Feedback ==
* Thread on fedora-devel in August 2022:
[https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
rpmautospec by default]
** open issues: a bunch have been fixed.
** maintenance: Nils will add some co-maintainers.
** compatibility with rpmdevtools, fedpkg/rpkg, fedora-review: see
Scope section.
== Benefit to Fedora ==
Various packaging workflows become smoother for packagers and contributors:
* packagers don't need to touch the `Release` field on updates
* packagers describe changes just once in the git commit message, the
`%changelog` entry is autogenerated
* patches to the spec file can be cherry-picked between branches
without trivial conflicts
* pull requests on src.fedoraproject.org can be merged without trivial conflicts
* in workflows that regenerate the spec file (rust2rpm, pip2rpm, …)
`%changelog` section doesn't need to be copied over
== Scope ==
* Proposal owners:
** provide pull requests to Packaging Guidelines and other docs to
make `%autorelease`+`%autochangelog` the default
** implement fixes for issues reported by packagers
** make semi-regular releases of rpmautospec
([https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
0.3.1 was released on November 17th, and should be deployed to
production in about two weeks])
* Other developers:
** provide pull requests to other docs as appropriate
** accept the changes to documentation
** update other spec file generators (pip2rpm, others?)
* Somebody (TBD):
** `fedora-review` — https://pagure.io/rpkg/issue/641
** `fedpkg import` — with https://pagure.io/rpkg/c/3087dd7, the
command will fail. A replacement workflow that instead restores
`%autorelease`+`%autochangelog` in the file committed to dist-git
needs to be implemented.
* Related work
** https://pagure.io/rpkg/c/3087dd7
** https://src.fedoraproject.org/rpms/fedora-packager/pull-request/4
* Release engineering: [https://pagure.io/releng/issues #Releng issue number]
* Policies and guidelines: a list of places to be updated
** https://docs.fedoraproject.org/en-US/packaging-guidelines/#changelogs
** https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning
** https://docs.fedoraproject.org/en-US/package-maintainers/Packaging_Tutori...
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A
== Upgrade/compatibility impact ==
Rpmautospec is already used by a decent number of packages, so any
issues are already being seen and need to be fixed anyway.
== How To Test ==
* Convert an existing package: `rpmautospec convert`. Ideally this
step is done right before a version bump so that the release numbers
restart at `-1`.
* Do local builds (`fedpkg local`, `fedpkg mockbuild`). Verify
correctness of version-release (`rpm -qpi`) and the changelog (`rpm
-qp --changelog`).
* Do builds in koji. Verify correctness of version-release and the changelog.
* For new packages, use `%autorelease`+`%autochangelog`. Repeat all
the tests listed above.
* Assume you are a newbie packager. Read the packaging docs and check
that the workflow is clear and the intructions are sufficient to use
rpmautospec tooling correctly.
== User Experience ==
No changes visible to end users.
== Dependencies ==
None.
== Contingency Plan ==
If it turns out that the rpmautospec workflows have unknown problems,
we can revert changes to documentation.
* Contingency mechanism: Revert changes to documentation by reverting
the appropriate commits. This can be done easily by FPC.
* Contingency deadline: Any time.
* Blocks release? No.
== Documentation ==
This page and any changes to Packaging Guidelines and other documents.
== Release Notes ==
Not needed.
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
6 months, 2 weeks
zlib-ng as a compat replacement for zlib
by Richard W.M. Jones
The background to this is I've spent far too long trying to optimize
the conversion of qcow2 files to raw files. Most existing qcow2 files
that you can find online are zlib compressed, including the qcow2
images provided by Fedora. Each cluster in the file is separately
compressed as a zlib stream, and qemu uses zlib library functions to
decompress them. When downloading and decompressing these files, I
measured 40%+ of the total CPU time is doing zlib decompression.
[You don't need to tell me how great Zstd is, qcow2 supports this for
compression also, but it is not widely used by existing content.]
I was looking around for alternative zlib implementations and there
are several. This AWS blog is a decent summary:
https://aws.amazon.com/blogs/opensource/improving-zlib-cloudflare-and-com...
We already package zlib-ng in Fedora:
https://src.fedoraproject.org/rpms/zlib-ng/blob/rawhide/f/zlib-ng.spec
https://github.com/zlib-ng/zlib-ng
In my test zlib-ng is about 40% faster.
Sadly zlib-ng is not compiled with the ZLIB_COMPAT option. What this
means in practice is that the zlib functions have different names
(eg. zng_inflateInit instead of inflateInit). It is not a drop-in
replacement for zlib and software would need to be adjusted to use it.
However there is this bug / RFE to package the compat library.
https://bugzilla.redhat.com/show_bug.cgi?id=2145239
It literally replaces /usr/lib64/libz.so.1, so pretty high stakes. It
has the appropriate Provides/Conflicts.
Anyway, 40% (or whatever, but significant) performance improvement
sounds good for a very widely used operation.
So I'd like to ask Fedora ...
What do we think about the opt in approach of adding a patch similar
to the one proposed in bug 2145239, where I think you could "simply"
install zlib-ng to get better performance with existing software?
("simply" because it seems high risk of going wrong)
What about replacing zlib with zlib-ng?
Other ideas ...? Does anyone know what other distros are doing?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
6 months, 3 weeks
F39 Change Proposal: Allow Removal of tzdata (System-Wide)
by Aoife Moloney
https://fedoraproject.org/wiki/Changes/AllowRemovalOfTzdata
This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.
== Summary ==
Allow the removal of tzdata especially on containers in order to minimize size.
== Owner ==
* Name: Patsy Griffin (Franklin)
* Email: patsy(a)redhat.com
== Detailed Description ==
This change will allow the removal of tzdata. When tzdata is removed,
the system will default to UTC. In order to reduce overhead, many
container installations now remove the data associated with tzdata but
cannot fully remove the package due to dependencies by other packages.
This results in confusion regarding the expected timezone info.
In order for this to work, we need packages that use tzdata at run
time to switch from Require'ing tzdata to Recommend'ing tzdata. These
packages should also gracefully handle instances where tzdata has been
removed. For example, this would require recognizing that tzdata is
not present and providing an appropriate error, such as recommending
that the user install tzdata.
== Feedback ==
In June of 2021, we proposed creating a new tzdata sub-package that
would only provide the UTC timezone. As part of the discussion around
this proposal, it was recommended that we completely remove tzdata. We
appreciated this input and welcome additional feedback.
== Benefit to Fedora ==
This change will allow tzdata to be removed from containers without
leaving inconsistent package remnants.
== Scope ==
* Proposal owners: No changes are needed to tzdata.
* Other developers: Some packages need to change their spec files from
`Requires: tzdata` to `Recommends: tzdata`. It would be beneficial if
all packages switched in this way, but it is not required. Supporting
optional tzdata installation for as many workloads as possible allows
those workloads to minimize their container image size.
List of packages which need to be changed:
* glibc (glibc-common)
* gcc (libstdc++)
* python3.XX (3.9, 3.10, 3.11, 3.12)
List of packages which would be beneficial to be changed:
* python3-dateutil
* python3-pytz
* libical
Upon acceptance of the change request we will file bugs to fix each of
these packages for Fedora 39.
* Release engineering: No changes needed.
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Community Initiatives:
== Upgrade/compatibility impact ==
If tzdata is already installed, then it continues to be installed on the system.
Following the Fedora Weak Dependencies Policy dnf will treat the
Recommends on tzdata as if it were a Requires and tzdata will always
be installed in a default system
(https://docs.fedoraproject.org/en-US/packaging-guidelines/WeakDependencies/)
However, the recommends will allow tzdata to be correctly uninstalled
in a container build file rather than having to use ‘rm -rf’ to delete
the zone files to recover space.
== How To Test ==
Language runtimes were installed and A/B tests carried out with tzdata
present and tzdata removed. The intent of these tests was to ensure
that the system can use the language frameworks without tzdata present
and that when the data was required that meaningful errors were
presented to the user.
Packages tested were C (glibc), C++ (libstdc++), Python (Python 3.11),
using their time and date APIs. This testing led to the correction of
the libstdc++ implementation as noted here:
https://gcc.gnu.org/cgit/gcc/commit/?id=4abd5bc600193e821fbc41995a0b8d9ea...
Developers can test this by installing Rawhide and uninstalling tzdata
and verifying their package operates as expected.
If tzdata cannot be uninstalled then we recommend filling a bug
against the package that Requires: tzdata and having a discussion with
the maintainer to make tzdata optional e.g. Recommends: tzdata.
== User Experience ==
The user can remove tzdata to minimize the container size.
With tzdata removed the system is UTC only.
== Dependencies ==
Fixing glibc, gcc, and python3.xx at a minimum to make tzdata
Recommends instead of Requires.
== Contingency Plan ==
* Contingency mechanism: Back out the change. Don’t do it.
* Contingency deadline: Can be backed out at the last minute since we
are only dealing with conversions from Requires to Recommends.
* Blocks release? No
== Documentation ==
Document that the tzdata package may be removed if needed to reduce
space on containers. With tzdata removed, the system will default to
UTC.
== Release Notes ==
--
Aoife Moloney
Product Owner
Community Platform Engineering Team
Red Hat EMEA
Communications House
Cork Road
Waterford
7 months, 2 weeks
Potential changes to systemd RPM macros
by Andrea Bolognani
Hi,
this requires a bit of background, so please bear with me :)
Earlier this year, the libvirt project changed its RPM package to be
more granular[1] as a step towards a future where the legacy
monolithic daemon (libvirtd) is completely gone. This change has
happened upstream, but the Fedora package uses the same spec file so
it was quickly reflected in Rawhide.
As part of this change virtproxyd, a service which has historically
been part of the libvirt-daemon package, has been moved to the
newly-introduced libvirt-daemon-proxy package.
More recently we have realized that, despite our care in trying to
ensure smooth upgrades, splitting the package this way has
unfortunately caused unintended consequences[2][3]. Specifically,
since the libvirt-daemon-proxy package is a completely new one as far
as RPM is concerned, all its systemd units will get presets applied
during %post. In at least two completely valid deployment scenarios,
this results in some amount of brokenness due to local configuration
changes made by the admin getting discarded.
In order to avoid this problem, I have implemented a new set of RPM
macros[4][5] that base the decision of whether presets should be
applied for a systemd unit not on whether the package that contains
them is being newly installed, but rather on whether the unit itself
existed on the system before package installation. This should
address not only the scenario of services being split off to separate
packages, but also the one where an existing package starts shipping
additional services.
Now, since this is clearly not a libvirt-specific issue, I believe
this approach should be adopted across Fedora by way of these macros
(or comparable ones) being added to systemd. Packages would have to
migrate from the old macros over time, and at some point it should be
possible to get rid of them entirely.
Note that openSUSE already has its own distro-specific RPM macros for
dealing with systemd units, and in fact the approach that I have
implemented is partially inspired by theirs. Ideally, we'd be able to
collaborate with them to create a single set of macros that lives in
systemd and satisfies the needs of both Fedora and openSUSE.
The problem is that Fedora 39 and RHEL 9.3 are fast approaching and,
if we don't do anything about this issue before then, a subset of
libvirt users will see their deployments broken after upgrade. In the
interest of avoiding that, I would prefer to get the libvirt-specific
version of the macros merged as soon as possible, and focus on
upstreaming the work all the way to systemd as a follow-up.
Does this plan sound reasonable to the Fedora community? Are there
any serious concerns regarding the approach taken for the macros that
would cause them to be considered a complete no-go? Any scenarios
that I might have missed while implementing them?
Thanks for your patience in reading this far :) and looking forward
to your feedback!
[1] https://listman.redhat.com/archives/libvir-list/2023-January/237059.html
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2210058
[3] https://listman.redhat.com/archives/libvir-list/2023-June/240226.html
[4] https://listman.redhat.com/archives/libvir-list/2023-July/240723.html
[5] https://listman.redhat.com/archives/libvir-list/2023-July/240729.html
--
Andrea Bolognani / Red Hat / Virtualization
7 months, 4 weeks
Re: TSS maintainer volunteer
by Petr Pisar
V Thu, Feb 16, 2023 at 07:29:07PM +0000, Kenneth Goldman napsal(a):
> I think I followed all those steps - identifying the package, announcing that
> I want to be the packager, making an account, etc.
>
> What's next?
Submit an updated tss2 package for a package review. As far as I can see,
there is no review opened for tss2 now
<https://bugzilla.redhat.com/buglist.cgi?component=Package%20Review&produc...>.
How to do it is described at
<https://docs.fedoraproject.org/en-US/package-maintainers/Package_Review_P...>.
Especially pay attention to:
If you are not member of the packager group, you need a sponsor. Add
FE-NEEDSPONSOR to the bugs being blocked by your review request.
> Does someone approve me?
Based on the FE-NEEDSPONSOR blocker someone from sponsors should notice your
review request and start to communicate with you in the review request in
Bugzilla. (If that does not happen, approach you a sponsor of your choice as
recommended at
<https://docs.fedoraproject.org/en-US/package-maintainers/How_to_Get_Spons...>)
Once the sponsor finds your package looks good and you understand how to
maintain a package, he/she will sponsor you, i.e. adds you into a packagers
group. Then you will be able to continue from this item on the
Package_Review_Process document:
When your package passes the review you should use fedpkg to request a Git
repository for it.
> Move a git repo somewhere?
For the purpose of the package review, you need to publish the spec file and
the SRPM file somewhere on the Internet. (Once you become a packager, you can
also use <https://fedorapeople.org/> server for that purpose.)
Once the package review passes, the offical git repository (called dist-git in
Fedora) for the tss2 package will be reopened with completing this item:
Request a Git repository for the package
Then you will commit the new spec file into the reopen dist-git repository.
-- Petr
8 months
Bootstrapping package with circular dependencies in koji
by Jaroslav Skarvada
Hi,
I need to bootstrap package which has bootstrap support written
according to the [1]. I am able to bootstrap it locally (rpmbuild,
mock, ...) with the "--with bootstrap" or "-D '_with_bootstrap 1'". Is
there support for it in koji? E.g. something like:
koji build SIDE-TAG PACKAGE --bootstrap? Or do I have to manually do:
1. patch:
- %bcond_with bootstrap
+ %bcond_without bootstrap
2. koji build SIDE-TAG SCM
3. update&build the circular dep
4. unpatch:
- %bcond_without bootstrap
+ %bcond_with bootstrap
5. release bump
6. koji build SIDE-TAG SCM
Or is there some better way?
thanks & regards
Jaroslav
[1] https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping
8 months
Donate 1 minute of your time to test upgrades from F38 to F39
by Miroslav Suchý
Do you want to make Fedora 39 better? Please spend 1 minute of your time and try to run:
# Run this only if you use default Fedora modules
# next time you run any DNF command default modules will be enabled again
sudo dnf module reset '*'
dnf --releasever=39 --setopt=module_platform_id=platform:f39 \
--enablerepo=updates-testing \
$(rpm -q fedora-repos-modular >/dev/null && echo --enablerepo=updates-testing-modular) \
--assumeno distro-sync
This command does not replace `dnf system-upgrade`, but it will reveal potential problems.
You may also run `dnf upgrade` before running this command.
The `--assumeno` will just test the transaction, but does not make the actual upgrade.
In case you hit dependency issues, please report it against the appropriate package.
Or against fedora-obsolete-packages if that package should be removed in Fedora 39. Please check existing reports
against fedora-obsolete-packages first:
https://red.ht/2kuBDPu
and also there is already bunch of "Fails to install" (F39FailsToInstall) reports:
https://bugzilla.redhat.com/buglist.cgi?bug_id=2168845&bug_id_type=anddep...
Two notes:
* you may want to run the same command with dnf5 to help test new dnf.
* this command found zero issues on my personal system - great work all everybody!
Thank you
Miroslav
8 months
