January 2021 - devel - Fedora Mailing-Lists

[ELN] Is it really necessary to build a package 100+ times when there is a bootstrap issue?

by Miro Hrončok

This topic was brought up several times already, but I don't think there has been a satisfactory answer to it yet. So let's try again. We have recently updated python-elementpath-2.1.1-2.fc34 and python-xmlschema-1.4.1-1.fc34 -- it requires a very simple bootstrapping that can be seen in the git history. https://bodhi.fedoraproject.org/updates/FEDORA-2021-eb7554bd39 I know that the ELN rebuild tooling cannot handle bootstrapping and I know it has been told this will eventually be solved. I appreciate that it is planned. But in the meantime, is it really necessary to build the packages 100/111 times? https://src.fedoraproject.org/rpms/python-elementpath/c/5de3aa397bda55500... https://src.fedoraproject.org/rpms/python-xmlschema/c/0c1384c7aaed1d6404b... Could you please not do that? I realize that sometimes dependency issues are transient. I sometimes beat packages with a stick until they build as well, but more than 100 builds in less than two weeks feels a little far fetched. Please, make the automation stop and inspect the problem by a human after a reasonable amount of trials (I'd say 3, possibly 12, but not 100+). Alternatively at least slow down the firing rate with time. After two weeks of failures it might make sense to give it a try every other day, but not every other hour. Thanks. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

3 years, 2 months

3
2
0 / 0

Change in glibc about _STAT_VER in Fedora rawhide?

by Jan Pazdziora

Hello, it seems that both fakeroot and fakechroot fail to build in Fedora rawhide (at least partially) because _STAT_VER is no longer declared in the current glibc (or rather, its headers): https://bugzilla.redhat.com/show_bug.cgi?id=1889862 https://bugzilla.redhat.com/show_bug.cgi?id=1901049 The bugzillas are filed against their respective components but I wonder if we have any guidance from the glibc point of view about how these components should proceed. Or is the loss of _STAT_VER an omission and will it come back? -- Jan Pazdziora Product Owner, Platform Security Readiness, Red Hat

3 years, 2 months

3
5
0 / 0

Fedora 34 Change: Enable btrfs transparent zstd compression by default (System-Wide Change proposal)

by Ben Cotton

https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression == Summary == On variants using btrfs as the default filesystem, enable transparent compression using zstd. Compression saves space and can significantly increase the lifespan of flash-based media by reducing write amplification. It can also increase read and write performance. == Owners == * Name: [[User:salimma|Michel Salim]], [[User:dcavalca|Davide Cavalca]], [[User:josef|Josef Bacik]] * Email: michel(a)michel-slm.name, dcavalca(a)fb.com, josef(a)toxicpanda.com == Detailed description == Transparent compression is a btrfs feature that allows a btrfs filesystem to apply compression on a per-file basis. Of the three supported algorithms, zstd is the one with the best compression speed and ratio. Enabling compression saves space, but it also reduces write amplification, which is important for SSDs. Depending on the workload and the hardware, compression can also result in an increase in read and write performance. See https://pagure.io/fedora-btrfs/project/issue/5 for details. This was originally scoped as an optimization for https://fedoraproject.org/wiki/Changes/BtrfsByDefault during Fedora 33. == Benefit to Fedora == Better disk space usage, reduction of write amplification, which in turn helps increase lifespan and performance on SSDs and other flash-based media. It can also increase read and write performance. == Scope == * Proposal owners: ** Update anaconda to perform the installation using <code>mount -o compress=zstd:1</code> ** Set the proper option in fstab (alternatively: set the XATTR) ** Update disk image build tools to enable compression: *** lorax *** appliance-tools *** osbuild *** imagefactory ** [optional] Add support for [https://github.com/kdave/btrfs-progs/issues/328 setting compression level when defragmenting] ** [optional] Add support for [https://github.com/kdave/btrfs-progs/issues/329 setting compression level using `btrfs property`] * Other developers: ** anaconda: review PRs as needed * Release engineering: https://pagure.io/releng/issue/9920 * Policies and guidelines: N/A * Trademark approval: N/A == Upgrade/compatibility impact == This Change only applies to newly installed systems. Existing systems on upgrade will be unaffected, but can be converted manually with <code>btrfs filesystem defrag -czstd -r</code>, updating `/etc/fstab` and remounting. == How to test == Existing systems can be converted to use compression manually with <code>btrfs filesystem defrag -czstd -r</code>, updating `/etc/fstab` and remounting. == User experience == Compression will result in file sizes (e.g. as reported by du) not matching the actual space occupied on disk. The [https://src.fedoraproject.org/rpms/compsize compsize] utility can be used to examine the compression type, effective compression ration and actual size. == Dependencies == Anaconda will need to be updated to perform the installation using <code>mount -o compress=zstd:1</code> == Contingency plan == * Contingency mechanism: will not include PR patches if not merged upstream and will not enable * Contingency deadline: Final freeze * Blocks release? No * Blocks product? No == Documentation == https://btrfs.wiki.kernel.org/index.php/Compression == Release Notes == Transparent compression of the filesystem using zstd is now enabled by default. Use the compsize utility to find out the actual size on disk of a given file. -- Ben Cotton He / Him / His Senior Program Manager, Fedora & CentOS Stream Red Hat TZ=America/Indiana/Indianapolis

3 years, 2 months

9
17
0 / 0

Copr in 2020 and outlook for 2021

by Miroslav Suchý

Let me sum up what we - the Copr team - did in 2020: * We enabled CDN for repos. https://fedora-copr.github.io/posts/copr-cdn * We enabled runtime dependecies on repositories https://fedora-copr.github.io/posts/runtime-dependencies * We migrated all our servers from PHX datacenter to AWS. With zero downtime for your repos. * Pavel Raiskup became the new Mock project leader, and he released seven new versions of Mock during 2020. https://github.com/rpm-software-management/mock/wiki * Our small team has been renamed to Community Packaging Team (CPT) because beside Copr, we maintain other tools as well: Mock, dist-git, Tito... * We drove changes in createrepo_c, which allowed us better throughput. Now, Copr can build thousands of builds per day. * We worked on Fedora packages website, which was outdated and do not work on recent Fedora. Unfortunately, it will not be likely used, and static pages will be used as it will require less maintenance. * We introduced Project scoring for projects in Copr (up/down vote) http://frostyx.cz/posts/upvoting-projects-in-copr * We allowed you to delete multiple builds at once. https://fedora-copr.github.io/posts/deleting-list-of-builds * You can use Github Actions now https://pavel.raiskup.cz/blog/github-push-actions-and-copr.html * We created modulemd-tools for low-level handling of repositories and modules. And tool for generating module-build-macros. https://github.com/rpm-software-management/modulemd-tools * We worked hard on better EOL handling of Copr repositories. * We parallelized copr-dist-git imports. * We wrote code for Ansible DNF Copr module. This still needs to land in a proper place like Ansible Galaxy. https://pagure.io/copr/copr/blob/master/f/ansible * We provided --isolation=* and --bootstrap=* knob in Copr * We started an experimental "external dependencies" feature in Mock. https://github.com/rpm-software-management/mock/wiki/Feature-external-deps * We created a new python module "templated-dictionary" as a spin-off from Mock's code. https://github.com/xsuchy/templated-dictionary * We wrote four articles "4 cool new projects to try in COPR" https://fedoramagazine.org/tag/copr/ * Users now have the ability to change the build timeout up to a maximum of 48 hours * Users can group builds in batches now. https://pavel.raiskup.cz/blog/build-ordering-by-batches-in-copr.html We already have plans to do: * Verify GPG checks in DNF using DNSSEC. https://pagure.io/copr/copr/issue/406 * Improve hooks (in cooperation with Packit team) * We are going to EOL APIv1 and APIv2 https://fedora-copr.github.io/posts/EOL-APIv1-APIv2 What will we do in 2021? We have some ideas. Some of them are yours, some of them are ours. At the end of this email is a link to Google Form where you can vote what you would like to get. We will consider your vote. (listed in no specific order) * When you create a project, you may specify that it will be for Package Review. We do some settings for you, and `rpminspect` will be run automatically at the end of the build. We can guide users on how to file Package Review BZ. Or do that automatically on their behalf. * Contribute to fedpkg/Koji that tagged commits will be automatically built in Koji and may be automatically sent to Bodhi. * Help with automatic changelog entries. * Native support for Fedora in Travis. * Continue on external dependencies. https://github.com/rpm-software-management/mock/wiki/Feature-external-deps * Finish rpm-spec-wizard https://github.com/xsuchy/rpm-spec-wizard * Better integration with Koshei. * Automatic rebuilds in Copr when dependency change (likely with the help of Koshei) * Enhance release-monitoring and rebase-helper to automatically open PR in Pagure when new version is available. * Enhance `Mock --chain` to try to set %bootstrap when standard loop fails. When the set succeeds, rebuild the bootstrapped package again without %bootstrap macro. https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping * Improve the process of finding a sponsor for the first package. * Contribute to fedpkg/koji to have machine-readable output. Please vote here: https://forms.gle/UuH86ECzfZNHJgEr8 The results are intentionally publicly available. You may check them anytime. CPT team consist of Pavel Raiskup, Silvie Chlupova, Jakub Kadlcik and this year we also had in the team Dominik Turecek and Tomas Hrnciar. -- Miroslav Suchy, RHCA Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys

3 years, 2 months

5
9
0 / 0

Attempting to allow for a source of truth for Fedora release cycle information: releasestream

by Adam Williamson

Hi folks! Before the RH holiday shutdown, I started a new project designed to address a long-standing pain point in Fedora: we don't really have a good source of truth for release cycle information. I'm thinking of situations where something: * Needs to know what the "current stable" Fedora release(s) are * Needs to know if Fedora X is Branched * Needs to know whether Fedora X Beta happened yet * Needs to know whether Fedora X is frozen ...etc etc etc. Some of this information is available...sort of...from various different systems, with various awkward caveats that I won't dive into unless someone asks. But there isn't really a single source of truth for it, and some of it just isn't really available in any easily machine-consumable way. So I wrote releasestream: https://pagure.io/fedora-qa/releasestream releasestream is intended to be a system that will let us do this. It is a very very simple webapp with three capabilities: * It can read a simple record ("stream") of "release events" for a "release" and produce several static JSON representations of the information * It can write an entry to one of these streams in response to a properly-formatted POST request * It can publish a message when a new entry is received that is all it does. The "releases" and "events" are entirely arbitrary; a "release" can be any string, and so can the "state" for a given "event". An "event" is defined as a state being either reached or left; any number of events for the same state can be present for a release. The JSON outputs are basically "states by release" and "releases by state", as you might want either depending on what you're doing. You can conveniently look up "what releases are currently in state X?" or "what states is release X currently in?". This all works right now; the main thing that isn't implemented yet is any form of authentication. Right now if you can talk to the server you can submit events. I wanted to check if there is interest in moving forward with this, and discuss various options, before working on that. There is a ticket where I sketched out various possible approaches: https://pagure.io/fedora-qa/releasestream/issue/2 My idea for using this is basically that we deploy an 'official' releasestream instance in infra, and then find things that do the actual work of moving Fedora releases into and out of given "states" and tack on a bit at the end to tell releasestream about it. So when e.g. Mohan is putting Fedora 34 into the Beta freeze, we would add a "submit 'enter freeze' event to releasestream" to that process somehow - ideally something that has to be run as part of the process anyway would send the POST, but in a pinch a human could do it too. When Fedora 34 is being 'released', we tweak that process to include sending a releasestream event POST. And so on. The thing I like about this design is that there's a low barrier to entry, and can easily be adopted piecemeal but still be immediately useful for some things - as long as the event your code needs to watch for has been "onboarded", you're good. It also allows for the implementation details of a state change to change radically - it can go from being done by a human to being done by system X to being done by system Y, and all that needs to happen is to ensure the same dead simple POST request is sent to the same server. So, what do folks think? Does this seem like a good idea? Should I go ahead with trying to get it deployed and onboard things to it? Any comments, ideas, potential problems? Thanks! -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net

3 years, 2 months

6
10
0 / 0

Fedora 34 Change proposal: Remove make from BuildRoot (System-Wide Change)

by Ben Cotton

https://fedoraproject.org/wiki/Changes/Remove_make_from_BuildRoot == Summary == This change will remove make from the default buildroot in Koji and mock. == Owner == * Name: [[User:tstellar| Tom Stellard]] * Email: <tstellar(a)redhat.com> == Detailed Description == ===== Phase 1: Analysis ===== For this change, we will start by creating a list of all packages that have a build-time dependency on make. This will be done by analyzing spec files and also by rebuilding all packages in Fedora with make removed from the buildroot to see which packages fail to build. Once we have this list, we will remove packages that already have BuildRequires:make in their spec file, and this final list[1] will be all the packages that need to be updated in Phase 2. [1] https://github.com/tstellar/fedora-change-make-buildroot/blob/main/needs_... ===== Phase 2: Package Updates ===== Once we have the list of packages that need to be updated, we will proceed with adding BuildRequires: make to all spec files that require it. This new BuildRequires will be added to the line after the last BuildRequires in the spec file. The release number for packages will '''*not*''' be incremented for this change. The spec file updates will be automated and changes will be pushed directly to dist-git once they are ready. ===== Phase 3: Update Buildroot ===== Once package spec files have been updated, then we can proceed with removing make from the BuildRoot. This will be done by removing make from the build group in Koji and by removing make from the buildsys-build group in comps (fedora-comps). In order to avoid issues with Koschei, this change will need to be made as close as possible to the start of the mass rebuild. If possible, we will try to first remove make from the mass rebuild side-tag and then remove it from rawhide after the rebuild completes. ===== Phase 4: Monitor Failures ===== Once all the changes are in place, we will continue to monitor koji builds to see if there are any failures related to this change. We expect that the analysis done in Phase 1 would give us the complete list of packages that need to be updated, but it is always possible that something will be missed. == Feedback == * Removing make from the Buildroot without rebuilding the packages first has the potential to cause mass failures in Koschei. This is because Koschei builds from the latest SRPM and not from dist-git. We can minimize this problem by removing make from the buildroot as close as possible to the mass rebuild. The proposal has been updated now to account for this issue. == Benefit to Fedora == Based on my research[1], Fedora Rawhide has 22,309 packages, and there are approximately 10,378 packages that either use make explicitly or fail to build when make is removed form the buildroot. This means that there are around 11,931 packages that don't need make. Removing make from the BuildRoot will reduce the network traffic, download times, and disk usage for these builds in Koji and also for users doing builds with mock. Removing make (and its dependencies) will: * Reduce the BuildRoot download size by: 7.3 MB [2]: ** make: 539k ** gc: 104k ** guile22: 6.6M ** libtool-ltdl: 36k * Reduce the BuildRoot install size by; 46 MB [2]: ** make: 1.6M ** gc: 229k ** guile22: 44M ** libtool-ltdl: 71k [1] https://github.com/tstellar/fedora-change-make-buildroot [2] Package sizes are from the x86_64 packages. == Scope == * '''Proposal owners:''' Tom Stellard * '''Package Maintainers:''' Fedora package maintainers should report bugs if they think there is a problem caused by this change, but otherwise there will be no action required by them. * '''Proven Packager:''' The proposal owner will need to either apply for provenpackager status or get the help of someone with provenpackager status in order to make the spec file changes that are required. * '''Release engineering:''' [https://pagure.io/releng/issue/9829 #9829] (a check of an impact with Release Engineering is needed) * '''Policies and guidelines:''' The packager guidelines will need to be updated to mention that BuildRequires: make is now required for all packages that need make. * '''Trademark approval:''' N/A (not needed for this Change) * '''Alignment with Objectives:''' This aligns with the Fedora Minimization [https://pagure.io/minimization/issue/12 objective]. == Upgrade/compatibility impact == This should not impact upgrades. == How To Test == This change can be tested by rebuilding affected packages. The goal is to complete this before the mass rebuild to ensure maximum testing coverage. == User Experience == Fedora users will not notice any difference with this change. This will only impact Fedora package maintainers. == Dependencies == gcc < 11 will fail if the -flto=auto option is used when make is not installed. Phase 3 cannot be completed until gcc 11 lands in rawhide, but Phase 1 and Phase 2 are not blocked by this. == Contingency Plan == * '''Contingency mechanism:''' If we discover critical issues during phase 4, then Fedora Release Engineering will need to re-add make into the buildroot. * '''Contingency deadline:''' 2021-02-23 (F34 Beta Freeze) * '''Blocks release?''' No * '''Blocks product?''' No == Documentation == The packager guidelines will need to be updated to mention that BuildRequires: make is now required for all packages that need make. -- Ben Cotton He / Him / His Senior Program Manager, Fedora & CentOS Stream Red Hat TZ=America/Indiana/Indianapolis

3 years, 2 months

23
56
0 / 0

git push permission denied

by Neal Becker

I just had to setup a new machine, and new ssh keys. I visited https://admin.fedoraproject.org/accounts/user/edit/xxxx I chose my new id_rsa.pub to upload. But I get: git push --verbose Pushing to ssh://nbecker@pkgs.fedoraproject.org/mercurial Permission denied (publickey). fatal: The remote end hung up unexpectedly Hints?

3 years, 2 months

4
5
2 / 0

our containers with alias vim=vi

by clime

Hello, could Fedora and CentOS containers for docker and podman come with `alias vim=vi` in ~/.bashrc? I would very much welcome it as I am used to type vim everywhere but if vi starts instead I am happy too. I know that the solution is to create a customized container but often I want to try something on vanilla containers from the whole range. Didn't want to write about this first but maybe there are more people with the same problem. clime

3 years, 2 months

9
24
0 / 0

Fedora 34 Change: Deprecate python-mock (Self-Contained change proposal)

by Ben Cotton

https://fedoraproject.org/wiki/Changes/DeprecatePythonMock == Summary == The {{package|python-mock}} ({{package|python3-mock}}) package will be [https://docs.fedoraproject.org/en-US/packaging-guidelines/deprecating-pac... deprecated] in [[Releases/34|Fedora 34]]. The package is a standard library backport for older Pythons, Fedora packages should use `unittest.mock` instead. Many still depend on `mock`, so we cannot remove it yet. Packagers are encouraged to work with upstream to switch to `unittest.mock` when available. A simple `sed` can be applied in `%prep` as a temporary (or even permanent) downstream solution. == Owner == * Name: [[User:Churchyard|Miro Hrončok]] * Email: mhroncok(a)redhat.com == Detailed Description == The {{package|python-mock}} package is the [https://pypi.org/project/mock/ 3rd party backport of the standard library `unittest.mock` module]. > mock is now part of the Python standard library, available as [https://docs.python.org/dev/library/unittest.mock.html unittest.mock] in Python 3.3 onwards. > This package contains a rolling backport of the standard library mock code compatible with Python 3.6 and up. Fedora has recent enough versions of Python, hence using a library backport is redundant. Many packages only use it out of habit. We'd like to encourage both downstream packages and upstreams to switch to [https://docs.python.org/dev/library/unittest.mock.html unittest.mock] instead. Eventually, we'd like to drop {{package|python-mock}} from Fedora entirely, if possible. Before we attempt to remove the package, we need to stop new packages to (Build)Require {{package|python3-mock}}, hence we want to have it [https://docs.fedoraproject.org/en-US/packaging-guidelines/deprecating-pac... deprecated]. Note that the change owner does not currently officially maintain {{package|python-mock}} but the Python Maintenance team maintains the package in RHEL and contributes to the Fedora package when needed. The Fedora package maintainers have been contacted without response. === How to migrate to unittest.mock === In most cases, performing the following replacement should be enough: s/^(\s*)import mock/\1from unittest import mock/ s/^(\s*)from mock import /\1from unittest.mock import / If upstream really needs to support Python versions without `unittest.mock`, we recommend using a `try-import` mechanism, such as: try: from unittest import mock except ImportError: import mock If dual support for `unittest.mock` and `mock` is required, and the `mock` package is required in the metadata (such as in the testing ''extras''), conditionalize it, with: mock;python_version<"3.3" == Feedback == In the past, we've managed to migrate some packages away from python-mock, without a push back: * https://src.fedoraproject.org/rpms/python-urllib3/pull-request/13 * https://src.fedoraproject.org/rpms/python-freezegun/pull-request/10 * https://src.fedoraproject.org/rpms/python-hypothesis/c/65a4191709 == Benefit to Fedora == Eventually, we might be able to no longer maintain a standard library backport in a separate package. == Scope == * Proposal owners: Deprecate {{package|python3-mock}} and update the package description. Provide help migrating to `unittest.mock` to other packagers who ask for it. * Other developers: No action needed. Don't add new dependencies on {{package|python3-mock}}. If interested, migrate existing packages to `unittest.mock` (feel free to ask for help) * Release engineering: no impact on Release Engineering is anticipated * Policies and guidelines: N/A (not a System Wide Change) * Trademark approval: N/A (not needed for this Change) * Alignment with Objectives: N/A (not needed for this Change) == Upgrade/compatibility impact == The package will remain available. Only new packages cannot depend on it. Once retired (in distant future), we don't plan to obsolete/provide <code>python3-mock</code> from {{package|python3-libs}}, because it cannot work as drop-in replacement (the import name is different). The package will eventually be obsoleted by {{package|fedora-obsolete-packages}} once Python is updated to 3.N+1 after the removal to avoid broken upgrades. == How To Test == $ repoquery --repo=rawhide --provides python3-mock ... deprecated() ... == User Experience == No changes. == Dependencies == N/A (not a System Wide Change) == Contingency Plan == * Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change) * Contingency deadline: N/A (not a System Wide Change) * Blocks release? N/A (not a System Wide Change) == Documentation == N/A (not a System Wide Change) -- Ben Cotton He / Him / His Senior Program Manager, Fedora & CentOS Stream Red Hat TZ=America/Indiana/Indianapolis

3 years, 2 months

6
15
0 / 0

Fedora 34 Mass Rebuild

by Mohan Boddu

Hi all, Per the Fedora 34 schedule[1] we will start a mass rebuild for Fedora 34 on Jan 20th 2021. We will run a mass rebuild for Fedora 34 for the changes listed in: https://pagure.io/releng/issues?status=Open&tags=mass+rebuild Mass rebuild will be done in a side tag (f34-rebuild) and moved over when completed. Failures can be seen https://kojipkgs.fedoraproject.org/mass-rebuild/f34-failures.html Things still needing rebuilt https://kojipkgs.fedoraproject.org/mass-rebuild/f34-need-rebuild.html FTBFS bugs will be filed shortly. Please be sure to let releng know if you see any bugs in the reporting. You can contact releng in #fedora-releng on freenode, by dropping an email to our list[2] or filing an issue in pagure[3] Regards, Mohan Boddu. [1] https://fedorapeople.org/groups/schedule/f-34/f-34-key-tasks.html [2] https://lists.fedoraproject.org/admin/lists/rel-eng.lists.fedoraproject.org/ [3] https://pagure.io/releng/ _______________________________________________ devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedora...

3 years, 2 months

13
31
0 / 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

devel January 2021