#6171: Support --isfinal for Atomic installer
by Fedora Release Engineering
#6171: Support --isfinal for Atomic installer
-----------------------------+------------------------
Reporter: walters | Owner: rel-eng@…
Type: task | Status: new
Milestone: Fedora 22 Final | Component: mash
Keywords: | Blocked By:
Blocking: |
-----------------------------+------------------------
I'm not seeing any code in the current scripts that would conditionally
pass --isfinal to lorax for the final Atomic installer. I'm not sure how
that should work, otherwise I'd write a patch. If it exists please feel
free to close this as invalid.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/6171>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
8 years, 11 months
[releng] Updated tag rawhide-stable
by Till Maas
The lightweight tag 'rawhide-stable' was updated to point to:
9c30085... block_retired: Simplify/Fix packages check
It previously pointed to:
bf9e7a8... hardlink the whole tree for now. when we fix mash drop back
NOTE: People pulling from the repository will not get the new tag.
For more information, please see:
http://live.gnome.org/Git/Help/TagUpdates
8 years, 11 months
[releng] block_retired: Simplify/Fix packages check
by Till Maas
commit 9c300851977833a9633fd9711c2b7011711ced53
Author: Till Maas <opensource(a)till.name>
Date: Thu May 7 20:14:52 2015 +0200
block_retired: Simplify/Fix packages check
- Use the package listing to ensure that inheritance is honored
Signed-off-by: Till Maas <opensource(a)till.name>
scripts/block_retired.py | 30 +++++++++++++-----------------
1 files changed, 13 insertions(+), 17 deletions(-)
---
diff --git a/scripts/block_retired.py b/scripts/block_retired.py
index 4e58335..41dd5d5 100755
--- a/scripts/block_retired.py
+++ b/scripts/block_retired.py
@@ -70,14 +70,17 @@ class ReleaseMapper(object):
return None
-def blocked_packages(branch="master", staging=False):
+def unblocked_packages(branch="master", staging=False):
+ """
+ Get a list of all unblocked pacakges in a branch.
+ """
mapper = ReleaseMapper(staging=staging)
tag = mapper.koji_tag(branch)
url = PRODUCTION_KOJI if not staging else STAGING_KOJI
kojisession = koji.ClientSession(url)
pkglist = kojisession.listPackages(tagID=tag, inherited=True)
- blocked = [p["package_name"] for p in pkglist if p.get("blocked")]
- return blocked
+ unblocked = [p["package_name"] for p in pkglist if not p.get("blocked")]
+ return unblocked
def get_retired_packages(branch="master", staging=False):
@@ -171,17 +174,6 @@ def block_package(packages, branch="master", staging=False):
tag = mapper.koji_tag(branch)
epel_build_tag = mapper.epel_build_tag(branch)
- # Due to race conditions, puppet:configs/system/owner-sync-pkgdb might not
- # have added the package to the tag, e.g. for EPEL only packages. Therefore
- # process only packages that are actually in the tag
- for package in list(packages):
- tags = run_koji(["list-tags", "--package", package],
- output=True).splitlines()
- if tag not in tags:
- packages.remove(package)
- if not packages:
- return None
-
# Untag builds first due to koji/mash bug:
# https://fedorahosted.org/koji/ticket/299
# FIXME: This introduces a theoretical race condition when a package is
@@ -230,11 +222,15 @@ def block_all_retired(branches=RETIRING_BRANCHES, staging=False):
log.warning('%s not handled in staging..' % branch)
continue
retired = get_retired_packages(branch, staging)
- blocked = blocked_packages(branch, staging)
-
unblocked = []
+
+ # Check which packages are included in a tag but not blocked, this
+ # ensures that no packages not included in a tag are tried to be
+ # blocked. Packages might not be in the rawhide tag if they are retired
+ # too fast, e.g. because they are EPEL-only
+ allunblocked = unblocked_packages(branch, staging)
for pkg in retired:
- if pkg not in blocked:
+ if pkg in allunblocked:
unblocked.append(pkg)
if unblocked:
8 years, 11 months
#6168: Create f23 tags on staging Koji
by Fedora Release Engineering
#6168: Create f23 tags on staging Koji
----------------------+------------------------
Reporter: mizdebsk | Owner: rel-eng@…
Type: task | Status: new
Milestone: | Component: koji
Keywords: | Blocked By:
Blocking: |
----------------------+------------------------
Please create missing f23 tags on staging Koji. They are needed by
Koschei.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/6168>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
8 years, 11 months
[releng] block_retired: Check whether packages are in tag
by Till Maas
commit c4dea7dc547bc0dcaa3112cc68d83feb5850ebb7
Author: Till Maas <opensource(a)till.name>
Date: Thu May 7 18:33:57 2015 +0200
block_retired: Check whether packages are in tag
Signed-off-by: Till Maas <opensource(a)till.name>
scripts/block_retired.py | 18 ++++++++++++++++--
1 files changed, 16 insertions(+), 2 deletions(-)
---
diff --git a/scripts/block_retired.py b/scripts/block_retired.py
index 0fb5000..4e58335 100755
--- a/scripts/block_retired.py
+++ b/scripts/block_retired.py
@@ -157,17 +157,31 @@ def block_package(packages, branch="master", staging=False):
if len(packages) == 0:
return None
- def run_koji(koji_params):
+ def run_koji(koji_params, output=False):
url = PRODUCTION_KOJI if not staging else STAGING_KOJI
koji_cmd = ["koji", "--server", url]
cmd = koji_cmd + koji_params
log.debug("Running: %s", " ".join(cmd))
- return subprocess.check_call(cmd)
+ if output:
+ return subprocess.check_output(cmd)
+ else:
+ return subprocess.check_call(cmd)
mapper = ReleaseMapper(staging=staging)
tag = mapper.koji_tag(branch)
epel_build_tag = mapper.epel_build_tag(branch)
+ # Due to race conditions, puppet:configs/system/owner-sync-pkgdb might not
+ # have added the package to the tag, e.g. for EPEL only packages. Therefore
+ # process only packages that are actually in the tag
+ for package in list(packages):
+ tags = run_koji(["list-tags", "--package", package],
+ output=True).splitlines()
+ if tag not in tags:
+ packages.remove(package)
+ if not packages:
+ return None
+
# Untag builds first due to koji/mash bug:
# https://fedorahosted.org/koji/ticket/299
# FIXME: This introduces a theoretical race condition when a package is
8 years, 11 months
Mash slowness update
by Dennis Gilmore
Hi all,
So in debugging why mash is so slow. we had a realisation, one thing it was
supposed to always be doing never worked. and that is hardlinking the
packages in /mnt/koji/mash to /mnt/koji/packages/ this is due to apache owning
everything under /mnt/koji/packages and the compose processes all running as
masher. the hardlink failed due to permission errors
the rawhide mash today took ~3 hours to put all the files into place in
/mnt/koji/mash as shown by some snippets from the mash.log
2015-05-05 05:25:51 mash: Writing out files for
/mnt/koji/mash/rawhide-20150505/rawhide/x86_64/os/Packages...
2015-05-05 08:11:43 mash: createrepo: starting
/mnt/koji/mash/rawhide-20150505/rawhide/x86_64/os/Packages...
when run as apache in a test putting all the files in place in /mnt/koji/mash
took ~4 minutes
2015-05-05 19:27:07 mash: Writing out files for
/mnt/koji/mash/rawhide-20150505.01/rawhide/x86_64/os/Packages...
2015-05-05 19:30:54 mash: createrepo: starting
/mnt/koji/mash/rawhide-20150505.01/rawhide/x86_64/os/Packages...
So that is a massive massive savings, both in the time it will take to mash
the tree, but also the disk we use on /mnt/koji right now we are using 27T of
disk, there will be large chunks of duplicated data. so now we need to adjust
a bunch of processes. we need to have bodhi run mash as apache, we need to do
the nightly branched and rawhide as apache, and we need to change how we do
TC's and RC's so that they run as apache also. however for TC's and RC's I
will propose we make the change for F23 so as to not shake up things too much
for F22 final. we do not want to be responsible for any slip.
There is likely other areas of mash that can do with improvement. we need to
run some mashes with debugging enabled to see how exactly drpm creation goes
as It seems to be a big slowdown point.
Dennis
8 years, 11 months
Redo the interaction with the lookaside cache
by Mathieu Bridon
This patch series moves the code which talks to the remote lookaside cache,
rewriting parts of it. This makes the code more maintainable, and provides a
nicer, easier to reuse and extend API.
At the same time, some features are added which could allow some downstreams
(fedpkg for example, but also probably centpkg) to remove some of their code
completely, without losing any functionality.
Speaking of downstream users of the pyrpkg API, this patch series does not
break the API, downstreams don't actually require any code change to keep
working as they currently do.
All of this is thoroughly tested, the newly introduced modules are fully
covered by unit tests. As a result, global test coverage goes from 26% to 36%.
In order to make this more maintainable and better tested, the code now does
not call the 'curl' command as a subprocess any more. Instead, everything uses
pycurl, which was already used in pyrpkg and as such no new runtime dependency
is introduced.
Some code paths are marked as deprecated, and will emit messages when they are
executed, so that downstream users of the pyrpkg API can learn about the
deprecations, and port their code. That code could then be removed from pyrpkg
eventually, but this patch series does not do that.
The new APIs are entirely documented. Try this for example:
>>> from pyrpkg.lookaside import CGILookasideCache
>>> help(CGILookasideCache)
As a cherry on the cake, the new modules (and their unit tests) are fully
compatible with Python 3, so we'll have less porting work when the time comes.
Of course, the code still is compatible with Python 2.6, so things should
still work on EL6.
Finally, these changes are going to help immensely when we try to move away
from MD5 checksums for the lookaside cache in Fedora.
All in all, this is a lot of changes:
src/pyrpkg/__init__.py | 263 +++++++-------------
src/pyrpkg/cli.py | 8 +-
src/pyrpkg/errors.py | 49 ++++
src/pyrpkg/lookaside.py | 299 ++++++++++++++++++++++
src/pyrpkg/sources.py | 14 +-
src/pyrpkg/utils.py | 63 +++++
test/test_lookaside.py | 591 ++++++++++++++++++++++++++++++++++++++++++++
test/test_utils.py | 159 ++++++++++++
8 files changed, 1258 insertions(+), 188 deletions(-)
I've made a copr where I've pushed packages built with these changes, and a
couple of packager-friends have been using them for a few days already. They
are heavy-users of pyrpkg (through fedpkg, rhpkg and rdopkg) and all the
problems they could find are fixed in this patch series.
8 years, 11 months
RFC: rel-eng tooling development workflow
by Adam Miller
Hello all,
It's recently become somewhat of a recurring theme in the
#fedora-releng channel to discuss the Fedora rel-eng tooling
development workflow, changes that would like to be seen, and
decisions on tools/infra to deliver them. I thought I'd shoot out an
email to the list to kick off the conversation in a more formal manner
with an initial proposal for feedback.
I'm personally a fan of what I'll simply refer to as "the GitHub
workflow" by virtue of it's rise in popularity but it boils down to
effectively this:
1) Upstream project has a 'master' or 'devel' branch were dev happens
2) Developers fork the repository
3) Developers create "topic branches" (a branch for a specific dev
feature/fix)
4) Developer then send a pull request to 'master' or 'devel' for code review
5) Once reviewed and signed off on it is approved and merged
Now a couple things to note:
First and foremost is that this is by no means the only workflow
but just one that has become increasingly popular that I'm a fan of.
However, I'm certainly open to other suggestions and opinions but for
the sake of my initial proposal and the rest of this email I'll be
continuing on with this workflow in mind.
Secondly, it has been brought up that the Fedora Rel-Eng team at
large does not want to use GitHub because it is non-free software and
I'm on board with that.
With that second note in mind, there are a decent handful of open
source solutions for this and of those there are two "front runners"
in my mind for our use since they are written in python and the Fedora
team at large has plenty of experience developing and
hosting/deploying/administering software written in python. They are:
- pagure (http://pagure.dev.fedoraproject.org/)
This has been written by Fedora's very own pingou and it satisfies
all immediate requirements that I know of (including FAS account
integration) and would offer the ability for us to fine-tune it for
our needs since it's effectively been developed "in house" but we
could also work to make it more popular in the broader community so
see if others would like to join and help add even more features to
it.
- kallithea (https://kallithea-scm.org/)
This project has been growing interest and to the best of my
knowledge is that it's slated towards being used officially by
upstream CPython developers at python.org, however some have mentioned
in #fedora-releng that there are missing features that we'd need/want
so this might require a bit of initial heavy lifting to add upstream.
(also no current support for FAS integration)
Alright, with all that said. I'd like to kind of round this back to a
set of requirements for the group in terms of the new workflow and
tooling:
I think everyone agrees that we need:
1) Code Review
2) Easy viewing/reference of code review history
3) Accountability for 1 & 2
If there are others, or things I have left out please let me know.
OK, that's it! If you made it this far, I owe you a cookie or something.
Please provide feedback, questions, and general snide remarks.
Thank you,
-AdamM
8 years, 11 months
#6161: soname version bump of gnutls,nettle in rawhide
by Fedora Release Engineering
#6161: soname version bump of gnutls,nettle in rawhide
------------------+------------------------
Reporter: nmav | Owner: rel-eng@…
Type: task | Status: new
Milestone: | Component: koji
Keywords: | Blocked By:
Blocking: |
------------------+------------------------
Hello,
I open this ticket due to Updates_Policy. I'd like to update nettle and
gnutls to their latest versions in rawhide. This would require a mass
rebuild of the packages shown by:
repoquery --whatrequires gnutls
repoquery --whatrequires nettle
Please advise on the process. I plan to update the packages next week.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/6161>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
8 years, 11 months
