infrastructure March 2018

infrastructure@lists.fedoraproject.org

38 participants
74 discussions

Freeze Break Request: update koji and oz_timeout on buildvm-s390x-02
by Kevin Fenzi 19 Mar '18

19 Mar '18

Greetings. image composes using oz take a long time on our s390x image builder, (due to them being in another datacenter). They take so long, they timeout. See: https://pagure.io/koji/pull-request/837 from sinny and https://pagure.io/koji/pull-request/841 which makes this value configurable in kojid.conf I would like to: 1. Update koji on buildvm-s390x-02 to the version I just built in rawhide/f28 that has a backport of the patch that makes this value configurable. 2. Apply the following ansible patch to set the timeout only on the buildvm-s390x-02 builder: diff --git a/roles/koji_builder/templates/kojid.conf b/roles/koji_builder/templates/kojid.conf index 832d80c..9f10d9c 100644 --- a/roles/koji_builder/templates/kojid.conf +++ b/roles/koji_builder/templates/kojid.conf @@ -20,7 +20,12 @@ maxjobs=25 keepalive=False rpmbuild_timeout=172800 - +{% if ansible_hostname.startswith('buildvm-s390x-02') %} +; Set oz timeout higher on x390x image builder to allow it to finish. +; Install timeout(seconds) for image build +; if it's unset, use the number in /etc/oz/oz.cfg, supported since oz-0.16.0 +oz_install_timeout=14400 +{% endif %} use_createrepo_c=True {% if host in groups['buildvm-s390x'] %} 3. Run the playbook on it to update the config and restart kojid. +1s? kevin

5 7

Goodbye nvr.rsplit('-', 2), hello modularity
by Patrick Uiterwijk 19 Mar '18

19 Mar '18

Hi all, If you maintain any application in Fedora Infra (or outside) that tries to parse things out of RPM names, you might be interested in this. For those wondering where I've been spending most of my time the last few weeks: I've been deep in the internals of Bodhi fixing all kinds of issues I've found between it and Modularity (turns out enabling composing was just the beginning). At some point, I was informed that for modules, they are throwing all kinds of old assumptions out of the door. One of those is the good old Name-Version-Release in RPM land. Modules do not have an NVR, they have an NSVC: Name, Stream, Version, Context. Name is the name of the module ("nodejs"). Stream is a stream, like "6", "7" or "8", these are crucial for the parallel availability that modules are trying to accomplish. Version is a date/time stamp, used to indicate new versions in the same name-stream. Context is a hash of some information in the built module that makes it so that the nodejs-6-20170314 built for Fedora 28 has a different identifier than the one built for Fedora 29. There are some additional fields for installed things (arch and profile), but those aren't really important for things trying to parse/show module names I think. One of the interesting things they wanted to also allow: dashes in streams. As a consequence, when you get an N-S-V.C as modules are represented in Koji builds, doing a .rsplit('-', 2) will not give you Name, Stream, Version.Context per se. You could totally have a module called nodejs-my-stream-5-20170314.abcd, with name=nodejs, stream=my-stream-5, version=20170314, context=abcd. There is no way for you to independently figure out what the NSVC components are, you will need to ask Koji, and use its name, version and release fields (with name=name, version=stream, release=version(.context)). Also as a consequence, modules should not be dash-separated in anything the users see. For user consumption, they are Name:Stream:Version:Context, so you may need to manually convert between one representation and the other if you need to look up in koji or other systems versus display to users. Also, note that if you are touching older modules in Koji (e.g. because you go through all koji builds), modules before February 19th will be lacking the Context field. However, I've been told that those modules will not be used, so unless you or your app goes spelunking in koji (for fun and profit), you should never see those. I hope that this is useful information for anyone else finding themselves having to parse NVRs/NSVs. Regards, Patrick

4 3

Freeze break request: refresh fedmsg crl.pem
by Kevin Fenzi 18 Mar '18

18 Mar '18

Greetings. Our current crl.pem for fedmsg (cert revocation list) expires on monday. I'd like to regenerate it with a expire time out 6 months and push it out to all hosts that use fedmsg, hopefully asap. So, this is: * Generate new crl.pem for fedmsg in ansible-private. * Run proxy playbooks with -t fedmsg/proxy to copy the new crl.pem to https://fedoraproject.org/fedmsg/crl.pem * fedmsg using services should download it and use it. +1s? kevin

3 2

Freeze Break Request: fix git hooks on new packages
by Kevin Fenzi 16 Mar '18

16 Mar '18

Greetings. It seems the git-check-perms cron job is still not 100% right. ;) Currently it's running as 'nobody' and thus doesn't have perms to setup right right hooks. ;( I think the following should fix it (also removes a MAILTO that doesn't matter anymore). diff --git a/roles/gitolite/check_fedmsg_hooks/tasks/main.yml b/roles/gitolite/check_fedmsg_hooks/tasks/main.yml index a22018e..51bff93 100644 --- a/roles/gitolite/check_fedmsg_hooks/tasks/main.yml +++ b/roles/gitolite/check_fedmsg_hooks/tasks/main.yml @@ -7,8 +7,8 @@ cron_file=ansible-git-check-perms minute=10 hour="0, 12" - user=nobody - job="MAILTO=root /usr/local/bin/git-check-perms /srv/git/repositories --check=fedmsg-hook -f" + user=root + job="/usr/local/bin/git-check-perms /srv/git/repositories --check=fedmsg-hook -f" tags: - git - gitolite +1s? kevin

3 3

FBR: Update download servers to new kernel.org mirrors
by Stephen John Smoogen 16 Mar '18

16 Mar '18

Add the following to the tier1 mirrors [smooge@batcave01 ansible (master)]$ git diff origin/master diff --git a/inventory/group_vars/download b/inventory/group_vars/download index 36b605e..e754529 100644 --- a/inventory/group_vars/download +++ b/inventory/group_vars/download @@ -66,3 +66,6 @@ dl_tier1: - ultra.linux.cz # 195.113.15.27 - wpi.edu # 130.215.36.26 - zaphod.gtlib.gatech.edu # 128.61.111.12 + - sjc.edge.kernel.org # 147.75.69.165 + - ewr.edge.kernel.org # 147.75.197.195 + - ams.edge.kernel.org # 147.75.101.1 -- Stephen J Smoogen.

3 2

FBR: To update pungi on branched-composer.phx2.fedoraproject.org and rawhide-composer.phx2.fedoraproject.org to pungi-4.1.22-10.fc27
by Mohan Boddu 16 Mar '18

16 Mar '18

pungi-4.1.22-10 should finally fix the issue with modular rpms getting into Everything repo Pungi Bug: https://pagure.io/pungi/issue/862 Fix: https://pagure.io/pungi/issue/862#comment-499692 Koji Build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1059055

3 2

Weekly Infrastructure Meeting 2018-03-15
by Stephen John Smoogen 14 Mar '18

14 Mar '18

Join us during the Ides of March as we greet Juliu.. oh sorry different meeting This shared document is for the next fedora infrastructure meeting. = Preamble = The infrastructure team will be having its weekly meeting tomorrow, 2018-03-14 at 18:00 UTC in #fedora-meeting on the freenode network. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of today is included in this email. If you have something to discuss, add the topic to the discussion area with your name. If you would like to teach other folks about some application or setup in our infrastructure, please add that topic and your name to the learn about section. = Introduction = We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2018-03-14) #meetingname infrastructure #topic aloha #chair smooge relrod nirik pingou puiterwijk tflink = Let new people say hello = #topic New folks introductions #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves = Status / Information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #topic announcements and information #info Infrastructure hackfest in april: https://fedoraproject.org/wiki/Infrastructure_Hackathon_2018 #info We are in F28 Beta Freeze! All changes to frozen hosts must get +2 on list before being applied - kevin #info = Things we should discuss = We use this section to bring up discussion topics. Things we want to talk about as a group and come up with some consensus /suor decision or just brainstorm a problem or issue. If there are none of these we skip this section. (Use #topic your discussion topic - your username) #topic Ticket cleanup #info none this week. #topic Turning off pkgdb #info It has been a depracated service but some things still refer to it #info what needs to be done to finish it off? #topic Darkserver and tagger retirements #info next steps/action items? = Apprentice office hours = #topic Apprentice Open office minutes #info A time where apprentices may ask for help or look at problems. Here we will discuss any apprentice questions, try and match up people looking for things to do with things to do, progress, testing anything like that. = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the i/nfo in this section. In the event we don't find someone to teach about something, we skip this section and just move on to open floor.) #topic Learn about: #info none this week = Meeting end stuff = #topic Open Floor #endmeeting -- Stephen J Smoogen.

1 0

FBR - OSBS authoritative registry
by Clement Verna 14 Mar '18

14 Mar '18

Hi, We currently set the authoritative registry value to registry.example.com this patch fixes that to set it to registry.fp.o. This is closing the following tickets [0][1] +1 to apply ? [0] - https://pagure.io/atomic-wg/issue/272 [1] - https://pagure.io/releng/issue/6780 --- playbooks/groups/buildvm.yml | 4 ++-- playbooks/groups/osbs-cluster.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/playbooks/groups/buildvm.yml b/playbooks/groups/buildvm.yml index e6d80eb0f..7343a2f9b 100644 --- a/playbooks/groups/buildvm.yml +++ b/playbooks/groups/buildvm.yml @@ -99,7 +99,7 @@ koji_hub: 'https://koji.stg.fedoraproject.org/kojihub', sources_command: 'fedpkg sources', build_type: 'prod', - authoritative_registry: 'registry.example.com', + authoritative_registry: 'registry.fedoraproject.org', vendor: 'Fedora Project', verify_ssl: true, use_auth: true, @@ -131,7 +131,7 @@ koji_hub: 'https://koji.fedoraproject.org/kojihub', sources_command: 'fedpkg sources', build_type: 'prod', - authoritative_registry: 'registry.example.com', + authoritative_registry: 'registry.fedoraproject.org', vendor: 'Fedora Project', verify_ssl: true, use_auth: true, diff --git a/playbooks/groups/osbs-cluster.yml b/playbooks/groups/osbs-cluster.yml index 39105b03b..476934d2a 100644 --- a/playbooks/groups/osbs-cluster.yml +++ b/playbooks/groups/osbs-cluster.yml @@ -731,7 +731,7 @@ koji_hub: 'https://{{koji_url}}/kojihub', sources_command: 'fedpkg sources', build_type: 'prod', - authoritative_registry: 'registry.example.com', + authoritative_registry: 'registry.fedoraproject.org', vendor: 'Fedora Project', verify_ssl: true, use_auth: true, @@ -763,7 +763,7 @@ koji_hub: 'https://{{koji_url}}/kojihub', sources_command: 'fedpkg sources', build_type: 'prod', - authoritative_registry: 'registry.example.com', + authoritative_registry: 'registry.fedoraproject.org', vendor: 'Fedora Project', verify_ssl: true, use_auth: true, --

3 4

FBR: To update pungi on branched-composer.phx2.fedoraproject.org and rawhide-composer.phx2.fedoraproject.org to pungi-4.1.22-9.fc27
by Mohan Boddu 14 Mar '18

14 Mar '18

This will fix the issue with empty module variant and also skipping buildinstall phase on variants. Koji build - https://koji.fedoraproject.org/koji/buildinfo?buildID=1058071

3 2

FBR: Fix the trigger upload script in to create AMIs manually
by Sayan Chowdhury 13 Mar '18

13 Mar '18

Hi, fedimg contains a trigger upload script that helps in uploading the raw images in case the upload fails. This patch fixes the script to use the third parameter in the function that would push the compose metadata. +1s? From: Sayan Chowdhury <sayan.chowdhury2012(a)gmail.com> Date: Tue, 13 Mar 2018 17:07:53 +0530 Subject: [PATCH 1/1] fedimg: Fix the trigger_upload script to create AMIs manually --- roles/fedimg/files/trigger_upload.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/roles/fedimg/files/trigger_upload.py b/roles/fedimg/files/trigger_upload.py index 5a5208d..29907f1 100644 --- a/roles/fedimg/files/trigger_upload.py +++ b/roles/fedimg/files/trigger_upload.py @@ -16,7 +16,7 @@ from fedimg.services.ec2 import EC2Service, EC2ServiceException import fedimg.uploader from fedimg.util import virt_types_from_url -if len(sys.argv) != 2: +if len(sys.argv) != 3: print 'Usage: trigger_upload.py <rawxz_image_url>' sys.exit(1) @@ -26,5 +26,10 @@ log = logging.getLogger('fedmsg') upload_pool = multiprocessing.pool.ThreadPool(processes=4) url = sys.argv[1] +compose_id = sys.argv[2] -fedimg.uploader.upload(upload_pool, [url]) +compose_meta = { + 'compose_id': compose_id +} + +fedimg.uploader.upload(upload_pool, [url], compose_meta=compose_meta) -- 2.9.4 -- Sayan Chowdhury <https://sayanchowdhury.dgplug.org/> Senior Software Engineer, Fedora Engineering - Emerging Platform GPG Fingerprint : 0F16 E841 E517 225C 7D13 AB3C B023 9931 9CD0 5C8B

3 4

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure March 2018