Goodbye nvr.rsplit('-', 2), hello modularity
by Patrick Uiterwijk
Hi all,
If you maintain any application in Fedora Infra (or outside) that
tries to parse things out of RPM names, you might be interested in
this.
For those wondering where I've been spending most of my time the last
few weeks: I've been deep in the internals of Bodhi fixing all kinds
of issues I've found between it and Modularity (turns out enabling
composing was just the beginning).
At some point, I was informed that for modules, they are throwing all
kinds of old assumptions out of the door.
One of those is the good old Name-Version-Release in RPM land.
Modules do not have an NVR, they have an NSVC: Name, Stream, Version, Context.
Name is the name of the module ("nodejs").
Stream is a stream, like "6", "7" or "8", these are crucial for the
parallel availability that modules are trying to accomplish.
Version is a date/time stamp, used to indicate new versions in the
same name-stream.
Context is a hash of some information in the built module that makes
it so that the nodejs-6-20170314 built for Fedora 28 has a different
identifier than the one built for Fedora 29.
There are some additional fields for installed things (arch and
profile), but those aren't really important for things trying to
parse/show module names I think.
One of the interesting things they wanted to also allow: dashes in streams.
As a consequence, when you get an N-S-V.C as modules are represented
in Koji builds, doing a .rsplit('-', 2) will not give you Name,
Stream, Version.Context per se.
You could totally have a module called
nodejs-my-stream-5-20170314.abcd, with name=nodejs,
stream=my-stream-5, version=20170314, context=abcd.
There is no way for you to independently figure out what the NSVC
components are, you will need to ask Koji, and use its name, version
and release fields (with name=name, version=stream,
release=version(.context)).
Also as a consequence, modules should not be dash-separated in
anything the users see.
For user consumption, they are Name:Stream:Version:Context, so you may
need to manually convert between one representation and the other if
you need to look up in koji or other systems versus display to users.
Also, note that if you are touching older modules in Koji (e.g.
because you go through all koji builds), modules before February 19th
will be lacking the Context field.
However, I've been told that those modules will not be used, so unless
you or your app goes spelunking in koji (for fun and profit), you
should never see those.
I hope that this is useful information for anyone else finding
themselves having to parse NVRs/NSVs.
Regards,
Patrick
5 years, 6 months
Freeze break request: refresh fedmsg crl.pem
by Kevin Fenzi
Greetings.
Our current crl.pem for fedmsg (cert revocation list) expires on monday.
I'd like to regenerate it with a expire time out 6 months and push it
out to all hosts that use fedmsg, hopefully asap.
So, this is:
* Generate new crl.pem for fedmsg in ansible-private.
* Run proxy playbooks with -t fedmsg/proxy to copy the new crl.pem to
https://fedoraproject.org/fedmsg/crl.pem
* fedmsg using services should download it and use it.
+1s?
kevin
5 years, 6 months
Freeze Break Request: fix git hooks on new packages
by Kevin Fenzi
Greetings.
It seems the git-check-perms cron job is still not 100% right. ;)
Currently it's running as 'nobody' and thus doesn't have perms to setup
right right hooks. ;(
I think the following should fix it (also removes a MAILTO that doesn't
matter anymore).
diff --git a/roles/gitolite/check_fedmsg_hooks/tasks/main.yml
b/roles/gitolite/check_fedmsg_hooks/tasks/main.yml
index a22018e..51bff93 100644
--- a/roles/gitolite/check_fedmsg_hooks/tasks/main.yml
+++ b/roles/gitolite/check_fedmsg_hooks/tasks/main.yml
@@ -7,8 +7,8 @@
cron_file=ansible-git-check-perms
minute=10
hour="0, 12"
- user=nobody
- job="MAILTO=root /usr/local/bin/git-check-perms
/srv/git/repositories --check=fedmsg-hook -f"
+ user=root
+ job="/usr/local/bin/git-check-perms /srv/git/repositories
--check=fedmsg-hook -f"
tags:
- git
- gitolite
+1s?
kevin
5 years, 6 months
VDO
by Miroslav Suchý
After I seen the talk about VDO:
https://www.youtube.com/watch?v=7CGr5LEAfRY
I went ahead and tried it.
I tried it on small (12GB) sample of Copr data and I saved 20-30% data.
I then deployed it on production server retrace.fedoraproject.org and saved there 15% out of 2TB.
Several notes: The RPM packages are only available for RHEL. Recently the public github has been updated with version of
VDO, which should work on Fedora. No RPM available for Fedora yet though.
Here are my notes:
# vdo create --name=vdo1 --device=/dev/vdb --vdoLogicalSize=1T
# mkfs.ext4 /dev/mapper/vdo1
Some slides suggest to use -E discard. This is not needed for new volumes. See vdo-devel mailing list for reasoning.
# mount -o discard /dev/mapper/vdo1 /mnt/a
Probably best scenario is to create vdo on whole disk and then make PV from whole VDO disk and put LVM on top of that.
When you already have data, you cannot convert your disk to VDO. You need create new VDO disk, format it, mount it and
copy the data.
In my case, due limited space for migration I created a small LVM volume "vdosrv":
# lvcreate -L 80G -n vdosrv vol0
I converted it to VDO which claims to have 2TB of vitual total size
# vdo create --name=srv18 --device=/dev/vol0/vdosrv --vdoLogicalSize=2T
# mkfs.ext4 /dev/mapper/srv18
See how many space is *actually* free.
# vdostats
# mkdir /mnt/srv18
# mount -o discard /dev/mapper/srv18 /mnt/srv18
Now I moved some data from old volume to /mnt/srv18. I shrank the FS of old volume, I shrank the old LV. And then I
enlarged the new LV.
# lvresize -L +400G /dev/vol0/vdosrv
and let know vdo that it can use the new space:
# vdo growPhysical -n srv18
I repeated this several times.
I have to say that my experience is so far good.
I hope that my notes help someone.
Miroslav
5 years, 6 months
Weekly Infrastructure Meeting 2018-03-15
by Stephen John Smoogen
Join us during the Ides of March as we greet Juliu.. oh sorry different meeting
This shared document is for the next fedora infrastructure meeting.
= Preamble =
The infrastructure team will be having its weekly meeting tomorrow,
2018-03-14 at 18:00 UTC in #fedora-meeting on the freenode network.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
= Introduction =
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2018-03-14)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik pingou puiterwijk tflink
= Let new people say hello =
#topic New folks introductions
#info This is a place where people who are interested in Fedora
Infrastructure can introduce themselves
= Status / Information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need
to discuss)
(Please use #info <the thing> - your name)
#topic announcements and information
#info Infrastructure hackfest in april:
https://fedoraproject.org/wiki/Infrastructure_Hackathon_2018
#info We are in F28 Beta Freeze! All changes to frozen hosts must get
+2 on list before being applied - kevin
#info
= Things we should discuss =
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus /suor decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)
#topic Ticket cleanup
#info none this week.
#topic Turning off pkgdb
#info It has been a depracated service but some things still refer to it
#info what needs to be done to finish it off?
#topic Darkserver and tagger retirements
#info next steps/action items?
= Apprentice office hours =
#topic Apprentice Open office minutes
#info A time where apprentices may ask for help or look at problems.
Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.
= Learn about some application or setup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for
improvement,
etc. Whoever would like to do this, just add the i/nfo in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)
#topic Learn about:
#info none this week
= Meeting end stuff =
#topic Open Floor
#endmeeting
--
Stephen J Smoogen.
5 years, 6 months
FBR - OSBS authoritative registry
by Clement Verna
Hi,
We currently set the authoritative registry value to
registry.example.com this patch fixes that to set it to registry.fp.o.
This is closing the following tickets [0][1]
+1 to apply ?
[0] - https://pagure.io/atomic-wg/issue/272
[1] - https://pagure.io/releng/issue/6780
---
playbooks/groups/buildvm.yml | 4 ++--
playbooks/groups/osbs-cluster.yml | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/playbooks/groups/buildvm.yml b/playbooks/groups/buildvm.yml
index e6d80eb0f..7343a2f9b 100644
--- a/playbooks/groups/buildvm.yml
+++ b/playbooks/groups/buildvm.yml
@@ -99,7 +99,7 @@
koji_hub: 'https://koji.stg.fedoraproject.org/kojihub',
sources_command: 'fedpkg sources',
build_type: 'prod',
- authoritative_registry: 'registry.example.com',
+ authoritative_registry: 'registry.fedoraproject.org',
vendor: 'Fedora Project',
verify_ssl: true,
use_auth: true,
@@ -131,7 +131,7 @@
koji_hub: 'https://koji.fedoraproject.org/kojihub',
sources_command: 'fedpkg sources',
build_type: 'prod',
- authoritative_registry: 'registry.example.com',
+ authoritative_registry: 'registry.fedoraproject.org',
vendor: 'Fedora Project',
verify_ssl: true,
use_auth: true,
diff --git a/playbooks/groups/osbs-cluster.yml
b/playbooks/groups/osbs-cluster.yml
index 39105b03b..476934d2a 100644
--- a/playbooks/groups/osbs-cluster.yml
+++ b/playbooks/groups/osbs-cluster.yml
@@ -731,7 +731,7 @@
koji_hub: 'https://{{koji_url}}/kojihub',
sources_command: 'fedpkg sources',
build_type: 'prod',
- authoritative_registry: 'registry.example.com',
+ authoritative_registry: 'registry.fedoraproject.org',
vendor: 'Fedora Project',
verify_ssl: true,
use_auth: true,
@@ -763,7 +763,7 @@
koji_hub: 'https://{{koji_url}}/kojihub',
sources_command: 'fedpkg sources',
build_type: 'prod',
- authoritative_registry: 'registry.example.com',
+ authoritative_registry: 'registry.fedoraproject.org',
vendor: 'Fedora Project',
verify_ssl: true,
use_auth: true,
--
5 years, 6 months
FBR: Fix the trigger upload script in to create AMIs manually
by Sayan Chowdhury
Hi,
fedimg contains a trigger upload script that helps in uploading the
raw images in case the upload fails. This patch fixes the script to
use the third parameter in the function that would push the compose
metadata.
+1s?
From: Sayan Chowdhury <sayan.chowdhury2012(a)gmail.com>
Date: Tue, 13 Mar 2018 17:07:53 +0530
Subject: [PATCH 1/1] fedimg: Fix the trigger_upload script to create AMIs
manually
---
roles/fedimg/files/trigger_upload.py | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/roles/fedimg/files/trigger_upload.py
b/roles/fedimg/files/trigger_upload.py
index 5a5208d..29907f1 100644
--- a/roles/fedimg/files/trigger_upload.py
+++ b/roles/fedimg/files/trigger_upload.py
@@ -16,7 +16,7 @@ from fedimg.services.ec2 import EC2Service,
EC2ServiceException
import fedimg.uploader
from fedimg.util import virt_types_from_url
-if len(sys.argv) != 2:
+if len(sys.argv) != 3:
print 'Usage: trigger_upload.py <rawxz_image_url>'
sys.exit(1)
@@ -26,5 +26,10 @@ log = logging.getLogger('fedmsg')
upload_pool = multiprocessing.pool.ThreadPool(processes=4)
url = sys.argv[1]
+compose_id = sys.argv[2]
-fedimg.uploader.upload(upload_pool, [url])
+compose_meta = {
+ 'compose_id': compose_id
+}
+
+fedimg.uploader.upload(upload_pool, [url], compose_meta=compose_meta)
--
2.9.4
--
Sayan Chowdhury <https://sayanchowdhury.dgplug.org/>
Senior Software Engineer, Fedora Engineering - Emerging Platform
GPG Fingerprint : 0F16 E841 E517 225C 7D13 AB3C B023 9931 9CD0 5C8B
5 years, 6 months
