12:16 p.m.
sinnykumari reported a new issue against the project: `releng` that you are following:
``
** Describe the issue**
Fedora CoreOS will have [various
streams](https://github.com/coreos/fedora-coreos-tracker/blob/master/stre....
All streams will use packages which are built in Fedora koji. Mechanical streams will use
packages available in Fedora repos from tags f$(release), f$(release)-updates,
f$(release)-updates-testing. Development and production streams will get built from
packages available under koji tag `coreos-pool`. Packages which make it to production will
also get tagged to `coreos-release` tag.
These tags will ensure that (1) packages are not automatically garbage collected (2)
stream builds are reproducible (up to the GC retention policy we agree upon), and (3)
packages are added to the pool (and thus into the production streams) in a controlled
manner.
To build FCOS for different streams we need:
- coreos-pool and coreos-release koji tags created (include arches aarch64, ppc64le,
x86_64)
- Permission to add packages to requested tags (ACLs for bgilbert, dustymabe, jlebon and
sinnykumari, until we have bot setup done)
- Tag builds needs to be signed
- Generate dist repo for requested tags with option `--non-latest` to koji dist-repo (to
perform build with desired NVRA)
**Question**
As we have streams like next-devel and testing-devel, packages will be from multiple
Fedora releases in coreos-pool and coreos-release tags. Since we will be using signed
packages, does koji dist-repo allows to specify multiple keys for generating dist repos?
This question is because, in [infra ansible
tag2distrepo](https://infrastructure.fedoraproject.org/cgit/ansible.git/t...,
so far single key has been used.
** When do you need this? (YYYY/MM/DD) **
ASAP
** When is this no longer needed or useful? (YYYY/MM/DD) **
Always needed
** If we cannot complete your request, what is the impact? **
we won't be able to build FCOS for different streams
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
12:18 p.m.
sinnykumari added a new comment to an issue you are following:
``
@ausil @rfonseca Do we need to include s390x here?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
12:25 p.m.
sinnykumari added a new comment to an issue you are following:
``
@ausil @rdossant Do we need to include s390x here?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
2:26 p.m.
dustymabe added a new comment to an issue you are following:
``
@ausil @rdossant Do we need to include s390x here?
I'd prefer to start with the three architectures we're already targeting and
succeed there before we add s390 personally. It will be hard enough just to pull it
together on those three. Once we work out the kinks then it shouldn't be a big deal
adding a 4th arch.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
8:17 a.m.
ausil added a new comment to an issue you are following:
``
@sinnykumari yes we need to, we are starting tow ork on s390x now in addition to aarch64
and ppc64le
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
9:12 a.m.
sinnykumari added a new comment to an issue you are following:
``
@mohanboddu Any additional info needed or we are good to proceed with this request?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
10:47 a.m.
mohanboddu added a new comment to an issue you are following:
``
This will be discussed in Fedora RelEng meeting tomorrow Apr 24 at 16:00 UTC
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
11:22 a.m.
kevin added a new comment to an issue you are following:
``
Some questions:
* You would tag all the packages for a compose into coreos-pool and all the ones in a
released compose into coreos-release. Could you perhaps just use fedora GA/fedora-updates
tags and only tag in the builds that are not in those and use those 2 repos on top of
coreos-pool/coreos-release? That would make them much smaller and more manageable. Of
course the fedora ones would be signed with fedora keys.
* What keys should these tags builds be signed with? If with the fedora ones, since they
aren't versioned how can we tell what key to use?
If we have to sign a large pile of packages with a new key often I am a bit worried at our
signing bandwith.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
11:37 a.m.
mohanboddu added a new comment to an issue you are following:
``
Can we use fxx-coreos-release and fxx-coreos-pool tags which follow a Fedora release and
can make use of same fedora release keys?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
11:39 a.m.
kevin added a new comment to an issue you are following:
``
Another thought we had was to version the tags? ie, coreos-pool-30, coreos-pool-31, etc...
and we could then just sign them with the fedora keys and just always make them when we
branch.
But is everything being signed by a different key a requrement?
Can you re-use the existing fedora GA/updates repos signed with the fedora key in addition
to these? or does it need to be one dist-repo?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
12:37 p.m.
dustymabe added a new comment to an issue you are following:
``
We just discussed this ticket in the releng meeting today. Here is what I think we decided
on:
- we create a `coreos-pool` tag - this is an aggregate of all the packages used to build
the various streams of FCOS. The package NVRAs for each build will be picked specifically
using input to the rpm-ostree build tooling.
- we create a `coreos-release` tag - this tag is simply for us to tag what rpms we have
actually shipped to end users in FCOS. There are no consumers of this tag. It's
purpose is to have a more conservative garbage collection policy on a smaller subset of
packages than what is in `coreos-pool`
- we create a `fxx-coreos-signing-pending` tag for each major version of fedora. When we
build backports rpms will go into this tag and robosignatory will then sign the rpms and
tag them into the `coreos-pool` tag and untag them from `fxx-coreos-signing-pending`.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
5:56 a.m.
sinnykumari added a new comment to an issue you are following:
``
Thanks all for the valuable input during [releng
meeting](https://meetbot-raw.fedoraproject.org/teams/releng/releng.2019-0...
! Unless we have any unanswered questions, can we get tags finalized in
https://pagure.io/releng/issue/8294#comment-567284 and dist repo for `coreos-pool` tags
with option `--non-latest` created?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
8:35 a.m.
mohanboddu added a new comment to an issue you are following:
``
@sinnykumari Can it wait until Wed?, since this requires some koji hub policy changes and
robosig changes, since we are in infra freeze we can make the changes once the freeze
lifts off.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
8:39 a.m.
sinnykumari added a new comment to an issue you are following:
``
Sure, thanks @mohanboddu
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
2:44 p.m.
mohanboddu added a new comment to an issue you are following:
``
Made the necessary changes to [koji hub
policy](https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/...
and [robosignatory
config](https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/...
and added the koji tags needed.
```
$ koji add-tag coreos-pool
$ koji add-tag coreos-pool --parent=coreos-release
$ koji add-tag f30-coreos-signing-pending --parent=coreos-pool
$ koji list-tag-inheritance coreos-release --reverse
coreos-release (8633)
└─coreos-pool (8632)
└─f30-coreos-signing-pending (8634)
```
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
3:21 p.m.
dustymabe added a new comment to an issue you are following:
``
hey @mohanboddu - thanks for working on this. Quick question related to the
implementation:
From the releng meeting (last week, I think) we discussed this and I think we had ended up
deciding to not use koji inheritance for any of these tags. My understanding is that we
should have `coreos-release`, `coreos-pool`, and `f30-coreos-signing-pending` all be
separate with no inheritance.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
9:19 a.m.
mohanboddu added a new comment to an issue you are following:
``
hey @mohanboddu - thanks for working on this. Quick question related
to the implementation:
From the releng meeting (last week, I think) we discussed this and I think we had ended
up deciding to not use koji inheritance for any of these tags. My understanding is that we
should have coreos-release, coreos-pool, and f30-coreos-signing-pending all be separate
with no inheritance.
@dustymabe The reason why I thought inheritance is, you dont need to add packages multiple
times. Once you add the package to coreos-release they will be available for all other
tags. And you can compose either from coreos-release or coreos-pool.
Now that I am thinking out loud, coreos-pool will pick all the builds in coreos-release,
which I think might not be desirable. Can you confirm this and I can remove the
inheritance between coreos-pool and coreos-release.
But I think we should leave the coreos-pool and fxx-coreos-signing-pending inheritance as
it is, since you can add a package to coreos-pool and it will be available in
fxx-core-signing-pending and if they are separate you need to add the package to both tags
and also we need to change koji hub permissions as well. Since fxx-coreos-signing-pending
is just a holding tag, it wont be of a problem.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
9:25 a.m.
dustymabe added a new comment to an issue you are following:
``
@dustymabe The reason why I thought inheritance is, you dont need to
add packages multiple times. Once you add the package to coreos-release they will be
available for all other tags. And you can compose either from coreos-release or
coreos-pool.
Now that I am thinking out loud, coreos-pool will pick all the builds
in coreos-release, which I think might not be desirable. Can you confirm this and I can
remove the inheritance between coreos-pool and coreos-release.
We aren't planning on composing from `coreos-release`. Just from `coreos-pool`. The
rationale for the need for `coreos-release` is in my earlier
comment](https://pagure.io/releng/issue/8294#comment-567284). Yes, please remove the
inheritance. Also note that we only need one distrepo set up and that is against
`coreos-pool`.
But I think we should leave the coreos-pool and
fxx-coreos-signing-pending inheritance as it is, since you can add a package to
coreos-pool and it will be available in fxx-core-signing-pending and if they are separate
you need to add the package to both tags and also we need to change koji hub permissions
as well. Since fxx-coreos-signing-pending is just a holding tag, it wont be of a problem.
My understanding is that the package goes to `fxx-coreos-signing-pending` when it is built
and then robosig moves it to `coreos-pool` when the package has been signed. I don't
think we need inheritance there? What would be the benefit?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
9:33 a.m.
mohanboddu added a new comment to an issue you are following:
``
We aren't planning on composing from coreos-release. Just from
coreos-pool. The rationale for the need for coreos-release is in my earlier comment. Yes,
please remove the inheritance. Also note that we only need one distrepo set up and that is
against coreos-pool.
I will remove the inheritance here.
My understanding is that the package goes to
fxx-coreos-signing-pending when it is built and then robosig moves it to coreos-pool when
the package has been signed. I don't think we need inheritance there? What would be
the benefit?
Without the inheritance, you need to add the package to both the coreos-pool and
fxx-coreos-signing-pending tags, just like you did
[here](https://pagure.io/releng/issue/8165#comment-560145) since both the tags have no
idea about the packages. Using inheritance, thats not needed and once the build gets
signed it will moved to coreos-pool tag from fxx-coreos-signing-pending tag.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
9:34 a.m.
mohanboddu added a new comment to an issue you are following:
``
Removed the tag inheritance between coreos-release and coreos-pool
```
$ koji remove-tag-inheritance coreos-pool coreos-release
```
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
9:52 a.m.
dustymabe added a new comment to an issue you are following:
``
Without the inheritance, you need to add the package to both the
coreos-pool and fxx-coreos-signing-pending tags, just like you did here since both the
tags have no idea about the packages. Using inheritance, thats not needed and once the
build gets signed it will moved to coreos-pool tag from fxx-coreos-signing-pending tag.
OK. One or two questions here:
- is there any chance for an unsigned build to get into the resulting dist-repo?
- once we have a `f31-coreos-signing-pending` will this break? We need this setup to work
equally for `f30-coreos-signing-pending` and `f31-coreos-signing-pending` at the same
time.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
10:11 a.m.
mohanboddu added a new comment to an issue you are following:
``
OK. One or two questions here:
is there any chance for an unsigned build to get into the resulting dist-repo?
Nope
once we have a f31-coreos-signing-pending will this break? We need
this setup to work equally for f30-coreos-signing-pending and f31-coreos-signing-pending
at the same time.
I think this will make a perfect choice then, since we can add f31-coreos-signing-pending
tag to the inheritance and both f30-c-s-p and f31-c-s-p will work.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
10:12 a.m.
dustymabe added a new comment to an issue you are following:
``
ok Thanks @mohanboddu !
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
11:02 a.m.
dustymabe added a new comment to an issue you are following:
``
@mohanboddu - can you link me to the distrepo for `coreos-pool` once you've set it
up?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
12:36 p.m.
mohanboddu added a new comment to an issue you are following:
``
@mohanboddu - can you link me to the distrepo for coreos-pool once
you've set it up?
@dustymabe What do you mean? You can tag a build to f30-coreos-signing-pending, and once
signed, it will be available in coreos-pool tag and I think currently tag2distrepo is
still not working, so after tagging a build please let me know and I can run the koji
dist-repo for you.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
12:44 p.m.
sinnykumari added a new comment to an issue you are following:
``
@mohanboddu - can you link me to the distrepo for coreos-pool once you've set it
up?
@dustymabe What do you mean? You can tag a build to f30-coreos-signing-pending, and once
signed, it will be available in coreos-pool tag and I think currently tag2distrepo is
still not working, so after tagging a build please let me know and I can run the koji
dist-repo for you.
As per @mizdebsk
[FBR](https://lists.fedoraproject.org/archives/list/infrastructure@lists.f...,
automatic dist-repo generation with tag2distrepo doesn't work only in case of unsigned
packages get tagged. Since, coreos-pool uses signed packages, it should be fine.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
1:14 p.m.
mohanboddu added a new comment to an issue you are following:
``
As per @mizdebsk FBR, automatic dist-repo generation with
tag2distrepo doesn't work only in case of unsigned packages get tagged. Since,
coreos-pool uses signed packages, it should be fine.
Ah nvm then, this
[patch](https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/r...
should fix this and I completed running the playbook.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
1:39 p.m.
jlebon added a new comment to an issue you are following:
``
Just to clear up re. that patch,
```
+ 'coreos-pool': [cfc659b9],
```
When we're going to be sourcing from both f30 and f31 versions of the signing pending
tags, we'd just have to list both keys here, right?
(Also shouldn't that keyid be in quotes?)
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
1:42 p.m.
mohanboddu added a new comment to an issue you are following:
``
Just to clear up re. that patch,
+ 'coreos-pool': [cfc659b9],
When we're going to be sourcing from both f30 and f31 versions of the signing pending
tags, we'd just have to list both keys here, right?
Yes
(Also shouldn't that keyid be in quotes?)
Thanks, I will rerun the playbook fixing it :disappointed:
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
4:33 a.m.
sinnykumari added a new comment to an issue you are following:
``
To check necessary permissions, I did:
```
$ koji add-pkg coreos-pool rpm-ostree --owner sinnykumari
$ koji tag-build coreos-pool rpm-ostree-2019.3-1.fc30
```
It went fine, but I don't see a coreos-pool dist-repo being created at
https://kojipkgs.fedoraproject.org/repos-dist/ . What are we missing here?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
3:10 p.m.
kevin added a new comment to an issue you are following:
``
I restarted fedmsg-hub-3 on bodhi-backend02, it looks like it might not have picked up the
new config yet.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
3:57 p.m.
mohanboddu added a new comment to an issue you are following:
``
Added the arches to coreos-pool tag
```
$ koji edit-tag coreos-pool --arches='aarch64 ppc64le s390x x86_64'
```
I am guessing its not required for coreos-release since it will just hold the builds from
the released FCOS. Please let me know if I am wrong.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
3:58 p.m.
dustymabe added a new comment to an issue you are following:
``
https://kojipkgs.fedoraproject.org/repos-dist/coreos-pool/latest/
thanks all who helped!
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
5:21 a.m.
sinnykumari added a new comment to an issue you are following:
``
Awesome, thanks all!
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
12:34 p.m.
sinnykumari added a new comment to an issue you are following:
``
Did few more checks to make sure we have everything inplace:
- coreos-pool dist-repo allows to have multiple version of same package. Added
openssh-7.9p1-5.fc30 and openssh-8.0p1-1.fc30 and we do have packages from both nvr
available in latst generated dist-repo.
- Can add packages to coreos-release tag (tested by adding rpm-ostree)
- Can tag-build and untag-build on specific pakcages in coreos-release tag (tested with
rpm-ostree-2019.3-1.fc30)
I think we have all requested items in this ticket done. Did we miss anything before we
close this ticket?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
12:39 p.m.
mohanboddu added a new comment to an issue you are following:
``
Added the arches to coreos-pool tag
$ koji edit-tag coreos-pool --arches='aarch64 ppc64le s390x x86_64'
I am guessing its not required for coreos-release since it will just hold the builds from
the released FCOS. Please let me know if I am wrong.
@sinnykumari Do you know answer for this ^?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
12:41 p.m.
dustymabe added a new comment to an issue you are following:
``
Did we miss anything before we close this ticket?
One thing that is missing is special permissions surrounding tagging for the kernel. Only
people who are allowed to build the kernel can tag it apparently. We need to be able to
tag the kernel into our various tags using a bot. How can we achieve this goal?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
12:45 p.m.
sinnykumari added a new comment to an issue you are following:
``
Added the arches to coreos-pool tag
$ koji edit-tag coreos-pool --arches='aarch64 ppc64le s390x x86_64'
I am guessing its not required for coreos-release since it will just hold the builds from
the released FCOS. Please let me know if I am wrong.
That's correct, we don't need this since coreos-release tag is only to tag
specific nvr builds which will make it to the FCOS release. we won't creating a
dist-repo out of it.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
9:16 a.m.
mohanboddu added a new comment to an issue you are following:
``
Did we miss anything before we close this ticket?
One thing that is missing is special permissions surrounding tagging for the kernel. Only
people who are allowed to build the kernel can tag it apparently. We need to be able to
tag the kernel into our various tags using a bot. How can we achieve this goal?
@dustymabe This
[commit](https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/...
should fix that issue.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
9:57 a.m.
dustymabe added a new comment to an issue you are following:
``
It worked!
```
[dustymabe@hattop ~]$ koji tag-build coreos-pool kernel-5.0.11-300.fc30
Created task 34767868
Watching tasks (this may be safely interrupted)...
34767868 tagBuild (noarch): free
34767868 tagBuild (noarch): free -> closed
0 free 0 open 1 done 0 failed
34767868 tagBuild (noarch) completed successfully
```
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
12:51 p.m.
mohanboddu added a new comment to an issue you are following:
``
We think we got everything we need here.
Closing the ticket.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
12:51 p.m.
The status of the issue: `coreos-pool and coreos-release koji tags request for FCOS` of
project: `releng` has been updated to: Closed as Fixed by mohanboddu.
https://pagure.io/releng/issue/8294
2:03 p.m.
dustymabe added a new comment to an issue you are following:
``
A couple additional requests:
- can we get a few more signing-pending tags (maybe we can add this to a branching SOP
somewhere) that have the same setup (inherits from `coreos-pool`) as the
`f30-coreos-signing-pending` tag?
- `f29-coreos-signing-pending`
- `f31-coreos-signing-pending`
- right now the distrepo config for `coreos-pool` has the f29 and f30 keys. Can we add the
f31 key? Should be able to update with: `koji edit-tag coreos-pool -x
tag2distrepo.keys="429476b4 cfc659b9 3c3359c4"`
```
$ koji taginfo coreos-pool
Tag: coreos-pool [8632]
Arches: aarch64 ppc64le s390x x86_64
Groups:
Tag options:
tag2distrepo.enabled : 'true'
tag2distrepo.keys : '429476b4 cfc659b9'
Inheritance:
```
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
2:14 p.m.
mohanboddu added a new comment to an issue you are following:
``
Done.
```
$ koji add-tag f29-coreos-signing-pending --parent=coreos-pool
$ koji add-tag f31-coreos-signing-pending --parent=coreos-pool
$ koji edit-tag coreos-pool -x tag2distrepo.keys="429476b4 cfc659b9 3c3359c4"
$ koji taginfo coreos-pool
Tag: coreos-pool [8632]
Arches: aarch64 ppc64le s390x x86_64
Groups:
Tag options:
tag2distrepo.enabled : 'true'
tag2distrepo.keys : '429476b4 cfc659b9 3c3359c4'
Inheritance:
```
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
2:43 p.m.
mohanboddu added a new comment to an issue you are following:
``
robosig change commit -
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?h=maste...
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
2:45 p.m.
dustymabe added a new comment to an issue you are following:
``
robosig change commit -
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?h=maste...
Thanks @mohanboddu. @kevin - can I get you to run the playbook that would get that change
to apply?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
12:52 p.m.
kevin added a new comment to an issue you are following:
``
Done.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
12:54 p.m.
dustymabe added a new comment to an issue you are following:
``
❤️
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
10:35 a.m.
dustymabe added a new comment to an issue you are following:
``
now that there is a key for f32 can we get the `coreos-pool` tag updated to support the
new key:
```
$ koji edit-tag coreos-pool -x tag2distrepo.keys="429476b4 cfc659b9 3c3359c4
12c944d0"
```
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
10:36 a.m.
kevin added a new comment to an issue you are following:
``
Done.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
10:37 a.m.
dustymabe added a new comment to an issue you are following:
``
```
$ koji taginfo coreos-pool
Tag: coreos-pool [8632]
Arches: aarch64 ppc64le s390x x86_64
Groups:
Tag options:
tag2distrepo.enabled : 'true'
tag2distrepo.keys : '429476b4 cfc659b9 3c3359c4 12c944d0'
Inheritance:
```
Thanks @kevin:
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8294
1715
days inactive
1834
days old
rel-eng@lists.fedoraproject.org
50 comments
6 participants
participants (6)
-
Dennis Gilmore -
Dusty Mabe
-
Jonathan Lebon
-
Kevin Fenzi
-
Mohan Boddu
-
Sinny Kumari