sinnykumari reported a new issue against the project: `releng` that you are following: `` ** Describe the issue**
Fedora CoreOS will have [various streams](https://github.com/coreos/fedora-coreos-tracker/blob/master/stream-tooling.m...). All streams will use packages which are built in Fedora koji. Mechanical streams will use packages available in Fedora repos from tags f$(release), f$(release)-updates, f$(release)-updates-testing. Development and production streams will get built from packages available under koji tag `coreos-pool`. Packages which make it to production will also get tagged to `coreos-release` tag.
These tags will ensure that (1) packages are not automatically garbage collected (2) stream builds are reproducible (up to the GC retention policy we agree upon), and (3) packages are added to the pool (and thus into the production streams) in a controlled manner.
To build FCOS for different streams we need: - coreos-pool and coreos-release koji tags created (include arches aarch64, ppc64le, x86_64) - Permission to add packages to requested tags (ACLs for bgilbert, dustymabe, jlebon and sinnykumari, until we have bot setup done) - Tag builds needs to be signed - Generate dist repo for requested tags with option `--non-latest` to koji dist-repo (to perform build with desired NVRA)
**Question**
As we have streams like next-devel and testing-devel, packages will be from multiple Fedora releases in coreos-pool and coreos-release tags. Since we will be using signed packages, does koji dist-repo allows to specify multiple keys for generating dist repos? This question is because, in [infra ansible tag2distrepo](https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/bodhi2/...), so far single key has been used.
** When do you need this? (YYYY/MM/DD) **
ASAP
** When is this no longer needed or useful? (YYYY/MM/DD) **
Always needed
** If we cannot complete your request, what is the impact? **
we won't be able to build FCOS for different streams ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
sinnykumari added a new comment to an issue you are following: `` @ausil @rfonseca Do we need to include s390x here? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
sinnykumari added a new comment to an issue you are following: ``
@ausil @rdossant Do we need to include s390x here?
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
dustymabe added a new comment to an issue you are following: ``
@ausil @rdossant Do we need to include s390x here?
I'd prefer to start with the three architectures we're already targeting and succeed there before we add s390 personally. It will be hard enough just to pull it together on those three. Once we work out the kinks then it shouldn't be a big deal adding a 4th arch. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
ausil added a new comment to an issue you are following: `` @sinnykumari yes we need to, we are starting tow ork on s390x now in addition to aarch64 and ppc64le ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
sinnykumari added a new comment to an issue you are following: `` @mohanboddu Any additional info needed or we are good to proceed with this request? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: `` This will be discussed in Fedora RelEng meeting tomorrow Apr 24 at 16:00 UTC ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
kevin added a new comment to an issue you are following: `` Some questions:
* You would tag all the packages for a compose into coreos-pool and all the ones in a released compose into coreos-release. Could you perhaps just use fedora GA/fedora-updates tags and only tag in the builds that are not in those and use those 2 repos on top of coreos-pool/coreos-release? That would make them much smaller and more manageable. Of course the fedora ones would be signed with fedora keys.
* What keys should these tags builds be signed with? If with the fedora ones, since they aren't versioned how can we tell what key to use?
If we have to sign a large pile of packages with a new key often I am a bit worried at our signing bandwith.
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: `` Can we use fxx-coreos-release and fxx-coreos-pool tags which follow a Fedora release and can make use of same fedora release keys? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
kevin added a new comment to an issue you are following: `` Another thought we had was to version the tags? ie, coreos-pool-30, coreos-pool-31, etc... and we could then just sign them with the fedora keys and just always make them when we branch.
But is everything being signed by a different key a requrement?
Can you re-use the existing fedora GA/updates repos signed with the fedora key in addition to these? or does it need to be one dist-repo? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
dustymabe added a new comment to an issue you are following: `` We just discussed this ticket in the releng meeting today. Here is what I think we decided on:
- we create a `coreos-pool` tag - this is an aggregate of all the packages used to build the various streams of FCOS. The package NVRAs for each build will be picked specifically using input to the rpm-ostree build tooling. - we create a `coreos-release` tag - this tag is simply for us to tag what rpms we have actually shipped to end users in FCOS. There are no consumers of this tag. It's purpose is to have a more conservative garbage collection policy on a smaller subset of packages than what is in `coreos-pool` - we create a `fxx-coreos-signing-pending` tag for each major version of fedora. When we build backports rpms will go into this tag and robosignatory will then sign the rpms and tag them into the `coreos-pool` tag and untag them from `fxx-coreos-signing-pending`. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
sinnykumari added a new comment to an issue you are following: `` Thanks all for the valuable input during [releng meeting](https://meetbot-raw.fedoraproject.org/teams/releng/releng.2019-04-24-15.59.l...) ! Unless we have any unanswered questions, can we get tags finalized in https://pagure.io/releng/issue/8294#comment-567284 and dist repo for `coreos-pool` tags with option `--non-latest` created? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: `` @sinnykumari Can it wait until Wed?, since this requires some koji hub policy changes and robosig changes, since we are in infra freeze we can make the changes once the freeze lifts off. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
sinnykumari added a new comment to an issue you are following: `` Sure, thanks @mohanboddu ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: `` Made the necessary changes to [koji hub policy](https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/roles?id=6c...) and [robosignatory config](https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/roles?id=c9...) and added the koji tags needed.
``` $ koji add-tag coreos-pool $ koji add-tag coreos-pool --parent=coreos-release $ koji add-tag f30-coreos-signing-pending --parent=coreos-pool $ koji list-tag-inheritance coreos-release --reverse coreos-release (8633) └─coreos-pool (8632) └─f30-coreos-signing-pending (8634) ``` ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
dustymabe added a new comment to an issue you are following: `` hey @mohanboddu - thanks for working on this. Quick question related to the implementation:
From the releng meeting (last week, I think) we discussed this and I think we had ended up deciding to not use koji inheritance for any of these tags. My understanding is that we should have `coreos-release`, `coreos-pool`, and `f30-coreos-signing-pending` all be separate with no inheritance. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: ``
hey @mohanboddu - thanks for working on this. Quick question related to the implementation: From the releng meeting (last week, I think) we discussed this and I think we had ended up deciding to not use koji inheritance for any of these tags. My understanding is that we should have coreos-release, coreos-pool, and f30-coreos-signing-pending all be separate with no inheritance.
@dustymabe The reason why I thought inheritance is, you dont need to add packages multiple times. Once you add the package to coreos-release they will be available for all other tags. And you can compose either from coreos-release or coreos-pool.
Now that I am thinking out loud, coreos-pool will pick all the builds in coreos-release, which I think might not be desirable. Can you confirm this and I can remove the inheritance between coreos-pool and coreos-release.
But I think we should leave the coreos-pool and fxx-coreos-signing-pending inheritance as it is, since you can add a package to coreos-pool and it will be available in fxx-core-signing-pending and if they are separate you need to add the package to both tags and also we need to change koji hub permissions as well. Since fxx-coreos-signing-pending is just a holding tag, it wont be of a problem. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
dustymabe added a new comment to an issue you are following: ``
@dustymabe The reason why I thought inheritance is, you dont need to add packages multiple times. Once you add the package to coreos-release they will be available for all other tags. And you can compose either from coreos-release or coreos-pool.
Now that I am thinking out loud, coreos-pool will pick all the builds in coreos-release, which I think might not be desirable. Can you confirm this and I can remove the inheritance between coreos-pool and coreos-release.
We aren't planning on composing from `coreos-release`. Just from `coreos-pool`. The rationale for the need for `coreos-release` is in my earlier comment](https://pagure.io/releng/issue/8294#comment-567284). Yes, please remove the inheritance. Also note that we only need one distrepo set up and that is against `coreos-pool`.
But I think we should leave the coreos-pool and fxx-coreos-signing-pending inheritance as it is, since you can add a package to coreos-pool and it will be available in fxx-core-signing-pending and if they are separate you need to add the package to both tags and also we need to change koji hub permissions as well. Since fxx-coreos-signing-pending is just a holding tag, it wont be of a problem.
My understanding is that the package goes to `fxx-coreos-signing-pending` when it is built and then robosig moves it to `coreos-pool` when the package has been signed. I don't think we need inheritance there? What would be the benefit? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: ``
We aren't planning on composing from coreos-release. Just from coreos-pool. The rationale for the need for coreos-release is in my earlier comment. Yes, please remove the inheritance. Also note that we only need one distrepo set up and that is against coreos-pool.
I will remove the inheritance here.
My understanding is that the package goes to fxx-coreos-signing-pending when it is built and then robosig moves it to coreos-pool when the package has been signed. I don't think we need inheritance there? What would be the benefit?
Without the inheritance, you need to add the package to both the coreos-pool and fxx-coreos-signing-pending tags, just like you did [here](https://pagure.io/releng/issue/8165#comment-560145) since both the tags have no idea about the packages. Using inheritance, thats not needed and once the build gets signed it will moved to coreos-pool tag from fxx-coreos-signing-pending tag. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: `` Removed the tag inheritance between coreos-release and coreos-pool
``` $ koji remove-tag-inheritance coreos-pool coreos-release ``` ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
dustymabe added a new comment to an issue you are following: ``
Without the inheritance, you need to add the package to both the coreos-pool and fxx-coreos-signing-pending tags, just like you did here since both the tags have no idea about the packages. Using inheritance, thats not needed and once the build gets signed it will moved to coreos-pool tag from fxx-coreos-signing-pending tag.
OK. One or two questions here:
- is there any chance for an unsigned build to get into the resulting dist-repo? - once we have a `f31-coreos-signing-pending` will this break? We need this setup to work equally for `f30-coreos-signing-pending` and `f31-coreos-signing-pending` at the same time. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: ``
OK. One or two questions here:
is there any chance for an unsigned build to get into the resulting dist-repo?
Nope
once we have a f31-coreos-signing-pending will this break? We need this setup to work equally for f30-coreos-signing-pending and f31-coreos-signing-pending at the same time.
I think this will make a perfect choice then, since we can add f31-coreos-signing-pending tag to the inheritance and both f30-c-s-p and f31-c-s-p will work. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
dustymabe added a new comment to an issue you are following: `` ok Thanks @mohanboddu ! ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
dustymabe added a new comment to an issue you are following: `` @mohanboddu - can you link me to the distrepo for `coreos-pool` once you've set it up? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: ``
@mohanboddu - can you link me to the distrepo for coreos-pool once you've set it up?
@dustymabe What do you mean? You can tag a build to f30-coreos-signing-pending, and once signed, it will be available in coreos-pool tag and I think currently tag2distrepo is still not working, so after tagging a build please let me know and I can run the koji dist-repo for you. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
sinnykumari added a new comment to an issue you are following: ``
@mohanboddu - can you link me to the distrepo for coreos-pool once you've set it up?
@dustymabe What do you mean? You can tag a build to f30-coreos-signing-pending, and once signed, it will be available in coreos-pool tag and I think currently tag2distrepo is still not working, so after tagging a build please let me know and I can run the koji dist-repo for you.
As per @mizdebsk [FBR](https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...), automatic dist-repo generation with tag2distrepo doesn't work only in case of unsigned packages get tagged. Since, coreos-pool uses signed packages, it should be fine. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: ``
As per @mizdebsk FBR, automatic dist-repo generation with tag2distrepo doesn't work only in case of unsigned packages get tagged. Since, coreos-pool uses signed packages, it should be fine.
Ah nvm then, this [patch](https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/roles/bodhi...) should fix this and I completed running the playbook. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
jlebon added a new comment to an issue you are following: `` Just to clear up re. that patch,
``` + 'coreos-pool': [cfc659b9], ```
When we're going to be sourcing from both f30 and f31 versions of the signing pending tags, we'd just have to list both keys here, right?
(Also shouldn't that keyid be in quotes?) ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: ``
Just to clear up re. that patch,
- 'coreos-pool': [cfc659b9],
When we're going to be sourcing from both f30 and f31 versions of the signing pending tags, we'd just have to list both keys here, right?
Yes
(Also shouldn't that keyid be in quotes?)
Thanks, I will rerun the playbook fixing it :disappointed: ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
sinnykumari added a new comment to an issue you are following: `` To check necessary permissions, I did:
``` $ koji add-pkg coreos-pool rpm-ostree --owner sinnykumari $ koji tag-build coreos-pool rpm-ostree-2019.3-1.fc30 ``` It went fine, but I don't see a coreos-pool dist-repo being created at https://kojipkgs.fedoraproject.org/repos-dist/ . What are we missing here? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
kevin added a new comment to an issue you are following: `` I restarted fedmsg-hub-3 on bodhi-backend02, it looks like it might not have picked up the new config yet.
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: `` Added the arches to coreos-pool tag
``` $ koji edit-tag coreos-pool --arches='aarch64 ppc64le s390x x86_64' ```
I am guessing its not required for coreos-release since it will just hold the builds from the released FCOS. Please let me know if I am wrong. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
dustymabe added a new comment to an issue you are following: `` https://kojipkgs.fedoraproject.org/repos-dist/coreos-pool/latest/
thanks all who helped! ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
sinnykumari added a new comment to an issue you are following: `` Awesome, thanks all! ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
sinnykumari added a new comment to an issue you are following: `` Did few more checks to make sure we have everything inplace:
- coreos-pool dist-repo allows to have multiple version of same package. Added openssh-7.9p1-5.fc30 and openssh-8.0p1-1.fc30 and we do have packages from both nvr available in latst generated dist-repo. - Can add packages to coreos-release tag (tested by adding rpm-ostree) - Can tag-build and untag-build on specific pakcages in coreos-release tag (tested with rpm-ostree-2019.3-1.fc30)
I think we have all requested items in this ticket done. Did we miss anything before we close this ticket?
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: ``
Added the arches to coreos-pool tag $ koji edit-tag coreos-pool --arches='aarch64 ppc64le s390x x86_64'
I am guessing its not required for coreos-release since it will just hold the builds from the released FCOS. Please let me know if I am wrong.
@sinnykumari Do you know answer for this ^? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
dustymabe added a new comment to an issue you are following: ``
Did we miss anything before we close this ticket?
One thing that is missing is special permissions surrounding tagging for the kernel. Only people who are allowed to build the kernel can tag it apparently. We need to be able to tag the kernel into our various tags using a bot. How can we achieve this goal? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
sinnykumari added a new comment to an issue you are following: ``
Added the arches to coreos-pool tag $ koji edit-tag coreos-pool --arches='aarch64 ppc64le s390x x86_64'
I am guessing its not required for coreos-release since it will just hold the builds from the released FCOS. Please let me know if I am wrong.
That's correct, we don't need this since coreos-release tag is only to tag specific nvr builds which will make it to the FCOS release. we won't creating a dist-repo out of it. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: ``
Did we miss anything before we close this ticket?
One thing that is missing is special permissions surrounding tagging for the kernel. Only people who are allowed to build the kernel can tag it apparently. We need to be able to tag the kernel into our various tags using a bot. How can we achieve this goal?
@dustymabe This [commit](https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=29c16b8) should fix that issue. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
dustymabe added a new comment to an issue you are following: ``
It worked!
``` [dustymabe@hattop ~]$ koji tag-build coreos-pool kernel-5.0.11-300.fc30 Created task 34767868 Watching tasks (this may be safely interrupted)... 34767868 tagBuild (noarch): free 34767868 tagBuild (noarch): free -> closed 0 free 0 open 1 done 0 failed
34767868 tagBuild (noarch) completed successfully ``` ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: `` We think we got everything we need here.
Closing the ticket. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
The status of the issue: `coreos-pool and coreos-release koji tags request for FCOS` of project: `releng` has been updated to: Closed as Fixed by mohanboddu.
dustymabe added a new comment to an issue you are following: `` A couple additional requests:
- can we get a few more signing-pending tags (maybe we can add this to a branching SOP somewhere) that have the same setup (inherits from `coreos-pool`) as the `f30-coreos-signing-pending` tag? - `f29-coreos-signing-pending` - `f31-coreos-signing-pending` - right now the distrepo config for `coreos-pool` has the f29 and f30 keys. Can we add the f31 key? Should be able to update with: `koji edit-tag coreos-pool -x tag2distrepo.keys="429476b4 cfc659b9 3c3359c4"`
``` $ koji taginfo coreos-pool Tag: coreos-pool [8632] Arches: aarch64 ppc64le s390x x86_64 Groups: Tag options: tag2distrepo.enabled : 'true' tag2distrepo.keys : '429476b4 cfc659b9' Inheritance: ```
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: `` Done.
``` $ koji add-tag f29-coreos-signing-pending --parent=coreos-pool $ koji add-tag f31-coreos-signing-pending --parent=coreos-pool $ koji edit-tag coreos-pool -x tag2distrepo.keys="429476b4 cfc659b9 3c3359c4" $ koji taginfo coreos-pool Tag: coreos-pool [8632] Arches: aarch64 ppc64le s390x x86_64 Groups: Tag options: tag2distrepo.enabled : 'true' tag2distrepo.keys : '429476b4 cfc659b9 3c3359c4' Inheritance: ``` ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
mohanboddu added a new comment to an issue you are following: `` robosig change commit - https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?h=master&a... ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
dustymabe added a new comment to an issue you are following: ``
robosig change commit - https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?h=master&a...
Thanks @mohanboddu. @kevin - can I get you to run the playbook that would get that change to apply? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
kevin added a new comment to an issue you are following: `` Done. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
dustymabe added a new comment to an issue you are following: `` ❤️ ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
dustymabe added a new comment to an issue you are following: `` now that there is a key for f32 can we get the `coreos-pool` tag updated to support the new key:
``` $ koji edit-tag coreos-pool -x tag2distrepo.keys="429476b4 cfc659b9 3c3359c4 12c944d0" ```
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
kevin added a new comment to an issue you are following: `` Done. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
dustymabe added a new comment to an issue you are following: `` ``` $ koji taginfo coreos-pool Tag: coreos-pool [8632] Arches: aarch64 ppc64le s390x x86_64 Groups: Tag options: tag2distrepo.enabled : 'true' tag2distrepo.keys : '429476b4 cfc659b9 3c3359c4 12c944d0' Inheritance: ```
Thanks @kevin:
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8294
rel-eng@lists.fedoraproject.org
-
Dennis Gilmore -
Dusty Mabe
-
Jonathan Lebon
-
Kevin Fenzi
-
Mohan Boddu
-
Sinny Kumari