Greetings fellow Fedorans!
I would like to kick off a general discussion about how we might gate packages in Rawhide. I think it would be nice to get something in place for the Fedora 29 timeframe.
As one of the Bodhi contributors, I am inclined to suggest that we could use Bodhi on Rawhide, similar to how we use it for our stable/branched releases, with more relaxed rules (perhaps 1 day in testing or something simple).
It may be possible to automate the process a bit to make it less heavy for developers, though there is some complication for multi-package updates (more on that in a bit). For simple package updates, we may be able to detect new commits on dist-git, and react to those by automatically starting a Koji build, and automatically filing a Bodhi update when that build is complete. I think that would be pretty nice, and pingou created a PoC[0] to do this about a year ago.
Multi-package updates won't be so easy though. It's not uncommon for us to need to update packages together, and the above workflow would be problematic since it would result in updates with single packages in them rather than updates with multiple packages. Of course, buildroot overrides would be a problem too, since multi-package updates often depend on each other at build time too.
We could create some way for packagers to indicate that a commit (or possibly even a whole repo) is not intended to be "autobuilt/updated". If the packager indicates this then their builds would go into a rawhide-pending (similar to what we do for f27 today). Once they have all their builds (and buildroot overrides) the way they want them, they can create the update.
Another idea that was tossed around in some chats I had with people about this involved a system for packagers to use to create Koji side tags. Bodhi manages BuildRoot Overrides today (with expirations), so perhaps Bodhi could be expanded to also manage Koji side tags (also with expirations). I can't remember all the details about this approach or why it was suggested over the former approach, but I wanted to list it to invite others to chew on it and see if it could work.
If you have other suggestions on how we might gate packages in Rawhide that are wildly different than the above, please feel free to share!
On 03/08/2018 12:00 PM, Randy Barlow wrote:
If you have other suggestions on how we might gate packages in Rawhide that are wildly different than the above, please feel free to share!
Why must the process change? It seems every bit of Fedora in the past year has been changing... for the sake of change. I'm going to speak up now and be a little more involved.
Do you want Rawhide used the same as Fedora N releases? If you want that I propose we remove Fedora N-1 releases. When branched occurs I have to switch branches four times. I'd much rather invest time in removing the need to support Fedora N-1 and if that means gating for Rawhide I can get on board.
On 03/08/2018 01:32 PM, Michael Cronenworth wrote:
Why must the process change? It seems every bit of Fedora in the past year has been changing... for the sake of change.
We've had extreme issues with stability in Rawhide lately that could have been prevented with automated testing. These issues cost a lot of time from infrastructure, QA, and release engineering members. They also caused us to be unable to compose Rawhide and Fedora 28 for quite some time. Not all of these issues would have been caught by package gating, but some would.
I'm going to speak up now and be a little more involved.
+1!
Do you want Rawhide used the same as Fedora N releases? If you want that I propose we remove Fedora N-1 releases. When branched occurs I have to switch branches four times. I'd much rather invest time in removing the need to support Fedora N-1 and if that means gating for Rawhide I can get on board.
I don't think making Rawhide used the same as Fedora N releases is the goal here, though I think it would also improve things for people who do choose to use Rawhide. I personally think we should continue to support N-1.
On 03/08/2018 10:00 AM, Randy Barlow wrote:
Greetings fellow Fedorans!
I would like to kick off a general discussion about how we might gate packages in Rawhide. I think it would be nice to get something in place for the Fedora 29 timeframe.
I was hoping we could come up with a more detailed proposal for the community at the upcoming Infrastructure hackfest, but sure we could start the discussion now.
As one of the Bodhi contributors, I am inclined to suggest that we could use Bodhi on Rawhide, similar to how we use it for our stable/branched releases, with more relaxed rules (perhaps 1 day in testing or something simple).
I don't think a testing will be of help here. Who would run with it enabled? I'm much prefer we just use the gating to run automated tests and get the update out as quickly as we can after that.
It may be possible to automate the process a bit to make it less heavy for developers, though there is some complication for multi-package updates (more on that in a bit). For simple package updates, we may be able to detect new commits on dist-git, and react to those by automatically starting a Koji build, and automatically filing a Bodhi update when that build is complete. I think that would be pretty nice, and pingou created a PoC[0] to do this about a year ago.
This has nice appeal for simple standalone updates...
Multi-package updates won't be so easy though. It's not uncommon for us to need to update packages together, and the above workflow would be problematic since it would result in updates with single packages in them rather than updates with multiple packages. Of course, buildroot overrides would be a problem too, since multi-package updates often depend on each other at build time too.
We could create some way for packagers to indicate that a commit (or possibly even a whole repo) is not intended to be "autobuilt/updated". If the packager indicates this then their builds would go into a rawhide-pending (similar to what we do for f27 today). Once they have all their builds (and buildroot overrides) the way they want them, they can create the update.
having to file overrides would be a pretty big drag for people like the KDE/Gnome teams... but I suppose they do it somehow in stable updates now.
Another idea that was tossed around in some chats I had with people about this involved a system for packagers to use to create Koji side tags. Bodhi manages BuildRoot Overrides today (with expirations), so perhaps Bodhi could be expanded to also manage Koji side tags (also with expirations). I can't remember all the details about this approach or why it was suggested over the former approach, but I wanted to list it to invite others to chew on it and see if it could work.
This approach has the advantage of not needing buildroot overrides, you just build all the things into the side tag in whatever order needed. The downside is that these would take koji cycles to keep doing newrepos for.
I was org not in favor of using bodhi, but it was noted that this is the one place we have now for test results and gating, so I kind of think we do need to use it now. That allows us to reuse it and waverdb, etc.
I really don't want to slow rawhide down or put barriers in place for maintainers, so I think we need to make things as painless as we can.
kevin
Il 08/03/2018 19:00, Randy Barlow ha scritto:
As one of the Bodhi contributors, I am inclined to suggest that we could use Bodhi on Rawhide, similar to how we use it for our stable/branched releases, with more relaxed rules (perhaps 1 day in testing or something simple).
Rawhide is a place where ABI changes are (quite) common. Having the builds pushed to a testing repo and managing them by body means that we would have more work to create buildroot overrides.
I think this change would imply changing the default tag assigned to a Rawhide build from (for example) 'f29' to 'f29-testing', doesn't it? So, what about giving maintainers the capability to force back the tag to 'f29', overriding the default directly from the 'fedpkg build' command (only for Rawhide)? This way, if a maintainer needs that build to be pushed directly to Rawhide for rebuilding dependent packages he can do that without going all over the buildroot override steps.
What about making side tags easily available to packagers directly? I could build my package that includes an ABI breaking update and rebuild the dependencies within that side tag and then submit when everything is complete.
Richard
On 03/08/2018 11:11 AM, Richard Shaw wrote:
What about making side tags easily available to packagers directly? I could build my package that includes an ABI breaking update and rebuild the dependencies within that side tag and then submit when everything is complete.
Yes, teaching bodhi about side tag management was one idea talked about.
Then you would do:
* get a side tag from bodhi * build all your stuff, test it, etc. * submit update with all those builds * CI/automated tests run on it, and if all is ok goes out in the next rawhide compose. * If not ok, you could wave the results or build more things/edit the update until it passes.
The only downsides to this is that it means more code for bodhi, and more newrepos for koji.
kevin
On Thu, 2018-03-08 at 11:24 -0800, Kevin Fenzi wrote:
- CI/automated tests run on it, and if all is ok goes out in the next
rawhide compose.
- If not ok, you could wave the results or build more things/edit the
update until it passes.
Hi, so it looks like you are going to remove chain-build from fedpkg, right? It's kind of pain it doesn't work for stable, but if you want to get rid of it for rawhide too, or add some unnecessary obstacles for its usage, then it's a downgrade like the kerberos usage (it's a pain to write my password all the time when I want to do something in packaging, with compare of once-per-year replace of the certificate, but that's another story, which only adds to the feeling of making process more and more painful instead of simpler (I heard some time ago something like this "security by obscurity", with which I fully agree)). I chain-build like this "A : B : C D", which means in stable that I build A, then fill override, then I build B, then fill an override, the I build C and D and then I fill the update. I hope the difference in stable and chain-build usage is obvious.
Anyway, could you enlighten what you call "if all is ok"? That's pretty important measure.
What do I do if it's not ok?
I do maintain a package which is in critpath. I'm not a proven packager, thus I cannot touch anyone's package (I hesitate to do it anyway, even there are cases where are not many other options). If my critpath package changes API and soname versions, then other packages, for which I do not have commit rights, will be broken by that update, but the update as such will build and look just fine. What do I do now? Will I hunt for respective maintainers and co-maintainers kindly asking them to do something about it? The paper work around this (finding who it is, their email addresses, writing them a mail) would be a pain on its own, but more painful would be the delay in the delivery. It will not be rawhide anymore, from my point of view.
I'd rather prefer to have detected issues in updates early (though it's understandable that doing API/ABI checks after each package update is not doable at least due to resource requirements - thus you'd not detect it with the gating too, right?), or at least like once a day.
I did a soname version bump in one of my packages recently and announced it, with a list of "possibly affected packages". I know my work flow is wrong in this regard, but I kind of rely on the notifications of broken dependencies to rebuild what is really needed to be rebuilt, because the packages sometimes requires something bigger in the build time, which is not actually used in the run time (just my feeling, no prove available). I didn't receive any single notification of broken dependencies this time. While it's possible someone was just quicker than me, I believe it's highly unlikely. I also believe I didn't filter out any such "broken dependencies" mail notifications from my notification settings, they used to come in the past.
That's also another disadvantage of the gating. I recall seeing broken dependencies messages for package I've no commit rights for for weeks. Either the maintainer (or co-maintainer) didn't receive those messages similar like me, or they just didn't care. How would I force them to rebuild/correct (I am definitely willing to help to correct the packages when an API change requires it) their packages, when they already ignore/filter out broken dependencies notifications? Should I hunt for a proven packager to move things forward, thus other packages can rely on the provided change? Proven packagers have their own work to do to, they cannot be disrupted from their work every other day by hundreds of packages to help with the packages their update broke.
By the way, why was the compose of rawhide broken for several days? Corresponding people/packagers not being available? It looks to me that you just want to move the issue to a slightly upper level, but then you'll have working rawhide compose, which will not use the recent/bleeding code. It is, from my point of view, the main credit of Rawhide, to be on the bleeding edge.
It would be very sad to turn Rawhide from 'package update' to 'people hunt' task.
Just my opinion. Bye, Milan
Milan Crha wrote:
so it looks like you are going to remove chain-build from fedpkg, right? It's kind of pain it doesn't work for stable, but if you want to get rid of it for rawhide too, or add some unnecessary obstacles for its usage, then it's a downgrade like the kerberos usage (it's a pain to write my password all the time when I want to do something in packaging, with compare of once-per-year replace of the certificate, but that's another story, which only adds to the feeling of making process more and more painful instead of simpler (I heard some time ago something like this "security by obscurity", with which I fully agree)). I chain-build like this "A : B : C D", which means in stable that I build A, then fill override, then I build B, then fill an override, the I build C and D and then I fill the update. I hope the difference in stable and chain-build usage is obvious.
+1
Anyway, could you enlighten what you call "if all is ok"? That's pretty important measure.
What do I do if it's not ok?
I guess then you will need to fix whatever the gating is complaining about and start all over again. :-(
I do maintain a package which is in critpath. I'm not a proven packager, thus I cannot touch anyone's package (I hesitate to do it anyway, even there are cases where are not many other options). If my critpath package changes API and soname versions, then other packages, for which I do not have commit rights, will be broken by that update, but the update as such will build and look just fine. What do I do now? Will I hunt for respective maintainers and co-maintainers kindly asking them to do something about it? The paper work around this (finding who it is, their email addresses, writing them a mail) would be a pain on its own, but more painful would be the delay in the delivery. It will not be rawhide anymore, from my point of view.
+1
I did a soname version bump in one of my packages recently and announced it, with a list of "possibly affected packages". I know my work flow is wrong in this regard, but I kind of rely on the notifications of broken dependencies to rebuild what is really needed to be rebuilt, because the packages sometimes requires something bigger in the build time, which is not actually used in the run time (just my feeling, no prove available). I didn't receive any single notification of broken dependencies this time. While it's possible someone was just quicker than me, I believe it's highly unlikely. I also believe I didn't filter out any such "broken dependencies" mail notifications from my notification settings, they used to come in the past.
Broken dependency notifications are disabled because the scripts are broken: they still use yum, which does not support the boolean dependencies. It is sad that this was still not fixed after months. Instead of wasting their time on annoyances such as update batching and now Rawhide gating, the infrastructure team should instead spend it to fix the existing essential QA tools.
That's also another disadvantage of the gating. I recall seeing broken dependencies messages for package I've no commit rights for for weeks. Either the maintainer (or co-maintainer) didn't receive those messages similar like me, or they just didn't care. How would I force them to rebuild/correct (I am definitely willing to help to correct the packages when an API change requires it) their packages, when they already ignore/filter out broken dependencies notifications? Should I hunt for a proven packager to move things forward, thus other packages can rely on the provided change? Proven packagers have their own work to do to, they cannot be disrupted from their work every other day by hundreds of packages to help with the packages their update broke.
+1. As a provenpackager, I have other things to do than rebuilding packages 24/7 because everything blocks on the rebuild.
By the way, why was the compose of rawhide broken for several days?
Because a broken dependency in ANY package included on ANY release-critical deliverable now fails the ENTIRE compose process. It is clear that this does not scale.
Corresponding people/packagers not being available? It looks to me that you just want to move the issue to a slightly upper level, but then you'll have working rawhide compose, which will not use the recent/bleeding code. It is, from my point of view, the main credit of Rawhide, to be on the bleeding edge.
It would be very sad to turn Rawhide from 'package update' to 'people hunt' task.
And I have to +1 these last 2 paragraphs as well.
Kevin Kofler
On 03/09/2018 05:42 AM, Kevin Kofler wrote:
Broken dependency notifications are disabled because the scripts are broken: they still use yum, which does not support the boolean dependencies. It is sad that this was still not fixed after months. Instead of wasting their time on annoyances such as update batching and now Rawhide gating, the infrastructure team should instead spend it to fix the existing essential QA tools.
Note that this script was disabled only 16 days ago. True it didn't understand rich deps, but until then it was reporting all the rest correctly. Also that tool would fall under releng, and as far as I know infrastructure has "wasted" no time on update batching or rawhide gating, nor really spent any time on it at all. I have definitely personally spent time answering emails and tickets and talking in fesco meetings, but thats as a fesco member and has nothing to do with infrastructure.
If this tool is important to you, perhaps you would like to step up and help fix it? Or as a workaround you could run dnf repoclosure or look at: https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20180308.n...
https://pagure.io/releng/issue/6365 is the releng ticket tracking this work.
...snip...
Because a broken dependency in ANY package included on ANY release-critical deliverable now fails the ENTIRE compose process. It is clear that this does not scale.
That is not what is causing the broken composes. See my last email.
kevin
Kevin Fenzi wrote:
On 03/09/2018 05:42 AM, Kevin Kofler wrote:
Because a broken dependency in ANY package included on ANY release-critical deliverable now fails the ENTIRE compose process. It is clear that this does not scale.
That is not what is causing the broken composes. See my last email.
At least one of the daily composes failed due to that (the last before things started working again). And I think several previous ones either also failed the same way, or the error was masked by a failure earlier in the process. The KDE Spin unfortunately had several issues with broken dependencies in those 2 weeks, e.g., I had to fight with a qt5-qtwebengine FTBFS caused by a GCC 8 regression that took us a few days to work around (and the GCC team a few more days to fix in GCC, but at least the workaround worked), which prevented rebuilding it for a new Qt and a new soname of some other library. (I do not follow the other release-blocking deliverables closely enough to know what their state was in those 2 weeks.)
Kevin Kofler
On 03/08/2018 11:10 PM, Milan Crha wrote:
On Thu, 2018-03-08 at 11:24 -0800, Kevin Fenzi wrote:
- CI/automated tests run on it, and if all is ok goes out in the next
rawhide compose.
- If not ok, you could wave the results or build more things/edit the
update until it passes.
Hi, so it looks like you are going to remove chain-build from fedpkg,
yeah, but you would have even better workflow with a side tag.
right? It's kind of pain it doesn't work for stable, but if you want to get rid of it for rawhide too, or add some unnecessary obstacles for its usage, then it's a downgrade like the kerberos usage (it's a pain to write my password all the time when I want to do something in packaging, with compare of once-per-year replace of the certificate, but that's another story, which only adds to the feeling of making process more and more painful instead of simpler (I heard some time ago something like this "security by obscurity", with which I fully agree)).
Note that if you use KCM (now default) and it works (It has some bugs), you only need to enter your password once a week. Also note that it's much more industry standard, and its much harder to have someone just take your cert and do things as you.
I chain-build like this "A : B : C D", which means in stable
that I build A, then fill override, then I build B, then fill an override, the I build C and D and then I fill the update. I hope the difference in stable and chain-build usage is obvious.
Sure, with this proposal you would:
* request a side tag * build a, wait for it to be added to the repo, build b, etc. You would not need to file overrides, just build them in the right order with wait-repo between them.
Anyway, could you enlighten what you call "if all is ok"? That's pretty important measure.
If all the tests we determine should gate packages pass.
I'd like to see a 'no new broken deps' test/check... but we could change the tests over time.
What do I do if it's not ok?
Fix your package(s) and resubmit for new tests, or if you feel the test is wrong file a bug on it and waive the results.
I do maintain a package which is in critpath. I'm not a proven packager, thus I cannot touch anyone's package (I hesitate to do it anyway, even there are cases where are not many other options). If my critpath package changes API and soname versions, then other packages, for which I do not have commit rights, will be broken by that update, but the update as such will build and look just fine. What do I do now?
You request a side tag, build your new package in that, then tell all the dependent package maintainers to use that side tag to rebuild all their packages. Once everyone is done, you (or someone else) submits it for testing as a unit.
Will I hunt for respective maintainers and co-maintainers kindly asking them to do something about it? The paper work around this (finding who it is, their email addresses, writing them a mail) would be a pain on its own, but more painful would be the delay in the delivery. It will not be rawhide anymore, from my point of view.
yes. You can use dnf repoquery to tell what uses your package and needs rebuilding. All those people should be watching your package or at least easy to communicate to.
sure, it might not be rawhide, it will be much more sane.
I'd rather prefer to have detected issues in updates early (though it's understandable that doing API/ABI checks after each package update is not doable at least due to resource requirements - thus you'd not detect it with the gating too, right?), or at least like once a day.
Not sure what you mean here... yes, we could do a check right before a compose, but then you have a lag after you build your package until it gets kicked out before the compose. Much better to check it as soon as we can so you can know and act on it.
I did a soname version bump in one of my packages recently and announced it, with a list of "possibly affected packages". I know my work flow is wrong in this regard, but I kind of rely on the notifications of broken dependencies to rebuild what is really needed to be rebuilt, because the packages sometimes requires something bigger in the build time, which is not actually used in the run time (just my feeling, no prove available). I didn't receive any single notification of broken dependencies this time. While it's possible someone was just quicker than me, I believe it's highly unlikely. I also believe I didn't filter out any such "broken dependencies" mail notifications from my notification settings, they used to come in the past.
As mentioned downstream the broken deps checking script was turned off because it doesn't understand rich deps. Also, we have had very few f28 or rawhide composes this last week.
That's also another disadvantage of the gating. I recall seeing broken dependencies messages for package I've no commit rights for for weeks.
Yep. Or longer.
Either the maintainer (or co-maintainer) didn't receive those messages similar like me, or they just didn't care. How would I force them to rebuild/correct (I am definitely willing to help to correct the packages when an API change requires it) their packages, when they already ignore/filter out broken dependencies notifications? Should I hunt for a proven packager to move things forward, thus other packages can rely on the provided change? Proven packagers have their own work to do to, they cannot be disrupted from their work every other day by hundreds of packages to help with the packages their update broke.
You could ask for commit on those packages you need commit on. If those maintainers don't approve it, ask for unresponsive maintainer process and you will get commit rights. If they still don't show up, you could look at retiring those packages so they drop from the collection and don't bother you anymore.
By the way, why was the compose of rawhide broken for several days? Corresponding people/packagers not being available? It looks to me that you just want to move the issue to a slightly upper level, but then you'll have working rawhide compose, which will not use the recent/bleeding code. It is, from my point of view, the main credit of Rawhide, to be on the bleeding edge.
It's been a long trail of things that needed to get worked out. it has nothing to do with availability. A number of people have been spending lots of time fixing things. I personally spent most of my week on this.
Some of the issues:
* new systemd landed in both rawhide and f28 right before freeze. It added a fsync call after writing/creating the /etc/machine-id file. However, in a installer chroot /proc isn't mounted so fsync fails to write the file, so systemd thinks /etc isn't writable and bind mounts /run/machine-id over /etc/machine-id, which results in lorax failing in a cp -alx command it uses to copy the install tree it just created. That was a fun one.. took up my wed tracking that down.
* Modular composes have had a number of issues in pungi. First, adding a Modular compose made things take 36 hours or so because it was gathering modules and trying to sort them into non modular repos. Then that was fixed, but then module signing was messed up before there are no modules made for f29/rawhide at all (There's a module build service bug about that), so we needed to disable them there. Then pungi gathered incorrectly rpms and noarch rpms were not placed in the right repos. There are composes running now with fixes for all that (we hope).
* Bodhi enablement was also this week... lots of issues around modules updating that were not figured since f27's fully modular approach, lots of signing issues around how to tell what module is part of what stream, etc.
Probibly more things I am blanking out on now. ;)
It would be very sad to turn Rawhide from 'package update' to 'people hunt' task.
Well, more communication is better. If people aren't responding we should know that and fix it, not just toss packages over a wall and wander off.
Just my opinion.
Thanks for the feedback!
Bye, Milan _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
kevin
Kevin Fenzi wrote:
yeah, but you would have even better workflow with a side tag.
The problem with a workflow relying entirely on side tags is that it is going to increase the risk of conflicts with concurrent changes (a phenomenon already somewhat present where side tags are used, but using side tags for everything is going to make it worse).
If my package depends on liba and libb, and if we have side tags f29-liba and f29-libb at the same time, when they are merged, the rebuild merged later will replace the one merged earlier and we will still end up with a half-broken package, or if we're lucky, the gating will fail the second merge (but only if the gated merges are serialized, otherwise there is a possible race condition there too).
Kevin Kofler
On Sat, Mar 10, 2018 at 04:09:41AM +0100, Kevin Kofler wrote:
Kevin Fenzi wrote:
yeah, but you would have even better workflow with a side tag.
The problem with a workflow relying entirely on side tags is that it is going to increase the risk of conflicts with concurrent changes (a phenomenon already somewhat present where side tags are used, but using side tags for everything is going to make it worse).
If my package depends on liba and libb, and if we have side tags f29-liba and f29-libb at the same time, when they are merged, the rebuild merged later will replace the one merged earlier and we will still end up with a half-broken package, or if we're lucky, the gating will fail the second merge (but only if the gated merges are serialized, otherwise there is a possible race condition there too).
Do note that the uniqueness constraint that koji has will mitigate things there. For a package to be built for -liba and then -libb, the package will need to have its release field bumped twice.
I figure it wouldn't be too hard to check when we merge a side-tag if there is a newer build of a package already available and if so block the merge (I guess that would be very similar to the upgrade path checks between releases, but here it would be between koji tags).
Pierre
Pierre-Yves Chibon wrote:
Do note that the uniqueness constraint that koji has will mitigate things there. For a package to be built for -liba and then -libb, the package will need to have its release field bumped twice.
Sure, but that has not prevented conflicts in the past either, because it does not ensure that the second bump is actually built against the new library for which the first bump was committed.
Kevin Kofler
On Mon, Mar 12, 2018 at 03:58:13PM +0100, Kevin Kofler wrote:
Pierre-Yves Chibon wrote:
Do note that the uniqueness constraint that koji has will mitigate things there. For a package to be built for -liba and then -libb, the package will need to have its release field bumped twice.
Sure, but that has not prevented conflicts in the past either, because it does not ensure that the second bump is actually built against the new library for which the first bump was committed.
Thinking about this some: package A (release 1) has two dependencies B and C. A-2 is built in side-tag #1 for B' A-3 is built in side-tag #2 for C'
Case #1 ------- Side-tag #2 is merged, A-3 is in rawhide for C'. Side-tag #1 is asked to be merged: - A-2 < A-3, so "uprade path" is not ok -> Merge is blocked, a new build A-4 is required A-4 is built against B' and C' --> all good
Case #2 ------- Side-tag #1 is merged, A-2 is in rawhide for B'. Side-tag #2 is asked to be merged: - A-3 > A-2, so "upgrade path" is ok -> Merge is ok, but A-3 was built against C', not B'
Solutions: - Do not allow one package to be in two side-tags at the same time? - Ensure the tests for Case #2 would prevent the merge (doable?) - Figure out if between build and side-tag merge-request one of the package in that side tags has been part of another side-tag (definitely doable, "just" a little more logic) - Another idea?
Pierre
On 03/12/2018 09:35 AM, Pierre-Yves Chibon wrote:
On Mon, Mar 12, 2018 at 03:58:13PM +0100, Kevin Kofler wrote:
Pierre-Yves Chibon wrote:
Do note that the uniqueness constraint that koji has will mitigate things there. For a package to be built for -liba and then -libb, the package will need to have its release field bumped twice.
Sure, but that has not prevented conflicts in the past either, because it does not ensure that the second bump is actually built against the new library for which the first bump was committed.
...snip...
Solutions:
- Do not allow one package to be in two side-tags at the same time?
I think this would be difficult...
- Ensure the tests for Case #2 would prevent the merge (doable?)
Yeah, that might be nice. In fact if it's a library so change it would easy because there would be broken deps.
- Figure out if between build and side-tag merge-request one of the package in that side tags has been part of another side-tag (definitely doable, "just" a little more logic)
- Another idea?
I think these cases are rare and also that maintainers should communicate, so they should be detected.
Notifications of builds and/or commits should definitely tell a maintainer that two people are rebuilding things in two side tags.
Perhaps when creating a side tag we could display the existing ones so you know if what you are trying to do might overlap with an existing one? For that we might need a short description of the side tag...
f29-kevin-xfce "Side tag to build the Xfce stack" or f29-exampleuser-boost "Side tag to rebuild against new boost x.y.z"
kevin
On Mon, Mar 12, 2018 at 12:44:39PM -0700, Kevin Fenzi wrote:
On 03/12/2018 09:35 AM, Pierre-Yves Chibon wrote:
On Mon, Mar 12, 2018 at 03:58:13PM +0100, Kevin Kofler wrote:
Pierre-Yves Chibon wrote:
Do note that the uniqueness constraint that koji has will mitigate things there. For a package to be built for -liba and then -libb, the package will need to have its release field bumped twice.
Sure, but that has not prevented conflicts in the past either, because it does not ensure that the second bump is actually built against the new library for which the first bump was committed.
...snip...
Solutions:
- Do not allow one package to be in two side-tags at the same time?
I think this would be difficult...
- Ensure the tests for Case #2 would prevent the merge (doable?)
Yeah, that might be nice. In fact if it's a library so change it would easy because there would be broken deps.
- Figure out if between build and side-tag merge-request one of the package in that side tags has been part of another side-tag (definitely doable, "just" a little more logic)
- Another idea?
I think these cases are rare and also that maintainers should communicate, so they should be detected.
Actually even the changelog entry should be enough to "communicate" this: When I see "- rebuilt for foo-n", it's pretty obvious that I need to make sure to include foo-n in my build environment.
Zbyszek
Hi,
On Fri, 2018-03-09 at 11:35 -0800, Kevin Fenzi wrote:
Sure, with this proposal you would:
- request a side tag
- build a, wait for it to be added to the repo, build b, etc.
You would not need to file overrides, just build them in the right order with wait-repo between them.
well, it's not better, it's worse. I could not just run one command and let the computers do the task on their own, I just check from time to time whether anything broken, but I'd be supposed to babysit whole build process with the packages from the "chain-build"? That's truly worse, needs more of my time, while the koji can do it on its own. Computers are here to help people, right? :)
By the way, I just successfully ran chain-build from an f28 branch. https://koji.fedoraproject.org/koji/taskinfo?taskID=25654934 That surprised me, I expected to be kicked of, as it used to do, but it didn't do it (it used to kick me off in the past, especially after branching/bodhi activation point). I wanted to try whether chain-build with --target would make the trick, it also used to work.
I'd like to see a 'no new broken deps' test/check... but we could change the tests over time.
Yeah, that's it, without such test you just make obstacles to packagers with no gain at all (if broken deps are recognized only after pushing the side tag into the Rawhide, then it's really only about unneeded obstacles).
You request a side tag,
Who is going to respond to it? Will it be automated, with no specific people being involved? Automated side tags sound wrong, from my point of view. Hunting for people in different timezones to let me do my duty sounds even worse.
build your new package in that, then tell all the dependent package maintainers to use that side tag to rebuild all their packages.
Oh, having multi-process synchronization is kind of unsolvable problem (if I recall my college study properly), how you'd like to synchronize the process between people in various countries and time zones in practice, while limiting the delay to a minimum?
All those people should be watching your package or at least easy to communicate to.
Watching a package and/or have commit rights usually brings in tons of unrelated stuff in your Inbox. Yes, there are settings, but I cannot decide which changes are relevant to my package and which not unless I read the info about the change. After some time one can just ignore most of the messages anyway (people do lack of time, the more those volunteers in the community).
That's also another disadvantage of the gating. I recall seeing broken dependencies messages for package I've no commit rights for for weeks.
Yep. Or longer.
Definitely, and it's the main point against gating Rawhide. If people cannot react in the timely manner right now, with already working infrastructure, how is the gating going to help and address this problem at the first place?
The gating will help to have buildable composes all the time. That's its main purpose, right? Then it'll help with what? What is it good for to have buildable composes, when they are using outdated software? All the QA work will be wasted on those composes, because of delay of some package update due to the gating.
You could ask for commit on those packages you need commit on. If those maintainers don't approve it, ask for unresponsive maintainer process and you will get commit rights.
Maybe they have a reason why they want to have as less people on the package as they can. Or they can miss some notification from "pkgdb" (I know, it's not pkgdb these days) or have it lost in the spam folder... And it's a bad idea to expect that other people have time to work on Fedora just at the moment when I decide to push in some giant API change, not talking that many of the packagers just do "proxies" for upstream, they possibly never touched the upstream code to be able to fix anything in the code, thus they rely on response from the upstream, which can add even more delay in some cases. Not talking that unresponsive maintainer process should be started only after some time of inactivity of the package maintainer.
If they still don't show up, you could look at retiring those packages so they drop from the collection and don't bother you anymore.
Oh, I'd not like to go that far, for basically the same reasons as in the previous paragraph.
It's been a long trail of things that needed to get worked out. it has nothing to do with availability. A number of people have been spending lots of time fixing things.
I see, that's understandable. How is rawhide gating going to help with this in practice? (Search for 'outdated software' above.) There will be still needed some people looking into the issue and spend their time on it, maybe they will be in less pressure, but I'm afraid it'll be just a feeling. Nobody should expect infrastructure folks understand each package (like you mentioned systemd here), at least I do not expect it, thus I'd expect you had some coordination with systemd package maintainer and its upstream to find out the root cause of the problem. Was it any smooth? Did it even happen? Might each maintainer of a crit- path package (or a package which has some users (libraries)) do the same round trip as you did for the last week?
- Bodhi enablement was also this week...
Yeah, I've been always wondering why the timing with Beta freeze and Bodhi activation is so bad. There is a GNOME stable release this Wednesday, the 3.28.0 version, and GNOME is the desktop environment of Fedora Workstation, still these two schedules collide in a way that the Beta contains development version of packages which turned into stable only few days after Beta freeze/Bodhi activation (I've here similar concerns as with 'outdated software' usage above). Anyway, I do not want to change subject of this (sub-)thread, there are other places where it had been discussed in the past, all of this only reminded me of it.
Well, more communication is better. If people aren't responding we should know that and fix it, not just toss packages over a wall and wander off.
Yes, definitely, though there are still legitimate cases when people do not have time and nobody can do with it anything. That's just life. Bye, Milan
On Mon, Mar 12, 2018 at 11:33:36AM +0100, Milan Crha wrote:
Hi,
On Fri, 2018-03-09 at 11:35 -0800, Kevin Fenzi wrote:
Sure, with this proposal you would:
- request a side tag
- build a, wait for it to be added to the repo, build b, etc.
You would not need to file overrides, just build them in the right order with wait-repo between them.
well, it's not better, it's worse. I could not just run one command and let the computers do the task on their own, I just check from time to time whether anything broken, but I'd be supposed to babysit whole build process with the packages from the "chain-build"? That's truly worse, needs more of my time, while the koji can do it on its own. Computers are here to help people, right? :)
Tooling can be built no? Why do you assume we can't make chain-build work with side-tags? Is there a technical reason we can't just have the tool submit a build, wait for the new repo, submit the second build and so on? If whether this needs to be a `chain-build --side-tag=<foo>` or a `side-build <foo>...` is up for discussion :)
I'd like to see a 'no new broken deps' test/check... but we could change the tests over time.
Yeah, that's it, without such test you just make obstacles to packagers with no gain at all (if broken deps are recognized only after pushing the side tag into the Rawhide, then it's really only about unneeded obstacles).
The side tags would only be used if you need to update multiple packages, so having the test run while you're in the middle of it and thus you know it's not going to pass, seems a little useless no? Seems to me much more interesting to say: I have a few packages to update, let's update them, ok I'm done, could you check if I missed any? yes -> fix, no -> merge
You request a side tag,
Who is going to respond to it?
The idea has been submitted a few times already in this thread, the side tags would be managed in bodhi very much like buildroot overrides are currently.
Will it be automated, with no specific people being involved?
It would be automated via bodhi, likely have a rather short life by default (exactly to avoid the race-condition issue that Kevin Kofler mentioned in another email) but that life period could be expanded for larger rebuilds.
(if I recall my college study properly), how you'd like to synchronize the process between people in various countries and time zones in practice, while limiting the delay to a minimum?
How would this be different from today? If it takes a week to solve a rebuild today, the side tag tomorrow would remain open for a week. The difference is that during that week, we will still be able to compose rawhide tomorrow (and thus test all the other changes), while we can't today.
That's also another disadvantage of the gating. I recall seeing broken dependencies messages for package I've no commit rights for for weeks.
Yep. Or longer.
Definitely, and it's the main point against gating Rawhide. If people cannot react in the timely manner right now, with already working infrastructure, how is the gating going to help and address this problem at the first place?
Because that problem will remain isolated from the rest of the distro. So yes, as you're pointing out below some package will get a little behind and others won't. But the formers won't prevent testing/integrating the laters.
The gating will help to have buildable composes all the time. That's its main purpose, right? Then it'll help with what? What is it good for to have buildable composes, when they are using outdated software? All the QA work will be wasted on those composes, because of delay of some package update due to the gating.
You could ask for commit on those packages you need commit on. If those maintainers don't approve it, ask for unresponsive maintainer process and you will get commit rights.
Maybe they have a reason why they want to have as less people on the package as they can. Or they can miss some notification from "pkgdb" (I know, it's not pkgdb these days) or have it lost in the spam folder... And it's a bad idea to expect that other people have time to work on Fedora just at the moment when I decide to push in some giant API
It seems to me this is even worth today. You decide a giant API change, the other people don't have time to work on Fedora just at that moment, the entire Fedora is stuck because things are burning. Tomorrow: you decide a giant API change, the other people don't have time to work on Fedora just at that moment: the rest of Fedora is happily moving forward. Yes your change will be delayed in its integration, but it won't delay or impact other integrations.
change, not talking that many of the packagers just do "proxies" for upstream, they possibly never touched the upstream code to be able to fix anything in the code, thus they rely on response from the upstream, which can add even more delay in some cases. Not talking that unresponsive maintainer process should be started only after some time of inactivity of the package maintainer.
It's been a long trail of things that needed to get worked out. it has nothing to do with availability. A number of people have been spending lots of time fixing things.
I see, that's understandable. How is rawhide gating going to help with this in practice?
Because if the packager want to push their update, they will need to investigate why it's not passing the tests, not infra. So in theory Kevin Fenzi would not have to dig into systemd just before freeze to find out what changed and broke the world. The systemd maintainer (with help if necessary, of course) would. So that'd be handling back the issue to the expert on the package, the maintainer rather than the infrastructure or releng folks trying to get a compose out.
(Search for 'outdated software' above.) There will be still needed some people looking into the issue and spend their time on it, maybe they will be in less pressure, but I'm afraid it'll be just a feeling. Nobody should expect infrastructure folks understand each package (like you mentioned systemd here), at least I do not expect it, thus I'd expect you had some coordination with systemd package maintainer and its upstream to find out the root cause of the problem. Was it any smooth? Did it even happen? Might each maintainer of a crit- path package (or a package which has some users (libraries)) do the same round trip as you did for the last week?
Pierre
Le 2018-03-12 12:10, Pierre-Yves Chibon a écrit :
On Mon, Mar 12, 2018 at 11:33:36AM +0100, Milan Crha wrote:
Hi,
On Fri, 2018-03-09 at 11:35 -0800, Kevin Fenzi wrote:
Sure, with this proposal you would:
- request a side tag
- build a, wait for it to be added to the repo, build b, etc.
You would not need to file overrides, just build them in the right order with wait-repo between them.
well, it's not better, it's worse. I could not just run one command and let the computers do the task on their own, I just check from time to time whether anything broken, but I'd be supposed to babysit whole build process with the packages from the "chain-build"? That's truly worse, needs more of my time, while the koji can do it on its own. Computers are here to help people, right? :)
Tooling can be built no? Why do you assume we can't make chain-build work with side-tags?
Also current chain-build is too primitive to handle complex chains
https://github.com/rpm-software-management/mock/issues/170
Regards,
On 2018-03-12 12:19, Nicolas Mailhot wrote:
Le 2018-03-12 12:10, Pierre-Yves Chibon a écrit :
On Mon, Mar 12, 2018 at 11:33:36AM +0100, Milan Crha wrote:
Hi,
On Fri, 2018-03-09 at 11:35 -0800, Kevin Fenzi wrote:
Sure, with this proposal you would:
- request a side tag
- build a, wait for it to be added to the repo, build b, etc.
You would not need to file overrides, just build them in the right order with wait-repo between them.
well, it's not better, it's worse. I could not just run one command and let the computers do the task on their own, I just check from time to time whether anything broken, but I'd be supposed to babysit whole build process with the packages from the "chain-build"? That's truly worse, needs more of my time, while the koji can do it on its own. Computers are here to help people, right? :)
Tooling can be built no? Why do you assume we can't make chain-build work with side-tags?
Also current chain-build is too primitive to handle complex chains
Koji chain-builds and mockchain are unrelated (both are primitive, I don't disagree with that).
Michael
Le lundi 12 mars 2018 à 14:51 +0100, Michael Šimáček a écrit :
On 2018-03-12 12:19, Nicolas Mailhot wrote:
Also current chain-build is too primitive to handle complex chains
Koji chain-builds and mockchain are unrelated (both are primitive, I don't disagree with that).
Yes that was mostly point to a document specifying how chain build should be implemented to handle boostraps and make sure the result is self-hosting.
If the rawhide workflow is going to depend on side tags and chain builds, they better be automated, robust and correct
Regards,
Hi,
On Mon, 2018-03-12 at 12:10 +0100, Pierre-Yves Chibon wrote:
On Mon, Mar 12, 2018 at 11:33:36AM +0100, Milan Crha wrote:
On Fri, 2018-03-09 at 11:35 -0800, Kevin Fenzi wrote:
Sure, with this proposal you would:
- request a side tag
- build a, wait for it to be added to the repo, build b, etc.
You would not need to file overrides, just build them in the right order with wait-repo between them.
Tooling can be built no? Why do you assume we can't make chain-build work with side-tags?
That wasn't my suggestion, that's what Kevin Fenzi suggested and I sort of questioned it. :)
Seems to me much more interesting to say: I have a few packages to update, let's update them, ok I'm done, could you check if I missed any? yes -> fix, no -> merge
Yes, definitely. How that "checked if I missed any" question will be implemented is important. It's necessary to have it done before merging the side tag, otherwise there will be no gain in gating at all, right? The best time is to have done the final check automatically when the merge of the side tag is requested, if I understand it correctly.
The idea has been submitted a few times already in this thread,...
Oh, right, I'm sorry for repeating it. My fault.
The difference is that during that week, we will still be able to compose rawhide tomorrow (and thus test all the other changes), while we can't today.
I see, that makes perfect sense. Some packages won't stop the rest of the distribution. I'm only afraid people will care less when the problem will be in a side tag, rather than in the main stream.
Because if the packager want to push their update, they will need to investigate why it's not passing the tests, not infra.
Definitely, that's my point too. I'd expect it working that way already, with or without gating. People should be responsible for changes they do in their packages (generally speaking, I do not mean any offense to systemd folks, not talking that I can imagine that finding such kind of bugs is a nightmare even for the developers of the software).
Thanks and bye, Milan
On Mon, Mar 12, 2018 at 12:48:02PM +0100, Milan Crha wrote:
Hi,
On Mon, 2018-03-12 at 12:10 +0100, Pierre-Yves Chibon wrote:
On Mon, Mar 12, 2018 at 11:33:36AM +0100, Milan Crha wrote:
On Fri, 2018-03-09 at 11:35 -0800, Kevin Fenzi wrote:
Sure, with this proposal you would:
- request a side tag
- build a, wait for it to be added to the repo, build b, etc.
You would not need to file overrides, just build them in the right order with wait-repo between them.
Tooling can be built no? Why do you assume we can't make chain-build work with side-tags?
That wasn't my suggestion, that's what Kevin Fenzi suggested and I sort of questioned it. :)
Seems to me much more interesting to say: I have a few packages to update, let's update them, ok I'm done, could you check if I missed any? yes -> fix, no -> merge
Yes, definitely. How that "checked if I missed any" question will be implemented is important. It's necessary to have it done before merging the side tag, otherwise there will be no gain in gating at all, right? The best time is to have done the final check automatically when the merge of the side tag is requested, if I understand it correctly.
The way I envision this would be: you request the side tag to be merged, the test will be run, report its outcome to you. If the outcome is positive (the test passed) it will automatically merge the tag, and otherwise it won't touch the tag, giving you the opportunity to fix the issue.
Does that sound right?
The difference is that during that week, we will still be able to compose rawhide tomorrow (and thus test all the other changes), while we can't today.
I see, that makes perfect sense. Some packages won't stop the rest of the distribution. I'm only afraid people will care less when the problem will be in a side tag, rather than in the main stream.
I guess that could happen, but on the other side, since the problem is isolated in a side-tag it's not in the main stream and thus not available, even to the people who started working on this.
Pierre
On 03/12/2018 04:48 AM, Milan Crha wrote:
Hi,
On Mon, 2018-03-12 at 12:10 +0100, Pierre-Yves Chibon wrote:
On Mon, Mar 12, 2018 at 11:33:36AM +0100, Milan Crha wrote:
On Fri, 2018-03-09 at 11:35 -0800, Kevin Fenzi wrote:
Sure, with this proposal you would:
- request a side tag
- build a, wait for it to be added to the repo, build b, etc.
You would not need to file overrides, just build them in the right order with wait-repo between them.
Tooling can be built no? Why do you assume we can't make chain-build work with side-tags?
That wasn't my suggestion, that's what Kevin Fenzi suggested and I sort of questioned it. :)
To be clear I didn't suggest chain build wouldn't work, I just outlined a workflow I could see myself using. I suspect chain build would work on a side tag just fine with --target passed to it. But of course we would want to test and note this use case.
So, thanks for bringing it up...
kevin
On 03/12/2018 11:33 AM, Milan Crha wrote:
By the way, I just successfully ran chain-build from an f28 branch. https://koji.fedoraproject.org/koji/taskinfo?taskID=25654934 That surprised me, I expected to be kicked of, as it used to do, but it didn't do it (it used to kick me off in the past, especially after branching/bodhi activation point). I wanted to try whether chain-build with --target would make the trick, it also used to work.
It should work if you use '--target f28-gnome'
Kalev
On Mon, 2018-03-12 at 12:12 +0100, Kalev Lember wrote:
On 03/12/2018 11:33 AM, Milan Crha wrote:
I wanted to try whether chain-build with --target would make the trick, it also used to work.
It should work if you use '--target f28-gnome'
Hi, yes, I know, that's why I wrote "it also used to work". I also used to be kicked off when trying chain-build after bodhi activation (something about "fXX-candidate not deriving from fXX-build", but I do not recall it precisely), which seems to be accepted right now, except after the build of the first package the next in the chain would wait "ad infinity" to have the first package appear in f28-build.
It's only that `koji cancel` doesn't work here (with python3, it does when using python2), thus it took me longer to find out what to do to stop the previous chain build, mark the one finished package with proper tag and start the chain-build in the side tag. Thanks and bye, Milan
On Thu, 2018-03-08 at 11:24 -0800, Kevin Fenzi wrote:
On 03/08/2018 11:11 AM, Richard Shaw wrote:
What about making side tags easily available to packagers directly? I could build my package that includes an ABI breaking update and rebuild the dependencies within that side tag and then submit when everything is complete.
Yes, teaching bodhi about side tag management was one idea talked about.
Then you would do:
- get a side tag from bodhi
- build all your stuff, test it, etc.
- CI/automated tests run on it, and if all is ok goes out in the next
rawhide compose.
- If not ok, you could wave the results or build more things/edit the
update until it passes.
That would be really nice! There are often cases where a fix or enhancement needs to land in at least two components at once (in our cases usually pykickstart & anaconda and/or blivet & anaconda) and the only "atomic" way of doing that is doing a chainbuild, which, frankly, sucks:
- cryptic CLI syntax, no Web UI - totally different process from branched releases - you need to have commit right to all involved packages - no automated tests run on the results whatosoever - no way to tell list inidividual scratchbuilds in the past (AFAIK)
So having the option to use Bodhi to group atomic package updates for Rawhide to a Bodho update would be really useful!
- consistent with branched Fedora process - no need to be commiter for all packages, just tell the other maintainers to add their new build to the update - tests could be easily hooked to the update - Web UI - ability to see & link people to pending & past updates
I might be missing some of the technicalities but this really seems like a win-win solution while keeping the current process in place due to the updates being optional.
The only downsides to this is that it means more code for bodhi, and more newrepos for koji.
kevin
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
Martin Kolman wrote:
That would be really nice! There are often cases where a fix or enhancement needs to land in at least two components at once (in our cases usually pykickstart & anaconda and/or blivet & anaconda) and the only "atomic" way of doing that is doing a chainbuild, which, frankly, sucks:
- cryptic CLI syntax, no Web UI
- totally different process from branched releases
- you need to have commit right to all involved packages
- no automated tests run on the results whatosoever
- no way to tell list inidividual scratchbuilds in the past (AFAIK)
So having the option to use Bodhi to group atomic package updates for Rawhide to a Bodho update would be really useful!
- consistent with branched Fedora process
- no need to be commiter for all packages, just tell the other maintainers
to add their new build to the update
- tests could be easily hooked to the update
- Web UI
- ability to see & link people to pending & past updates
I might be missing some of the technicalities but this really seems like a win-win solution while keeping the current process in place due to the updates being optional.
A chain build is much more effective than the Bodhi process though. See Milan Crha's reply.
Kevin Kofler
Il 08/03/2018 20:11, Richard Shaw ha scritto:
What about making side tags easily available to packagers directly? I could build my package that includes an ABI breaking update and rebuild the dependencies within that side tag and then submit when everything is complete.
+1
Maybe something like 'fedpkg build --side-tag=pkgname-abichange', that would mean to build that package using all packages already with the same side tag and tag the build with side-tag. After all the packages are rebuilt, a simple command like 'fedpkg enable side-tag' will push all the packages with side-tag to testing.
On 08/03/18 19:25, Mattia Verga wrote:
Il 08/03/2018 20:11, Richard Shaw ha scritto:
What about making side tags easily available to packagers directly? I could build my package that includes an ABI breaking update and rebuild the dependencies within that side tag and then submit when everything is complete.
+1
Maybe something like 'fedpkg build --side-tag=pkgname-abichange', that would mean to build that package using all packages already with the same side tag and tag the build with side-tag.
Well if a side tag exists then you can already choose to build in it using the --target switch
What would be needed is a way to create a side tag and then to merge it back.
Tom
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On Thu, 2018-03-08 at 13:00 -0500, Randy Barlow wrote:
Greetings fellow Fedorans!
I would like to kick off a general discussion about how we might gate packages in Rawhide. I think it would be nice to get something in place for the Fedora 29 timeframe.
As one of the Bodhi contributors, I am inclined to suggest that we could use Bodhi on Rawhide, similar to how we use it for our stable/branched releases, with more relaxed rules (perhaps 1 day in testing or something simple).
It may be possible to automate the process a bit to make it less heavy for developers, though there is some complication for multi-package updates (more on that in a bit). For simple package updates, we may be able to detect new commits on dist-git, and react to those by automatically starting a Koji build, and automatically filing a Bodhi update when that build is complete. I think that would be pretty nice, and pingou created a PoC[0] to do this about a year ago.
Multi-package updates won't be so easy though. It's not uncommon for us to need to update packages together, and the above workflow would be problematic since it would result in updates with single packages in them rather than updates with multiple packages. Of course, buildroot overrides would be a problem too, since multi-package updates often depend on each other at build time too.
We could create some way for packagers to indicate that a commit (or possibly even a whole repo) is not intended to be "autobuilt/updated". If the packager indicates this then their builds would go into a rawhide-pending (similar to what we do for f27 today). Once they have all their builds (and buildroot overrides) the way they want them, they can create the update.
Another idea that was tossed around in some chats I had with people about this involved a system for packagers to use to create Koji side tags. Bodhi manages BuildRoot Overrides today (with expirations), so perhaps Bodhi could be expanded to also manage Koji side tags (also with expirations). I can't remember all the details about this approach or why it was suggested over the former approach, but I wanted to list it to invite others to chew on it and see if it could work.
If you have other suggestions on how we might gate packages in Rawhide that are wildly different than the above, please feel free to share!
I would not mind this, but making buildroot overrides is something what makes me to not update N-1 release.. Especially when I need to build 15-20 packages which have some dependency loops. - -- - -Igor Gnatenko
Randy Barlow wrote:
As one of the Bodhi contributors, I am inclined to suggest that we could use Bodhi on Rawhide, similar to how we use it for our stable/branched releases, with more relaxed rules (perhaps 1 day in testing or something simple).
IMHO, this is highly impractical. Having to file Bodhi updates for every Rawhide build will make development cringe to a halt.
It may be possible to automate the process a bit to make it less heavy for developers, though there is some complication for multi-package updates
Some automation would help, sure. But if we are going to do things automatically, why bother with Bodhi at all?
We are already building into a pending tag for the autosigning, from which the packages are moved into the final tag when they are signed. The same process should work for autotesting. Just add it before or after the autosigning in the pipeline, possibly with another intermediate tag.
I think that Bodhi is really the wrong tool for the job here. What you are trying to hit with your shiny hammer may look like a nail to you, but it is really a screw. ;-)
(That said, I am not even convinced gating Rawhide is going to be workable to begin with. Temporary breakage is just sometimes necessary to make things move forward. I would much rather see the compose process made more robust so it does not fail on every small breakage.)
Another idea that was tossed around in some chats I had with people about this involved a system for packagers to use to create Koji side tags.
This idea is interesting, and it could be also used without gating or with a gating process not relying on Bodhi. The side tag creation, and possibly the side tag merge request, would then be the only thing(s) done through Bodhi.
Kevin Kofler
On 03/08/2018 07:53 PM, Kevin Kofler wrote:
Randy Barlow wrote:
It may be possible to automate the process a bit to make it less heavy for developers, though there is some complication for multi-package updates
Some automation would help, sure. But if we are going to do things automatically, why bother with Bodhi at all?
Well, we don't currently have another tool to show results and allow waving of them. We could come up with another app and have yet a different place and workflow from regular releases, but why not stick to the same workflow and avoid making yet another app.
We are already building into a pending tag for the autosigning, from which the packages are moved into the final tag when they are signed. The same process should work for autotesting. Just add it before or after the autosigning in the pipeline, possibly with another intermediate tag.
I think that Bodhi is really the wrong tool for the job here. What you are trying to hit with your shiny hammer may look like a nail to you, but it is really a screw. ;-)
I don't think thats the case here, it's more than we don't want to build another different tool for something that could work with the hammer we have.
To be fair, there was a suggestion that we show results in pagure/src.fedoraproject.org, but to me, that definitely sounds like the wrong place for such things. We want tests tied to a specific update, not the entire package as a whole. (Even thought we could fudge this by having git tags or something to tie results to).
kevin
On Thu, Mar 08, 2018 at 08:26:30PM -0800, Kevin Fenzi wrote:
On 03/08/2018 07:53 PM, Kevin Kofler wrote:
Randy Barlow wrote:
It may be possible to automate the process a bit to make it less heavy for developers, though there is some complication for multi-package updates
Some automation would help, sure. But if we are going to do things automatically, why bother with Bodhi at all?
Well, we don't currently have another tool to show results and allow waving of them. We could come up with another app and have yet a different place and workflow from regular releases, but why not stick to the same workflow and avoid making yet another app.
We are already building into a pending tag for the autosigning, from which the packages are moved into the final tag when they are signed. The same process should work for autotesting. Just add it before or after the autosigning in the pipeline, possibly with another intermediate tag.
I think that Bodhi is really the wrong tool for the job here. What you are trying to hit with your shiny hammer may look like a nail to you, but it is really a screw. ;-)
I don't think thats the case here, it's more than we don't want to build another different tool for something that could work with the hammer we have.
To be fair, there was a suggestion that we show results in pagure/src.fedoraproject.org, but to me, that definitely sounds like the wrong place for such things. We want tests tied to a specific update, not the entire package as a whole. (Even thought we could fudge this by having git tags or something to tie results to).
For regular Fedora releases I definitely agree with you, but for rawhide where the only time we bundle package together would be via side-tags, I think reporting test results to src.fp.o would be a valid approach.
Pierre
On Fri, 2018-03-09 at 10:06 +0100, Pierre-Yves Chibon wrote:
On Thu, Mar 08, 2018 at 08:26:30PM -0800, Kevin Fenzi wrote:
On 03/08/2018 07:53 PM, Kevin Kofler wrote:
Randy Barlow wrote:
It may be possible to automate the process a bit to make it less heavy for developers, though there is some complication for multi- package updates
Some automation would help, sure. But if we are going to do things automatically, why bother with Bodhi at all?
Well, we don't currently have another tool to show results and allow waving of them. We could come up with another app and have yet a different place and workflow from regular releases, but why not stick to the same workflow and avoid making yet another app.
We are already building into a pending tag for the autosigning, from which the packages are moved into the final tag when they are signed. The same process should work for autotesting. Just add it before or after the autosigning in the pipeline, possibly with another intermediate tag.
I think that Bodhi is really the wrong tool for the job here. What you are trying to hit with your shiny hammer may look like a nail to you, but it is really a screw. ;-)
I don't think thats the case here, it's more than we don't want to build another different tool for something that could work with the hammer we have.
To be fair, there was a suggestion that we show results in pagure/src.fedoraproject.org, but to me, that definitely sounds like the wrong place for such things. We want tests tied to a specific update, not the entire package as a whole. (Even thought we could fudge this by having git tags or something to tie results to).
For regular Fedora releases I definitely agree with you, but for rawhide where the only time we bundle package together would be via side-tags, I think reporting test results to src.fp.o would be a valid approach.
We could take a hint from how we make software upstream these days:
1. submit a pull request for the package in src.fp.o on the master branch 2. a CI automatically builds this package in Koji 3. merge the pull request into master 4. the build (from step 2) is tagged for Rawhide, without a rebuild
Gating is achieved by preventing merges if the build/tests fail.
On Fri, Mar 09, 2018 at 10:12:37AM +0100, Mathieu Bridon wrote:
We could take a hint from how we make software upstream these days:
- submit a pull request for the package in src.fp.o on the master branch
- a CI automatically builds this package in Koji
- merge the pull request into master
- the build (from step 2) is tagged for Rawhide, without a rebuild
Gating is achieved by preventing merges if the build/tests fail.
This has been discussed but requires a way for koji to "promote" scratch builds to real builds and this isn't a small task. Long term, I do think this is a good idea anyway yes.
Pierre
On Fri, 2018-03-09 at 10:38 +0100, Pierre-Yves Chibon wrote:
On Fri, Mar 09, 2018 at 10:12:37AM +0100, Mathieu Bridon wrote:
We could take a hint from how we make software upstream these days:
- submit a pull request for the package in src.fp.o on the master branch
- a CI automatically builds this package in Koji
- merge the pull request into master
- the build (from step 2) is tagged for Rawhide, without a rebuild
Gating is achieved by preventing merges if the build/tests fail.
This has been discussed but requires a way for koji to "promote" scratch builds to real builds and this isn't a small task. Long term, I do think this is a good idea anyway yes.
It doesn't have to be scratch builds, it could be real builds in a side tag.
On Fri, Mar 09, 2018 at 06:48:56PM +0100, Mathieu Bridon wrote:
On Fri, 2018-03-09 at 10:38 +0100, Pierre-Yves Chibon wrote:
On Fri, Mar 09, 2018 at 10:12:37AM +0100, Mathieu Bridon wrote:
We could take a hint from how we make software upstream these days:
- submit a pull request for the package in src.fp.o on the master branch
- a CI automatically builds this package in Koji
- merge the pull request into master
- the build (from step 2) is tagged for Rawhide, without a rebuild
Gating is achieved by preventing merges if the build/tests fail.
This has been discussed but requires a way for koji to "promote" scratch builds to real builds and this isn't a small task. Long term, I do think this is a good idea anyway yes.
It doesn't have to be scratch builds, it could be real builds in a side tag.
That would register in koji's DB the unique NEVRA which means, the PR can't be updated without bumping the release field.
Pierre
On Mon, Mar 12, 2018, at 5:32 AM, Pierre-Yves Chibon wrote:
That would register in koji's DB the unique NEVRA which means, the PR can't be updated without bumping the release field.
Yep, that problem is exactly (along with the conflict-fest that is %changelog) what holds back any big steps in making PRs-on-distgit more useful.
After removing %changelog, the next step is to make it so e.g. Koji just auto-increments Release, or whatever. In the end, that sort of workflow is what everyone ends up reinventing on *top* of the existing system in one of many versions: https://fedoraproject.org/wiki/Layered_build_scripts_for_package_maintainers
But the PR workflow is what really brings this issue to the fore.
On 03/09/2018 04:06 AM, Pierre-Yves Chibon wrote:
To be fair, there was a suggestion that we show results in pagure/src.fedoraproject.org, but to me, that definitely sounds like the wrong place for such things. We want tests tied to a specific update, not the entire package as a whole. (Even thought we could fudge this by having git tags or something to tie results to).
For regular Fedora releases I definitely agree with you, but for rawhide where the only time we bundle package together would be via side-tags, I think reporting test results to src.fp.o would be a valid approach.
I think this would be great when the intended update has only one package in it. For times when we need to update n packages together it might be tricky, but perhaps we can think of a way to indicate that a set of patches to different repositories are meant to be a single atomic change.
Kevin Fenzi wrote:
To be fair, there was a suggestion that we show results in pagure/src.fedoraproject.org, but to me, that definitely sounds like the wrong place for such things.
I think they should be displayed in Koji on the build info page for the particular build that was tested. That seems the most logical place to me. It is where I would look for it first, in any case.
Kevin Kofler
On 03/08/2018 10:53 PM, Kevin Kofler wrote:
IMHO, this is highly impractical. Having to file Bodhi updates for every Rawhide build will make development cringe to a halt.
The proposal is to automate the filing of Bodhi updates, so in most cases you would not really interact with them unless tests fail. I hinted at a "perhaps 1 day", but it was suggested elsewhere that we could also not have any time restrictions and just send it to stable as soon as automated tests pass. I don't have strong feelings about the policy there, so if we went that direction you might not even notice Bodhi is involved unless the tests fail (in which case you should take action anyway).
Some automation would help, sure. But if we are going to do things automatically, why bother with Bodhi at all?
As nirik said, because we have it. Bodhi's mission in life is gating[0], and we want to gate Rawhide. We have a tool that does that now, and it's easier to use it (even if it would need a few changes) than it is to create another service.
We are already building into a pending tag for the autosigning, from which the packages are moved into the final tag when they are signed. The same process should work for autotesting. Just add it before or after the autosigning in the pipeline, possibly with another intermediate tag.
The "it" is the thing we don't have, outside of a conveniently already created tool.
I think that Bodhi is really the wrong tool for the job here. What you are trying to hit with your shiny hammer may look like a nail to you, but it is really a screw. ;-)
After having been in Bodhi's JSON rendering code today, it doesn't look so shiny to me, but thanks for the compliment.
As I said, Bodhi's mission is gating, and gating is the proposal here, so I think it is actually a good tool to use.
[0] I more generally think of Bodhi as a community driven Koji admin.
On Thu, Mar 08, 2018 at 01:00:32PM -0500, Randy Barlow wrote:
I would like to kick off a general discussion about how we might gate packages in Rawhide. I think it would be nice to get something in place for the Fedora 29 timeframe.
I was waiting to have some more time to work on it to kick off that discussion. This is cool as it means there are more people interested in having this :)
As one of the Bodhi contributors, I am inclined to suggest that we could use Bodhi on Rawhide, similar to how we use it for our stable/branched releases, with more relaxed rules (perhaps 1 day in testing or something simple).
It may be possible to automate the process a bit to make it less heavy for developers, though there is some complication for multi-package updates (more on that in a bit). For simple package updates, we may be able to detect new commits on dist-git, and react to those by automatically starting a Koji build, and automatically filing a Bodhi update when that build is complete. I think that would be pretty nice, and pingou created a PoC[0] to do this about a year ago.
Multi-package updates won't be so easy though. It's not uncommon for us to need to update packages together, and the above workflow would be problematic since it would result in updates with single packages in them rather than updates with multiple packages. Of course, buildroot overrides would be a problem too, since multi-package updates often depend on each other at build time too.
We could create some way for packagers to indicate that a commit (or possibly even a whole repo) is not intended to be "autobuilt/updated". If the packager indicates this then their builds would go into a rawhide-pending (similar to what we do for f27 today). Once they have all their builds (and buildroot overrides) the way they want them, they can create the update.
Another idea that was tossed around in some chats I had with people about this involved a system for packagers to use to create Koji side tags. Bodhi manages BuildRoot Overrides today (with expirations), so perhaps Bodhi could be expanded to also manage Koji side tags (also with expirations). I can't remember all the details about this approach or why it was suggested over the former approach, but I wanted to list it to invite others to chew on it and see if it could work.
If you have other suggestions on how we might gate packages in Rawhide that are wildly different than the above, please feel free to share!
I had a different idea in mind, basically try to keep the experience as close as what it is now. for single package: - packager commit - packager build - build is tagged into a specfic koji tag - test are run on this tag -> results go to src.fp.o - if tests pass - package is moved to another koji tag - package is signed - package is pushed to rawhide - if tests do not pass - do nothing - if the user submits a waiver - move the package to the next koji tag for signing it If a packager does not want to run the tests, we could have a fedpkg build --noci that would directly build the package in the koji tag used for signing the package (useful for mass-rebuild for example)
for multi-package: - packager requests a new side-tag (I'd propose via bodhi) - packager build all the different packages in that side-tag - packager asks for the side tag to be merged - tests are run on all these packages -> results go to src.fp.o --> We probably want to also report them on the bodhi request to merge the tag as otherwise the packager will have to go through all the package to find the one(s) that failed - if tests pass -> cf above
For this to work we could need: - one new koji tag on rawhide (easy) - support for side-tags in bodhi - likely improving the side-tags generation in koji - a few services, likely fedmsg-based, to link the different services (tests results to src.fp.o/bodhi, migrate package to the next koji tag when test results is published, when a waiver is submitted push the package to the next koji tag...)
The idea to automatically build upon commit is neat and something we can look into, but I think it can be approached separately from gating rawhide, one doesn't require the other. Speaking of both changes in one go might confuse things a little.
Pierre
Thanks for starting this discussion and I really want to thank you for offering Bodhi to handle the Rawhide. It would be really sad I some other system was used for Rawhide then for stable versions.
Dne 9.3.2018 v 10:20 Pierre-Yves Chibon napsal(a):
On Thu, Mar 08, 2018 at 01:00:32PM -0500, Randy Barlow wrote:
I would like to kick off a general discussion about how we might gate packages in Rawhide. I think it would be nice to get something in place for the Fedora 29 timeframe.
I was waiting to have some more time to work on it to kick off that discussion. This is cool as it means there are more people interested in having this :)
As one of the Bodhi contributors, I am inclined to suggest that we could use Bodhi on Rawhide, similar to how we use it for our stable/branched releases, with more relaxed rules (perhaps 1 day in testing or something simple).
It may be possible to automate the process a bit to make it less heavy for developers, though there is some complication for multi-package updates (more on that in a bit). For simple package updates, we may be able to detect new commits on dist-git, and react to those by automatically starting a Koji build, and automatically filing a Bodhi update when that build is complete. I think that would be pretty nice, and pingou created a PoC[0] to do this about a year ago.
Multi-package updates won't be so easy though. It's not uncommon for us to need to update packages together, and the above workflow would be problematic since it would result in updates with single packages in them rather than updates with multiple packages. Of course, buildroot overrides would be a problem too, since multi-package updates often depend on each other at build time too.
We could create some way for packagers to indicate that a commit (or possibly even a whole repo) is not intended to be "autobuilt/updated". If the packager indicates this then their builds would go into a rawhide-pending (similar to what we do for f27 today). Once they have all their builds (and buildroot overrides) the way they want them, they can create the update.
Another idea that was tossed around in some chats I had with people about this involved a system for packagers to use to create Koji side tags. Bodhi manages BuildRoot Overrides today (with expirations), so perhaps Bodhi could be expanded to also manage Koji side tags (also with expirations). I can't remember all the details about this approach or why it was suggested over the former approach, but I wanted to list it to invite others to chew on it and see if it could work.
If you have other suggestions on how we might gate packages in Rawhide that are wildly different than the above, please feel free to share!
I had a different idea in mind, basically try to keep the experience as close as what it is now. for single package:
- packager commit
- packager build
- build is tagged into a specfic koji tag
- test are run on this tag -> results go to src.fp.o
- if tests pass
- package is moved to another koji tag
- package is signed
- package is pushed to rawhide
- if tests do not pass
- do nothing
- if the user submits a waiver
- move the package to the next koji tag for signing it
If a packager does not want to run the tests, we could have a fedpkg build --noci that would directly build the package in the koji tag used for signing the package (useful for mass-rebuild for example)
for multi-package:
- packager requests a new side-tag (I'd propose via bodhi)
- packager build all the different packages in that side-tag
- packager asks for the side tag to be merged
- tests are run on all these packages -> results go to src.fp.o --> We probably want to also report them on the bodhi request to merge the tag as otherwise the packager will have to go through all the package to find the one(s) that failed
- if tests pass -> cf above
I more or less like your proposal. But I have to highlight one danger of sidetags and that is you have use the "--target" option. If you forget it by accident (and it is quite easy to forget), you will unintentionally mess the main Rawhide repository. From this POV is much safer to use build overrides. Also, using buildroot overrides is probably more infrastructure resources friendly. BTW the buildroot overrides could be submitted automatically by "fedpkg build", with possible opt-in/out (depends what will qualify to be better default).
Also, using side-tags without appropriate branches is wasted opportunity. If I could have "master-mysidetag" branch, I would use to build my packages and side-tag merge would mean to merge dist-git as well as the repository, it would be even more meaningful.
For this to work we could need:
- one new koji tag on rawhide (easy)
- support for side-tags in bodhi
- likely improving the side-tags generation in koji
- a few services, likely fedmsg-based, to link the different services (tests results to src.fp.o/bodhi, migrate package to the next koji tag when test results is published, when a waiver is submitted push the package to the next koji tag...)
The idea to automatically build upon commit is neat and something we can look into, but I think it can be approached separately from gating rawhide, one doesn't require the other.
I really don't like the idea of building something based on commit. Each commit should be atomic and meaningful. If you will build based on commit, that could motivate maintainers to put more unrelated changes into single commit. And of course, you should be speaking rather about "build unpon push", since push can contain more commits.
V.
Speaking of both changes in one go might confuse things a little.
Pierre
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Fri, Mar 09, 2018 at 10:52:27AM +0100, Vít Ondruch wrote:
I had a different idea in mind, basically try to keep the experience as close as what it is now. for single package:
- packager commit
- packager build
- build is tagged into a specfic koji tag
- test are run on this tag -> results go to src.fp.o
- if tests pass
- package is moved to another koji tag
- package is signed
- package is pushed to rawhide
- if tests do not pass
- do nothing
- if the user submits a waiver
- move the package to the next koji tag for signing it
If a packager does not want to run the tests, we could have a fedpkg build --noci that would directly build the package in the koji tag used for signing the package (useful for mass-rebuild for example)
for multi-package:
- packager requests a new side-tag (I'd propose via bodhi)
- packager build all the different packages in that side-tag
- packager asks for the side tag to be merged
- tests are run on all these packages -> results go to src.fp.o --> We probably want to also report them on the bodhi request to merge the tag as otherwise the packager will have to go through all the package to find the one(s) that failed
- if tests pass -> cf above
I more or less like your proposal. But I have to highlight one danger of sidetags and that is you have use the "--target" option. If you forget it by accident (and it is quite easy to forget), you will unintentionally mess the main Rawhide repository. From this POV is much safer to use build
You wouldn't mess the main repo as it would not have been built against the new version of the library you updated and are trying to rebuild against. So for that package there is no change compared to what was already in rawhide. And for the side-tag when trying to merge it changes are that the tests would fail no?
overrides. Also, using buildroot overrides is probably more infrastructure resources friendly. BTW the buildroot overrides could be submitted automatically by "fedpkg build", with possible opt-in/out (depends what will qualify to be better default).
Buildroot override means we gate the packages for an unknown time. The buildroot override means: adding to the buildroot something that isn't already there. So how would we know when to wait (depending libraries need a rebuild) and when not to wait (single-package update)? If we always push, we end up with the current rawhide situation no? If we default to wait, we end up with what we have currently in stable releases and are changing quite our packaging workflow. I'm not saying it's not an option, just that it needs to be made clear to everyone.
Also, using side-tags without appropriate branches is wasted opportunity. If I could have "master-mysidetag" branch, I would use to build my packages and side-tag merge would mean to merge dist-git as well as the repository, it would be even more meaningful.
This seems a lot of work without necessary a lot of gain tbh.
Pierre
Dne 9.3.2018 v 11:24 Pierre-Yves Chibon napsal(a):
On Fri, Mar 09, 2018 at 10:52:27AM +0100, Vít Ondruch wrote:
I had a different idea in mind, basically try to keep the experience as close as what it is now. for single package:
- packager commit
- packager build
- build is tagged into a specfic koji tag
- test are run on this tag -> results go to src.fp.o
- if tests pass
- package is moved to another koji tag
- package is signed
- package is pushed to rawhide
- if tests do not pass
- do nothing
- if the user submits a waiver
- move the package to the next koji tag for signing it
If a packager does not want to run the tests, we could have a fedpkg build --noci that would directly build the package in the koji tag used for signing the package (useful for mass-rebuild for example)
for multi-package:
- packager requests a new side-tag (I'd propose via bodhi)
- packager build all the different packages in that side-tag
- packager asks for the side tag to be merged
- tests are run on all these packages -> results go to src.fp.o --> We probably want to also report them on the bodhi request to merge the tag as otherwise the packager will have to go through all the package to find the one(s) that failed
- if tests pass -> cf above
I more or less like your proposal. But I have to highlight one danger of sidetags and that is you have use the "--target" option. If you forget it by accident (and it is quite easy to forget), you will unintentionally mess the main Rawhide repository. From this POV is much safer to use build
You wouldn't mess the main repo as it would not have been built against the new version of the library you updated and are trying to rebuild against.
It depends what you are building. If are building package foo with soname bump and its dependencies, the if you build the foo by accident into the official tag, then you are in troubles.
And for that dependencies, it depends. If you bum some dependency in the middle of the chain, to be compatible with the new soname, but it requires some other runtime dependencies, and you build it in the official tag instread of the side tag, you are in trouble again.
So for that package there is no change compared to what was already in rawhide. And for the side-tag when trying to merge it changes are that the tests would fail no?
overrides. Also, using buildroot overrides is probably more infrastructure resources friendly. BTW the buildroot overrides could be submitted automatically by "fedpkg build", with possible opt-in/out (depends what will qualify to be better default).
Buildroot override means we gate the packages for an unknown time. The buildroot override means: adding to the buildroot something that isn't already there. So how would we know when to wait (depending libraries need a rebuild) and when not to wait (single-package update)? If we always push, we end up with the current rawhide situation no?
The buildroot override can be sumbitted or expired. You can't do that in Rawhide now. So there are two possibilities:
1) We automatically submit override, but it could be expired based on testing. 2) If more packages are needed to build, the override can be submitted manually.
If we default to wait, we end up with what we have currently in stable releases and are changing quite our packaging workflow. I'm not saying it's not an option, just that it needs to be made clear to everyone.
Also, using side-tags without appropriate branches is wasted opportunity. If I could have "master-mysidetag" branch, I would use to build my packages and side-tag merge would mean to merge dist-git as well as the repository, it would be even more meaningful.
This seems a lot of work without necessary a lot of gain tbh.
It was just me dreaming :) I realize it is a lot of work so this would be just NTH.
V.
Pierre
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On 03/09/2018 04:20 AM, Pierre-Yves Chibon wrote:
I had a different idea in mind, basically try to keep the experience as close as what it is now. for single package:
- packager commit
- packager build
- build is tagged into a specfic koji tag
- test are run on this tag -> results go to src.fp.o
- if tests pass
- package is moved to another koji tag
- package is signed
- package is pushed to rawhide
- if tests do not pass
- do nothing
- if the user submits a waiver
- move the package to the next koji tag for signing it
If a packager does not want to run the tests, we could have a fedpkg build --noci that would directly build the package in the koji tag used for signing the package (useful for mass-rebuild for example)
I would really like to have tests in src.fpo, but I think they would be most helpfully displayed on pull requests, rather than after I've already made a Koji build. This is more similar to the workflow I'm accustomed to when I work on Bodhi's upstream code - I submit a PR and CentOS CI (which is very nice btw) chews on it and tells me if I'm good to merge. For a single package update (which is most of what I personally do) it would be very nice to have this before I merge my PR in Pagure.
So all that to say that I like the above, but I'd like to propose that there also be some tests run before I commit (i.e., merge the PR) and build in Koji (for cases where we use PRs. I'm not proposing we require PRs, but that it be available for those of us who would like to use that workflow). PRs do get the simplekojici build today, which is cool. I'd really just like a few more tests than that to be run as well.
for multi-package:
- packager requests a new side-tag (I'd propose via bodhi)
- packager build all the different packages in that side-tag
- packager asks for the side tag to be merged
- tests are run on all these packages -> results go to src.fp.o
At this point there aren't pull requests, right? Where would we display this in src.fp.o that would make it clear which change the test results pertain to?
--> We probably want to also report them on the bodhi request to merge the tag as otherwise the packager will have to go through all the package to find the one(s) that failed
Do you mean that Bodhi would display them on a side tag page? Or in this scenario are you suggesting that a Bodhi update has been created? The update would be less friction since we already have code to display test results on Bodhi updates.
The idea to automatically build upon commit is neat and something we can look into, but I think it can be approached separately from gating rawhide, one doesn't require the other. Speaking of both changes in one go might confuse things a little.
True, it's orthogonal.
On Mon, Mar 12, 2018 at 06:05:22PM -0400, Randy Barlow wrote:
On 03/09/2018 04:20 AM, Pierre-Yves Chibon wrote:
I had a different idea in mind, basically try to keep the experience as close as what it is now. for single package:
- packager commit
- packager build
- build is tagged into a specfic koji tag
- test are run on this tag -> results go to src.fp.o
- if tests pass
- package is moved to another koji tag
- package is signed
- package is pushed to rawhide
- if tests do not pass
- do nothing
- if the user submits a waiver
- move the package to the next koji tag for signing it
If a packager does not want to run the tests, we could have a fedpkg build --noci that would directly build the package in the koji tag used for signing the package (useful for mass-rebuild for example)
I would really like to have tests in src.fpo, but I think they would be most helpfully displayed on pull requests, rather than after I've already made a Koji build. This is more similar to the workflow I'm accustomed to when I work on Bodhi's upstream code - I submit a PR and CentOS CI (which is very nice btw) chews on it and tells me if I'm good to merge. For a single package update (which is most of what I personally do) it would be very nice to have this before I merge my PR in Pagure.
I don't think the two are antagonist, one is the pre-merge/PR workflow with the tests ran on the PR, the other is post-merge. Both approach can already be supported today.
So all that to say that I like the above, but I'd like to propose that there also be some tests run before I commit (i.e., merge the PR) and build in Koji (for cases where we use PRs. I'm not proposing we require PRs, but that it be available for those of us who would like to use that workflow). PRs do get the simplekojici build today, which is cool. I'd really just like a few more tests than that to be run as well.
We're working on the pipeline that will run the tests present on all and every package having tests (instead of the restricted set of packages included in Fedora Atomic Host). The next step will be PR testing. Feel free to poke Dominik Perpeet if you want more details :)
for multi-package:
- packager requests a new side-tag (I'd propose via bodhi)
- packager build all the different packages in that side-tag
- packager asks for the side tag to be merged
- tests are run on all these packages -> results go to src.fp.o
At this point there aren't pull requests, right? Where would we display this in src.fp.o that would make it clear which change the test results pertain to?
In the commit page itself, for example: https://src.fedoraproject.org/rpms/perl-IO-HTML/c/69a460858b8c1f47bb6320faa6... (that's the fbom service: https://pagure.io/fedora-ci/fbom which flags in src.fp.o successful build made in koji, it's not currently running, we need to process the RFR: https://pagure.io/fedora-infrastructure/issue/6717 so don't look for it on every repo)
--> We probably want to also report them on the bodhi request to merge the tag as otherwise the packager will have to go through all the package to find the one(s) that failedDo you mean that Bodhi would display them on a side tag page?
That was my proposal yes.
Or in this scenario are you suggesting that a Bodhi update has been created?
I don't really see the need for one, so in this proposal, the bodhi update mechanism wasn't used. Bodhi was used for only for managing side-tags and since merging a side-tag will be impacted by the test results including that info there made sense to me.
The update would be less friction since we already have code to display test results on Bodhi updates.
That's fair, on the other side we're going to have to build an entirely new part in bodhi and we potentially may want to present the test results differently there (don't know, something to think about).
Pierre
On Fri, Mar 09, 2018 at 10:20:12AM +0100, Pierre-Yves Chibon wrote:
On Thu, Mar 08, 2018 at 01:00:32PM -0500, Randy Barlow wrote:
I would like to kick off a general discussion about how we might gate packages in Rawhide. I think it would be nice to get something in place for the Fedora 29 timeframe.
I was waiting to have some more time to work on it to kick off that discussion. This is cool as it means there are more people interested in having this :)
As one of the Bodhi contributors, I am inclined to suggest that we could use Bodhi on Rawhide, similar to how we use it for our stable/branched releases, with more relaxed rules (perhaps 1 day in testing or something simple).
It may be possible to automate the process a bit to make it less heavy for developers, though there is some complication for multi-package updates (more on that in a bit). For simple package updates, we may be able to detect new commits on dist-git, and react to those by automatically starting a Koji build, and automatically filing a Bodhi update when that build is complete. I think that would be pretty nice, and pingou created a PoC[0] to do this about a year ago.
Multi-package updates won't be so easy though. It's not uncommon for us to need to update packages together, and the above workflow would be problematic since it would result in updates with single packages in them rather than updates with multiple packages. Of course, buildroot overrides would be a problem too, since multi-package updates often depend on each other at build time too.
We could create some way for packagers to indicate that a commit (or possibly even a whole repo) is not intended to be "autobuilt/updated". If the packager indicates this then their builds would go into a rawhide-pending (similar to what we do for f27 today). Once they have all their builds (and buildroot overrides) the way they want them, they can create the update.
Another idea that was tossed around in some chats I had with people about this involved a system for packagers to use to create Koji side tags. Bodhi manages BuildRoot Overrides today (with expirations), so perhaps Bodhi could be expanded to also manage Koji side tags (also with expirations). I can't remember all the details about this approach or why it was suggested over the former approach, but I wanted to list it to invite others to chew on it and see if it could work.
If you have other suggestions on how we might gate packages in Rawhide that are wildly different than the above, please feel free to share!
I had a different idea in mind, basically try to keep the experience as close as what it is now. for single package:
- packager commit
- packager build
- build is tagged into a specfic koji tag
- test are run on this tag -> results go to src.fp.o
- if tests pass
- package is moved to another koji tag
- package is signed
- package is pushed to rawhide
- if tests do not pass
- do nothing
- if the user submits a waiver
- move the package to the next koji tag for signing it
If a packager does not want to run the tests, we could have a fedpkg build --noci that would directly build the package in the koji tag used for signing the package (useful for mass-rebuild for example)
In the graphs, the red arrows are always full, i.e. signify manual steps. Is this an oversight, or are so many manual steps actually needed in that workflow?
for multi-package:
- packager requests a new side-tag (I'd propose via bodhi)
- packager build all the different packages in that side-tag
- packager asks for the side tag to be merged
- tests are run on all these packages
Are those tests run will all the packages in the side-tag, or with each package separately installed onto current rawhide? I assume the first, but please clarify that.
-> results go to src.fp.o --> We probably want to also report them on the bodhi request to merge the tag as otherwise the packager will have to go through all the package to find the one(s) that failed
- if tests pass -> cf above
I see two advantages of the workflow with bodhi: - we can reuse existing reporting interfaces, for tests results, but also for user comments. In particular I expect that the integration of bodhi with greenwave and gating CI will keep changing and increasing, as required by updates for stable releases, and this would have to be duplicated in pagure, if bodhi is not used. Such features don't seem to be applicable to upstream pagure, but mostly to the src.fp.o, and it'll be an ongoing burden.
- using bodhi updates gives some additional flexibility, for example we (in the future) could make the "push bodhi updates" step *non*-automatic, so that for example a positive karma from another maintainer would be required. This could be used for example as "send mail to fedora-devel, ask for checking update FEDORA-1223345464" on especially tricky or big changes.
All in all, the simpler workflow looks attractive as first blush, but I wouldn't discard the more complicated option just yet. If all the steps were automated, the fact that bodhi is involved would not cause additional work to the maintainer.
Zbyszek
For this to work we could need:
- one new koji tag on rawhide (easy)
- support for side-tags in bodhi
- likely improving the side-tags generation in koji
- a few services, likely fedmsg-based, to link the different services (tests results to src.fp.o/bodhi, migrate package to the next koji tag when test results is published, when a waiver is submitted push the package to the next koji tag...)
The idea to automatically build upon commit is neat and something we can look into, but I think it can be approached separately from gating rawhide, one doesn't require the other. Speaking of both changes in one go might confuse things a little.
Pierre
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Wed, Mar 28, 2018 at 09:09:09PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Fri, Mar 09, 2018 at 10:20:12AM +0100, Pierre-Yves Chibon wrote:
On Thu, Mar 08, 2018 at 01:00:32PM -0500, Randy Barlow wrote:
I would like to kick off a general discussion about how we might gate packages in Rawhide. I think it would be nice to get something in place for the Fedora 29 timeframe.
I was waiting to have some more time to work on it to kick off that discussion. This is cool as it means there are more people interested in having this :)
As one of the Bodhi contributors, I am inclined to suggest that we could use Bodhi on Rawhide, similar to how we use it for our stable/branched releases, with more relaxed rules (perhaps 1 day in testing or something simple).
It may be possible to automate the process a bit to make it less heavy for developers, though there is some complication for multi-package updates (more on that in a bit). For simple package updates, we may be able to detect new commits on dist-git, and react to those by automatically starting a Koji build, and automatically filing a Bodhi update when that build is complete. I think that would be pretty nice, and pingou created a PoC[0] to do this about a year ago.
Multi-package updates won't be so easy though. It's not uncommon for us to need to update packages together, and the above workflow would be problematic since it would result in updates with single packages in them rather than updates with multiple packages. Of course, buildroot overrides would be a problem too, since multi-package updates often depend on each other at build time too.
We could create some way for packagers to indicate that a commit (or possibly even a whole repo) is not intended to be "autobuilt/updated". If the packager indicates this then their builds would go into a rawhide-pending (similar to what we do for f27 today). Once they have all their builds (and buildroot overrides) the way they want them, they can create the update.
Another idea that was tossed around in some chats I had with people about this involved a system for packagers to use to create Koji side tags. Bodhi manages BuildRoot Overrides today (with expirations), so perhaps Bodhi could be expanded to also manage Koji side tags (also with expirations). I can't remember all the details about this approach or why it was suggested over the former approach, but I wanted to list it to invite others to chew on it and see if it could work.
If you have other suggestions on how we might gate packages in Rawhide that are wildly different than the above, please feel free to share!
I had a different idea in mind, basically try to keep the experience as close as what it is now. for single package:
- packager commit
- packager build
- build is tagged into a specfic koji tag
- test are run on this tag -> results go to src.fp.o
- if tests pass
- package is moved to another koji tag
- package is signed
- package is pushed to rawhide
- if tests do not pass
- do nothing
- if the user submits a waiver
- move the package to the next koji tag for signing it
If a packager does not want to run the tests, we could have a fedpkg build --noci that would directly build the package in the koji tag used for signing the package (useful for mass-rebuild for example)
In the graphs, the red arrows are always full, i.e. signify manual steps. Is this an oversight, or are so many manual steps actually needed in that workflow?
The one between fedpkg build --noci and koji build should be dashed indeed, the others are plain as they should I believe.
for multi-package:
- packager requests a new side-tag (I'd propose via bodhi)
- packager build all the different packages in that side-tag
- packager asks for the side tag to be merged
- tests are run on all these packages
Are those tests run will all the packages in the side-tag, or with each package separately installed onto current rawhide? I assume the first, but please clarify that.
They should run with all the packages in the side-tag I think.
-> results go to src.fp.o --> We probably want to also report them on the bodhi request to merge the tag as otherwise the packager will have to go through all the package to find the one(s) that failed
- if tests pass -> cf above
I see two advantages of the workflow with bodhi:
- we can reuse existing reporting interfaces, for tests results, but also for user comments. In particular I expect that the integration of bodhi with greenwave and gating CI will keep changing and increasing, as required by updates for stable releases, and this would have to be duplicated in pagure, if bodhi is not used. Such features don't seem to be applicable to upstream pagure, but mostly to the src.fp.o, and it'll be an ongoing burden.
Pagure already support reporting CI results (both on commits and on PRs) and in the workflow without bodhi, the gating wouldn't be done in pagure itself so less burden there.
- using bodhi updates gives some additional flexibility, for example we (in the future) could make the "push bodhi updates" step *non*-automatic, so that for example a positive karma from another maintainer would be required. This could be used for example as "send mail to fedora-devel, ask for checking update FEDORA-1223345464" on especially tricky or big changes.
This should be doable, though it will need a community decision for this. At first, I don't think we should allow karma on rawhide's packages, just rely on automated tests. But yes, using bodhi would allow for such a workflow if we one day desire it.
All in all, the simpler workflow looks attractive as first blush, but I wouldn't discard the more complicated option just yet. If all the steps were automated, the fact that bodhi is involved would not cause additional work to the maintainer.
Agreed on both accounts.
Pierre
A more modern architecture would be something closer to pull requests on dist-git; tests report into the PR the same way github works today.
This would work *particularly* well if it was easy to make a pull request across multiple packages, which would be obvious and easy if we just one git repo rather than thousands. (Or perhaps core/extras git repos).
Examples of systems that work this way:
https://github.com/NixOS/nixpkgs https://github.com/openembedded/openembedded-core https://github.com/clearlinux/clr-bundles
On Fri, Mar 9, 2018, at 9:23 AM, Colin Walters wrote:
A more modern architecture would be something closer to pull requests on dist-git; tests report into the PR the same way github works today.
This would work *particularly* well if it was easy to make a pull request across multiple packages, which would be obvious and easy if we just one git repo rather than thousands. (Or perhaps core/extras git repos).
Examples of systems that work this way:
https://github.com/NixOS/nixpkgs https://github.com/openembedded/openembedded-core https://github.com/clearlinux/clr-bundles
Oh and how could I forget:
Dne 9.3.2018 v 15:23 Colin Walters napsal(a):
A more modern architecture would be something closer to pull requests on dist-git; tests report into the PR the same way github works today.
May be if we had all .spec files in single repository .....
V.
This would work *particularly* well if it was easy to make a pull request across multiple packages, which would be obvious and easy if we just one git repo rather than thousands. (Or perhaps core/extras git repos).
Examples of systems that work this way:
https://github.com/NixOS/nixpkgs https://github.com/openembedded/openembedded-core https://github.com/clearlinux/clr-bundles _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
Greetings fellow Fedorans!
I would like to kick off a general discussion about how we might gate packages in Rawhide. I think it would be nice to get something in place for the Fedora 29 timeframe.
As one of the Bodhi contributors, I am inclined to suggest that we could use Bodhi on Rawhide, similar to how we use it for our stable/branched releases, with more relaxed rules (perhaps 1 day in testing or something simple).
We need a better approach for this, since I am not exactly sure what this solve as it just waiting for only 1 day.
It may be possible to automate the process a bit to make it less heavy for developers, though there is some complication for multi-package updates (more on that in a bit). For simple package updates, we may be able to detect new commits on dist-git, and react to those by automatically starting a Koji build, and automatically filing a Bodhi update when that build is complete. I think that would be pretty nice, and pingou created a PoC[0] to do this about a year ago.
I am not sure if this is possible with arbitrary branches. People can now build from different branches that does not follow fedora release branches, so it will be difficult to identify which buildroot to build it against. And pingou is planning to update pagure over dist-git to able to build from one branch rather than different branches.
Multi-package updates won't be so easy though. It's not uncommon for us to need to update packages together, and the above workflow would be problematic since it would result in updates with single packages in them rather than updates with multiple packages. Of course, buildroot overrides would be a problem too, since multi-package updates often depend on each other at build time too.
We could create some way for packagers to indicate that a commit (or possibly even a whole repo) is not intended to be "autobuilt/updated". If the packager indicates this then their builds would go into a rawhide-pending (similar to what we do for f27 today). Once they have all their builds (and buildroot overrides) the way they want them, they can create the update.
Same as my above comment.
Another idea that was tossed around in some chats I had with people about this involved a system for packagers to use to create Koji side tags. Bodhi manages BuildRoot Overrides today (with expirations), so perhaps Bodhi could be expanded to also manage Koji side tags (also with expirations). I can't remember all the details about this approach or why it was suggested over the former approach, but I wanted to list it to invite others to chew on it and see if it could work.
If you have other suggestions on how we might gate packages in Rawhide that are wildly different than the above, please feel free to share!
On 03/15/2018 04:57 PM, Mohan Boddu wrote:
As one of the Bodhi contributors, I am inclined to suggest that we could use Bodhi on Rawhide, similar to how we use it for our stable/branched releases, with more relaxed rules (perhaps 1 day in testing or something simple).
We need a better approach for this, since I am not exactly sure what this solve as it just waiting for only 1 day.
It will introduce automated test gating, which will help identify a few common problems (like missing dependencies). As mentioned elsewhere in the thread, we wouldn't even have to make it wait a day if we don't want to - we could send it on its way once tests pass (this would simply be a matter of policy).
Good Morning Everyone,
Based on the outcome of this discussion, I started trying to draw how the process to update a package in rawhide would look like with rawhide being gated on tests.
There are currently two proposals: - one that does not involve bodhi updates - one that does
Both proposals have a different flow for single-package update and multi-packages update.
Here is what I came up with.
Without bodhi: - Single package update https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide.png - Multi-packages update https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_MultiplePkgs.pn...
With bodhi: - Single package update https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_bodhi.png - Multi-packages update https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_MultiplePkgs_bo...
Outside of this workflow things we know we want to have/keep working: - Keep chain-build working -> would one of the proposal facilitate that? - Have a way to run tests against an entire side-tag before merging it - Have a way to ask the tools to re-check greenwave's decision (so that false negative can be waived and the process continued)
mizdebsk suggested on IRC two ideas which I think would be worth looking into a little bit down the road once we got the basis done: - new side-tag could be automatically generated from fedpkg build command - packagers could define a list of packages that should be rebuilt in the side-tag and when all of these packages have been successfully rebuilt, the request to merge the side-tag is automatically created This would allow to use chain-build and let the entire process be automated.
What do you think?
Pierre
On Wed, Mar 28, 2018, 12:53 Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone,
Good morning!
Based on the outcome of this discussion, I started trying to draw how the process to update a package in rawhide would look like with rawhide being gated on tests.
There are currently two proposals:
- one that does not involve bodhi updates
- one that does
Both proposals have a different flow for single-package update and multi-packages update.
Here is what I came up with.
Without bodhi:
- Single package update https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide.png
- Multi-packages update
https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_MultiplePkgs.pn...
With bodhi:
- Single package update https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_bodhi.png
- Multi-packages update
https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_MultiplePkgs_bo...
Outside of this workflow things we know we want to have/keep working:
- Keep chain-build working -> would one of the proposal facilitate that?
- Have a way to run tests against an entire side-tag before merging it
- Have a way to ask the tools to re-check greenwave's decision (so that
false negative can be waived and the process continued)
mizdebsk suggested on IRC two ideas which I think would be worth looking into a little bit down the road once we got the basis done:
- new side-tag could be automatically generated from fedpkg build command
- packagers could define a list of packages that should be rebuilt in the side-tag and when all of these packages have been successfully rebuilt,
the request to merge the side-tag is automatically created This would allow to use chain-build and let the entire process be automated.
What do you think?
Looking at your two(four) suggested workflows, they look very similar - there's just bodhi sitting between components in the second case, acting as a MITM, but not really adding anything useful compared to the case without bodhi (please correct me if I am wrong).
So, just to state my opinion (with my packager hat on), I would prefer the first case: without involving additional bodhi updates for rawhide builds. Additionally, these workflows also closely resemble the current workflows for submitting packages to rawhide, which is nice.
Whichever direction fedora will choose in the future, I think being able to test single updates before tagging them to rawhide, and especially being able to run tests against whole side-tags, would be a huge improvement over the status quo, and would especially improve the stability and "composability" of rawhide (and would probably cut down on the notorious number of fedora release delays, as a result).
Fabio
Pierre _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Wed, Mar 28, 2018 at 11:16:35AM +0000, Fabio Valentini wrote:
On Wed, Mar 28, 2018, 12:53 Pierre-Yves Chibon pingou@pingoured.fr wrote: Based on the outcome of this discussion, I started trying to draw how the process to update a package in rawhide would look like with rawhide being gated on tests.
There are currently two proposals: - one that does not involve bodhi updates - one that does Both proposals have a different flow for single-package update and multi-packages update. Here is what I came up with. Without bodhi: - Single package update  https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide.png - Multi-packages update  https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_MultiplePkgs.png With bodhi: - Single package update  https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_bodhi.png - Multi-packages update  https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_MultiplePkgs_bodhi.png Outside of this workflow things we know we want to have/keep working: - Keep chain-build working  -> would one of the proposal facilitate that? - Have a way to run tests against an entire side-tag before merging it - Have a way to ask the tools to re-check greenwave's decision (so that false  negative can be waived and the process continued) mizdebsk suggested on IRC two ideas which I think would be worth looking into a little bit down the road once we got the basis done: - new side-tag could be automatically generated from fedpkg build command - packagers could define a list of packages that should be rebuilt in the  side-tag and when all of these packages have been successfully rebuilt, the  request to merge the side-tag is automatically created  This would allow to use chain-build and let the entire process be automated. What do you think?Looking at your two(four) suggested workflows, they look very similar - there's just bodhi sitting between components in the second case, acting as a MITM, but not really adding anything useful compared to the case without bodhi (please correct me if I am wrong).
They are quite close indeed.
So, just to state my opinion (with my packager hat on), I would prefer the first case: without involving additional bodhi updates for rawhide builds. Additionally, these workflows also closely resemble the current workflows for submitting packages to rawhide, which is nice.
That is also my take on it, but including bodhi has also some benefit, one of which is that test results for rawhide and stable releases end up there instead of having test results for rawhide in src.fp.o (and bodhi for the case of multi-packages) vs bodhi for stable releases.
Whichever direction fedora will choose in the future, I think being able to test single updates before tagging them to rawhide, and especially being able to run tests against whole side-tags
I think being able to run tests on a side-tag before request for it to be merged is a good feature, I am less sure about this for single-package. Could you expand? My take is that for single package: you build it, if tests pass it goes into rawhide (as today) if they fail, it doesn't go anywhere and as packager we can either fix the issue, waive their results and/or fix the test.
Pierre
On Wed, Mar 28, 2018, 14:51 Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Wed, Mar 28, 2018 at 11:16:35AM +0000, Fabio Valentini wrote:
On Wed, Mar 28, 2018, 12:53 Pierre-Yves Chibon pingou@pingoured.fr wrote: Based on the outcome of this discussion, I started trying to draw
how
the process to update a package in rawhide would look like with rawhide being gated on tests. There are currently two proposals: - one that does not involve bodhi updates - one that does Both proposals have a different flow for single-package update and multi-packages update. Here is what I came up with. Without bodhi: - Single package update  https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide.png - Multi-packages update Âhttps://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_MultiplePkgs.pn...
With bodhi: - Single package update Âhttps://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_bodhi.png
- Multi-packages update Âhttps://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_MultiplePkgs_bo...
Outside of this workflow things we know we want to have/keepworking:
- Keep chain-build working  -> would one of the proposal facilitate that? - Have a way to run tests against an entire side-tag before mergingit
- Have a way to ask the tools to re-check greenwave's decision (sothat
false  negative can be waived and the process continued) mizdebsk suggested on IRC two ideas which I think would be worthlooking
into a little bit down the road once we got the basis done: - new side-tag could be automatically generated from fedpkg build command - packagers could define a list of packages that should be rebuiltin
the  side-tag and when all of these packages have been successfully rebuilt, the  request to merge the side-tag is automatically created  This would allow to use chain-build and let the entire process be automated. What do you think?Looking at your two(four) suggested workflows, they look very similar
there's just bodhi sitting between components in the second case,
acting
as a MITM, but not really adding anything useful compared to the case without bodhi (please correct me if I am wrong).
They are quite close indeed.
So, just to state my opinion (with my packager hat on), I would
prefer the
first case: without involving additional bodhi updates for rawhide
builds.
Additionally, these workflows also closely resemble the current
workflows
for submitting packages to rawhide, which is nice.
That is also my take on it, but including bodhi has also some benefit, one of which is that test results for rawhide and stable releases end up there instead of having test results for rawhide in src.fp.o (and bodhi for the case of multi-packages) vs bodhi for stable releases.
That's true, but I'm not sure making the update process more complicated is worth the benefit of having test results in one place.
Whichever direction fedora will choose in the future, I think being
able
to test single updates before tagging them to rawhide, and especially being able to run tests against whole side-tags
I think being able to run tests on a side-tag before request for it to be merged is a good feature, I am less sure about this for single-package. Could you expand? My take is that for single package: you build it, if tests pass it goes into rawhide (as today) if they fail, it doesn't go anywhere and as packager we can either fix the issue, waive their results and/or fix the test.
One example where running tests against a single-package update would be nice IMO would be for toolchain and base packages, for example, updates to annobin or binutils, where the answer to "Does this update break compilation with GCC?" (which could be added as a test case) would be vital in determining if the package should be pushed to rawhide or not.
Hope that makes it more clear what I meant by "it also would be nice for single-package updates".
Pierre _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Wed, Mar 28, 2018 at 01:39:34PM +0000, Fabio Valentini wrote:
One example where running tests against a single-package update would be nice IMO would be for toolchain and base packages, for example, updates to annobin or binutils, where the answer to "Does this update break compilation with GCC?" (which could be added as a test case) would be vital in determining if the package should be pushed to rawhide or not. Hope that makes it more clear what I meant by "it also would be nice for single-package updates".
I think I follow you there, what I don't follow is the difference between this and the build not landing in rawhide because it failed its tests.
Or are you referring to: pre-commit testing, in other words pull-request testing? Knowing if pushing annobin or binutils break compiling GCC before pushing the commit into the git repo? So, opening a PR against annobin/binutils, running the tests and if they pass then push to the git repo and build in rawhide?
If that's what you have in mind, this is definitively in the roadmap (PR testing) but will be tracked separately from gating rawhide packages since PR testing will concern all branches not just rawhide.
Pierre
On Wed, Mar 28, 2018, 15:47 Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Wed, Mar 28, 2018 at 01:39:34PM +0000, Fabio Valentini wrote:
One example where running tests against a single-package update would
be
nice IMO would be for toolchain and base packages, for example,
updates to
annobin or binutils, where the answer to "Does this update break compilation with GCC?" (which could be added as a test case) would be vital in determining if the package should be pushed to rawhide or
not.
Hope that makes it more clear what I meant by "it also would be nice
for
single-package updates".
I think I follow you there, what I don't follow is the difference between this and the build not landing in rawhide because it failed its tests.
Or are you referring to: pre-commit testing, in other words pull-request testing?
No, that's not what I meant (although testing PRs would be nice for the future, too).
I just wanted to express that gating rawhide updates depending on test results is meaningful not only for the proposed merging of side-tags, but also for single important packages.
Knowing if pushing annobin or binutils break compiling GCC before pushing
the commit into the git repo? So, opening a PR against annobin/binutils, running the tests and if they pass then push to the git repo and build in rawhide?
If that's what you have in mind, this is definitively in the roadmap (PR testing) but will be tracked separately from gating rawhide packages since PR testing will concern all branches not just rawhide.
Pierre _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Wed, Mar 28, 2018 at 01:51:26PM +0000, Fabio Valentini wrote:
On Wed, Mar 28, 2018, 15:47 Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Wed, Mar 28, 2018 at 01:39:34PM +0000, Fabio Valentini wrote: >Â Â One example where running tests against a single-package update would be >Â Â nice IMO would be for toolchain and base packages, for example, updates to >Â Â annobin or binutils, where the answer to "Does this update break >Â Â compilation with GCC?" (which could be added as a test case) would be >Â Â vital in determining if the package should be pushed to rawhide or not. >Â Â Hope that makes it more clear what I meant by "it also would be nice for >Â Â single-package updates". I think I follow you there, what I don't follow is the difference between this and the build not landing in rawhide because it failed its tests. Or are you referring to: pre-commit testing, in other words pull-request testing?No, that's not what I meant (although testing PRs would be nice for the future, too). I just wanted to express that gating rawhide updates depending on test results is meaningful not only for the proposed merging of side-tags, but also for single important packages.
Ok then I think we're just agreeing :) It is what these diagrams are trying represent: the process of gating single package: https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide.png https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_bodhi.png
Pierre
Why are you stressing MultiplePkgs vs single pkgs. Single pgk process is just subcase of multiplepkgs process.
And if you can do chain build in Rawhide, I can't see any reason why it should not be possible for stable branch.
IOW the process for Rawhide should be as close to stable version as possible. Using buildroot override for stable while using sidetags for Rawhide does not make any sense.
Vít
Dne 28.3.2018 v 12:52 Pierre-Yves Chibon napsal(a):
Good Morning Everyone,
Based on the outcome of this discussion, I started trying to draw how the process to update a package in rawhide would look like with rawhide being gated on tests.
There are currently two proposals:
- one that does not involve bodhi updates
- one that does
Both proposals have a different flow for single-package update and multi-packages update.
Here is what I came up with.
Without bodhi:
- Single package update https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide.png
- Multi-packages update https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_MultiplePkgs.pn...
With bodhi:
- Single package update https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_bodhi.png
- Multi-packages update https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_MultiplePkgs_bo...
Outside of this workflow things we know we want to have/keep working:
- Keep chain-build working -> would one of the proposal facilitate that?
- Have a way to run tests against an entire side-tag before merging it
- Have a way to ask the tools to re-check greenwave's decision (so that false negative can be waived and the process continued)
mizdebsk suggested on IRC two ideas which I think would be worth looking into a little bit down the road once we got the basis done:
- new side-tag could be automatically generated from fedpkg build command
- packagers could define a list of packages that should be rebuilt in the side-tag and when all of these packages have been successfully rebuilt, the request to merge the side-tag is automatically created This would allow to use chain-build and let the entire process be automated.
What do you think?
Pierre
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Wed, Mar 28, 2018 at 06:46:50PM +0200, Vít Ondruch wrote:
Why are you stressing MultiplePkgs vs single pkgs. Single pgk process is just subcase of multiplepkgs process.
And because it is just a subcase, it can be simplified.
And if you can do chain build in Rawhide, I can't see any reason why it should not be possible for stable branch.
I'm not entirely sure where I spoke about that.
IOW the process for Rawhide should be as close to stable version as possible. Using buildroot override for stable while using sidetags for Rawhide does not make any sense.
I believe we already discussed this and I provided some of the drawbacks this would have.
Pierre
Dne 28.3.2018 v 19:06 Pierre-Yves Chibon napsal(a):
On Wed, Mar 28, 2018 at 06:46:50PM +0200, Vít Ondruch wrote:
Why are you stressing MultiplePkgs vs single pkgs. Single pgk process is just subcase of multiplepkgs process.
And because it is just a subcase, it can be simplified.
And if you can do chain build in Rawhide, I can't see any reason why it should not be possible for stable branch.
I'm not entirely sure where I spoke about that.
You are proposing to use side tags for chain builds, which is not how the things are done in stable branches, unless you are proposing the same changes for stable branches.
IOW the process for Rawhide should be as close to stable version as possible. Using buildroot override for stable while using sidetags for Rawhide does not make any sense.
I believe we already discussed this and I provided some of the drawbacks this would have.
It is sad you still see Rawhide as something completely different then stable (and stabilizing) Fedoras. Rawhide should use the same process as other branches. Only the built in timeouts should be different, e.g. by default, the package should spent no time in testing etc.
So go with buildroot overrides or sidetags, but use them everywhere. If chainbuilds are testing of mutliple packages are important for Rawhide, they are of the same importance for other branches.
And please rename this thread to "Gating packages in Fedora", because Rawhide should be just Fedora.
V.
On Wed, Mar 28, 2018 at 07:34:51PM +0200, Vít Ondruch wrote:
Dne 28.3.2018 v 19:06 Pierre-Yves Chibon napsal(a):
On Wed, Mar 28, 2018 at 06:46:50PM +0200, Vít Ondruch wrote:
Why are you stressing MultiplePkgs vs single pkgs. Single pgk process is just subcase of multiplepkgs process.
And because it is just a subcase, it can be simplified.
And if you can do chain build in Rawhide, I can't see any reason why it should not be possible for stable branch.
I'm not entirely sure where I spoke about that.
You are proposing to use side tags for chain builds, which is not how the things are done in stable branches, unless you are proposing the same changes for stable branches.
IOW the process for Rawhide should be as close to stable version as possible. Using buildroot override for stable while using sidetags for Rawhide does not make any sense.
I believe we already discussed this and I provided some of the drawbacks this would have.
It is sad you still see Rawhide as something completely different then stable (and stabilizing) Fedoras.
Rawhide should use the same process as other branches. Only the built in timeouts should be different, e.g. by default, the package should spent no time in testing etc.
So go with buildroot overrides or sidetags, but use them everywhere. If chainbuilds are testing of mutliple packages are important for Rawhide, they are of the same importance for other branches.
And again, this is an option, but that will affect the current workflow of all packagers. If we are fine with this then maybe it is doable but this needs to be acknowledged.
And please rename this thread to "Gating packages in Fedora", because Rawhide should be just Fedora.
You do realize that there is already gating in place for stable releases right? So no this thread is specifically about rawhide.
Pierre
Dne 28.3.2018 v 19:55 Pierre-Yves Chibon napsal(a):
On Wed, Mar 28, 2018 at 07:34:51PM +0200, Vít Ondruch wrote:
And please rename this thread to "Gating packages in Fedora", because Rawhide should be just Fedora.
You do realize that there is already gating in place for stable releases right?
Yes I do.
So no this thread is specifically about rawhide.
And I want the same gating for Rawhide. Just the timing should be different.
V.
On 03/28/2018 12:46 PM, Vít Ondruch wrote:
And if you can do chain build in Rawhide, I can't see any reason why it should not be possible for stable branch.
It's more that it's tricky and not that it's impossible, because of the Koji buildroot. Packages built into Rawhide become part of the buildroot automatically, but packages build into f27 do not, until the package either goes stable or until a buildroot override is created. In either case, a koji chain-build will not work like it does on Rawhide, since Koji doesn't create buildroot overrides itself.
Le 2018-03-28 12:52, Pierre-Yves Chibon a écrit :
Hi
With bodhi:
- Single package update
https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_bodhi.png
- Multi-packages update
Not terribly fond of anything that offloads multi-build intellignece coordination on packagers, forcing everyone that needs multi-build to either do lots of manual operations, to reinvent its own local tooling, or just give up on some updates.
The correct packager-friendly way to do it is
eat-this <list of srpms> → ok/nok
With the tooling chain-building <list of srpms>, automatically applying boostrap sections when present, automatically rebuilding everything in the set against the final state of other parts of the set, and applying integration tests to the end result and not to the intermediary steps.
And it should not matter if <list of srpms> has a count of one or many.
Anything that posits list of srpms is a special case that can be bolted on the single srpm is broken by design and produces broken packaging, because people will try to workaround tooling defects by stashing everything in a single srpm to make tools happy.
(that's also a huge defect our the review workflow, unbundle and produce a clean and maintenable set of packages → lots of review and tooling misery, keep it all in a single srpm → no problems)
Regards,
On Thu, Mar 29, 2018 at 10:47:50AM +0200, Nicolas Mailhot wrote:
Le 2018-03-28 12:52, Pierre-Yves Chibon a écrit :
Hi
With bodhi:
- Single package update
https://pingou.fedorapeople.org/gating_rawhide/GatingRawhide_bodhi.png
- Multi-packages update
Not terribly fond of anything that offloads multi-build intellignece coordination on packagers, forcing everyone that needs multi-build to either do lots of manual operations, to reinvent its own local tooling, or just give up on some updates.
I am not sure where you're seeing: - lots of manual operations - reinventing its own tooling - giving up on some updates
There would be 2 new operation: ask for a new side-tag, ask for it to be merged. These already exists for large rebuilds and need to go via rel-eng while the new tooling would allow self-service.
I think we made it clear that we want to get chain-build to work/keep working
For the last point, I'm very unclear on where you got this impression.
The correct packager-friendly way to do it is
eat-this <list of srpms> → ok/nok
How is this different from chain-build?
With the tooling chain-building <list of srpms>, automatically applying boostrap sections when present, automatically rebuilding everything in the set against the final state of other parts of the set, and applying integration tests to the end result and not to the intermediary steps.
And it should not matter if <list of srpms> has a count of one or many.
Agreed in theory, in practice having a list of one item allows to simplify things.
Anything that posits list of srpms is a special case
I think it's the other way around: single srpm are a special case, they are a sub-case of the multi-srpm which allow taking shortcut.
(that's also a huge defect our the review workflow, unbundle and produce a clean and maintenable set of packages → lots of review and tooling misery, keep it all in a single srpm → no problems)
One could argue than 1 is always simpler than > 1 so this is intrinsically a state of things.
That being said, I think it is a fair critic one we may want to address but I think that's broader than the discussion at hand.
Pierre
Hello again!
There's been a lot of discussion on this thread, and some useful feedback. Thanks!
I would like to make a clearer proposal about how gating in Rawhide would work:
Multi-package update workflow =============================
When a packager wants to update multiple interdependent packages (such as a Gnome or KDE update), they will request a new side tag from Bodhi (we can make a fedpkg feature for ease), and then they will use fedpkg build --target=my_side_tag for each package they want to include in the same update. The packager could also use chain building to do this if preferred.
When the packager has finished the builds they intend to group together, they will request the side tag to be merged via Bodhi/fedpkg. This will create a Bodhi update and will trigger tests to run on the set of packages included in the side tag. If the tests pass, Bodhi will merge the side tag into Rawhide, and the packager's work is done. If they fail, the packager will need to take action to get them to pass, or submit a waiver to waiverdb to ignore them.
Once the side tag is merged, the update will be marked stable automatically by Bodhi (no need for karma or time in testing) and will be tagged into the rawhide tag.
Single package update workflow ==============================
If a packager wants to update a single package, they will use fedpkg build, without specifying a side tag. fedpkg can see that the user did not specify a side tag, so it will go ahead and prompt the user for update details and create an update (or we could make this opt-in, either way).
Once the Bodhi update exists, tests will run. If the tests pass, Bodhi will automatically mark the update for stable and the packager's work is done. If the tests fail, the packager will need to either fix the problem or waive the update.
Benefits ========
* We increase the quality of packages in Rawhide through automated testing. * By giving Bodhi the ability to manage side tags, we can also give side tags to packagers for stable releases. This will allow packagers to chain build for stable releases in addition to rawhide, which is not possible today. * The workflow isn't so different from what we have today. Single package updates are nearly the same, and multi-package updates often manually request side tags from releng.
Buildroot overrides ===================
If Bodhi can provide side tags to packagers, the use case for buildroot overrides becomes diminished. I am not able to immediately think of a purpose for them that would remain, but it would be bold to say they would have no purpose with this design. If we go with the above design, we should monitor buildroot override usage and remove it if it is unused. We may even make it an admin-only feature (you would request it from releng).
Thanks to pingou for creating the nice flowcharts for us.
-
Colin Walters -
Fabio Valentini -
Igor Gnatenko -
Kalev Lember -
Kevin Fenzi -
Kevin Kofler -
Martin Kolman -
Mathieu Bridon -
Mattia Verga -
Michael Cronenworth -
Michael Šimáček -
Milan Crha -
Mohan Boddu -
Nicolas Mailhot -
Pierre-Yves Chibon -
Randy Barlow -
Richard Shaw -
Tom Hughes -
Vít Ondruch -
Zbigniew Jędrzejewski-Szmek