Hi All,
The kernel team has been running an exploded git tree since Flock last year. This has been mostly a "nice to have", but several people have commented on how useful it is for them when debugging issues, etc. Having the exploded sources with the patches we carry included lets them easily see what we have changed and how.
While not a huge burden, this has been mostly an additional effort on top of what we already do. In order to make us a bit more efficient, we've been looking at ways to automate or ease this, as well as make it possible for more than one person to maintain it. This has lead to some discussion around moving to making the exploded git tree the canonical source location for Fedora kernel work. I'm sending this email to start a conversation around that.
So what would this mean? It means we'd maintain the exploded source tree very similar to today, with it rebasing on top of whatever upstream we're using as our base. For info on how that works, see the blog post I did about that a while ago.
Using this exploded tree, I've created some scripts to generate the pkg-git content. This is probably the biggest workflow change involved. We still need to use pkg-git to do builds in koji, but direct commits to that repo would be disabled outside of a few of us needed to do the syncs. Patches and config changes could be done via email to this list, or possibly via pull request to the branch maintainer of the exploded tree.
Why would we do this? A few reasons. First, it allows us to eliminate duplicate maintenance in multiple spots while still providing all the same things we are right now. Second, it moves us to more of an upstream kernel development model and hopefully increases both visibility and communication in terms of what patches/changes go in, etc. It also makes our job easier during rebases, as we can literally use 'git rebase' and work via that instead of having to copy around patches to different trees or manually adjust patches outside of a normal git workflow.
Who does this impact? Honestly, not many. There are exactly 4 people that commit to pkg-git today on anything resembling a regular basis. However, it is my hope that changing the workflow a bit makes this somewhat easier for others to work against a tree and submit patches and changes. I'm not under any impressions that it will exponentially increase our contributors, but it will quite possibly make things a bit more transparent.
Is this a done decision? No. Not at all. In fact, the scripts aren't even finished yet. I wanted to get the discussion started before really investing heavily in finishing them off. Justin and Laura have copies of the (unfinished) scripts and a howto document on various tasks. I'll likely send the howto out once I get a few of the last remaining steps working for people to read over. But the general idea isn't going to change much, which is why we're talking about this now.
When would this happen? Unclear. Certainly not this month as one of the primary stakeholders is on PTO. If the discussion goes well and this direction is something we're going to do, then it might be next month or at some natural cut-over point.
Will the exploded tree go away if we don't change to this model? Also unclear. I'd hesitantly say it would stick around, but it might change in the way it is created and published.
I'm sure there will be more questions, so please ask away. Looking forward to hearing other's thoughts.
josh
On Wed, Jun 17, 2015 at 12:45 PM, Josh Boyer jwboyer@fedoraproject.org wrote:
Hi All,
The kernel team has been running an exploded git tree since Flock last year. This has been mostly a "nice to have", but several people have commented on how useful it is for them when debugging issues, etc. Having the exploded sources with the patches we carry included lets them easily see what we have changed and how.
While not a huge burden, this has been mostly an additional effort on top of what we already do. In order to make us a bit more efficient, we've been looking at ways to automate or ease this, as well as make it possible for more than one person to maintain it. This has lead to some discussion around moving to making the exploded git tree the canonical source location for Fedora kernel work. I'm sending this email to start a conversation around that.
So what would this mean? It means we'd maintain the exploded source tree very similar to today, with it rebasing on top of whatever upstream we're using as our base. For info on how that works, see the blog post I did about that a while ago.
Using this exploded tree, I've created some scripts to generate the pkg-git content. This is probably the biggest workflow change involved. We still need to use pkg-git to do builds in koji, but direct commits to that repo would be disabled outside of a few of us needed to do the syncs. Patches and config changes could be done via email to this list, or possibly via pull request to the branch maintainer of the exploded tree.
Why would we do this? A few reasons. First, it allows us to eliminate duplicate maintenance in multiple spots while still providing all the same things we are right now. Second, it moves us to more of an upstream kernel development model and hopefully increases both visibility and communication in terms of what patches/changes go in, etc. It also makes our job easier during rebases, as we can literally use 'git rebase' and work via that instead of having to copy around patches to different trees or manually adjust patches outside of a normal git workflow.
Who does this impact? Honestly, not many. There are exactly 4 people that commit to pkg-git today on anything resembling a regular basis. However, it is my hope that changing the workflow a bit makes this somewhat easier for others to work against a tree and submit patches and changes. I'm not under any impressions that it will exponentially increase our contributors, but it will quite possibly make things a bit more transparent.
Is this a done decision? No. Not at all. In fact, the scripts aren't even finished yet. I wanted to get the discussion started before really investing heavily in finishing them off. Justin and Laura have copies of the (unfinished) scripts and a howto document on various tasks. I'll likely send the howto out once I get a few of the last remaining steps working for people to read over. But the general idea isn't going to change much, which is why we're talking about this now.
When would this happen? Unclear. Certainly not this month as one of the primary stakeholders is on PTO. If the discussion goes well and this direction is something we're going to do, then it might be next month or at some natural cut-over point.
Will the exploded tree go away if we don't change to this model? Also unclear. I'd hesitantly say it would stick around, but it might change in the way it is created and published.
I'm sure there will be more questions, so please ask away. Looking forward to hearing other's thoughts.
So. Given that it has been a week and we've heard nothing back, I'm going to assume that either nobody cares how we maintain our tree (which tbh is a fair point of view), or all of this sounds great to everyone.
So here is our plan going forward. We need a place to try this out to make sure it is actually a net positive for us. We intend to drive the master branch in pkg-git (rawhide) using the tree and scripts for the 4.2 merge window. What that means for other people is basically this:
1) Please do not commit directly to the master branch in pkg-git. If you have patches or changes you'd like to see, please send them to this list. If you commit something directly to the master branch, it will get lost on the next sync. I don't want to close the ACLs right now since we're still evaluating, so please be careful.
2) The exploded tree will be at https://pagure.io/linux . The scripts and howto document are in the fedora directory there. There is still work to be done on them, so they'll change over the course of the next couple of weeks. Remember, just like the RPM, the tree rebases on top of upstream linux.git daily. You'll need to account for this if you track it.
josh
Hi!
On 25.06.2015 18:02, Josh Boyer wrote:
On Wed, Jun 17, 2015 at 12:45 PM, Josh Boyer jwboyer@fedoraproject.org wrote:
The kernel team has been running an exploded git tree since Flock last year. This has been mostly a "nice to have", but several people have commented on how useful it is for them when debugging issues, etc. Having the exploded sources with the patches we carry included lets them easily see what we have changed and how.
[...] So. Given that it has been a week and we've heard nothing back, I'm going to assume that either nobody cares how we maintain our tree (which tbh is a fair point of view), or all of this sounds great to everyone.
FWIW, It matters to me, but I can't see any problem with the planed approach.
There is only one thing I fear a little bit: sooner or later some things might be done in the exploded git tree that can't be reversed easily in pkg-git content or the srpm. For example, reversing a single patch, adding your own patch or building a vanilla kernel should continue to be as easy as today.
I'm sure it's not the intent to make these things harder, but you didn't write that down. It might be worth it, as doing things directly in git without caring about the spec file/the srpm at all sometimes is a whole lot easier(¹). That's why I fear sooner or later someone will likely do that, which would make other peoples life harder.
Cu, knurd
(¹) fwiw, the "make debug"/"make nodebug" stuff we have in pkg-git today is one such thing; that's why I many moons ago worked on removing that, but than totally lost track; sorry for that :-/
On Thu, Jun 25, 2015 at 12:02 PM, Josh Boyer jwboyer@fedoraproject.org wrote:
Will the exploded tree go away if we don't change to this model? Also unclear. I'd hesitantly say it would stick around, but it might change in the way it is created and published.
I'm sure there will be more questions, so please ask away. Looking forward to hearing other's thoughts.
So. Given that it has been a week and we've heard nothing back, I'm going to assume that either nobody cares how we maintain our tree (which tbh is a fair point of view), or all of this sounds great to everyone.
So here is our plan going forward. We need a place to try this out to make sure it is actually a net positive for us. We intend to drive the master branch in pkg-git (rawhide) using the tree and scripts for the 4.2 merge window. What that means for other people is basically this:
The team and I were discussing this topic yesterday, and it really isn't working as we'd hoped. The exploded tree is great for rebasing patches. It's trivial and integrated into the workflow just fine. However, taking that source code and producing an RPM is pretty cumbersome now. The scripts work, but they can be fragile in certain places. It's also still double commits in the long run, one in pkg-git and another in the exploded tree for all the junk that is needed for tracking and updating pkg-git. Over time, a 'git rebase' call will take forever as those commits pile up because they're never going upstream. Thorsten and Peter also pointed out that changes "sneak" into pkg-git and only are mentioned in the RPM changelog, not as separate commits.
So going from exploded tree -> kernel.spec just isn't paying off. In the spirit of fail fast, we're probably going to scrap this experiment in the next few days. However, we still want the exploded tree available so we're going to look at making it easier to go from kernel.spec -> exploded tree. Primarily, we are likely going to change how we apply patches in the spec file to use git-am. What does this mean for contributors?
a) Your patch _must_ conform to something git-am accepts. Ironically, git-show does not produce such a patch so if you are using a git tree to create a patch, use git format-patch. Plain diff won't work either. It needs a From: a Date: and a Subject: at a minimum.
b) The spec will undergo some surgery and you will no longer need to specify an ApplyPatch line. This means the order of patch applications is determined by the order they are listed in the PatchXXX section. Do not change this order randomly.
c) 'fedpkg prep' time on a fresh tree will take a bit longer. Subsequent 'fedpkg prep' runs will be faster, but still a bit slower than today's method. Creating the initial git tree is somewhat time consuming. Still, we're talking on the order of just over a minute for a fresh tree, and about 40 seconds for subsequent runs. This is not world ending.
This will be tried in rawhide to start. Aside from the above changes, the workflow will go back to what we had for years. I'll look at creating the exploded tree on the side and it shouldn't impact anyone aside from the above requirements.
josh
On Thu, Jul 09, 2015 at 07:42:08AM -0400, Josh Boyer wrote:
On Thu, Jun 25, 2015 at 12:02 PM, Josh Boyer jwboyer@fedoraproject.org wrote:
Will the exploded tree go away if we don't change to this model? Also unclear. I'd hesitantly say it would stick around, but it might change in the way it is created and published.
I'm sure there will be more questions, so please ask away. Looking forward to hearing other's thoughts.
So. Given that it has been a week and we've heard nothing back, I'm going to assume that either nobody cares how we maintain our tree (which tbh is a fair point of view), or all of this sounds great to everyone.
So here is our plan going forward. We need a place to try this out to make sure it is actually a net positive for us. We intend to drive the master branch in pkg-git (rawhide) using the tree and scripts for the 4.2 merge window. What that means for other people is basically this:
The team and I were discussing this topic yesterday, and it really isn't working as we'd hoped. The exploded tree is great for rebasing patches. It's trivial and integrated into the workflow just fine. However, taking that source code and producing an RPM is pretty cumbersome now. The scripts work, but they can be fragile in certain places. It's also still double commits in the long run, one in pkg-git and another in the exploded tree for all the junk that is needed for tracking and updating pkg-git. Over time, a 'git rebase' call will take forever as those commits pile up because they're never going upstream. Thorsten and Peter also pointed out that changes "sneak" into pkg-git and only are mentioned in the RPM changelog, not as separate commits.
Hi Josh,
Thanks for the effort! Sad to hear it isn't working out, but I understand your reasons.
While it is still fresh in your mind, can I ask a few questions for curiousity.
Not that submodules is a good solution (from what I hear) but would a solution like that (if made easier) work for those fedora scripts? The idea would be a rebase and a single commit on top to add the submodule-like directory/area that pulls in a separate tree full of those changes?
This would sidestep the rebase growing larger over time, sort of (one initial commit, but a growing set of commits brought in later??).
It might address the 'sneaky' changes problem too.
The only thing it doesn't handle is the extra Fedora kernel patches themselves that is carried (but I think you have that minimized and under control anyway).
I don't expect you to hold your breathe for that solution to arise, but if it did, would it help do you think?
Cheers, Don
So going from exploded tree -> kernel.spec just isn't paying off. In the spirit of fail fast, we're probably going to scrap this experiment in the next few days. However, we still want the exploded tree available so we're going to look at making it easier to go from kernel.spec -> exploded tree. Primarily, we are likely going to change how we apply patches in the spec file to use git-am. What does this mean for contributors?
a) Your patch _must_ conform to something git-am accepts. Ironically, git-show does not produce such a patch so if you are using a git tree to create a patch, use git format-patch. Plain diff won't work either. It needs a From: a Date: and a Subject: at a minimum.
b) The spec will undergo some surgery and you will no longer need to specify an ApplyPatch line. This means the order of patch applications is determined by the order they are listed in the PatchXXX section. Do not change this order randomly.
c) 'fedpkg prep' time on a fresh tree will take a bit longer. Subsequent 'fedpkg prep' runs will be faster, but still a bit slower than today's method. Creating the initial git tree is somewhat time consuming. Still, we're talking on the order of just over a minute for a fresh tree, and about 40 seconds for subsequent runs. This is not world ending.
This will be tried in rawhide to start. Aside from the above changes, the workflow will go back to what we had for years. I'll look at creating the exploded tree on the side and it shouldn't impact anyone aside from the above requirements.
josh _______________________________________________ kernel mailing list kernel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/kernel
On Thu, Jul 9, 2015 at 10:17 AM, Don Zickus dzickus@redhat.com wrote:
On Thu, Jul 09, 2015 at 07:42:08AM -0400, Josh Boyer wrote:
On Thu, Jun 25, 2015 at 12:02 PM, Josh Boyer jwboyer@fedoraproject.org wrote:
Will the exploded tree go away if we don't change to this model? Also unclear. I'd hesitantly say it would stick around, but it might change in the way it is created and published.
I'm sure there will be more questions, so please ask away. Looking forward to hearing other's thoughts.
So. Given that it has been a week and we've heard nothing back, I'm going to assume that either nobody cares how we maintain our tree (which tbh is a fair point of view), or all of this sounds great to everyone.
So here is our plan going forward. We need a place to try this out to make sure it is actually a net positive for us. We intend to drive the master branch in pkg-git (rawhide) using the tree and scripts for the 4.2 merge window. What that means for other people is basically this:
The team and I were discussing this topic yesterday, and it really isn't working as we'd hoped. The exploded tree is great for rebasing patches. It's trivial and integrated into the workflow just fine. However, taking that source code and producing an RPM is pretty cumbersome now. The scripts work, but they can be fragile in certain places. It's also still double commits in the long run, one in pkg-git and another in the exploded tree for all the junk that is needed for tracking and updating pkg-git. Over time, a 'git rebase' call will take forever as those commits pile up because they're never going upstream. Thorsten and Peter also pointed out that changes "sneak" into pkg-git and only are mentioned in the RPM changelog, not as separate commits.
Hi Josh,
Thanks for the effort! Sad to hear it isn't working out, but I understand your reasons.
While it is still fresh in your mind, can I ask a few questions for curiousity.
Not that submodules is a good solution (from what I hear) but would a solution like that (if made easier) work for those fedora scripts? The idea would be a rebase and a single commit on top to add the submodule-like directory/area that pulls in a separate tree full of those changes?
This would sidestep the rebase growing larger over time, sort of (one initial commit, but a growing set of commits brought in later??).
I thought about this when I originally started, but avoided it for a few different reasons. The main one was my unfamiliarity with submodules. Another was now you're going from 2 "trees" to manage (pkg-git, linux.git) to 3 (pkg-git, linux.git, submodule). It would help the rebase issue, but that is about all it would help.
It might address the 'sneaky' changes problem too.
I'm not sure about that one. The issue there stems from multiple things happening in linux.git, and then when we sync to pkg-git it all just gets sucked into one large commit. Unless you held to a strict 1:1 linux.git -> pkg-git rule, you're always going to face that problem. And if you go to 1:1, you aren't really treating linux.git as the canonical source at that point. It is just an exploded tree mirror essentially.
In some scenarios, this isn't a big deal. Some distros don't expose their package SCM and their users/customers are used to the SRPM as being the canonical source, with %changelog being the equivalent of the SCM commit logs. However, since the Core+Extras merge Fedora has always used the package SCM as the transparent location to see "what changed and why". Deviating from that, while it makes things easier for my team, is somewhat against Fedora's transparency.
(It's also the reason why I bothered to spit out separate patches and tarballs. The easiest solution is to just suck up the output of git-archive of the linux.git tree and use that as a tarball and build it, but not having patches would probably cause people to lose their minds in the Fedora world.)
The only thing it doesn't handle is the extra Fedora kernel patches themselves that is carried (but I think you have that minimized and under control anyway).
I don't expect you to hold your breathe for that solution to arise, but if it did, would it help do you think?
I think the only thing that would really make working from an exploded tree viable is if koji could be pointed at it directly. Even then, the mechanics of source -> spec -> srpm -> rpm(s) isn't straightforward.
josh
On Thu, Jul 09, 2015 at 11:17:29AM -0400, Josh Boyer wrote:
On Thu, Jul 9, 2015 at 10:17 AM, Don Zickus dzickus@redhat.com wrote:
On Thu, Jul 09, 2015 at 07:42:08AM -0400, Josh Boyer wrote:
On Thu, Jun 25, 2015 at 12:02 PM, Josh Boyer jwboyer@fedoraproject.org wrote:
Will the exploded tree go away if we don't change to this model? Also unclear. I'd hesitantly say it would stick around, but it might change in the way it is created and published.
I'm sure there will be more questions, so please ask away. Looking forward to hearing other's thoughts.
So. Given that it has been a week and we've heard nothing back, I'm going to assume that either nobody cares how we maintain our tree (which tbh is a fair point of view), or all of this sounds great to everyone.
So here is our plan going forward. We need a place to try this out to make sure it is actually a net positive for us. We intend to drive the master branch in pkg-git (rawhide) using the tree and scripts for the 4.2 merge window. What that means for other people is basically this:
The team and I were discussing this topic yesterday, and it really isn't working as we'd hoped. The exploded tree is great for rebasing patches. It's trivial and integrated into the workflow just fine. However, taking that source code and producing an RPM is pretty cumbersome now. The scripts work, but they can be fragile in certain places. It's also still double commits in the long run, one in pkg-git and another in the exploded tree for all the junk that is needed for tracking and updating pkg-git. Over time, a 'git rebase' call will take forever as those commits pile up because they're never going upstream. Thorsten and Peter also pointed out that changes "sneak" into pkg-git and only are mentioned in the RPM changelog, not as separate commits.
Hi Josh,
Thanks for the effort! Sad to hear it isn't working out, but I understand your reasons.
While it is still fresh in your mind, can I ask a few questions for curiousity.
Not that submodules is a good solution (from what I hear) but would a solution like that (if made easier) work for those fedora scripts? The idea would be a rebase and a single commit on top to add the submodule-like directory/area that pulls in a separate tree full of those changes?
This would sidestep the rebase growing larger over time, sort of (one initial commit, but a growing set of commits brought in later??).
I thought about this when I originally started, but avoided it for a few different reasons. The main one was my unfamiliarity with submodules. Another was now you're going from 2 "trees" to manage (pkg-git, linux.git) to 3 (pkg-git, linux.git, submodule). It would help the rebase issue, but that is about all it would help.
Ok.
It might address the 'sneaky' changes problem too.
I'm not sure about that one. The issue there stems from multiple things happening in linux.git, and then when we sync to pkg-git it all just gets sucked into one large commit. Unless you held to a strict 1:1 linux.git -> pkg-git rule, you're always going to face that problem. And if you go to 1:1, you aren't really treating linux.git as the canonical source at that point. It is just an exploded tree mirror essentially.
In some scenarios, this isn't a big deal. Some distros don't expose their package SCM and their users/customers are used to the SRPM as being the canonical source, with %changelog being the equivalent of the SCM commit logs. However, since the Core+Extras merge Fedora has always used the package SCM as the transparent location to see "what changed and why". Deviating from that, while it makes things easier for my team, is somewhat against Fedora's transparency.
(It's also the reason why I bothered to spit out separate patches and tarballs. The easiest solution is to just suck up the output of git-archive of the linux.git tree and use that as a tarball and build it, but not having patches would probably cause people to lose their minds in the Fedora world.)
Sure. We had a 'create-patches.sh' script that did the 1:1 thing based on an initial tarball in RHEL that worked well for years. Just throwing it out there.
The only thing it doesn't handle is the extra Fedora kernel patches themselves that is carried (but I think you have that minimized and under control anyway).
I don't expect you to hold your breathe for that solution to arise, but if it did, would it help do you think?
I think the only thing that would really make working from an exploded tree viable is if koji could be pointed at it directly. Even then, the mechanics of source -> spec -> srpm -> rpm(s) isn't straightforward.
We do this in RHEL, not sure if koji was passed that technology or not but we have already solved this years ago.
Again, I am not trying to persuade you to try your mind, just trying to understand the pitfalls for my personal reasons. :-)
Thanks for the feedback.
Cheers, Don
josh _______________________________________________ kernel mailing list kernel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/kernel
kernel@lists.fedoraproject.org
-
Don Zickus -
Josh Boyer -
Thorsten Leemhuis