9:54 a.m.
On Tue, Jun 18, 2019 at 1:31 PM Neal Gompa <ngompa13(a)gmail.com> wrote:
On Tue, Jun 18, 2019 at 5:21 AM Aleksandra Fedorova <alpha(a)bookwar.info> wrote:
>
> Hi,
> On Tue, Jun 18, 2019 at 3:05 AM Neal Gompa <ngompa13(a)gmail.com> wrote:
> >
> > On Mon, Jun 17, 2019 at 8:53 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
> > >
> > > On 6/17/19 4:47 PM, Kevin Kofler wrote:
> > > > Kevin Fenzi wrote:
> > > >> I disagree. I think we need gating to block as much stuff that
breaks
> > > >> things from landing as we can and then we should find that
keeping
> > > >> composes going is much easier on all of us. Then things can be
fixed
> > > >> when gating catches them and it's on the person who broke
things.
> > > >
> > > > And that is going to make development completely cringe to a halt. It
is the
> > > > nature of a distribution branch under development that things will
sometimes
> > > > be completely broken for a couple weeks. There needs to be a place to
do
> > > > development that can cause such temporary breakage.
> > >
> > > I again completely disagree. There is no reason for weeks of breakage.
> > > Most of the issues that break composes are unannounced abi bumps where
> > > just rebuilding dependent packages fixes it. Or broken deps (likewise).
> > > Or mistakes made in kickstarts/comps. Or something that doesn't even
> > > run. What good does having everyone broken for weeks do?
> > >
> >
> > And that comes down to people shouldn't need to have to think about
> > these things when working in Rawhide. While I don't disagree that
> > Rawhide should be usable, I fundamentally disagree with making it
> > harder for people to put things in Rawhide. We should be developing
> > our tooling to make it _easier_ for stuff to go into Rawhide, and have
> > Rawhide fix itself when the issues are relatively trivial to fix (such
> > as reverse dependency rebuilding).
>
> I think this is exactly what gating is supposed to do.
>
> Let's compare it in this way:
>
> currently to add a feature which may break Rawhide and some unknown
> dependencies of the component you need to write a HEADS-UP e-mail, or,
> better, submit a Change request with the analysis of the change.
> People who read this e-mail would need to make a guess on whether or
> not his change affects them, then they have to fetch it and test it
> somehow, then they have to provide the feedback back to you. You need
> to wait for feedback, then you get the reports, in best cases - bugs,
> which you need to debug, requesting more info. Then you implement the
> change, hit the unexpected bug, which was unnoticed, and block others
> from building their packages and implementing their changes for
> unknown amount of time.
>
> With gating you can submit a code change, the tooling will take care
> of building it, building its dependencies, informing you of possible
> breakages, giving you the list of actual issues with all the debug
> logs. And then based on this data you can proceed or stop and rework
> the change a bit more in collaboration with exactly those people
> affected.
>
> I also think that we need a second point of view here: you were
> talking about not driving away the developer, who makes the change.
> But there are also other people who we shouldn't drive away. For
> example developers who depend on the change. Or QA who need to react
> on such changes. These people are have their part in the process.
>
>
> But, to be honest, I think there is a bit of overreacting on the
> entire Gating topic.
>
> It doesn't do a hard block. It is included in the design that gating
> can be bypassed. But it supposed to provide better analysis of the
> change. Bypassing of the gate can happen, The key here is that it
> won't be a surprise, rather informed decision.
>
I think this is the first time I've heard a coherent description of
the intent of this stuff. If it really is intended to work this way,
it'd be very helpful.
> > > > The side tag approach already does not scale, as evidenced by this
thread.
> > >
> > > It does. You just need to communicate with others working in the same
> > > area, IMHO. I don't think we need some technical thing for something
> > > that happens rarely and can be solved by more communication.
> > >
> >
> > Side tags will happen a lot more often because the tooling is pushing
> > us to do it that way. Don't discount the potential for future
> > insanity. I'm still not sure side-tags are enough. Could we have a
> > concept of a "scratch side tag"? Something like a scratch build, but
> > contains a collection of builds and creates an overlay repo that can
> > be used to run checks on for auto-merging? If they're good, then it
> > would get auto-built properly into the main rawhide tag (or even
> > stable tag!).
>
> Afaik, this is exactly the concept of a dynamic sidetag as Fedora
> Infra is currently implementing. Sidetag in koji as a sort of
> pull-request: you create sidetag, get repos and composes built out of
> it, run tests, get results. Then "merge" this sidetag into the main
> repo.
>
> And while clashing sidetags is a problem, it is the same problem as it
> happens with overlapping pull-requests when several people are working
> on the source code.
>
> There are several ways to address it:
>
> 1) create smaller pull-requests and merge them more often
>
> This is one of the core points of CI workflows and it also seems to be
> most often underestimated. There is an overhead in creating smaller
> changes, which we are trying to address with tooling. But the overhead
> is _linear_. While the complexity of one merge grows exponentially
> with the size of the branch you are merging.
>
> The smaller the changes - the easier the rebase.
>
> 2) reorganize the codebase
>
> When working with source code, it usually means splitting files. If
> there are many changes targeting the same file again and again,
> causing conflicts, it might mean this file has too much logic bundled
> in it and needs a split up.
>
> It might also be merging files. For example if there are many changes
> targeting two files and it causes problems when these two files get
> out of sync.
>
> When applied to RPM packages, it may be also a good question: if there
> is circular dependency between packages, so that these are strictly
> tied to each other by version and can only be processed together - why
> these are separate packages and not subpackages?
>
> 3) Talk about changes in advance
>
> This of course is the default, and it doesn't go anywhere with gating
> or without it. Again I believe gating can make it easier. For example,
> we can add a note on conflicting "pull-requests" and put a rebase
> requirement as a gating check.
>
I think you kind of missed what I was asking: I want side tags that
are composed entirely of scratch builds. The result *must not* be used
to merge into the distribution. It *must* be thrown away and when the
pull request (or whatever) is merged, it should trigger the build
again, which would be a "real" side tag that would get auto-merged on
success.
You are right, I missed that part. And I agree this is needed. But I
am not sure if there is already a proposal for that.
There are several ways how one can implement it. I would personally
prefer pull requests to dist-git over scratch builds. I think it is
important to keep some visibility in the system, so that not just the
owner of a change, but also random lurkers can see what is being
prepared. (Also solves the )
And if we look into PR-based workflows, then Zuul team has a proof of
concept pipeline which builds packages on PR in Pagure.
Which is not your ordinary Jenkins PR-triggered pipeline.
The killer-feature of Zuul is that it can manage cross-project
dependencies. It understands links between pull-requests in different
git repositories, even in different Git forges (Pagure and GitHub for
example).
If, let's say, you mark PR#123 in rpms/dnf as dependent on PR#765 in
rpms/rpm. Zuul fetches source code for both projects into the
workspace, and runs the pipeline in this workspace. Results of the
pipeline will be applied to both PR's.
Zuul could also do the merges for you, then it will also ensure that
the chain merge of dependent patches happen in the right order.
I like this concept, and I wonder if we can map this workflow cleanly
on the sidetag process, which happens after code is merged in
dist-git.
> This is more or less the equivalent of a staging/integration overlay
> that is used to test, where the artifacts are not preserved because
> they aren't useful beyond that.