Defining the future of the packager workflow in Fedora

[Pierre-Yves Chibon]

Good Morning Everyone,

At Flock, a few of us met to discuss a future vision of the packager workflow.
This discussion was triggered by the realization that a number of initiatives
are happening around packaging in Fedora but there is no real shared vision on
what we want the packager UX/workflow to be.
The lack of vision on the packager workflow means we could deploy something
today, thinking it is the improvement over the current workflow but would
prevent us from (or make it harder to) doing other changes afterwords that would
be even more beneficial..

So once that concern was raised, we took some time during the Fedora
Infrastructure hackfest to gather the people interested around a white board and
brainstorm on what a future packager workflow could look like.

We tried not to link this process to any tool in particular as well as focus on
the what and why rather than any how.

Here is what the vision we came to and that we would like to discuss:

○ Every changes to dist-git is done via pull-requests
○ Pull-requests are automatically tested
○ Every commit to dist-git (ie: PR merged) is automatically built in koji
○ Every build in koji results automatically in an update in bodhi
○ Every update in bodhi is automatically tested
○ If the tests pass, the update is automatically pushed to the repository


For this workflow to work nicely we need to fix a few things first:

- We need to work on the change logs in the spec files, as otherwise
  pull-requests are going to conflict more often than not
- We need to fine a place to insert the end-user information about an update (in
  the PR description?)
- When a group of people want to collectively work on a change (involving one or
  more packages), they’ll need a place to collaborate. This could be someone’s
  fork or, preferably, a fork of the project collectively maintained by that
  group.
    For example: the python SIG would have a group with forks of the different
    python packages they are working on 
- We need to figure out how to enable opening a pull-request again multiple
  branches at one.


Why should we agree now on a long term vision?

A concrete example to this, I have been playing with the idea to get ride of git
branches in dist-git for a while now. Basically use git as it is meant to be
used: have one branch for development (most often master) and branch of master
when you need to, not simply because a new version of Fedora happened.
However, if we implemented this, we would break the workflow described above as
suddenly there would be no way to specify that a given PR is meant to land in a
certain version of Fedora or another (Is this PR for rawhide? F31? F30? All of
them? Some of them?).


Having this idea of where we want to go should give us a goal as well as way to
reflect if changes we think are good in isolation would actually help achieving
this vision or not.

This proposal was based on the brainstorming session a few of us had at flock,
but before we can see how to implement it we need more inputs and this is where
you come in :)

Do you like this vision? Would you change some pieces of it? Would you change it
entirely?
In an ideal world, what would packaging software look like to you?

Looking forward to the discussion,

Pierre
_______________________________________________
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...

[Remi Collet]

Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
>  Here is what the vision we came to and that we would like to
> discuss:
>  
>  ○ Every changes to dist-git is done via pull-requests 
IMHO Have to stay optional, making this mandatory being a terrible headache.





RFemi

[Pierre-Yves Chibon]

On Thu, Sep 26, 2019 at 03:01:25PM +0200, Remi Collet wrote:
>  Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
>  > Here is what the vision we came to and that we would like to discuss:
>  > 
>  > ○ Every changes to dist-git is done via pull-requests
>  
>  IMHO Have to stay optional, making this mandatory being a terrible headache. 
What makes it a headache? What can we do to not have this be a terrible
headache? Can we fix/improve the tooling?

Note: this is the long term vision, as we move towards it things will be
optional: opt-in, then (maybe) opt-out, then (if we want to) mandatory.


Thanks for your feedback :)
Pierre

[Troy Dawson]

On Thu, Sep 26, 2019 at 6:11 AM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:
>  On Thu, Sep 26, 2019 at 03:01:25PM +0200, Remi Collet wrote:
>  > Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
>  > > Here is what the vision we came to and that we would like to discuss:
>  > >
>  > > ○ Every changes to dist-git is done via pull-requests
>  >
>  > IMHO Have to stay optional, making this mandatory being a terrible headache.
>
>  What makes it a headache? What can we do to not have this be a terrible
>  headache? Can we fix/improve the tooling?
>
>  Note: this is the long term vision, as we move towards it things will be
>  optional: opt-in, then (maybe) opt-out, then (if we want to) mandatory.
>  
There has to be an easier way to do pull requests.  And maybe there
is, I'm not a git expert.
My biggest concern that I see isn't for my own packages, but as a
proven packager, there are times I need to "group change" things.  An
example is that I recently helped with the EPEL 7 python34 -> python36
changeover.  This required changing alot of packages.  For the sake of
simplicity, we'll say it's 100.
I now have to fork 100 packages.
Download those 100 git repo's.
And for each of those packages I have to then create my fork
pull-request branch.
After that, the steps wouldn't be too different.   I would make my
changes, commit and push.
I would then have to create the pull request (I currently don't know
how to do that from the command line), wait for the testing, get the
pull request pulled, and from there it's the same.

For me, it's those first three steps that are the headache.
Especially with having to fork the repo.  I look at that and think
that it's a waste of space, on both my machine, as well as the server
machines.

If there was a better, easier way of creating the pull request, then
it wouldn't be a headache for me.

Troy

[Pierre-Yves Chibon]

On Thu, Sep 26, 2019 at 06:38:45AM -0700, Troy Dawson wrote:
>  On Thu, Sep 26, 2019 at 6:11 AM Pierre-Yves Chibon
> <pingou(a)pingoured.fr&gt; wrote:
>  >
>  > On Thu, Sep 26, 2019 at 03:01:25PM +0200, Remi Collet wrote:
>  > > Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
>  > > > Here is what the vision we came to and that we would like to discuss:
>  > > >
>  > > > ○ Every changes to dist-git is done via pull-requests
>  > >
>  > > IMHO Have to stay optional, making this mandatory being a terrible headache.
>  >
>  > What makes it a headache? What can we do to not have this be a terrible
>  > headache? Can we fix/improve the tooling?
>  >
>  > Note: this is the long term vision, as we move towards it things will be
>  > optional: opt-in, then (maybe) opt-out, then (if we want to) mandatory.
>  >
>  
>  There has to be an easier way to do pull requests.  And maybe there
>  is, I'm not a git expert.
>  My biggest concern that I see isn't for my own packages, but as a
>  proven packager, there are times I need to "group change" things.  An
>  example is that I recently helped with the EPEL 7 python34 -> python36
>  changeover.  This required changing alot of packages.  For the sake of
>  simplicity, we'll say it's 100.
>  I now have to fork 100 packages.
>  Download those 100 git repo's.
>  And for each of those packages I have to then create my fork
>  pull-request branch.
>  After that, the steps wouldn't be too different.   I would make my
>  changes, commit and push.
>  I would then have to create the pull request (I currently don't know
>  how to do that from the command line), wait for the testing, get the
>  pull request pulled, and from there it's the same.
>  
>  For me, it's those first three steps that are the headache.
>  Especially with having to fork the repo.  I look at that and think
>  that it's a waste of space, on both my machine, as well as the server
>  machines.
>  
>  If there was a better, easier way of creating the pull request, then
>  it wouldn't be a headache for me. 
All what you are describing here should be achievable with some tooling.
I mean, we could have all the magic be in `fedpkg push` which suddenly, checks
if you have a clone, it not creates it, adds it as a remote to your git repo and
do the git push to that remote. We could then have a `fedpkg new-pr` to create
the PR or even a `fedpkg push --pr` to do both actions in one step :)

Would that make it less of a headache?

Pierre

[Troy Dawson]

On Thu, Sep 26, 2019 at 6:56 AM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:
>  On Thu, Sep 26, 2019 at 06:38:45AM -0700, Troy Dawson wrote:
>  > On Thu, Sep 26, 2019 at 6:11 AM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt;
> wrote:
>  > >
>  > > On Thu, Sep 26, 2019 at 03:01:25PM +0200, Remi Collet wrote:
>  > > > Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
>  > > > > Here is what the vision we came to and that we would like to
> discuss:
>  > > > >
>  > > > > ○ Every changes to dist-git is done via pull-requests
>  > > >
>  > > > IMHO Have to stay optional, making this mandatory being a terrible
> headache.
>  > >
>  > > What makes it a headache? What can we do to not have this be a terrible
>  > > headache? Can we fix/improve the tooling?
>  > >
>  > > Note: this is the long term vision, as we move towards it things will be
>  > > optional: opt-in, then (maybe) opt-out, then (if we want to) mandatory.
>  > >
>  >
>  > There has to be an easier way to do pull requests.  And maybe there
>  > is, I'm not a git expert.
>  > My biggest concern that I see isn't for my own packages, but as a
>  > proven packager, there are times I need to "group change" things.  An
>  > example is that I recently helped with the EPEL 7 python34 -> python36
>  > changeover.  This required changing alot of packages.  For the sake of
>  > simplicity, we'll say it's 100.
>  > I now have to fork 100 packages.
>  > Download those 100 git repo's.
>  > And for each of those packages I have to then create my fork
>  > pull-request branch.
>  > After that, the steps wouldn't be too different.   I would make my
>  > changes, commit and push.
>  > I would then have to create the pull request (I currently don't know
>  > how to do that from the command line), wait for the testing, get the
>  > pull request pulled, and from there it's the same.
>  >
>  > For me, it's those first three steps that are the headache.
>  > Especially with having to fork the repo.  I look at that and think
>  > that it's a waste of space, on both my machine, as well as the server
>  > machines.
>  >
>  > If there was a better, easier way of creating the pull request, then
>  > it wouldn't be a headache for me.
>
>  All what you are describing here should be achievable with some tooling.
>  I mean, we could have all the magic be in `fedpkg push` which suddenly, checks
>  if you have a clone, it not creates it, adds it as a remote to your git repo and
>  do the git push to that remote. We could then have a `fedpkg new-pr` to create
>  the PR or even a `fedpkg push --pr` to do both actions in one step :)
>
>  Would that make it less of a headache?
>  
Yes, if we can achieve that, then that would be good for me.
I personally would like to have my pushes checked in some way.  And if
this achieves that, then I would like to see it.

Troy

[Miroslav Suchý]

Dne 26. 09. 19 v 15:10 Pierre-Yves Chibon napsal(a):
>  On Thu, Sep 26, 2019 at 03:01:25PM +0200, Remi Collet wrote:
> > Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
> >> Here is what the vision we came to and that we would like to discuss:
> >>
> >> ○ Every changes to dist-git is done via pull-requests
> > IMHO Have to stay optional, making this mandatory being a terrible headache.
>  What makes it a headache? What can we do to not have this be a terrible
>  headache? 
I use PRs in Pagure a lot. And the Rebase/Merge functionality is broken most of the time.
For months.
Fortunately it is still a git, so when the team agrees, we can merge and push PR
manually.
Having PR a mandatory thing (relying on Pagure to do the merge) would be PITA for me.

Generally, I like it. I like PRs. But I like more then one way to do anything as well. If
one way is broken, I can use 
the other way, so it will not block me.

-- 
Miroslav Suchy, RHCA
Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys

[Pierre-Yves Chibon]

On Thu, Sep 26, 2019 at 03:40:49PM +0200, Miroslav Suchý wrote:
>  Dne 26. 09. 19 v 15:10 Pierre-Yves Chibon napsal(a):
>  > On Thu, Sep 26, 2019 at 03:01:25PM +0200, Remi Collet wrote:
>  > > Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
>  > > > Here is what the vision we came to and that we would like to discuss:
>  > > > 
>  > > > ○ Every changes to dist-git is done via pull-requests
>  > > IMHO Have to stay optional, making this mandatory being a terrible headache.
>  > What makes it a headache? What can we do to not have this be a terrible
>  > headache?
>  
>  I use PRs in Pagure a lot. And the Rebase/Merge functionality is broken most of the time.
> For months.
>  Fortunately it is still a git, so when the team agrees, we can merge and push PR
> manually.
>  Having PR a mandatory thing (relying on Pagure to do the merge) would be PITA for me.
You'll notice I didn't mention any tools in the proposal. I specifically
didn't
want to limit ourself to our current tooling :)
Before we discuss how we want to implement something let's see if we can agree
on what that thing is :)
 

Thanks!
Pierre

[Tristan Cacqueray]

On Thu, Sep 26, 2019 at 15:49 Pierre-Yves Chibon wrote:
>  On Thu, Sep 26, 2019 at 03:40:49PM +0200, Miroslav Suchý wrote:
> > Dne 26. 09. 19 v 15:10 Pierre-Yves Chibon napsal(a):
> > > On Thu, Sep 26, 2019 at 03:01:25PM +0200, Remi Collet wrote:
> > > > Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
> > > > > Here is what the vision we came to and that we would like to discuss:
> > > > > 
> > > > > ○ Every changes to dist-git is done via pull-requests
> > > > IMHO Have to stay optional, making this mandatory being a terrible
> headache.
> > > What makes it a headache? What can we do to not have this be a terrible
> > > headache?
> > 
> > I use PRs in Pagure a lot. And the Rebase/Merge functionality is broken most of the
> time. For months.
> > Fortunately it is still a git, so when the team agrees, we can merge and push PR
> manually.
> > Having PR a mandatory thing (relying on Pagure to do the merge) would be PITA for
> me.
>
>  You'll notice I didn't mention any tools in the proposal. I specifically
> didn't
>  want to limit ourself to our current tooling :)
>  Before we discuss how we want to implement something let's see if we can agree
>  on what that thing is :)
>    
Note that git-pull-requests now support pagure. The tool takes care of
creating the fork, pushing the local changes and opening the PR:
  https://github.com/Mergifyio/git-pull-request

Regards,
-Tristan

[Miro Hrončok]

On 26. 09. 19 22:37, Tristan Cacqueray wrote:
>  Note that git-pull-requests now support pagure. The tool takes care
> of
>  creating the fork, pushing the local changes and opening the PR:
>     https://github.com/Mergifyio/git-pull-request 
Will definitively test this soon, thanks!

If it works, we should make it a part of fedpkg, really.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Remi Collet]

Le 26/09/2019 à 15:10, Pierre-Yves Chibon a écrit :
>  On Thu, Sep 26, 2019 at 03:01:25PM +0200, Remi Collet wrote:
> > Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
> >> Here is what the vision we came to and that we would like to discuss:
> >>
> >> ○ Every changes to dist-git is done via pull-requests
> >
> > IMHO Have to stay optional, making this mandatory being a terrible headache.
>  
>  What makes it a headache? What can we do to not have this be a terrible
>  headache? Can we fix/improve the tooling? 
1/ I don't use Pagure, only git (from CLI)

Even for PR (I prefer to download the patch, test it locally, then "git
am" it, and IMHO, the git log is better, author vs commiter)


2/ having to wait for a build before merging will make all the process
twice slower than before, which is a real pain when you maintain lot of
packages, and sometime have to update lot of them


3/ for most of my packages, branches have diverged, and merge or
cherry-pick between branches doesn't work (have to manually resolves the
conflict)

Yes, mostly because I skip the 'mass rebuild' commits


Remi

[Vitaly Zaitsev]

On 26.09.2019 15:10, Pierre-Yves Chibon wrote:
>  What makes it a headache? What can we do to not have this be a
> terrible
>  headache? Can we fix/improve the tooling? 
I'm not going to log in into web, create a new pr, then merge it. This
is a terrible idea. Do not change current workflow.

-- 
Sincerely,
  Vitaly Zaitsev (vitaly(a)easycoding.org)

[Ken Dreyer]

On Thu, Sep 26, 2019 at 7:11 AM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:
>  On Thu, Sep 26, 2019 at 03:01:25PM +0200, Remi Collet wrote:
>  > Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
>  > > Here is what the vision we came to and that we would like to discuss:
>  > >
>  > > ○ Every changes to dist-git is done via pull-requests
>  >
>  > IMHO Have to stay optional, making this mandatory being a terrible headache.
>
>  What makes it a headache? What can we do to not have this be a terrible
>  headache? Can we fix/improve the tooling? 
One of the reasons I push directly to branches in Pagure instead of
opening Pull Requests is that the workflow is too clunky and slow. I
have to toggle between the browser and the console, make sure I have a
Fedora Kerberos ticket, etc.

Please get a stopwatch app and try the following workflows in GitHub
and Pagure, and compare the speed at which you can do these:

- Create a new personal fork of a repository

  This is almost instant on GitHub, and much slower in Pagure,
sometimes hanging entirely at the "waiting..." web page.

- Push a hundred commits to a topic branch

  GitHub can accept the ref updates very quickly, and Pagure has so
many hooks and operations that I've seen my Git client hang for
seconds or sometimes even minutes. I'm not sure what is going on, but
I suspect it involves ACL checks, Fedmesg, etc. I've seen this one
several times: I forked a project a year ago and I did not update my
forked copy of "master". I push a new topic branch to my clone, and
this topic branch has all the changes from todays' upstream master,
plus my new changes for the topic branch. When I run "git push" for my
new topic branch, Pagure.io will process all of those commits
one-by-one and emit fedmsgs for all of them *according to my old
version of master, not upstream*. I have no idea if that is single
reason for why I see my "git push" commits return very slowly, but it
is probably one of the reasons.

- Open a new pull request

  The GitHub web UI provides a lot of different buttons in a lot of
different places, allowing the user to fall into the "pit of success"
and submit the change upstream. GitHub always immediately knows "hey
this branch is ready to go upstream, do you want to open a PR?" With
Pagure, the UI to open a pull request requires many different clicks,
and if I open it from certain pages (like the origin repository), the
web UI makes some AJAX calls that just hang forever. GitHub also has a
really mature CLI ("hub") that makes opening PRs lightning quick
because the GitHub's REST API responds very fast.

You might wonder why I've not opened tickets for these, and it's
because I am not sure it's worth piling on your team when you guys are
already extremely overloaded. The UX problems I've encountered do not
strike me as trivial issues to resolve in an afternoon. Sometimes I
ask in #fedora-admin if it really looks like I've broken something.
Maybe that's lame and I should just open the tickets. I also have a
bigger concern though: I've played around with pygit2 in some smaller
applications in order to learn it, and I'm beginning to think that
it's simply not possible to get close to the performance and
functionality that GitHub has achieved.

I would like to hear if you see Pagure as still strategic, and if so,
how we can make these common user operations faster.

- Ken

[Robert Marcano]

On 9/26/19 12:57 PM, Ken Dreyer wrote:
>  On Thu, Sep 26, 2019 at 7:11 AM Pierre-Yves Chibon
> <pingou(a)pingoured.fr&gt; wrote:
> >
> > On Thu, Sep 26, 2019 at 03:01:25PM +0200, Remi Collet wrote:
> >> Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
> >>> Here is what the vision we came to and that we would like to discuss:
> >>>
> >>> ○ Every changes to dist-git is done via pull-requests
> >>
> >> IMHO Have to stay optional, making this mandatory being a terrible headache.
> >
> > What makes it a headache? What can we do to not have this be a terrible
> > headache? Can we fix/improve the tooling?
>  
>  One of the reasons I push directly to branches in Pagure instead of
>  opening Pull Requests is that the workflow is too clunky and slow. I
>  have to toggle between the browser and the console, make sure I have a
>  Fedora Kerberos ticket, etc. 
GitLab support creating merge request from a git push:

  https://docs.gitlab.com/ce/user/project/merge_requests/#git-push-options

Maybe something like this on Pagure could help alleviate the process.

>  
>  Please get a stopwatch app and try the following workflows in GitHub
>  and Pagure, and compare the speed at which you can do these:
>  
>  - Create a new personal fork of a repository
>  
>     This is almost instant on GitHub, and much slower in Pagure,
>  sometimes hanging entirely at the "waiting..." web page.
>  
>  - Push a hundred commits to a topic branch
>  
>     GitHub can accept the ref updates very quickly, and Pagure has so
>  many hooks and operations that I've seen my Git client hang for
>  seconds or sometimes even minutes. I'm not sure what is going on, but
>  I suspect it involves ACL checks, Fedmesg, etc. I've seen this one
>  several times: I forked a project a year ago and I did not update my
>  forked copy of "master". I push a new topic branch to my clone, and
>  this topic branch has all the changes from todays' upstream master,
>  plus my new changes for the topic branch. When I run "git push" for my
>  new topic branch, Pagure.io will process all of those commits
>  one-by-one and emit fedmsgs for all of them *according to my old
>  version of master, not upstream*. I have no idea if that is single
>  reason for why I see my "git push" commits return very slowly, but it
>  is probably one of the reasons.
>  
>  - Open a new pull request
>  
>     The GitHub web UI provides a lot of different buttons in a lot of
>  different places, allowing the user to fall into the "pit of success"
>  and submit the change upstream. GitHub always immediately knows "hey
>  this branch is ready to go upstream, do you want to open a PR?" With
>  Pagure, the UI to open a pull request requires many different clicks,
>  and if I open it from certain pages (like the origin repository), the
>  web UI makes some AJAX calls that just hang forever. GitHub also has a
>  really mature CLI ("hub") that makes opening PRs lightning quick
>  because the GitHub's REST API responds very fast.
>  
>  You might wonder why I've not opened tickets for these, and it's
>  because I am not sure it's worth piling on your team when you guys are
>  already extremely overloaded. The UX problems I've encountered do not
>  strike me as trivial issues to resolve in an afternoon. Sometimes I
>  ask in #fedora-admin if it really looks like I've broken something.
>  Maybe that's lame and I should just open the tickets. I also have a
>  bigger concern though: I've played around with pygit2 in some smaller
>  applications in order to learn it, and I'm beginning to think that
>  it's simply not possible to get close to the performance and
>  functionality that GitHub has achieved.
>  
>  I would like to hear if you see Pagure as still strategic, and if so,
>  how we can make these common user operations faster.
>  
>  - Ken
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Jeremy Cline]

On Thu, Sep 26, 2019 at 01:22:03PM -0400, Robert Marcano via devel wrote:
>  On 9/26/19 12:57 PM, Ken Dreyer wrote:
>  > On Thu, Sep 26, 2019 at 7:11 AM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt;
> wrote:
>  > > 
>  > > On Thu, Sep 26, 2019 at 03:01:25PM +0200, Remi Collet wrote:
>  > > > Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
>  > > > > Here is what the vision we came to and that we would like to
> discuss:
>  > > > > 
>  > > > > ○ Every changes to dist-git is done via pull-requests
>  > > > 
>  > > > IMHO Have to stay optional, making this mandatory being a terrible
> headache.
>  > > 
>  > > What makes it a headache? What can we do to not have this be a terrible
>  > > headache? Can we fix/improve the tooling?
>  > 
>  > One of the reasons I push directly to branches in Pagure instead of
>  > opening Pull Requests is that the workflow is too clunky and slow. I
>  > have to toggle between the browser and the console, make sure I have a
>  > Fedora Kerberos ticket, etc.
>  
>  GitLab support creating merge request from a git push:
>  
>   https://docs.gitlab.com/ce/user/project/merge_requests/#git-push-options
>  
>  Maybe something like this on Pagure could help alleviate the process.
>   
Or we could just use GitLab because it's pretty good and has many orders
of magnitude more developers and users. Let's not re-invent more wheels
that others have already built and continue to polish.

[Joe Doss]

On 9/26/19 11:57 AM, Ken Dreyer wrote:
>  I would like to hear if you see Pagure as still strategic, and if
> so,
>  how we can make these common user operations faster. 
I always wondered why Fedora home rolled their own Github/Gitlab clone. 
Using either Github or Gitlab would be better for new users to get 
started with helping to contribute back to Fedora without having to jump 
through hoops to figure out how to work around the pitfalls of Pagure.

Joe



-- 
Joe Doss
joe(a)solidadmin.com

[Stephen John Smoogen]

On Tue, 1 Oct 2019 at 10:18, Joe Doss <joe(a)solidadmin.com&gt; wrote:
>  On 9/26/19 11:57 AM, Ken Dreyer wrote:
>  > I would like to hear if you see Pagure as still strategic, and if so,
>  > how we can make these common user operations faster.
>
>  I always wondered why Fedora home rolled their own Github/Gitlab clone. 
Because when development started, neither was a market winner. When
development ideas started, Launchpad was where everything was moving
to and people kept saying that Fedora was falling behind because it
didn't have its own Launchpad and the hosted solution we were using
with Trac was so lame.

Then when GitHub started to take over, it is closed source and that
was a big nono for services from Fedora developers. [AKA we move to
Github various package owners were going to drop their packages and
leave.] Gitlab was also looked at but then there was a push to run it
ourselves and it is a ruby frame and we are a python shop. Once you
start to stand it up you find you need a lot more hardware/software
configs to make it scale for the amount we are pulling/pushing into
git. That then leads to needing more equipment which leads to...

Then there are problems with budgets and figuring out what exactly it
would cost. We fall outside of many of the 'caveats' that would allow
us to get free. We pull/push too much data (as people with coprs
regularly find out when throttled down to 0), we eat a lot of data, we
are not a non-profit etc. That means moving to anything costs money,
but figuring out what it costs vs random bill of the month is not
trivial. Which then means it takes a long time

So you could sit around and just let pagure wither while waiting for
Godot, or you could continue to improve it and make it something
people may want to use for other things.

>  Using either Github or Gitlab would be better for new users to get
>  started with helping to contribute back to Fedora without having to jump
>  through hoops to figure out how to work around the pitfalls of Pagure.
>
>  Joe
>
>
>
>  --
>  Joe Doss
>  joe(a)solidadmin.com
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail] 


-- 
Stephen J Smoogen.

[John M. Harris Jr]

On Tuesday, October 1, 2019 7:55:10 AM MST Stephen John Smoogen wrote:
>  Then when GitHub started to take over, it is closed source and that
>  was a big nono for services from Fedora developers. [AKA we move to
>  Github various package owners were going to drop their packages and
>  leave.]  
This is, most likely, still the case today as well.

I don't know why some people think it's a good idea to move to GitHub, a 
platform owned by Microsoft.

-- 
John M. Harris, Jr.
Splentity

[Felix Schwarz]

Am 01.10.19 um 16:55 schrieb Stephen John Smoogen:
>  Then there are problems with budgets and figuring out what exactly
> it
>  would cost. We fall outside of many of the 'caveats' that would allow
>  us to get free. 
IIRC at the time when Fedora evaluated its options the open source version of 
Gitlab was more limited than today. AFAIK Debian + FreeDesktop developers 
worked with Gitlab Inc. and finally succeeded in getting necessary features 
included in the open source version.

If the evaluation was done today and there were no Pagure I suspect Fedora 
would use gitlab as well.

(I'm also wondering if Fedora writes too much custom infrastructure when there 
are "open core" offerings which might provide more features - e.g. Gitlab 
instead of Pagure, sentry instead of abrt. But I'm aware of limited ressources 
and I trust my fellow Fedorians with their judgement.)

Felix

[Martin Kolman]

On Wed, 2019-10-02 at 09:39 +0200, Felix Schwarz wrote:
>  Am 01.10.19 um 16:55 schrieb Stephen John Smoogen:
>  > Then there are problems with budgets and figuring out what exactly it
>  > would cost. We fall outside of many of the 'caveats' that would allow
>  > us to get free.
>  
>  IIRC at the time when Fedora evaluated its options the open source version of 
>  Gitlab was more limited than today. AFAIK Debian + FreeDesktop developers 
>  worked with Gitlab Inc. and finally succeeded in getting necessary features 
>  included in the open source version.
>  
>  If the evaluation was done today and there were no Pagure I suspect Fedora 
>  would use gitlab as well.
>  
>  (I'm also wondering if Fedora writes too much custom infrastructure when there 
>  are "open core" offerings which might provide more features - e.g. Gitlab 
>  instead of Pagure, sentry instead of abrt. But I'm aware of limited ressources 
>  and I trust my fellow Fedorians with their judgement.) 
Personally I still think
it's not a very good idea to depend on any open-core infrastructure
software, due to the possibility of conflict of interrest in the future.

Eq. the company behind the software will not accept your patches as it clashes with their
business model & proprietary code additions they sell to their customers. Then you
will
have to maintain those patches pretty much indefinitely. Might as well use a fully open
source solution to avoid such future pitfalls.

>  
>  Felix
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Stephen John Smoogen]

On Wed, 2 Oct 2019 at 03:39, Felix Schwarz <fschwarz(a)fedoraproject.org&gt; wrote:
>  Am 01.10.19 um 16:55 schrieb Stephen John Smoogen:
>  > Then there are problems with budgets and figuring out what exactly it
>  > would cost. We fall outside of many of the 'caveats' that would allow
>  > us to get free.
>
>  IIRC at the time when Fedora evaluated its options the open source version of
>  Gitlab was more limited than today. AFAIK Debian + FreeDesktop developers
>  worked with Gitlab Inc. and finally succeeded in getting necessary features
>  included in the open source version.
>
>  If the evaluation was done today and there were no Pagure I suspect Fedora
>  would use gitlab as well.
>
>  (I'm also wondering if Fedora writes too much custom infrastructure when there
>  are "open core" offerings which might provide more features - e.g. Gitlab
>  instead of Pagure, sentry instead of abrt. But I'm aware of limited ressources
>  and I trust my fellow Fedorians with their judgement.)
>  
One of the goals many early Fedora participants had was that Fedora
was to be an incubator for open source software which isn't available
as FLOSS elsewhere. The issue though is that software does not spring
up instantly from the head of Zeus fully formed. It takes years and a
lot of effort. Other things can overtake what you are working on and
just be so much larger and bigger than what is now. It can also be
that what communities want changes faster than it takes for the
software to be written.



>  Felix 


-- 
Stephen J Smoogen.

[Neal Gompa]

On Wed, Oct 2, 2019 at 8:42 AM Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  On Wed, 2 Oct 2019 at 03:39, Felix Schwarz <fschwarz(a)fedoraproject.org&gt; wrote:
>  >
>  >
>  > Am 01.10.19 um 16:55 schrieb Stephen John Smoogen:
>  > > Then there are problems with budgets and figuring out what exactly it
>  > > would cost. We fall outside of many of the 'caveats' that would allow
>  > > us to get free.
>  >
>  > IIRC at the time when Fedora evaluated its options the open source version of
>  > Gitlab was more limited than today. AFAIK Debian + FreeDesktop developers
>  > worked with Gitlab Inc. and finally succeeded in getting necessary features
>  > included in the open source version.
>  >
>  > If the evaluation was done today and there were no Pagure I suspect Fedora
>  > would use gitlab as well.
>  >
>  > (I'm also wondering if Fedora writes too much custom infrastructure when there
>  > are "open core" offerings which might provide more features - e.g. Gitlab
>  > instead of Pagure, sentry instead of abrt. But I'm aware of limited ressources
>  > and I trust my fellow Fedorians with their judgement.)
>  >
>
>  One of the goals many early Fedora participants had was that Fedora
>  was to be an incubator for open source software which isn't available
>  as FLOSS elsewhere. The issue though is that software does not spring
>  up instantly from the head of Zeus fully formed. It takes years and a
>  lot of effort. Other things can overtake what you are working on and
>  just be so much larger and bigger than what is now. It can also be
>  that what communities want changes faster than it takes for the
>  software to be written.
>  
For those that might have forgotten, Ansible is a fantastic success
story from those principles (It started as a fedorahosted project
called func). It took a while for it to take off, but it did.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Kevin Kofler]

Pierre-Yves Chibon wrote:
>  On Thu, Sep 26, 2019 at 03:01:25PM +0200, Remi Collet wrote:
> > Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
> > > Here is what the vision we came to and that we would like to discuss:
> > > 
> > > ○ Every changes to dist-git is done via pull-requests
> > 
> > IMHO Have to stay optional, making this mandatory being a terrible
> > headache.
>  
>  What makes it a headache? 
The fact that this forces a specific workflow on everyone and makes it 
impossible to use different workflows. In particular, it breaks:
* using the git cli directly, without fedpkg (or some dedicated Pagure CLI),
* using a Git GUI frontend such as git-cola (as I do),
* doing more complex Git operations such as merges, fast forwards, cherry
  picks, etc. (I would actually argue for MORE flexibility, such as allowing
  force pushes, not less as in your proposal.)
and probably more things.

>  What can we do to not have this be a terrible headache? 
Nothing. This is a headache by design.

>  Can we fix/improve the tooling? 
No. No amount of tooling can make a forced pull-request workflow workable.

You need a different CI approach. Maybe:
* a push hook that just locks the repository and does the tests before
  validating the push, though I can see that becoming a PITA as well, or
* an asynchronous CI push hook that just notifies the maintainer on
  failures, but does not otherwise touch the repository, or
* an asynchronous CI push hook that automatically reverts commits that fail
  the CI tests, or even
* an asynchronous CI push hook that automatically uses a force push to
  revert the commits that fail the CI tests in such a way that they vanish
  from history entirely, as if they had never happened,
or something more clever that we have not thought of yet.

        Kevin Kofler

[Stephen John Smoogen]

On Thu, 26 Sep 2019 at 09:02, Remi Collet <Fedora(a)famillecollet.com&gt; wrote:
>  Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
>  > Here is what the vision we came to and that we would like to discuss:
>  >
>  > ○ Every changes to dist-git is done via pull-requests
>
>  IMHO Have to stay optional, making this mandatory being a terrible headache.
>  
I am going to agree in that it seems to go against what a large number
of packagers do currently to try and deal with the number of branches
and packages they try to keep up with. I say this from breaking their
workflow with EPEL-8 and getting an earful or two.

The workflow they follow is:
0. Go to first package in list
A. Make a change in master
B. Push that change
C. fedpkg switch-branch <first on N branches>
D. some variant of 'git merge master', 'git checkout master (some
flags I forget)', etc
E. fedpkg push
F. fedpkg build
G. fedpkg update
H. Iterate branch and GOTO C.
I. Iterate package and GOTO 0.

They might be doing that for FN-1, FN, FN+1, rawhide, and possibly
EPEL-6,7,8 so 7 branches. They can't do it with 8 because we added a
file which causes conflicts and they don't want to deal with pull
requests or anything like that. Some do this for dozens to hundreds of
packages. Now I am the first that is probably too many packages but it
is also one of those if they drop them then all of them go away.


-- 
Stephen J Smoogen.

[Pierre-Yves Chibon]

On Thu, Sep 26, 2019 at 09:14:38AM -0400, Stephen John Smoogen wrote:
>  On Thu, 26 Sep 2019 at 09:02, Remi Collet
> <Fedora(a)famillecollet.com&gt; wrote:
>  >
>  > Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
>  > > Here is what the vision we came to and that we would like to discuss:
>  > >
>  > > ○ Every changes to dist-git is done via pull-requests
>  >
>  > IMHO Have to stay optional, making this mandatory being a terrible headache.
>  >
>  
>  I am going to agree in that it seems to go against what a large number
>  of packagers do currently to try and deal with the number of branches
>  and packages they try to keep up with. I say this from breaking their
>  workflow with EPEL-8 and getting an earful or two.
>  
>  The workflow they follow is:
>  0. Go to first package in list
>  A. Make a change in master
>  B. Push that change
>  C. fedpkg switch-branch <first on N branches>
>  D. some variant of 'git merge master', 'git checkout master (some
>  flags I forget)', etc 
This is why I listed in the things to figure out: a way to open a pull-request
against different branch.
This way, from the master branch of your fork you could open a PR against
rawhide, Fn+1, Fn, EpelX, EpelX-1...
This would actually reduce the amount of work you'd have to do, you commit in a
single branch and ask for the PR to be automatically created in all the branches
of interest.
Tests then run
PR gets (reviewed and) merged
Your work is done, no more touching koji or bodhi, it's all automated :)


Pierre

[Zbigniew Jędrzejewski-Szmek]

On Thu, Sep 26, 2019 at 03:01:25PM +0200, Remi Collet wrote:
>  Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
>  > Here is what the vision we came to and that we would like to discuss:
>  > 
>  > ○ Every changes to dist-git is done via pull-requests
>  
>  IMHO Have to stay optional, making this mandatory being a terrible headache. 
Agreed. *How* commits are pushed to the repo is generally completely
orthogonal to the issue of automatically building stuff based on
commits to the repo. One the one hand, none of the other stuff in the
proposal requires PRs, and on the other hand, this requirement would
break other legitimate workflows: scripted cleanups to small things
(like the recent removal of Group:) that don't merite rebuilding,
scripted management of packages that people do, etc.

Zbyszek

[Pierre-Yves Chibon]

On Thu, Sep 26, 2019 at 01:15:03PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  On Thu, Sep 26, 2019 at 03:01:25PM +0200, Remi Collet wrote:
>  > Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
>  > > Here is what the vision we came to and that we would like to discuss:
>  > > 
>  > > ○ Every changes to dist-git is done via pull-requests
>  > 
>  > IMHO Have to stay optional, making this mandatory being a terrible headache.
>  
>  Agreed. *How* commits are pushed to the repo is generally completely
>  orthogonal to the issue of automatically building stuff based on
>  commits to the repo. One the one hand, none of the other stuff in the
>  proposal requires PRs,  
If you want to test changes before they are merged PR is the place to do this :)

>  and on the other hand, this requirement would
>  break other legitimate workflows: scripted cleanups to small things
>  (like the recent removal of Group:) that don't merite rebuilding,
>  scripted management of packages that people do, etc. 
If you don't bump the release, the automated build will stop because the NEVR
already exists in koji, so I don't quite see how scripting cleaning up is
impacted by this.
Just like today, you will clone the git repo, edit the master branch and instead
of doing just a git push you'd do a git push fork && open PR and if typing
two
commands is too much we can easily build tooling around this :)


Again, I'd like to reinforce that the idea is not to enforce any part of this
workflow tomorrow, it'll be a smooth, slow and long transition. My question is
whether this is a place where we want to go or can we come up with a
different/better one? :)

Thanks for your input :)
Pierre

[Michael Cronenworth]

On 9/26/19 8:42 AM, Pierre-Yves Chibon wrote:
>  Again, I'd like to reinforce that the idea is not to enforce any
> part of this
>  workflow tomorrow, it'll be a smooth, slow and long transition. My question is
>  whether this is a place where we want to go or can we come up with a
>  different/better one?:) 
The problem with this statement is that this is written as "come up with something 
different or else we're doing it my way anyway" and not "ok, we'll not
do this."

I don't want a build for every commit. Sometimes I cache changes for future 
releases. A commit does *not* always equal an update. This proposed workflow model 
doesn't help me. It hurts. It makes me think of dropping my roles from Fedora.

With that said I believe there are some good parts to it, but the PR for everything 
and builds for every commit are show-stoppers.

[Pierre-Yves Chibon]

On Thu, Sep 26, 2019 at 08:47:24AM -0500, Michael Cronenworth wrote:
>  On 9/26/19 8:42 AM, Pierre-Yves Chibon wrote:
>  > Again, I'd like to reinforce that the idea is not to enforce any part of this
>  > workflow tomorrow, it'll be a smooth, slow and long transition. My question is
>  > whether this is a place where we want to go or can we come up with a
>  > different/better one?:)
>  
>  The problem with this statement is that this is written as "come up with
>  something different or else we're doing it my way anyway" and not "ok,
> we'll
>  not do this." 
This is all very open for discussion but I'm going to defend my proposal to some
extent :)

There is a clear initial rejection of a PR-only contribution model. I hear that
and that may mean that we never go this way. I'm honestly fine with that :)
I do want to see why that is a show-stopper and if we can find ways to not have
it be a show-stopper.

When we work on upstream projects, I think it's pretty standard now to always go
via PRs, even for your own branch.
So that tests are run, so that other member of the community can see, comment,
review the change.
What is so different in Fedora that we cannot move to this model?
Is it a tooling issue?
Is it something else?

>  I don't want a build for every commit. Sometimes I cache changes
> for future
>  releases. A commit does *not* always equal an update.  
There are a few ways to approach this:
- don't bump the release in the spec file, the build will be triggered and will
  just never complete
- have a magic keyword in the commit message that prevents the build from
  happening at all
- is an extra build in rawhide such a big deal?
- gate at bodhi level builds that have the same rpm payload as the previous
  build shipped
  --> this is what OpenSUSE does btw, when you build an update, they rebuild the
  entire dependency tree but will filter out the RPM whose payload haven't
  changed from the update that is pushed to the users.

>  This proposed workflow model doesn't help me. It hurts. It makes
> me think of
>  dropping my roles from Fedora. 
Seeing you leave is really not the idea and would be the opposite result.

The way I thought of this was:
- is it easier to ask everyone for what their ideal workflow would be?
or
- present a workflow (which is hopefully not entirely insane) and encourage the
  community to patch it so we end up with something that is consensual between
  all of us?

I went with the later approach. Using the brainstorming session at flock, I'm
proposing *a* workflow, it's not perfect, it may not not ideal for everyone but
it's a start and we can patch it as much as we want.
We may end up with something entirely different from what I'm proposing and
that's fine, but at least we'll have an agreed upon idea of where we want to go
(ie: a long term vision)


Does that make sense?
I am sorry if the wording I've used didn't make this clear in my initial email
:(


Pierre

[Zbigniew Jędrzejewski-Szmek]

On Thu, Sep 26, 2019 at 04:07:59PM +0200, Pierre-Yves Chibon wrote:
>  On Thu, Sep 26, 2019 at 08:47:24AM -0500, Michael Cronenworth wrote:
>  > On 9/26/19 8:42 AM, Pierre-Yves Chibon wrote:
>  > > Again, I'd like to reinforce that the idea is not to enforce any part of
> this
>  > > workflow tomorrow, it'll be a smooth, slow and long transition. My question
> is
>  > > whether this is a place where we want to go or can we come up with a
>  > > different/better one?:)
>  > 
>  > The problem with this statement is that this is written as "come up with
>  > something different or else we're doing it my way anyway" and not "ok,
> we'll
>  > not do this."
>  
>  This is all very open for discussion but I'm going to defend my proposal to some
>  extent :)
>  
>  There is a clear initial rejection of a PR-only contribution model. I hear that
>  and that may mean that we never go this way. I'm honestly fine with that :)
>  I do want to see why that is a show-stopper and if we can find ways to not have
>  it be a show-stopper.
>  
>  When we work on upstream projects, I think it's pretty standard now to always go
>  via PRs, even for your own branch.
>  So that tests are run, so that other member of the community can see, comment,
>  review the change. 
That's completely fine...

>  What is so different in Fedora that we cannot move to this model?
... the problem is saying that this is the only approach.

In various upstream projects I maintain, PRs may be used in 99.5% of cases,
but there always are cases where a manual merge or a direct commit is the
best option.

Particular aspect of working in a distro is the number of repos one might
touch: it would be very unusual to commit to more than a handful of github
repos in a year. It is completely normal to commit to hundreds of dist-git
repos in one week when doing a mass change.

Zbyszek

[Ben Rosser]

On Thu, Sep 26, 2019 at 4:29 PM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:
>  There is a clear initial rejection of a PR-only contribution model. I
> hear that
>  and that may mean that we never go this way. I'm honestly fine with that :)
>  I do want to see why that is a show-stopper and if we can find ways to not have
>  it be a show-stopper.
>
>  When we work on upstream projects, I think it's pretty standard now to always go
>  via PRs, even for your own branch.
>  So that tests are run, so that other member of the community can see, comment,
>  review the change.
>  What is so different in Fedora that we cannot move to this model?
>  Is it a tooling issue?
>  Is it something else? 
Most packages in Fedora are effectively one-person projects (modulo
rebuild scripts and other automated tooling). My experience when
working on a personal project is that I don't use PRs for changes even
if I do develop a change in a branch, rather than master; it's a lot
of unnecessary overhead. There are no "other members" of the
community. No one is reviewing the change other than me.

For some critical, high-profile packages maintained by a team of
people, forcing pull requests seems reasonable enough. I'm very
skeptical it makes sense for most of the distribution. I'm glad you're
thinking about improving the packager experience (it's a very
important topic!), but I think it's critical that we keep in mind the
use case of the "ordinary packager", not the "expert packager". And I
think that's packages with limited divergence from upstream (only a
few patches) that only one, _maybe_ two, people regularly touch.

Ben Rosser

[Pierre-Yves Chibon]

On Thu, Sep 26, 2019 at 04:46:32PM +0200, Ben Rosser wrote:
>  On Thu, Sep 26, 2019 at 4:29 PM Pierre-Yves Chibon
> <pingou(a)pingoured.fr&gt; wrote:
>  > There is a clear initial rejection of a PR-only contribution model. I hear that
>  > and that may mean that we never go this way. I'm honestly fine with that :)
>  > I do want to see why that is a show-stopper and if we can find ways to not have
>  > it be a show-stopper.
>  >
>  > When we work on upstream projects, I think it's pretty standard now to always
> go
>  > via PRs, even for your own branch.
>  > So that tests are run, so that other member of the community can see, comment,
>  > review the change.
>  > What is so different in Fedora that we cannot move to this model?
>  > Is it a tooling issue?
>  > Is it something else?
>  
>  Most packages in Fedora are effectively one-person projects (modulo
>  rebuild scripts and other automated tooling). My experience when
>  working on a personal project is that I don't use PRs for changes even
>  if I do develop a change in a branch, rather than master; it's a lot
>  of unnecessary overhead. There are no "other members" of the
>  community. No one is reviewing the change other than me. 
Would this change if the PR was automatically tested for you without you having
to do anything?

Thanks,
Pierre

[Michael Cronenworth]

On 9/26/19 10:28 AM, Pierre-Yves Chibon wrote:
>  Would this change if the PR was automatically tested for you without
> you having
>  to do anything? 
I always run local mock builds prior to commits. Maybe not everyone likes to do this 
and wants Koji to do it for them, but I prefer local mock builds. Having it done for 
me is redundant. I don't think I would stop doing local mock builds.

[Pierre-Yves Chibon]

On Thu, Sep 26, 2019 at 11:23:21AM -0500, Michael Cronenworth wrote:
>  On 9/26/19 10:28 AM, Pierre-Yves Chibon wrote:
>  > Would this change if the PR was automatically tested for you without you having
>  > to do anything?
>  
>  I always run local mock builds prior to commits. Maybe not everyone likes to
>  do this and wants Koji to do it for them, but I prefer local mock builds.
>  Having it done for me is redundant. I don't think I would stop doing local
>  mock builds. 
There are more tests we can do than simply: does it build?
There is also the: Does it install? Does it upgrade? Does it uninstall?
Does the machine still boot after installing it?
Does it adhere to our packaging guidelines?

So, all of the work the CI SIG folks are trying to set up these days.

Pierre

[Ben Rosser]

On Thu, Sep 26, 2019 at 5:29 PM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:
>  On Thu, Sep 26, 2019 at 04:46:32PM +0200, Ben Rosser wrote:
>  > On Thu, Sep 26, 2019 at 4:29 PM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt;
> wrote:
>  > > There is a clear initial rejection of a PR-only contribution model. I hear
> that
>  > > and that may mean that we never go this way. I'm honestly fine with that
> :)
>  > > I do want to see why that is a show-stopper and if we can find ways to not
> have
>  > > it be a show-stopper.
>  > >
>  > > When we work on upstream projects, I think it's pretty standard now to
> always go
>  > > via PRs, even for your own branch.
>  > > So that tests are run, so that other member of the community can see, comment,
>  > > review the change.
>  > > What is so different in Fedora that we cannot move to this model?
>  > > Is it a tooling issue?
>  > > Is it something else?
>  >
>  > Most packages in Fedora are effectively one-person projects (modulo
>  > rebuild scripts and other automated tooling). My experience when
>  > working on a personal project is that I don't use PRs for changes even
>  > if I do develop a change in a branch, rather than master; it's a lot
>  > of unnecessary overhead. There are no "other members" of the
>  > community. No one is reviewing the change other than me.
>
>  Would this change if the PR was automatically tested for you without you having
>  to do anything? 
No, not really? For a personal project, a continuous integration
system can be set up to run tests on _all_ of my commits, regardless
of whether or not they're to master or to a development branch. What's
the benefit of creating a pull request here?

That being said, packages are slightly different. If it wasn't
necessary to use the web interface to make a pull request and if
fedpkg could do it for me [1], and automatically merge it when the
build succeeds... that might be nice. But if manual work is required
to create a pull request-- filling out a form on Pagure, manually
forking a project, etc.-- I think it's a lot of overhead. And I
wouldn't want to do it for most of my packages.

Ben Rosser

[1] And, in an ideal world, do it without needing an API key but
rather authenticating to Pagure via some other mechanism that doesn't
require manually downloading a key, but I digress...

[Martin Kolman]

On Thu, 2019-09-26 at 18:26 +0200, Ben Rosser wrote:
>  On Thu, Sep 26, 2019 at 5:29 PM Pierre-Yves Chibon
> <pingou(a)pingoured.fr&gt; wrote:
>  > On Thu, Sep 26, 2019 at 04:46:32PM +0200, Ben Rosser wrote:
>  > > On Thu, Sep 26, 2019 at 4:29 PM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt;
> wrote:
>  > > > There is a clear initial rejection of a PR-only contribution model. I hear
> that
>  > > > and that may mean that we never go this way. I'm honestly fine with
> that :)
>  > > > I do want to see why that is a show-stopper and if we can find ways to not
> have
>  > > > it be a show-stopper.
>  > > > 
>  > > > When we work on upstream projects, I think it's pretty standard now to
> always go
>  > > > via PRs, even for your own branch.
>  > > > So that tests are run, so that other member of the community can see,
> comment,
>  > > > review the change.
>  > > > What is so different in Fedora that we cannot move to this model?
>  > > > Is it a tooling issue?
>  > > > Is it something else?
>  > > 
>  > > Most packages in Fedora are effectively one-person projects (modulo
>  > > rebuild scripts and other automated tooling). My experience when
>  > > working on a personal project is that I don't use PRs for changes even
>  > > if I do develop a change in a branch, rather than master; it's a lot
>  > > of unnecessary overhead. There are no "other members" of the
>  > > community. No one is reviewing the change other than me.
>  > 
>  > Would this change if the PR was automatically tested for you without you having
>  > to do anything?
>  
>  No, not really? For a personal project, a continuous integration
>  system can be set up to run tests on _all_ of my commits, regardless
>  of whether or not they're to master or to a development branch. What's
>  the benefit of creating a pull request here?
>  
>  That being said, packages are slightly different. If it wasn't
>  necessary to use the web interface to make a pull request and if
>  fedpkg could do it for me [1], and automatically merge it when the
>  build succeeds... that might be nice. But if manual work is required
>  to create a pull request-- filling out a form on Pagure, manually
>  forking a project, etc.-- I think it's a lot of overhead. And I
>  wouldn't want to do it for most of my packages.
>  
>  Ben Rosser
>  
>  [1] And, in an ideal world, do it without needing an API key but
>  rather authenticating to Pagure via some other mechanism that doesn't
>  require manually downloading a key, but I digress... 
If we are talking packager
quality of live imporvements, I would vote for this as one of the
improvements - unification of the ~5 methods of authetication one might need to apply when
working
with fedpkg at the moment.

Ideally there should be one method, most likely kerberos, that autheticates the user with
FAS
and should be used for everything. While its true that some of the services (Pagure, COPR)
use
API keys, this should be an implementation detail in this case. You authorize Pagure &
COPR
with your FAS account once (or possibly this happens automatically) and don't have to
care
about API keys unless you are wiritng some automated tool that needs them.


>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Michael Cronenworth]

On 9/26/19 9:07 AM, Pierre-Yves Chibon wrote:
>  What is so different in Fedora that we cannot move to this model?
>  Is it a tooling issue?
>  Is it something else? 
As others have already stated that may work in projects with tens, hundreds, or 
thousands of contributors, but most of Fedora packages are owned and maintained by a 
single person. I'm not going to wait around for anyone to review my commit or 
comment on it. No one tests or comments on most of my Bodhi updates.

The same development model for upstream projects is not 1:1 to Fedora packaging. 
Stop trying to force it.

> > I don't want a build for every commit. Sometimes I cache
> changes for future
> > releases. A commit does*not*  always equal an update.
>  There are a few ways to approach this:
>  - don't bump the release in the spec file, the build will be triggered and will
>     just never complete 
It should be an opt-in feature. Who called for this feature? I don't remember seeing 
anyone ask for it.

>  - have a magic keyword in the commit message that prevents the build
> from
>     happening at all 
No, thanks. Requires way too much human interaction (memory).
>  - is an extra build in rawhide such a big deal? 
It may fail to
build because of a dependency on other features not yet in place... 
so it *must* be opt-in.
>  - gate at bodhi level builds that have the same rpm payload as the
> previous
>     build shipped
>     --> this is what OpenSUSE does btw, when you build an update, they rebuild the
>     entire dependency tree but will filter out the RPM whose payload haven't
>     changed from the update that is pushed to the users. 
If you want to spend time doing this go ahead, but it feels like a waste of 
resources and talent.

> > This proposed workflow model doesn't help me. It hurts. It
> makes me think of
> > dropping my roles from Fedora.
>  Seeing you leave is really not the idea and would be the opposite result.
>
>  The way I thought of this was:
>  - is it easier to ask everyone for what their ideal workflow would be?
>  or
>  - present a workflow (which is hopefully not entirely insane) and encourage the
>     community to patch it so we end up with something that is consensual between
>     all of us?
>
>  I went with the later approach. Using the brainstorming session at flock, I'm
>  proposing*a*  workflow, it's not perfect, it may not not ideal for everyone but
>  it's a start and we can patch it as much as we want.
>  We may end up with something entirely different from what I'm proposing and
>  that's fine, but at least we'll have an agreed upon idea of where we want to go
>  (ie: a long term vision)
>
>
>  Does that make sense?
>  I am sorry if the wording I've used didn't make this clear in my initial email
>  :( 
If you want to have fedpkg do all of this for you buy default, sure, fine. It should 
/also/ have a configuration able to be defined to disable all of this automatic 
stuff. The reverse could also happen. Bake all this functionality in and make it opt-in.

Again, who was asking for all of this?

The only interesting part is the automated testing and automated push to stable, but 
that would take years to implement distro wide.

Thanks,
Michael

[Pierre-Yves Chibon]

On Thu, Sep 26, 2019 at 11:24:28AM -0500, Michael Cronenworth wrote:
>  On 9/26/19 9:07 AM, Pierre-Yves Chibon wrote:
>  > What is so different in Fedora that we cannot move to this model?
>  > Is it a tooling issue?
>  > Is it something else?
>  
>  As others have already stated that may work in projects with tens, hundreds,
>  or thousands of contributors, but most of Fedora packages are owned and
>  maintained by a single person. I'm not going to wait around for anyone to
>  review my commit or comment on it. No one tests or comments on most of my
>  Bodhi updates.
>  
>  The same development model for upstream projects is not 1:1 to Fedora
>  packaging. Stop trying to force it. 
It may just be a wording issue, but I would really appreciate if you would not
attribute me thoughts I've never had.
I think I have been clear that this thread is about proposing something,
everything in it is up for debate and discussion.
If every discussion that is started is treated as something that is "forced"
then it's questioning (at least to me).

>  > > I don't want a build for every commit. Sometimes I
> cache changes for future
>  > > releases. A commit does*not*  always equal an update.
>  > There are a few ways to approach this:
>  > - don't bump the release in the spec file, the build will be triggered and will
>  >    just never complete
>  
>  It should be an opt-in feature. Who called for this feature? I don't
>  remember seeing anyone ask for it. 
If you want a name then I'll put mine, this is something that I've found would
be useful in more than one occasion.
Making it opt-in is definitely possible but the rest of your comment makes me
wonder if it's even wanted.

>  > - have a magic keyword in the commit message that prevents the
> build from
>  >    happening at all
>  No, thanks. Requires way too much human interaction (memory).
>  > - is an extra build in rawhide such a big deal?
>  It may fail to build because of a dependency on other features not yet in
>  place... so it *must* be opt-in.
>  > - gate at bodhi level builds that have the same rpm payload as the previous
>  >    build shipped
>  >    --> this is what OpenSUSE does btw, when you build an update, they rebuild
> the
>  >    entire dependency tree but will filter out the RPM whose payload haven't
>  >    changed from the update that is pushed to the users.
>  
>  If you want to spend time doing this go ahead, but it feels like a waste of
>  resources and talent. 
It's also what would provide one of the greatest return on investment. It should
basically solve all the broken-deps reports, the un-announced soname bumps and
so on.
 
>  > > This proposed workflow model doesn't help me. It hurts.
> It makes me think of
>  > > dropping my roles from Fedora.
>  > Seeing you leave is really not the idea and would be the opposite result.
>  > 
>  > The way I thought of this was:
>  > - is it easier to ask everyone for what their ideal workflow would be?
>  > or
>  > - present a workflow (which is hopefully not entirely insane) and encourage the
>  >    community to patch it so we end up with something that is consensual between
>  >    all of us?
>  > 
>  > I went with the later approach. Using the brainstorming session at flock, I'm
>  > proposing*a*  workflow, it's not perfect, it may not not ideal for everyone but
>  > it's a start and we can patch it as much as we want.
>  > We may end up with something entirely different from what I'm proposing and
>  > that's fine, but at least we'll have an agreed upon idea of where we want to
> go
>  > (ie: a long term vision)
>  > 
>  > 
>  > Does that make sense?
>  > I am sorry if the wording I've used didn't make this clear in my initial
> email
>  > :(
>  
>  If you want to have fedpkg do all of this for you buy default, sure, fine.
>  It should /also/ have a configuration able to be defined to disable all of
>  this automatic stuff. The reverse could also happen. Bake all this
>  functionality in and make it opt-in.
>  
>  Again, who was asking for all of this? 
Why do we need to have someone asking to propose something?
There are regularly people complaining on this very list about how hard
packaging has become. So here is a thread trying to see if you can come up with
a long term, ideal, vision of what the packager workflow should be so we can
work towards it.
I'm going to ask again what was in my original email: What is your ideal
workflow? How do you think things should work?
Is what we have today the ideal state of things?
If so, great!
If not, what can we improve and are there things we can easily change that will
make it easier for a majority of packagers?

>  The only interesting part is the automated testing and automated push
> to
>  stable, but that would take years to implement distro wide. 
The CI SIG folks are actively working towards this. It is a slow process but
hopefully we are improving things there.



Pierre

[Ben Rosser]

On Wed, Oct 2, 2019 at 1:59 PM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:
>  There are regularly people complaining on this very list about how
> hard
>  packaging has become. So here is a thread trying to see if you can come up with
>  a long term, ideal, vision of what the packager workflow should be so we can
>  work towards it. 
I'm such a person. I tried to put together an Objective on this topic
back in January before realizing I didn't have enough time to drive it
forwards due to real life commitments.

I may not have said it explicitly in my other replies on the thread,
but I _am_ glad to see people thinking seriously about ways to improve
the packager experience. So I appreciate your proposal, even if I
disagree with the proposed pull request workflow.

That being said...

>  I'm going to ask again what was in my original email: What is
> your ideal
>  workflow? How do you think things should work?
>  Is what we have today the ideal state of things?
>  If so, great!
>  If not, what can we improve and are there things we can easily change that will
>  make it easier for a majority of packagers? 
My feeling is that you've focused on the wrong part of the workflow.
My feeling is that the basic "commit, push, build, repeat" part of
packaging works reasonably well for most packages. Sure, it isn't
perfect, and it can be tedious to keep branches up to date across many
packages, and it'd be nice if there was more continuous integration
and running of a tests.

But as a packager, the things that frustrate _me_-- the things I was
proposing to help fix, before I realized tha are all the peripherals:
the bits of the infrastructure that don't feel like they interact as
well with the workflow as they could. At the moment, two of my biggest
complaints are:

1. Creating new packages has become (more of) a pain since the
retirement of pkgdb2. I know I keep complaining about needing to
manually fetch Pagure API keys, but it is actually extremely annoying
when I go to request a repo and realize I need to first request a new
API key before doing anything else. The problem isn't the workflow,
per se, but the infrastructure: reviews could really use a better
platform than bugzilla. If reviews were more integrated into the
tooling, automatic checks like fedora-review could maybe be ran
automatically. Maybe approving a new package could even automatically
create the repository, without the requestor having to do anything!

2. Release monitoring is a wonderful tool, but it's poorly integrated
with the rest of the project. As a packager maintaining probably more
packages than I should, getting release monitoring notifications to
tell me to pay attention to a particular package is incredibly useful.
But I feel like we could do more with the information. There are
nodejs packages out there, to take an ecosystem at random, that have
had open tickets created by release monitoring for four+ years, and
the only activity on those tickets is the release monitoring bot
detecting new versions. Eventually, maybe, a human comes across the
package and realizes it might be unmaintained, and proceeds with the
nonresponsive maintainer policy or manages to track down the
maintainer to find out why the package hasn't been updated. I don't
say this to criticize anyone in particular, but surely we could be
more proactive here?

Basically, I don't think we really need an entirely new packaging
workflow. (I would argue that attempts to impose an entirely new
packaging workflow-- like modularity-- are one of the reasons
packaging has gotten harder recently...). We need to improve the
contributor-facing _infrastructure_ to make the current workflow
better.

I would personally advocate starting with a serious look at the review
process, and the tooling around it. If for no other reason than it is
the first thing most new contributors will interact with, so perhaps
it is in our interest to make it as pleasant as possible.

Ben Rosser

[Ben Rosser]

On Wed, Oct 2, 2019 at 8:17 PM Ben Rosser <rosser.bjr(a)gmail.com&gt; wrote:
>  On Wed, Oct 2, 2019 at 1:59 PM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:
>  > There are regularly people complaining on this very list about how hard
>  > packaging has become. So here is a thread trying to see if you can come up with
>  > a long term, ideal, vision of what the packager workflow should be so we can
>  > work towards it.
>
>  I'm such a person. I tried to put together an Objective on this topic
>  back in January before realizing I didn't have enough time to drive it
>  forwards due to real life commitments.
>
>  I may not have said it explicitly in my other replies on the thread,
>  but I _am_ glad to see people thinking seriously about ways to improve
>  the packager experience. So I appreciate your proposal, even if I
>  disagree with the proposed pull request workflow.
>
>  That being said...
>
>  > I'm going to ask again what was in my original email: What is your ideal
>  > workflow? How do you think things should work?
>  > Is what we have today the ideal state of things?
>  > If so, great!
>  > If not, what can we improve and are there things we can easily change that will
>  > make it easier for a majority of packagers?
>
>  My feeling is that you've focused on the wrong part of the workflow.
>  My feeling is that the basic "commit, push, build, repeat" part of
>  packaging works reasonably well for most packages. Sure, it isn't
>  perfect, and it can be tedious to keep branches up to date across many
>  packages, and it'd be nice if there was more continuous integration
>  and running of a tests.
>
>  But as a packager, the things that frustrate _me_-- the things I was
>  proposing to help fix, before I realized tha are all the peripherals:
>  the bits of the infrastructure that don't feel like they interact as
>  well with the workflow as they could. At the moment, two of my biggest
>  complaints are: 
Whoops, I meant to write here:

"things I was proposing to help fix, before I realized that I didn't
have the time"

>
> 1. Creating new packages has become (more of) a pain since the
> retirement of pkgdb2. I know I keep complaining about needing to
> manually fetch Pagure API keys, but it is actually extremely annoying
> when I go to request a repo and realize I need to first request a new
> API key before doing anything else. The problem isn't the workflow,
> per se, but the infrastructure: reviews could really use a better
> platform than bugzilla. If reviews were more integrated into the
> tooling, automatic checks like fedora-review could maybe be ran
> automatically. Maybe approving a new package could even automatically
> create the repository, without the requestor having to do anything!
>
> 2. Release monitoring is a wonderful tool, but it's poorly integrated
> with the rest of the project. As a packager maintaining probably more
> packages than I should, getting release monitoring notifications to
> tell me to pay attention to a particular package is incredibly useful.
> But I feel like we could do more with the information. There are
> nodejs packages out there, to take an ecosystem at random, that have
> had open tickets created by release monitoring for four+ years, and
> the only activity on those tickets is the release monitoring bot
> detecting new versions. Eventually, maybe, a human comes across the
> package and realizes it might be unmaintained, and proceeds with the
> nonresponsive maintainer policy or manages to track down the
> maintainer to find out why the package hasn't been updated. I don't
> say this to criticize anyone in particular, but surely we could be
> more proactive here?
>
> Basically, I don't think we really need an entirely new packaging
> workflow. (I would argue that attempts to impose an entirely new
> packaging workflow-- like modularity-- are one of the reasons
> packaging has gotten harder recently...). We need to improve the
> contributor-facing _infrastructure_ to make the current workflow
> better.
>
> I would personally advocate starting with a serious look at the review
> process, and the tooling around it. If for no other reason than it is
> the first thing most new contributors will interact with, so perhaps
> it is in our interest to make it as pleasant as possible.
>
> Ben Rosser

[Zbigniew Jędrzejewski-Szmek]

On Wed, Oct 02, 2019 at 08:17:37PM +0200, Ben Rosser wrote:
>  > I'm going to ask again what was in my original email: What
> is your ideal
>  > workflow? How do you think things should work?
>  > Is what we have today the ideal state of things?
>  > If so, great!
>  > If not, what can we improve and are there things we can easily change that will
>  > make it easier for a majority of packagers?
>  
>  My feeling is that you've focused on the wrong part of the workflow.
>  My feeling is that the basic "commit, push, build, repeat" part of
>  packaging works reasonably well for most packages. Sure, it isn't
>  perfect, and it can be tedious to keep branches up to date across many
>  packages, and it'd be nice if there was more continuous integration
>  and running of a tests.
>
>  But as a packager, the things that frustrate _me_-- the things I was
>  proposing to help fix, before I realized tha are all the peripherals:
>  the bits of the infrastructure that don't feel like they interact as
>  well with the workflow as they could. At the moment, two of my biggest
>  complaints are: 
I very much agree with what Ben says. We need to improve the
integration between different components of the packaging infra.

>  1. Creating new packages has become (more of) a pain since the
>  retirement of pkgdb2. I know I keep complaining about needing to
>  manually fetch Pagure API keys, but it is actually extremely annoying
>  when I go to request a repo and realize I need to first request a new
>  API key before doing anything else. The problem isn't the workflow,
>  per se, but the infrastructure: reviews could really use a better
>  platform than bugzilla. If reviews were more integrated into the
>  tooling, automatic checks like fedora-review could maybe be ran
>  automatically. Maybe approving a new package could even automatically
>  create the repository, without the requestor having to do anything!
>  
>  2. Release monitoring is a wonderful tool, but it's poorly integrated
>  with the rest of the project. As a packager maintaining probably more
>  packages than I should, getting release monitoring notifications to
>  tell me to pay attention to a particular package is incredibly useful.
>  But I feel like we could do more with the information. There are
>  nodejs packages out there, to take an ecosystem at random, that have
>  had open tickets created by release monitoring for four+ years, and
>  the only activity on those tickets is the release monitoring bot
>  detecting new versions. Eventually, maybe, a human comes across the
>  package and realizes it might be unmaintained, and proceeds with the
>  nonresponsive maintainer policy or manages to track down the
>  maintainer to find out why the package hasn't been updated. I don't
>  say this to criticize anyone in particular, but surely we could be
>  more proactive here? 
Yep. Not every package is the same. For stuff like simple
python/nodejs/rust/ruby/perl/… packages, I know that the only thing
I do is mechanically bump the version and rebuild. I don't take a careful
look at the changes in the package, I don't do any non-automated checks;
if it builds, it gets shipped. For such packages, there really would be
no difference if a bot was doing all the steps I'm doing now.

But I don't think creating things from scratch is the answer. I think
we need to find ways to extend and integrate what we already have.
In particular, if we want to keep support for existing workflows
that people use, we cannot just add new services and deprecate
an old one. We need to grow new functionality in the existing ones.

Zbyszek

> Basically, I don't think we really need an entirely new packaging
> workflow. (I would argue that attempts to impose an entirely new
> packaging workflow-- like modularity-- are one of the reasons
> packaging has gotten harder recently...). We need to improve the
> contributor-facing _infrastructure_ to make the current workflow
> better.
> 
> I would personally advocate starting with a serious look at the review
> process, and the tooling around it. If for no other reason than it is
> the first thing most new contributors will interact with, so perhaps
> it is in our interest to make it as pleasant as possible.

[Pierre-Yves Chibon]

On Thu, Oct 03, 2019 at 08:07:33AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  Yep. Not every package is the same. For stuff like simple
>  python/nodejs/rust/ruby/perl/… packages, I know that the only thing
>  I do is mechanically bump the version and rebuild. I don't take a careful
>  look at the changes in the package, I don't do any non-automated checks;
>  if it builds, it gets shipped. For such packages, there really would be
>  no difference if a bot was doing all the steps I'm doing now. 
That's the thing which I think we're missing, how much of our packages are
simple: bump & rebuild? (40%, 50%, 75%?)
Automating the build from a commit and the update from a build was very much
something I thought would be useful for these packages.

The challenge is to find a way to improve this simple workflow without
penalizing the more complex ones.


Pierre

[Pierre-Yves Chibon]

On Wed, Oct 02, 2019 at 08:17:37PM +0200, Ben Rosser wrote:

Thanks for your words, I appreciate the support on the idea.

>  1. Creating new packages has become (more of) a pain since the
>  retirement of pkgdb2. I know I keep complaining about needing to
>  manually fetch Pagure API keys, but it is actually extremely annoying
>  when I go to request a repo and realize I need to first request a new
>  API key before doing anything else. The problem isn't the workflow,
>  per se, but the infrastructure: reviews could really use a better
>  platform than bugzilla. If reviews were more integrated into the
>  tooling, automatic checks like fedora-review could maybe be ran
>  automatically. Maybe approving a new package could even automatically
>  create the repository, without the requestor having to do anything! 
So I've been wondering a little bit how we could solve this and I've been
wondering if we couldn't leverage the PR workflow for this.
Imagine:
- You request a new repo to be created
- The new repo is automatically created from your request
- You fork it
- Push your spec file and patches to your fork
- Open a PR against the empty repo

The package review becomes then a basic PR. We could leverage the tools we are
working on for regular PRs.
If the PR is approved, you get access granted to it.
If the PR is denied, both repo are deleted.

>  2. Release monitoring is a wonderful tool, but it's poorly
> integrated
>  with the rest of the project. As a packager maintaining probably more
>  packages than I should, getting release monitoring notifications to
>  tell me to pay attention to a particular package is incredibly useful.
>  But I feel like we could do more with the information. There are
>  nodejs packages out there, to take an ecosystem at random, that have
>  had open tickets created by release monitoring for four+ years, and
>  the only activity on those tickets is the release monitoring bot
>  detecting new versions. Eventually, maybe, a human comes across the
>  package and realizes it might be unmaintained, and proceeds with the
>  nonresponsive maintainer policy or manages to track down the
>  maintainer to find out why the package hasn't been updated. I don't
>  say this to criticize anyone in particular, but surely we could be
>  more proactive here? 
So there are a few works in the pipes for this.
Dist-git in staging already offers a better integration between dist-git and
anitya, you can see it for example at:
https://src.stg.fedoraproject.org/rpms/guake on the left hand side column.
the-new-hotness is being adjusted so that it opens a pull-request rather than a
bugzilla ticket. This doesn't quite solve the issue of "there is no-one to
review the ticket/PR" though.
Packit ( https://github.com/packit-service/packit/ ) is also aimed at helping
with speeding up the flow from upstream to downstream.

>  I would personally advocate starting with a serious look at the
> review
>  process, and the tooling around it. If for no other reason than it is
>  the first thing most new contributors will interact with, so perhaps
>  it is in our interest to make it as pleasant as possible. 
Trying to improve the package review process is something that has been in my
radar for a while now but not high enough in the priority list.


Looking forward to hear your thoughts,
Pierre

[Fabio Valentini]

On Fri, Oct 4, 2019 at 5:54 PM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:
>  On Wed, Oct 02, 2019 at 08:17:37PM +0200, Ben Rosser wrote:
>
>  Thanks for your words, I appreciate the support on the idea.
>
>  > 1. Creating new packages has become (more of) a pain since the
>  > retirement of pkgdb2. I know I keep complaining about needing to
>  > manually fetch Pagure API keys, but it is actually extremely annoying
>  > when I go to request a repo and realize I need to first request a new
>  > API key before doing anything else. The problem isn't the workflow,
>  > per se, but the infrastructure: reviews could really use a better
>  > platform than bugzilla. If reviews were more integrated into the
>  > tooling, automatic checks like fedora-review could maybe be ran
>  > automatically. Maybe approving a new package could even automatically
>  > create the repository, without the requestor having to do anything!
>
>  So I've been wondering a little bit how we could solve this and I've been
>  wondering if we couldn't leverage the PR workflow for this.
>  Imagine:
>  - You request a new repo to be created
>  - The new repo is automatically created from your request
>  - You fork it
>  - Push your spec file and patches to your fork
>  - Open a PR against the empty repo
>
>  The package review becomes then a basic PR. We could leverage the tools we are
>  working on for regular PRs.
>  If the PR is approved, you get access granted to it.
>  If the PR is denied, both repo are deleted. 
This is an interesting idea. And it would definitely be better than
doing the bugzilla dance every time ...
It would be great if the initial commit could then be squashed when
it's finally approved and merged :)

>  > 2. Release monitoring is a wonderful tool, but it's poorly
> integrated
>  > with the rest of the project. As a packager maintaining probably more
>  > packages than I should, getting release monitoring notifications to
>  > tell me to pay attention to a particular package is incredibly useful.
>  > But I feel like we could do more with the information. There are
>  > nodejs packages out there, to take an ecosystem at random, that have
>  > had open tickets created by release monitoring for four+ years, and
>  > the only activity on those tickets is the release monitoring bot
>  > detecting new versions. Eventually, maybe, a human comes across the
>  > package and realizes it might be unmaintained, and proceeds with the
>  > nonresponsive maintainer policy or manages to track down the
>  > maintainer to find out why the package hasn't been updated. I don't
>  > say this to criticize anyone in particular, but surely we could be
>  > more proactive here?
>
>  So there are a few works in the pipes for this.
>  Dist-git in staging already offers a better integration between dist-git and
>  anitya, you can see it for example at:
>  https://src.stg.fedoraproject.org/rpms/guake on the left hand side column.
>  the-new-hotness is being adjusted so that it opens a pull-request rather than a
>  bugzilla ticket. This doesn't quite solve the issue of "there is no-one to
>  review the ticket/PR" though. 
I have never looked at the patches that the-new-hotness generates, but
automatically filed PRs would actually be helpful, and easily
actionable.
This sounds awesome, and would make simple "bump + build" updates
pretty much automatic and painless \o/

Thanks for working on all this,
Fabio

> Packit ( https://github.com/packit-service/packit/ ) is also aimed at helping
> with speeding up the flow from upstream to downstream.
>
> > I would personally advocate starting with a serious look at the review
> > process, and the tooling around it. If for no other reason than it is
> > the first thing most new contributors will interact with, so perhaps
> > it is in our interest to make it as pleasant as possible.
>
> Trying to improve the package review process is something that has been in my
> radar for a while now but not high enough in the priority list.
>
>
> Looking forward to hear your thoughts,
> Pierre
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Vít Ondruch]

Dne 04. 10. 19 v 18:10 Fabio Valentini napsal(a):
>  On Fri, Oct 4, 2019 at 5:54 PM Pierre-Yves Chibon
> <pingou(a)pingoured.fr&gt; wrote:
> > On Wed, Oct 02, 2019 at 08:17:37PM +0200, Ben Rosser wrote:
> >
> > Thanks for your words, I appreciate the support on the idea.
> >
> >> 1. Creating new packages has become (more of) a pain since the
> >> retirement of pkgdb2. I know I keep complaining about needing to
> >> manually fetch Pagure API keys, but it is actually extremely annoying
> >> when I go to request a repo and realize I need to first request a new
> >> API key before doing anything else. The problem isn't the workflow,
> >> per se, but the infrastructure: reviews could really use a better
> >> platform than bugzilla. If reviews were more integrated into the
> >> tooling, automatic checks like fedora-review could maybe be ran
> >> automatically. Maybe approving a new package could even automatically
> >> create the repository, without the requestor having to do anything!
> > So I've been wondering a little bit how we could solve this and I've been
> > wondering if we couldn't leverage the PR workflow for this.
> > Imagine:
> > - You request a new repo to be created
> > - The new repo is automatically created from your request
> > - You fork it
> > - Push your spec file and patches to your fork
> > - Open a PR against the empty repo
> >
> > The package review becomes then a basic PR. We could leverage the tools we are
> > working on for regular PRs.
> > If the PR is approved, you get access granted to it.
> > If the PR is denied, both repo are deleted.
>  This is an interesting idea. 

It is, but I am afraid that then we will have Foo, foo, FOO and fOo
repositories and we won't be able to get rid of them for similar reasons
we are not allowed to delete branches in current dist-git.

Also this ignores possible issues with patented or inappropriately
licensed code.


Vít



>  And it would definitely be better than
> doing the bugzilla dance every time ...
> It would be great if the initial commit could then be squashed when
> it's finally approved and merged :)
>
>>> 2. Release monitoring is a wonderful tool, but it's poorly integrated
>>> with the rest of the project. As a packager maintaining probably more
>>> packages than I should, getting release monitoring notifications to
>>> tell me to pay attention to a particular package is incredibly useful.
>>> But I feel like we could do more with the information. There are
>>> nodejs packages out there, to take an ecosystem at random, that have
>>> had open tickets created by release monitoring for four+ years, and
>>> the only activity on those tickets is the release monitoring bot
>>> detecting new versions. Eventually, maybe, a human comes across the
>>> package and realizes it might be unmaintained, and proceeds with the
>>> nonresponsive maintainer policy or manages to track down the
>>> maintainer to find out why the package hasn't been updated. I don't
>>> say this to criticize anyone in particular, but surely we could be
>>> more proactive here?
>> So there are a few works in the pipes for this.
>> Dist-git in staging already offers a better integration between dist-git and
>> anitya, you can see it for example at:
>> https://src.stg.fedoraproject.org/rpms/guake on the left hand side column.
>> the-new-hotness is being adjusted so that it opens a pull-request rather than a
>> bugzilla ticket. This doesn't quite solve the issue of "there is no-one
to
>> review the ticket/PR" though.
> I have never looked at the patches that the-new-hotness generates, but
> automatically filed PRs would actually be helpful, and easily
> actionable.
> This sounds awesome, and would make simple "bump + build" updates
> pretty much automatic and painless \o/
>
> Thanks for working on all this,
> Fabio
>
>> Packit ( https://github.com/packit-service/packit/ ) is also aimed at helping
>> with speeding up the flow from upstream to downstream.
>>
>>> I would personally advocate starting with a serious look at the review
>>> process, and the tooling around it. If for no other reason than it is
>>> the first thing most new contributors will interact with, so perhaps
>>> it is in our interest to make it as pleasant as possible.
>> Trying to improve the package review process is something that has been in my
>> radar for a while now but not high enough in the priority list.
>>
>>
>> Looking forward to hear your thoughts,
>> Pierre
>> _______________________________________________
>> devel mailing list -- devel(a)lists.fedoraproject.org
>> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Matthew Miller]

On Fri, Oct 04, 2019 at 06:57:55PM +0200, Vít Ondruch wrote:
>  >> The package review becomes then a basic PR. We could
> leverage the tools we are
>  >> working on for regular PRs.
>  >> If the PR is approved, you get access granted to it.
>  >> If the PR is denied, both repo are deleted.
>  > This is an interesting idea.
>  It is, but I am afraid that then we will have Foo, foo, FOO and fOo
>  repositories and we won't be able to get rid of them for similar reasons
>  we are not allowed to delete branches in current dist-git.
>   
Maybe we could have a namespace for unapproved ones, and move the repo if
approved?


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Pierre-Yves Chibon]

On Fri, Oct 04, 2019 at 06:57:55PM +0200, Vít Ondruch wrote:
>  
>  Dne 04. 10. 19 v 18:10 Fabio Valentini napsal(a):
>  > On Fri, Oct 4, 2019 at 5:54 PM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt;
> wrote:
>  >> On Wed, Oct 02, 2019 at 08:17:37PM +0200, Ben Rosser wrote:
>  >>
>  >> Thanks for your words, I appreciate the support on the idea.
>  >>
>  >>> 1. Creating new packages has become (more of) a pain since the
>  >>> retirement of pkgdb2. I know I keep complaining about needing to
>  >>> manually fetch Pagure API keys, but it is actually extremely annoying
>  >>> when I go to request a repo and realize I need to first request a new
>  >>> API key before doing anything else. The problem isn't the workflow,
>  >>> per se, but the infrastructure: reviews could really use a better
>  >>> platform than bugzilla. If reviews were more integrated into the
>  >>> tooling, automatic checks like fedora-review could maybe be ran
>  >>> automatically. Maybe approving a new package could even automatically
>  >>> create the repository, without the requestor having to do anything!
>  >> So I've been wondering a little bit how we could solve this and I've
> been
>  >> wondering if we couldn't leverage the PR workflow for this.
>  >> Imagine:
>  >> - You request a new repo to be created
>  >> - The new repo is automatically created from your request
>  >> - You fork it
>  >> - Push your spec file and patches to your fork
>  >> - Open a PR against the empty repo
>  >>
>  >> The package review becomes then a basic PR. We could leverage the tools we are
>  >> working on for regular PRs.
>  >> If the PR is approved, you get access granted to it.
>  >> If the PR is denied, both repo are deleted.
>  > This is an interesting idea.
>  
>  It is, but I am afraid that then we will have Foo, foo, FOO and fOo
>  repositories and we won't be able to get rid of them for similar reasons
>  we are not allowed to delete branches in current dist-git. 
You are allowed to force-push/delete branches in your fork.
So if the issue is found before the review is approved this should be fine, no?

>  Also this ignores possible issues with patented or inappropriately
>  licensed code. 
If the code is patented or with an inappropriate license, this
should be caught
by the review no?
If so, then once the fork and its destination project are deleted we should be
fine again, no?

Note: I'm not proposing that we allow/encourage uploading to the lookaside cache
with this workflow, this would indeed create more work for infra/releng to clean
things up if the PR is rejected (but I'd point out that nothing prevents early
uploads of sources to the lookaside already today).


Pierre

[Vít Ondruch]

Dne 07. 10. 19 v 12:00 Pierre-Yves Chibon napsal(a):
>  On Fri, Oct 04, 2019 at 06:57:55PM +0200, Vít Ondruch wrote:
> > Dne 04. 10. 19 v 18:10 Fabio Valentini napsal(a):
> >> On Fri, Oct 4, 2019 at 5:54 PM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt;
> wrote:
> >>> On Wed, Oct 02, 2019 at 08:17:37PM +0200, Ben Rosser wrote:
> >>>
> >>> Thanks for your words, I appreciate the support on the idea.
> >>>
> >>>> 1. Creating new packages has become (more of) a pain since the
> >>>> retirement of pkgdb2. I know I keep complaining about needing to
> >>>> manually fetch Pagure API keys, but it is actually extremely annoying
> >>>> when I go to request a repo and realize I need to first request a new
> >>>> API key before doing anything else. The problem isn't the workflow,
> >>>> per se, but the infrastructure: reviews could really use a better
> >>>> platform than bugzilla. If reviews were more integrated into the
> >>>> tooling, automatic checks like fedora-review could maybe be ran
> >>>> automatically. Maybe approving a new package could even automatically
> >>>> create the repository, without the requestor having to do anything!
> >>> So I've been wondering a little bit how we could solve this and I've
> been
> >>> wondering if we couldn't leverage the PR workflow for this.
> >>> Imagine:
> >>> - You request a new repo to be created
> >>> - The new repo is automatically created from your request
> >>> - You fork it
> >>> - Push your spec file and patches to your fork
> >>> - Open a PR against the empty repo
> >>>
> >>> The package review becomes then a basic PR. We could leverage the tools we
> are
> >>> working on for regular PRs.
> >>> If the PR is approved, you get access granted to it.
> >>> If the PR is denied, both repo are deleted.
> >> This is an interesting idea.
> > It is, but I am afraid that then we will have Foo, foo, FOO and fOo
> > repositories and we won't be able to get rid of them for similar reasons
> > we are not allowed to delete branches in current dist-git.
>  You are allowed to force-push/delete branches in your fork. 

To be able to have the fork, you have to have the repository to fork
from. So I can imagine there will be process like "open ticket for
releng and fill this template". But I can't imagine it will be reviewed
by relengs, when they currently don't review the repo names and
therefore we have issues like:


https://pagure.io/releng/issue/7523


>  So if the issue is found before the review is approved this should be
> fine, no?
>
> > Also this ignores possible issues with patented or inappropriately
> > licensed code.
>  If the code is patented or with an inappropriate license, this should be caught
>  by the review no?
>  If so, then once the fork and its destination project are deleted we should be
>  fine again, no?
>
>  Note: I'm not proposing that we allow/encourage uploading to the lookaside cache
>  with this workflow, 

But where will you get the sources to build the package then? Yes, for
simple cases, you can use the source URL to fetch the tarball, but I
assume that this will work approx just for 50 % packages.


Vít


>  this would indeed create more work for infra/releng to clean
>  things up if the PR is rejected (but I'd point out that nothing prevents early
>  uploads of sources to the lookaside already today).
>
>
>  Pierre

[Pierre-Yves Chibon]

On Mon, Oct 07, 2019 at 12:54:56PM +0200, Vít Ondruch wrote:
>  
>  Dne 07. 10. 19 v 12:00 Pierre-Yves Chibon napsal(a):
>  > On Fri, Oct 04, 2019 at 06:57:55PM +0200, Vít Ondruch wrote:
>  >> Dne 04. 10. 19 v 18:10 Fabio Valentini napsal(a):
>  >>> On Fri, Oct 4, 2019 at 5:54 PM Pierre-Yves Chibon
> <pingou(a)pingoured.fr&gt; wrote:
>  >>>> On Wed, Oct 02, 2019 at 08:17:37PM +0200, Ben Rosser wrote:
>  >>>>
>  >>>> Thanks for your words, I appreciate the support on the idea.
>  >>>>
>  >>>>> 1. Creating new packages has become (more of) a pain since the
>  >>>>> retirement of pkgdb2. I know I keep complaining about needing to
>  >>>>> manually fetch Pagure API keys, but it is actually extremely
> annoying
>  >>>>> when I go to request a repo and realize I need to first request a
> new
>  >>>>> API key before doing anything else. The problem isn't the
> workflow,
>  >>>>> per se, but the infrastructure: reviews could really use a better
>  >>>>> platform than bugzilla. If reviews were more integrated into the
>  >>>>> tooling, automatic checks like fedora-review could maybe be ran
>  >>>>> automatically. Maybe approving a new package could even
> automatically
>  >>>>> create the repository, without the requestor having to do anything!
>  >>>> So I've been wondering a little bit how we could solve this and
> I've been
>  >>>> wondering if we couldn't leverage the PR workflow for this.
>  >>>> Imagine:
>  >>>> - You request a new repo to be created
>  >>>> - The new repo is automatically created from your request
>  >>>> - You fork it
>  >>>> - Push your spec file and patches to your fork
>  >>>> - Open a PR against the empty repo
>  >>>>
>  >>>> The package review becomes then a basic PR. We could leverage the tools
> we are
>  >>>> working on for regular PRs.
>  >>>> If the PR is approved, you get access granted to it.
>  >>>> If the PR is denied, both repo are deleted.
>  >>> This is an interesting idea.
>  >> It is, but I am afraid that then we will have Foo, foo, FOO and fOo
>  >> repositories and we won't be able to get rid of them for similar reasons
>  >> we are not allowed to delete branches in current dist-git.
>  > You are allowed to force-push/delete branches in your fork.
>  
>  
>  To be able to have the fork, you have to have the repository to fork
>  from. So I can imagine there will be process like "open ticket for
>  releng and fill this template". But I can't imagine it will be reviewed
>  by relengs, when they currently don't review the repo names and
>  therefore we have issues like:
>  
>  https://pagure.io/releng/issue/7523 
Hm, that sounds plausible/likely but then it means we're not making it worst,
just as bad.
Any idea how we could improve on this?

>  > So if the issue is found before the review is approved this
> should be fine, no?
>  >
>  >> Also this ignores possible issues with patented or inappropriately
>  >> licensed code.
>  > If the code is patented or with an inappropriate license, this should be caught
>  > by the review no?
>  > If so, then once the fork and its destination project are deleted we should be
>  > fine again, no?
>  >
>  > Note: I'm not proposing that we allow/encourage uploading to the lookaside
> cache
>  > with this workflow,
>  
>  
>  But where will you get the sources to build the package then? Yes, for
>  simple cases, you can use the source URL to fetch the tarball, but I
>  assume that this will work approx just for 50 % packages. 
We could include it in the PR description as we're doing with bugzilla tickets
now.
And fetching from the URL automatically and running some tests for it would
still give us a 50% improvement then.


Pierre

[Ben Rosser]

On Fri, Oct 4, 2019 at 5:54 PM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:
>  So I've been wondering a little bit how we could solve this and
> I've been
>  wondering if we couldn't leverage the PR workflow for this.
>  Imagine:
>  - You request a new repo to be created
>  - The new repo is automatically created from your request
>  - You fork it
>  - Push your spec file and patches to your fork
>  - Open a PR against the empty repo
>
>  The package review becomes then a basic PR. We could leverage the tools we are
>  working on for regular PRs.
>  If the PR is approved, you get access granted to it.
>  If the PR is denied, both repo are deleted. 
This is a really interesting idea. I think having to version control
spec files while they're going through review would probably help
things a lot, and it definitely seems like the pull request workflow
is a much more natural fit here.

I'm a little uncomfortable with the "both repos are deleted" part of
this though. One nice thing we have right now is that there's an
archive of failed and/or dead reviews. In principle I could go through
FE-DEADREVIEW and pick up tickets that got closed because a submitter
vanished. Or search for a particular package to see if anyone had
tried to submit it before, find out what went wrong, take a look at
their spec, etc.  So I think it would be important to preserve some
record of failed reviews. As proposed here, it sounds like the pull
request just gets deleted.

But this is definitely a cool starting point for a better review process.

>  So there are a few works in the pipes for this.
>  Dist-git in staging already offers a better integration between dist-git and
>  anitya, you can see it for example at:
>  https://src.stg.fedoraproject.org/rpms/guake on the left hand side column. 
Yes, I have seen this-- glad to see it has been worked on.

>  the-new-hotness is being adjusted so that it opens a pull-request
> rather than a
>  bugzilla ticket. This doesn't quite solve the issue of "there is no-one to
>  review the ticket/PR" though. 
Having pull requests for this would be really nice. Like Fabio, I also
never look at the patches generated by release monitoring, but if they
were pull requests I'd definitely consider using them.

As for not having someone to review the pull request... I think this
is an area where we just need more monitoring, not necessarily
automation. It would be wonderful if there was a page somewhere where
I could see a list of all packages release monitoring thinks are
outdated. Or (and maybe this is easier) if the "monitoring status" on
the sidebar in Pagure showed that information too. If nothing else, it
would save me a click or two when trying to check if a package is up
to date.

>  Trying to improve the package review process is something that has
> been in my
>  radar for a while now but not high enough in the priority list. 
Sure, I understand. I appreciate that you're taking the time to
think/work on this stuff!

Ben Rosser

[Richard Shaw]

On Wed, Oct 2, 2019 at 1:18 PM Ben Rosser <rosser.bjr(a)gmail.com&gt; wrote:

>  1. Creating new packages has become (more of) a pain since the
>  retirement of pkgdb2. I know I keep complaining about needing to
>  manually fetch Pagure API keys, but it is actually extremely annoying
>  when I go to request a repo and realize I need to first request a new
>  API key before doing anything else. The problem isn't the workflow,
>  per se, but the infrastructure: reviews could really use a better
>  platform than bugzilla. If reviews were more integrated into the
>  tooling, automatic checks like fedora-review could maybe be ran
>  automatically. Maybe approving a new package could even automatically
>  create the repository, without the requestor having to do anything!
>  
This is a big one for me and I think part of the problem is we have both
src.fedoraproject.org (a pagure instance) and pagure.io, which also is
specifically for fedora but not in the fedoraproject.org domain. Then I get
emails telling me my API key is about to expire and there's no links i the
notification and I'm not entirely sure which pagure instance it is.

THEN once I create new keys, because it's often enough to be annoying but
not often enough to remember, I don't know which file to put it in...

.config/fedrepo_req/config.ini ?
or
.config/rpkg/fedpkg.conf ?

And google was almost useless this last time, "man fedpkg" was no help
either I had to run:

$ fedpkg request-repo --help

To get the answer...

Thanks,
Richard

[Per Bothner]

* Unclear work-
On 9/26/19 7:07 AM, Pierre-Yves Chibon wrote:

>  When we work on upstream projects, I think it's pretty standard
> now to always go
>  via PRs, even for your own branch.
>  So that tests are run, so that other member of the community can see, comment,
>  review the change.
>  What is so different in Fedora that we cannot move to this model?
>  Is it a tooling issue?
>  Is it something else? 
I find the pull request work-flow rather awkward.  Problems that come to mind.

* Patch creators have to create a private (temporary) fork and branch.
That seems like wasted effort.
* Pull requests don't help maintainers all that much with review and testing.
(I don't mean the commenting facilities, which are useful - I mean
testing and evaluating a pull request is no easier than a plain patch.)
* Poor integration between pull requests and issues.  I think it would be
better for a "pull request" to be a kind of issue that has an associated patch.
That is what one did in the pre-GitHub world.  All that is missing is a
simple "accept patch" button added to a patch associated with an issue,
which a maintainer can click when satisfied.
* Related to the former: Unclear work-flow: Are you supposed to create an
issue before you create a pull request?  That is neither enforced or encouraged
by GitHub or GitLab.

Pull requests make sense for merging forks.  I'm not sure they make sense
for smaller changes.
-- 
	--Per Bothner
per(a)bothner.com   http://per.bothner.com/

[Emmanuel Seyman]

* Pierre-Yves Chibon [26/09/2019 16:07] :
>  When we work on upstream projects, I think it's pretty standard now to always go
>  via PRs, even for your own branch. 
FWIW, this workflow makes sense for one that has a community but it sounds very
strange to do this on a project on which you are the only developer.

>  Is it a tooling issue? 
The biggest issue I have with this workflow is that I don't know of any means
through which I can merge a PR using only git and/or fedpkg.

Emmanuel

[Robert-André Mauchin]

On Thursday, 26 September 2019 15:01:25 CEST Remi Collet wrote:
>  Le 26/09/2019 à 11:36, Pierre-Yves Chibon a écrit :
>  
>  > Here is what the vision we came to and that we would like to discuss:
>  > 
>  > ○ Every changes to dist-git is done via pull-requests
>  
>  
>  IMHO Have to stay optional, making this mandatory being a terrible
>  headache. 
 
>  
>  
>  
>  
>  RFemi
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List
>  Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List
>  Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail] 
Indeed. With hundreds of packaqes to maintain, I don't see myself to create a 
PR for each change I have to make. This would make packaging even more 
complicated than it is today.

[Miroslav Suchý]

Dne 26. 09. 19 v 11:36 Pierre-Yves Chibon napsal(a):
>  - We need to work on the change logs in the spec files, as otherwise
>     pull-requests are going to conflict more often than not 
+1

Thou, I would love to have general solution. Not just Fedora-centric. Something which will
respect use-case of layered 
projects and maintainers outside of Fedora.

-- 
Miroslav Suchy, RHCA
Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys

[Daniel P. Berrangé]

On Thu, Sep 26, 2019 at 11:36:10AM +0200, Pierre-Yves Chibon wrote:
>  Good Morning Everyone,
>  
>  At Flock, a few of us met to discuss a future vision of the packager workflow.
>  This discussion was triggered by the realization that a number of initiatives
>  are happening around packaging in Fedora but there is no real shared vision on
>  what we want the packager UX/workflow to be.
>  The lack of vision on the packager workflow means we could deploy something
>  today, thinking it is the improvement over the current workflow but would
>  prevent us from (or make it harder to) doing other changes afterwords that would
>  be even more beneficial..
>  
>  So once that concern was raised, we took some time during the Fedora
>  Infrastructure hackfest to gather the people interested around a white board and
>  brainstorm on what a future packager workflow could look like.
>  
>  We tried not to link this process to any tool in particular as well as focus on
>  the what and why rather than any how.
>  
>  Here is what the vision we came to and that we would like to discuss:
>  
>  ○ Every changes to dist-git is done via pull-requests
>  ○ Pull-requests are automatically tested
>  ○ Every commit to dist-git (ie: PR merged) is automatically built in koji
>  ○ Every build in koji results automatically in an update in bodhi
>  ○ Every update in bodhi is automatically tested
>  ○ If the tests pass, the update is automatically pushed to the repository 
For packages I maintain, my preference is to touch dist-git as little
as possible. Ideally I would never touch dist-git at all & rather wish
it didn't exist at all in its current form of spec+patchfiles.

Instead I prefer a clone of the master upstream git repo and maintain a
branch with patches cherry-picked into it. This is used to auto-generate
patches for the Fedora RPM, at the same time updating the patch file list
in the RPM spec. The only manual step is adding the changelog entry &
bumping release number.

The ideas above around associating CI with pull requests make a lot of
conceptual sense & align with modern github/gitlab software development
best practices followed by non-distro software upstreams. Automatically
triggering builds from merged PRs similarly makes sense, especially
if that can automate bumping spec release number & adding a changelog
entry based on the pull request subject.


Obviously I can still use my upstream git repo branch and change the
scripts to generate a pull request to dist-git instead.  The downside
is if the PR fails, I now how to rebase my real git repo & re-submit
and new PR.

So if we're talking "ideal" long term vision though, I'd rather
eliminate
the dist-git repo intermediate step as IMHO it adds no value, only complexity
for the sake of historical compat with the way Red Hat distros has worked
dating back years. Its time to say goodbye to this historical way as the
world has moved on since this spec+patches approach was invented in the
days of CVS source control.

Allow packagers to have a clone of the upstraem git repo, and use the
pull-requests and run Fedora CI testing against the Fedora branch of
the upstream repo instead of against dist-git.

In this way, maintaining packages in Fedora would be functionally identical
to how an upstream project maintains their own stable branches. The only
Fedora difference would be that the branch would need to include the RPM
spec file in some well defined place.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Pierre-Yves Chibon]

On Thu, Sep 26, 2019 at 02:57:45PM +0100, Daniel P. Berrangé wrote:
>  On Thu, Sep 26, 2019 at 11:36:10AM +0200, Pierre-Yves Chibon wrote:
>  > Good Morning Everyone,
>  > 
>  > At Flock, a few of us met to discuss a future vision of the packager workflow.
>  > This discussion was triggered by the realization that a number of initiatives
>  > are happening around packaging in Fedora but there is no real shared vision on
>  > what we want the packager UX/workflow to be.
>  > The lack of vision on the packager workflow means we could deploy something
>  > today, thinking it is the improvement over the current workflow but would
>  > prevent us from (or make it harder to) doing other changes afterwords that would
>  > be even more beneficial..
>  > 
>  > So once that concern was raised, we took some time during the Fedora
>  > Infrastructure hackfest to gather the people interested around a white board and
>  > brainstorm on what a future packager workflow could look like.
>  > 
>  > We tried not to link this process to any tool in particular as well as focus on
>  > the what and why rather than any how.
>  > 
>  > Here is what the vision we came to and that we would like to discuss:
>  > 
>  > ○ Every changes to dist-git is done via pull-requests
>  > ○ Pull-requests are automatically tested
>  > ○ Every commit to dist-git (ie: PR merged) is automatically built in koji
>  > ○ Every build in koji results automatically in an update in bodhi
>  > ○ Every update in bodhi is automatically tested
>  > ○ If the tests pass, the update is automatically pushed to the repository
>  
>  For packages I maintain, my preference is to touch dist-git as little
>  as possible. Ideally I would never touch dist-git at all & rather wish
>  it didn't exist at all in its current form of spec+patchfiles.
>  
>  Instead I prefer a clone of the master upstream git repo and maintain a
>  branch with patches cherry-picked into it. This is used to auto-generate
>  patches for the Fedora RPM, at the same time updating the patch file list
>  in the RPM spec. The only manual step is adding the changelog entry &
>  bumping release number.
>  
>  The ideas above around associating CI with pull requests make a lot of
>  conceptual sense & align with modern github/gitlab software development
>  best practices followed by non-distro software upstreams. Automatically
>  triggering builds from merged PRs similarly makes sense, especially
>  if that can automate bumping spec release number & adding a changelog
>  entry based on the pull request subject.
>  
>  
>  Obviously I can still use my upstream git repo branch and change the
>  scripts to generate a pull request to dist-git instead.  The downside
>  is if the PR fails, I now how to rebase my real git repo & re-submit
>  and new PR.
>  
>  So if we're talking "ideal" long term vision though, I'd rather
> eliminate
>  the dist-git repo intermediate step as IMHO it adds no value, only complexity
>  for the sake of historical compat with the way Red Hat distros has worked
>  dating back years. Its time to say goodbye to this historical way as the
>  world has moved on since this spec+patches approach was invented in the
>  days of CVS source control.
>  
>  Allow packagers to have a clone of the upstraem git repo, and use the
>  pull-requests and run Fedora CI testing against the Fedora branch of
>  the upstream repo instead of against dist-git.
>  
>  In this way, maintaining packages in Fedora would be functionally identical
>  to how an upstream project maintains their own stable branches. The only
>  Fedora difference would be that the branch would need to include the RPM
>  spec file in some well defined place. 
I like this.
It means that the build-system would have to generate the tarball of the source
and put it into dist-git at srpm-build time (I believe we still want to store a
copy of the sources used for a build).

Three questions though:
- What about the upstream projects that still only publish a tarball at release?
  Should the packager explode that tarball into a git repo?
  Do we keep the current way of uploading the tarball?
- Won't this make pull-request for new update harder? Suddenly you have the diff
  of both all of the upstream changes as well as the spec file and potential
  patches.
  In most cases the spec file and the patches is what will matter but for large
  code-base the diff could be quite significant then.
- Do you have in mind that the copy/fork of upstream would be in our dist-git or
  in other places on the internet? If the later, how would you recommend
  contributors to find it?


Thanks for your inputs!

Pierre

[Daniel P. Berrangé]

On Thu, Sep 26, 2019 at 04:24:29PM +0200, Pierre-Yves Chibon wrote:
>  On Thu, Sep 26, 2019 at 02:57:45PM +0100, Daniel P. Berrangé wrote:
>  > On Thu, Sep 26, 2019 at 11:36:10AM +0200, Pierre-Yves Chibon wrote:
>  > > Good Morning Everyone,
>  > > 
>  > > At Flock, a few of us met to discuss a future vision of the packager workflow.
>  > > This discussion was triggered by the realization that a number of initiatives
>  > > are happening around packaging in Fedora but there is no real shared vision on
>  > > what we want the packager UX/workflow to be.
>  > > The lack of vision on the packager workflow means we could deploy something
>  > > today, thinking it is the improvement over the current workflow but would
>  > > prevent us from (or make it harder to) doing other changes afterwords that
> would
>  > > be even more beneficial..
>  > > 
>  > > So once that concern was raised, we took some time during the Fedora
>  > > Infrastructure hackfest to gather the people interested around a white board
> and
>  > > brainstorm on what a future packager workflow could look like.
>  > > 
>  > > We tried not to link this process to any tool in particular as well as focus
> on
>  > > the what and why rather than any how.
>  > > 
>  > > Here is what the vision we came to and that we would like to discuss:
>  > > 
>  > > ○ Every changes to dist-git is done via pull-requests
>  > > ○ Pull-requests are automatically tested
>  > > ○ Every commit to dist-git (ie: PR merged) is automatically built in koji
>  > > ○ Every build in koji results automatically in an update in bodhi
>  > > ○ Every update in bodhi is automatically tested
>  > > ○ If the tests pass, the update is automatically pushed to the repository
>  > 
>  > For packages I maintain, my preference is to touch dist-git as little
>  > as possible. Ideally I would never touch dist-git at all & rather wish
>  > it didn't exist at all in its current form of spec+patchfiles.
>  > 
>  > Instead I prefer a clone of the master upstream git repo and maintain a
>  > branch with patches cherry-picked into it. This is used to auto-generate
>  > patches for the Fedora RPM, at the same time updating the patch file list
>  > in the RPM spec. The only manual step is adding the changelog entry &
>  > bumping release number.
>  > 
>  > The ideas above around associating CI with pull requests make a lot of
>  > conceptual sense & align with modern github/gitlab software development
>  > best practices followed by non-distro software upstreams. Automatically
>  > triggering builds from merged PRs similarly makes sense, especially
>  > if that can automate bumping spec release number & adding a changelog
>  > entry based on the pull request subject.
>  > 
>  > 
>  > Obviously I can still use my upstream git repo branch and change the
>  > scripts to generate a pull request to dist-git instead.  The downside
>  > is if the PR fails, I now how to rebase my real git repo & re-submit
>  > and new PR.
>  > 
>  > So if we're talking "ideal" long term vision though, I'd rather
> eliminate
>  > the dist-git repo intermediate step as IMHO it adds no value, only complexity
>  > for the sake of historical compat with the way Red Hat distros has worked
>  > dating back years. Its time to say goodbye to this historical way as the
>  > world has moved on since this spec+patches approach was invented in the
>  > days of CVS source control.
>  > 
>  > Allow packagers to have a clone of the upstraem git repo, and use the
>  > pull-requests and run Fedora CI testing against the Fedora branch of
>  > the upstream repo instead of against dist-git.
>  > 
>  > In this way, maintaining packages in Fedora would be functionally identical
>  > to how an upstream project maintains their own stable branches. The only
>  > Fedora difference would be that the branch would need to include the RPM
>  > spec file in some well defined place.
>  
>  I like this.
>  It means that the build-system would have to generate the tarball of the source
>  and put it into dist-git at srpm-build time (I believe we still want to store a
>  copy of the sources used for a build). 
It doesn't have to generate a tarball. There's still the option of using the
lookaside cache to acquire the tarball as today. I don't mind the lookaside
cache since its rarely touched, doesn't cause inconvenience in the same way
patches in dist-git do.

>  Three questions though:
>  - What about the upstream projects that still only publish a tarball at release?
>    Should the packager explode that tarball into a git repo?
>    Do we keep the current way of uploading the tarball? 
I guess I answered that already.

>  - Won't this make pull-request for new update harder? Suddenly
> you have the diff
>    of both all of the upstream changes as well as the spec file and potential
>    patches.
>    In most cases the spec file and the patches is what will matter but for large
>    code-base the diff could be quite significant then. 
By "new update", I presume you mean rebasing to new upstream version.
Currently with my separate git repo/branch, I would just force push
the branch overwriting old history, but obviously that's no desirable
for Fedora's historical record.

It would make more sense to create a new branch when switching to
the new upstream release, as again this is what upstream would do
when maintaining stable branches for multiple releases. 

>  - Do you have in mind that the copy/fork of upstream would be in our
> dist-git or
>    in other places on the internet? If the later, how would you recommend
>    contributors to find it? 
Ideally I think Fedora would have a git server than host the Fedora
branches, as it is nice to have a single point of reference for all
Fedora additions. It could even be the current dist-git server, just
with the upstream repo mirrored into it, instead of our current patches+
spec approach.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Pierre-Yves Chibon]

On Thu, Sep 26, 2019 at 03:33:11PM +0100, Daniel P. Berrangé wrote:
>  On Thu, Sep 26, 2019 at 04:24:29PM +0200, Pierre-Yves Chibon wrote:
>  > On Thu, Sep 26, 2019 at 02:57:45PM +0100, Daniel P. Berrangé wrote:
>  > > On Thu, Sep 26, 2019 at 11:36:10AM +0200, Pierre-Yves Chibon wrote:
>  > > > Good Morning Everyone,
>  > > > 
>  > > > At Flock, a few of us met to discuss a future vision of the packager
> workflow.
>  > > > This discussion was triggered by the realization that a number of
> initiatives
>  > > > are happening around packaging in Fedora but there is no real shared
> vision on
>  > > > what we want the packager UX/workflow to be.
>  > > > The lack of vision on the packager workflow means we could deploy
> something
>  > > > today, thinking it is the improvement over the current workflow but would
>  > > > prevent us from (or make it harder to) doing other changes afterwords that
> would
>  > > > be even more beneficial..
>  > > > 
>  > > > So once that concern was raised, we took some time during the Fedora
>  > > > Infrastructure hackfest to gather the people interested around a white
> board and
>  > > > brainstorm on what a future packager workflow could look like.
>  > > > 
>  > > > We tried not to link this process to any tool in particular as well as
> focus on
>  > > > the what and why rather than any how.
>  > > > 
>  > > > Here is what the vision we came to and that we would like to discuss:
>  > > > 
>  > > > ○ Every changes to dist-git is done via pull-requests
>  > > > ○ Pull-requests are automatically tested
>  > > > ○ Every commit to dist-git (ie: PR merged) is automatically built in koji
>  > > > ○ Every build in koji results automatically in an update in bodhi
>  > > > ○ Every update in bodhi is automatically tested
>  > > > ○ If the tests pass, the update is automatically pushed to the repository
>  > > 
>  > > For packages I maintain, my preference is to touch dist-git as little
>  > > as possible. Ideally I would never touch dist-git at all & rather wish
>  > > it didn't exist at all in its current form of spec+patchfiles.
>  > > 
>  > > Instead I prefer a clone of the master upstream git repo and maintain a
>  > > branch with patches cherry-picked into it. This is used to auto-generate
>  > > patches for the Fedora RPM, at the same time updating the patch file list
>  > > in the RPM spec. The only manual step is adding the changelog entry &
>  > > bumping release number.
>  > > 
>  > > The ideas above around associating CI with pull requests make a lot of
>  > > conceptual sense & align with modern github/gitlab software development
>  > > best practices followed by non-distro software upstreams. Automatically
>  > > triggering builds from merged PRs similarly makes sense, especially
>  > > if that can automate bumping spec release number & adding a changelog
>  > > entry based on the pull request subject.
>  > > 
>  > > 
>  > > Obviously I can still use my upstream git repo branch and change the
>  > > scripts to generate a pull request to dist-git instead.  The downside
>  > > is if the PR fails, I now how to rebase my real git repo & re-submit
>  > > and new PR.
>  > > 
>  > > So if we're talking "ideal" long term vision though, I'd
> rather eliminate
>  > > the dist-git repo intermediate step as IMHO it adds no value, only complexity
>  > > for the sake of historical compat with the way Red Hat distros has worked
>  > > dating back years. Its time to say goodbye to this historical way as the
>  > > world has moved on since this spec+patches approach was invented in the
>  > > days of CVS source control.
>  > > 
>  > > Allow packagers to have a clone of the upstraem git repo, and use the
>  > > pull-requests and run Fedora CI testing against the Fedora branch of
>  > > the upstream repo instead of against dist-git.
>  > > 
>  > > In this way, maintaining packages in Fedora would be functionally identical
>  > > to how an upstream project maintains their own stable branches. The only
>  > > Fedora difference would be that the branch would need to include the RPM
>  > > spec file in some well defined place.
>  > 
>  > I like this.
>  > It means that the build-system would have to generate the tarball of the source
>  > and put it into dist-git at srpm-build time (I believe we still want to store a
>  > copy of the sources used for a build).
>  
>  It doesn't have to generate a tarball. There's still the option of using the
>  lookaside cache to acquire the tarball as today. I don't mind the lookaside
>  cache since its rarely touched, doesn't cause inconvenience in the same way
>  patches in dist-git do. 
I had in mind generating the tarball as a way to make sure the sources don't
disappear and we are able to reproduce the builds

>  > - Do you have in mind that the copy/fork of upstream would be in
> our dist-git or
>  >   in other places on the internet? If the later, how would you recommend
>  >   contributors to find it?
>  
>  Ideally I think Fedora would have a git server than host the Fedora
>  branches, as it is nice to have a single point of reference for all
>  Fedora additions. It could even be the current dist-git server, just
>  with the upstream repo mirrored into it, instead of our current patches+
>  spec approach. 
If the mirrored repo are somewhere that is somewhat under Fedora's control it
solves the tarball question from above since in this case we can make sure that
the git repo or a commit in it doesn't disappear suddenly solving the
reproducibility question.


Pierre

[Iñaki Ucar]

On Thu, 26 Sep 2019 at 16:46, Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:
>  On Thu, Sep 26, 2019 at 02:57:45PM +0100, Daniel P. Berrangé wrote:
>  > Allow packagers to have a clone of the upstraem git repo
>
>  - What about the upstream projects that still only publish a tarball at release? 
What about upstream projects that use other version control systems?

Iñaki

[Martin Kolman]

On Thu, 2019-09-26 at 16:24 +0200, Pierre-Yves Chibon wrote:
>  On Thu, Sep 26, 2019 at 02:57:45PM +0100, Daniel P. Berrangé wrote:
>  > On Thu, Sep 26, 2019 at 11:36:10AM +0200, Pierre-Yves Chibon wrote:
>  > > Good Morning Everyone,
>  > > 
>  > > At Flock, a few of us met to discuss a future vision of the packager workflow.
>  > > This discussion was triggered by the realization that a number of initiatives
>  > > are happening around packaging in Fedora but there is no real shared vision on
>  > > what we want the packager UX/workflow to be.
>  > > The lack of vision on the packager workflow means we could deploy something
>  > > today, thinking it is the improvement over the current workflow but would
>  > > prevent us from (or make it harder to) doing other changes afterwords that
> would
>  > > be even more beneficial..
>  > > 
>  > > So once that concern was raised, we took some time during the Fedora
>  > > Infrastructure hackfest to gather the people interested around a white board
> and
>  > > brainstorm on what a future packager workflow could look like.
>  > > 
>  > > We tried not to link this process to any tool in particular as well as focus
> on
>  > > the what and why rather than any how.
>  > > 
>  > > Here is what the vision we came to and that we would like to discuss:
>  > > 
>  > > ○ Every changes to dist-git is done via pull-requests
>  > > ○ Pull-requests are automatically tested
>  > > ○ Every commit to dist-git (ie: PR merged) is automatically built in koji
>  > > ○ Every build in koji results automatically in an update in bodhi
>  > > ○ Every update in bodhi is automatically tested
>  > > ○ If the tests pass, the update is automatically pushed to the repository
>  > 
>  > For packages I maintain, my preference is to touch dist-git as little
>  > as possible. Ideally I would never touch dist-git at all & rather wish
>  > it didn't exist at all in its current form of spec+patchfiles.
>  > 
>  > Instead I prefer a clone of the master upstream git repo and maintain a
>  > branch with patches cherry-picked into it. This is used to auto-generate
>  > patches for the Fedora RPM, at the same time updating the patch file list
>  > in the RPM spec. The only manual step is adding the changelog entry &
>  > bumping release number.
>  > 
>  > The ideas above around associating CI with pull requests make a lot of
>  > conceptual sense & align with modern github/gitlab software development
>  > best practices followed by non-distro software upstreams. Automatically
>  > triggering builds from merged PRs similarly makes sense, especially
>  > if that can automate bumping spec release number & adding a changelog
>  > entry based on the pull request subject.
>  > 
>  > 
>  > Obviously I can still use my upstream git repo branch and change the
>  > scripts to generate a pull request to dist-git instead.  The downside
>  > is if the PR fails, I now how to rebase my real git repo & re-submit
>  > and new PR.
>  > 
>  > So if we're talking "ideal" long term vision though, I'd rather
> eliminate
>  > the dist-git repo intermediate step as IMHO it adds no value, only complexity
>  > for the sake of historical compat with the way Red Hat distros has worked
>  > dating back years. Its time to say goodbye to this historical way as the
>  > world has moved on since this spec+patches approach was invented in the
>  > days of CVS source control.
>  > 
>  > Allow packagers to have a clone of the upstraem git repo, and use the
>  > pull-requests and run Fedora CI testing against the Fedora branch of
>  > the upstream repo instead of against dist-git.
>  > 
>  > In this way, maintaining packages in Fedora would be functionally identical
>  > to how an upstream project maintains their own stable branches. The only
>  > Fedora difference would be that the branch would need to include the RPM
>  > spec file in some well defined place.
>  
>  I like this.
>  It means that the build-system would have to generate the tarball of the source
>  and put it into dist-git at srpm-build time (I believe we still want to store a
>  copy of the sources used for a build). 
If we want to generate tarballs, we need to
think about translations as well.
While some projects do have translations as part of their source code repository,
others currently pull them in at tarball build time. This would have to be accounted
for or otherwise the resulting tarball would miss the translations.

>  
>  Three questions though:
>  - What about the upstream projects that still only publish a tarball at release?
>    Should the packager explode that tarball into a git repo?
>    Do we keep the current way of uploading the tarball?
>  - Won't this make pull-request for new update harder? Suddenly you have the diff
>    of both all of the upstream changes as well as the spec file and potential
>    patches.
>    In most cases the spec file and the patches is what will matter but for large
>    code-base the diff could be quite significant then.
>  - Do you have in mind that the copy/fork of upstream would be in our dist-git or
>    in other places on the internet? If the later, how would you recommend
>    contributors to find it?
>  
>  
>  Thanks for your inputs!
>  
>  Pierre
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Daniel P. Berrangé]

On Fri, Sep 27, 2019 at 12:40:42PM +0200, Martin Kolman wrote:
>  On Thu, 2019-09-26 at 16:24 +0200, Pierre-Yves Chibon wrote:
>  > On Thu, Sep 26, 2019 at 02:57:45PM +0100, Daniel P. Berrangé wrote:
>  > > On Thu, Sep 26, 2019 at 11:36:10AM +0200, Pierre-Yves Chibon wrote:
>  > > > Good Morning Everyone,
>  > > > 
>  > > > At Flock, a few of us met to discuss a future vision of the packager
> workflow.
>  > > > This discussion was triggered by the realization that a number of
> initiatives
>  > > > are happening around packaging in Fedora but there is no real shared
> vision on
>  > > > what we want the packager UX/workflow to be.
>  > > > The lack of vision on the packager workflow means we could deploy
> something
>  > > > today, thinking it is the improvement over the current workflow but would
>  > > > prevent us from (or make it harder to) doing other changes afterwords that
> would
>  > > > be even more beneficial..
>  > > > 
>  > > > So once that concern was raised, we took some time during the Fedora
>  > > > Infrastructure hackfest to gather the people interested around a white
> board and
>  > > > brainstorm on what a future packager workflow could look like.
>  > > > 
>  > > > We tried not to link this process to any tool in particular as well as
> focus on
>  > > > the what and why rather than any how.
>  > > > 
>  > > > Here is what the vision we came to and that we would like to discuss:
>  > > > 
>  > > > ○ Every changes to dist-git is done via pull-requests
>  > > > ○ Pull-requests are automatically tested
>  > > > ○ Every commit to dist-git (ie: PR merged) is automatically built in koji
>  > > > ○ Every build in koji results automatically in an update in bodhi
>  > > > ○ Every update in bodhi is automatically tested
>  > > > ○ If the tests pass, the update is automatically pushed to the repository
>  > > 
>  > > For packages I maintain, my preference is to touch dist-git as little
>  > > as possible. Ideally I would never touch dist-git at all & rather wish
>  > > it didn't exist at all in its current form of spec+patchfiles.
>  > > 
>  > > Instead I prefer a clone of the master upstream git repo and maintain a
>  > > branch with patches cherry-picked into it. This is used to auto-generate
>  > > patches for the Fedora RPM, at the same time updating the patch file list
>  > > in the RPM spec. The only manual step is adding the changelog entry &
>  > > bumping release number.
>  > > 
>  > > The ideas above around associating CI with pull requests make a lot of
>  > > conceptual sense & align with modern github/gitlab software development
>  > > best practices followed by non-distro software upstreams. Automatically
>  > > triggering builds from merged PRs similarly makes sense, especially
>  > > if that can automate bumping spec release number & adding a changelog
>  > > entry based on the pull request subject.
>  > > 
>  > > 
>  > > Obviously I can still use my upstream git repo branch and change the
>  > > scripts to generate a pull request to dist-git instead.  The downside
>  > > is if the PR fails, I now how to rebase my real git repo & re-submit
>  > > and new PR.
>  > > 
>  > > So if we're talking "ideal" long term vision though, I'd
> rather eliminate
>  > > the dist-git repo intermediate step as IMHO it adds no value, only complexity
>  > > for the sake of historical compat with the way Red Hat distros has worked
>  > > dating back years. Its time to say goodbye to this historical way as the
>  > > world has moved on since this spec+patches approach was invented in the
>  > > days of CVS source control.
>  > > 
>  > > Allow packagers to have a clone of the upstraem git repo, and use the
>  > > pull-requests and run Fedora CI testing against the Fedora branch of
>  > > the upstream repo instead of against dist-git.
>  > > 
>  > > In this way, maintaining packages in Fedora would be functionally identical
>  > > to how an upstream project maintains their own stable branches. The only
>  > > Fedora difference would be that the branch would need to include the RPM
>  > > spec file in some well defined place.
>  > 
>  > I like this.
>  > It means that the build-system would have to generate the tarball of the source
>  > and put it into dist-git at srpm-build time (I believe we still want to store a
>  > copy of the sources used for a build).
>  If we want to generate tarballs, we need to think about translations as well.
>  While some projects do have translations as part of their source code repository,
>  others currently pull them in at tarball build time. This would have to be accounted
>  for or otherwise the resulting tarball would miss the translations. 
Personally I'd avoid re-generating tarballs as it is a huge can of
worms IMHO. There are many ways upstreams generate their tarballs, often
including generated files for certain desirable reasons. This is very
common with autotools in particular. Just auto-generating patches is
likely to be more reliable.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Colin Walters]

On Thu, Sep 26, 2019, at 10:24 AM, Pierre-Yves Chibon wrote:

>  I like this.
>  It means that the build-system would have to generate the tarball of the source
>  and put it into dist-git at srpm-build time (I believe we still want to store a
>  copy of the sources used for a build). 
https://github.com/projectatomic/rpmdistro-gitoverlay/blob/master/doc/rew...
calls for defining mirrored git repositories as the canonical source - anything not stored
in git upstream would be imported into git.

[Zbigniew Jędrzejewski-Szmek]

On Thu, Sep 26, 2019 at 02:57:45PM +0100, Daniel P. Berrangé wrote:
>  Instead I prefer a clone of the master upstream git repo and maintain
> a
>  branch with patches cherry-picked into it. This is used to auto-generate
>  patches for the Fedora RPM, at the same time updating the patch file list
>  in the RPM spec. The only manual step is adding the changelog entry &
>  bumping release number. 
Quick note: this is essentially what debian does.

If you want to go this way, I hope we can avoid the need to generate
a "pristine" tarball and patches. Instead, the contents of the repo should
be directly used for a build. A tarball can be created, but this tarball
would always be the tip of the tree containing both the sources and
the packaging.

Zbyszek

[Fabio Valentini]

On Thu, Sep 26, 2019 at 4:47 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Thu, Sep 26, 2019 at 02:57:45PM +0100, Daniel P. Berrangé wrote:
>  > Instead I prefer a clone of the master upstream git repo and maintain a
>  > branch with patches cherry-picked into it. This is used to auto-generate
>  > patches for the Fedora RPM, at the same time updating the patch file list
>  > in the RPM spec. The only manual step is adding the changelog entry &
>  > bumping release number.
>
>  Quick note: this is essentially what debian does. 
Ugh. Can we please just agree that source-git (vs. dist-git) is almost
always a bad idea?

> If you want to go this way, I hope we can avoid the need to generate
> a "pristine" tarball and patches. Instead, the contents of the repo should
> be directly used for a build. A tarball can be created, but this tarball
> would always be the tip of the tree containing both the sources and
> the packaging.
>
> Zbyszek
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Jeremy Cline]

On Thu, Sep 26, 2019 at 04:49:31PM +0200, Fabio Valentini wrote:
>  On Thu, Sep 26, 2019 at 4:47 PM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > On Thu, Sep 26, 2019 at 02:57:45PM +0100, Daniel P. Berrangé wrote:
>  > > Instead I prefer a clone of the master upstream git repo and maintain a
>  > > branch with patches cherry-picked into it. This is used to auto-generate
>  > > patches for the Fedora RPM, at the same time updating the patch file list
>  > > in the RPM spec. The only manual step is adding the changelog entry &
>  > > bumping release number.
>  >
>  > Quick note: this is essentially what debian does.
>  
>  Ugh. Can we please just agree that source-git (vs. dist-git) is almost
>  always a bad idea?
>   
Can you expand on that? To me it is a much easier way to maintain
packages, but I don't have tons of experience doing that and I'm
perhaps not the "average package maintainer". Hearing other perspectives
would be helpful.

> > If you want to go this way, I hope we can avoid the need to generate
> > a "pristine" tarball and patches. Instead, the contents of the repo
should
> > be directly used for a build. A tarball can be created, but this tarball
> > would always be the tip of the tree containing both the sources and
> > the packaging.
> >
> > Zbyszek
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Fabio Valentini]

On Thu, Sep 26, 2019 at 4:57 PM Jeremy Cline <jeremy(a)jcline.org&gt; wrote:
>  On Thu, Sep 26, 2019 at 04:49:31PM +0200, Fabio Valentini wrote:
>  > On Thu, Sep 26, 2019 at 4:47 PM Zbigniew Jędrzejewski-Szmek
>  > <zbyszek(a)in.waw.pl&gt; wrote:
>  > >
>  > > On Thu, Sep 26, 2019 at 02:57:45PM +0100, Daniel P. Berrangé wrote:
>  > > > Instead I prefer a clone of the master upstream git repo and maintain a
>  > > > branch with patches cherry-picked into it. This is used to auto-generate
>  > > > patches for the Fedora RPM, at the same time updating the patch file list
>  > > > in the RPM spec. The only manual step is adding the changelog entry &
>  > > > bumping release number.
>  > >
>  > > Quick note: this is essentially what debian does.
>  >
>  > Ugh. Can we please just agree that source-git (vs. dist-git) is almost
>  > always a bad idea?
>  >
>
>  Can you expand on that? To me it is a much easier way to maintain
>  packages, but I don't have tons of experience doing that and I'm
>  perhaps not the "average package maintainer". Hearing other perspectives
>  would be helpful. 
Sorry, I should have expanded on that short statement.

I literally maintain hundreds of packages, mostly by myself (yes,
that's too many, but that's a problem that tooling can't fix).
I'd hazard to guess that 90% of these packages don't contain *any*
patches on top of an upstream release. So pointing the .spec file to
an upstream tarball is the *absolute easiest thing* possible, and I
don't want to have to deal with the upstream git repo contents for
these packages at all.
For another 9% of packages, I maintain a small set of local patches
(up to two patches or so), which I can "generate on the fly" for every
new upstream release (if necessary), and which is only a small amount
of work.
For the remaining 1% (about 2-3 of my packages), I maintain a local
git repository with branches for every release, where I maintain a
small downstream patch-set. This seems to correspond to a source-git
approach, and might be similar to what you're doing.

So, to summarize: I just don't want the absolute edge case to become
the general case. It would make things *massively* more complicated
for me, and I guess that applies to the "average packager" in fedora,
as well.

Fabio

> > > If you want to go this way, I hope we can avoid the need to generate
> > > a "pristine" tarball and patches. Instead, the contents of the
repo should
> > > be directly used for a build. A tarball can be created, but this tarball
> > > would always be the tip of the tree containing both the sources and
> > > the packaging.
> > >
> > > Zbyszek
> > > _______________________________________________
> > > devel mailing list -- devel(a)lists.fedoraproject.org
> > > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Jeremy Cline]

On Thu, Sep 26, 2019 at 05:14:59PM +0200, Fabio Valentini wrote:
>  On Thu, Sep 26, 2019 at 4:57 PM Jeremy Cline
> <jeremy(a)jcline.org&gt; wrote:
>  >
>  > On Thu, Sep 26, 2019 at 04:49:31PM +0200, Fabio Valentini wrote:
>  > > On Thu, Sep 26, 2019 at 4:47 PM Zbigniew Jędrzejewski-Szmek
>  > > <zbyszek(a)in.waw.pl&gt; wrote:
>  > > >
>  > > > On Thu, Sep 26, 2019 at 02:57:45PM +0100, Daniel P. Berrangé wrote:
>  > > > > Instead I prefer a clone of the master upstream git repo and maintain
> a
>  > > > > branch with patches cherry-picked into it. This is used to
> auto-generate
>  > > > > patches for the Fedora RPM, at the same time updating the patch file
> list
>  > > > > in the RPM spec. The only manual step is adding the changelog entry
> &
>  > > > > bumping release number.
>  > > >
>  > > > Quick note: this is essentially what debian does.
>  > >
>  > > Ugh. Can we please just agree that source-git (vs. dist-git) is almost
>  > > always a bad idea?
>  > >
>  >
>  > Can you expand on that? To me it is a much easier way to maintain
>  > packages, but I don't have tons of experience doing that and I'm
>  > perhaps not the "average package maintainer". Hearing other perspectives
>  > would be helpful.
>  
>  Sorry, I should have expanded on that short statement.
>  
>  I literally maintain hundreds of packages, mostly by myself (yes,
>  that's too many, but that's a problem that tooling can't fix).
>  I'd hazard to guess that 90% of these packages don't contain *any*
>  patches on top of an upstream release. So pointing the .spec file to
>  an upstream tarball is the *absolute easiest thing* possible, and I
>  don't want to have to deal with the upstream git repo contents for
>  these packages at all.
>  For another 9% of packages, I maintain a small set of local patches
>  (up to two patches or so), which I can "generate on the fly" for every
>  new upstream release (if necessary), and which is only a small amount
>  of work.
>  For the remaining 1% (about 2-3 of my packages), I maintain a local
>  git repository with branches for every release, where I maintain a
>  small downstream patch-set. This seems to correspond to a source-git
>  approach, and might be similar to what you're doing.
>  
>  So, to summarize: I just don't want the absolute edge case to become
>  the general case. It would make things *massively* more complicated
>  for me, and I guess that applies to the "average packager" in fedora,
>  as well.
>   
Ah right, that makes a lot of sense.

I can imagine automatically detecting the new upstream release, building
that, and presenting the packager with a easy-to-review PR that you just
click "merge" on instead of pointing the specfile at a new tarball.

Some of these could (I think) be solved with quality tooling, but I
don't see that kind of stuff appearing any time soon. I do think all
these things would have to be addressed, though, and I definitely agree
that the corner cases shouldn't drive the workflow.

- Jeremy

[Adam Williamson]

On Thu, 2019-09-26 at 15:46 +0000, Jeremy Cline wrote:
>  
>  Ah right, that makes a lot of sense.
>  
>  I can imagine automatically detecting the new upstream release, building
>  that, and presenting the packager with a easy-to-review PR that you just
>  click "merge" on instead of pointing the specfile at a new tarball. 
This already basically happens, at least for things that are hooked up
to anitya:
https://bugzilla.redhat.com/show_bug.cgi?id=1751432
https://fedoraproject.org/wiki/Upstream_release_monitoring

You get a bug report with a patch attached, and it runs a scratch
build. This is great for simple release bumps, but there's still all
sorts of cases where it's not enough or you're just doing something
else.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Jeremy Cline]

On Thu, Sep 26, 2019 at 09:08:16AM -0700, Adam Williamson wrote:
>  On Thu, 2019-09-26 at 15:46 +0000, Jeremy Cline wrote:
>  > 
>  > Ah right, that makes a lot of sense.
>  > 
>  > I can imagine automatically detecting the new upstream release, building
>  > that, and presenting the packager with a easy-to-review PR that you just
>  > click "merge" on instead of pointing the specfile at a new tarball.
>  
>  This already basically happens, at least for things that are hooked up
>  to anitya:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1751432
>  https://fedoraproject.org/wiki/Upstream_release_monitoring
>  
>  You get a bug report with a patch attached, and it runs a scratch
>  build. This is great for simple release bumps, but there's still all
>  sorts of cases where it's not enough or you're just doing something
>  else. 
I do have a little[0] experience with anitya, but it's not nearly as
hooked up as it could be and honestly Libraries.io generally does
better for tracking upstreams, it just lacks the mapping to downstreams.
Maybe we could add that and use it, or continue with anitya.

What I'd really like to see is a lot more "cleverness" from it regarding
versions and automatic backporting to various Fedora releases based on
semantic versioning. Also there's still manual steps you need to do as a
maintainer like downloading and then uploading the tarball.

[0] https://github.com/release-monitoring/anitya/graphs/contributors

[Michal Konecny]

On 2019-09-26 20:35, Jeremy Cline wrote:
>  On Thu, Sep 26, 2019 at 09:08:16AM -0700, Adam Williamson wrote:
> > On Thu, 2019-09-26 at 15:46 +0000, Jeremy Cline wrote:
> >> Ah right, that makes a lot of sense.
> >>
> >> I can imagine automatically detecting the new upstream release, building
> >> that, and presenting the packager with a easy-to-review PR that you just
> >> click "merge" on instead of pointing the specfile at a new tarball.
> > This already basically happens, at least for things that are hooked up
> > to anitya:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1751432
> > https://fedoraproject.org/wiki/Upstream_release_monitoring
> >
> > You get a bug report with a patch attached, and it runs a scratch
> > build. This is great for simple release bumps, but there's still all
> > sorts of cases where it's not enough or you're just doing something
> > else.
>  I do have a little[0] experience with anitya, but it's not nearly as
>  hooked up as it could be and honestly Libraries.io generally does
>  better for tracking upstreams, it just lacks the mapping to downstreams.
>  Maybe we could add that and use it, or continue with anitya.
>
>  What I'd really like to see is a lot more "cleverness" from it regarding
>  versions and automatic backporting to various Fedora releases based on
>  semantic versioning. Also there's still manual steps you need to do as a
>  maintainer like downloading and then uploading the tarball. 
As a current maintainer
I thought about this one and I have already work 
in progress [0] that adds ability to create PR in dist-git directly 
using packit. With Fedora CI in place, this will be automatically 
tested. To get this work I'm missing mostly time to do it and new 
version of Pagure to be deployed.

Anitya still needs plenty of love, but right now Fedora Infra has 
priorities elsewhere. There is still possibility to use libraries.io 
instead of Anitya, but there are some issues:
- lack of downstream mapping (this could be easily solved by some 
database with only downstream mapping)
- lack of custom project additions (libraries.io can only watch projects 
in sources they have already implemented)

Michal

[0] - https://github.com/fedora-infra/the-new-hotness/pull/235
>  [0] https://github.com/release-monitoring/anitya/graphs/contributors
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Randy Barlow]

On Fri, 2019-09-27 at 10:26 +0200, Michal Konecny wrote:
>  There is still possibility to use libraries.io 
>  instead of Anitya, but there are some issues:
>  - lack of downstream mapping (this could be easily solved by some 
>  database with only downstream mapping)
>  - lack of custom project additions (libraries.io can only watch
>  projects 
>  in sources they have already implemented) 
libraries.io is open source:

https://github.com/librariesio

It would be good to work with them to add these features there, rather
than to make our own database with the downstream mapping. This way all
distros can benefit, and we also don't have to do all the work of
creating and maintaining our own project.

[Michal Konecny]

On 2019-09-27 17:46, Randy Barlow wrote:
>  On Fri, 2019-09-27 at 10:26 +0200, Michal Konecny wrote:
> > There is still possibility to use libraries.io
> > instead of Anitya, but there are some issues:
> > - lack of downstream mapping (this could be easily solved by some
> > database with only downstream mapping)
> > - lack of custom project additions (libraries.io can only watch
> > projects
> > in sources they have already implemented)
>  libraries.io is open source:
>
>  https://github.com/librariesio
>
>  It would be good to work with them to add these features there, rather
>  than to make our own database with the downstream mapping. This way all
>  distros can benefit, and we also don't have to do all the work of
>  creating and maintaining our own project. 
Yes, this will need to be done, if we
decide to replace Anitya with 
libraries.io.
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Steven A. Falco]

On 9/26/19 11:14 AM, Fabio Valentini wrote:
>  On Thu, Sep 26, 2019 at 4:57 PM Jeremy Cline
> <jeremy(a)jcline.org&gt; wrote:
> >
> > On Thu, Sep 26, 2019 at 04:49:31PM +0200, Fabio Valentini wrote:
> >> On Thu, Sep 26, 2019 at 4:47 PM Zbigniew Jędrzejewski-Szmek
> >> <zbyszek(a)in.waw.pl&gt; wrote:
> >>>
> >>> On Thu, Sep 26, 2019 at 02:57:45PM +0100, Daniel P. Berrangé wrote:
> >>>> Instead I prefer a clone of the master upstream git repo and maintain a
> >>>> branch with patches cherry-picked into it. This is used to auto-generate
> >>>> patches for the Fedora RPM, at the same time updating the patch file
> list
> >>>> in the RPM spec. The only manual step is adding the changelog entry
> &
> >>>> bumping release number.
> >>>
> >>> Quick note: this is essentially what debian does.
> >>
> >> Ugh. Can we please just agree that source-git (vs. dist-git) is almost
> >> always a bad idea?
> >>
> >
> > Can you expand on that? To me it is a much easier way to maintain
> > packages, but I don't have tons of experience doing that and I'm
> > perhaps not the "average package maintainer". Hearing other perspectives
> > would be helpful.
>  
>  Sorry, I should have expanded on that short statement.
>  
>  I literally maintain hundreds of packages, mostly by myself (yes,
>  that's too many, but that's a problem that tooling can't fix).
>  I'd hazard to guess that 90% of these packages don't contain *any*
>  patches on top of an upstream release. So pointing the .spec file to
>  an upstream tarball is the *absolute easiest thing* possible, and I
>  don't want to have to deal with the upstream git repo contents for
>  these packages at all.
>  For another 9% of packages, I maintain a small set of local patches
>  (up to two patches or so), which I can "generate on the fly" for every
>  new upstream release (if necessary), and which is only a small amount
>  of work.
>  For the remaining 1% (about 2-3 of my packages), I maintain a local
>  git repository with branches for every release, where I maintain a
>  small downstream patch-set. This seems to correspond to a source-git
>  approach, and might be similar to what you're doing.
>  
>  So, to summarize: I just don't want the absolute edge case to become
>  the general case. It would make things *massively* more complicated
>  for me, and I guess that applies to the "average packager" in fedora,
>  as well.
>  
>  Fabio 
KiCAD is the main project I help maintain.  It consists of seven separate upstream
repositories.  One is on launchpad; the rest are on github.  I would not want to have to
mirror them all into Fedora.  The tarball / dist-git approach works very well for this
project.

	Steve

>  
> >>> If you want to go this way, I hope we can avoid the need to generate
> >>> a "pristine" tarball and patches. Instead, the contents of the repo
> should
> >>> be directly used for a build. A tarball can be created, but this tarball
> >>> would always be the tip of the tree containing both the sources and
> >>> the packaging.
> >>>
> >>> Zbyszek
> >>> _______________________________________________
> >>> devel mailing list -- devel(a)lists.fedoraproject.org
> >>> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> >>> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> >> _______________________________________________
> >> devel mailing list -- devel(a)lists.fedoraproject.org
> >> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> >> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Ben Rosser]

On Thu, Sep 26, 2019 at 4:50 PM Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  On Thu, Sep 26, 2019 at 4:47 PM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > On Thu, Sep 26, 2019 at 02:57:45PM +0100, Daniel P. Berrangé wrote:
>  > > Instead I prefer a clone of the master upstream git repo and maintain a
>  > > branch with patches cherry-picked into it. This is used to auto-generate
>  > > patches for the Fedora RPM, at the same time updating the patch file list
>  > > in the RPM spec. The only manual step is adding the changelog entry &
>  > > bumping release number.
>  >
>  > Quick note: this is essentially what debian does.
>
>  Ugh. Can we please just agree that source-git (vs. dist-git) is almost
>  always a bad idea? 
I can see the benefit for some very complex packages that have lots of
patches, so I think it's not unreasonable for source-git to be an
_option_. (Maybe one that should require approval?)

But I'll echo what I wrote in my other message: for the majority of
packages which are relatively simple and don't deviate much from
upstream, or try not to deviate from upstream, I think it's a bad idea
and won't improve things. We need to keep the simple/everyday use
cases in mind here.

I like that I can build a Fedora package with just a specfile--
essentially, a bit of metadata about how to build and install a
package-- and don't need to modify an upstream archive. I would be
disappointed if we moved towards a Debian-like way of doing packaging.

Ben Rosser

[Brian C. Lane]

On Thu, Sep 26, 2019 at 02:57:45PM +0100, Daniel P. Berrangé wrote:

[snip]

>  Allow packagers to have a clone of the upstraem git repo, and use
> the
>  pull-requests and run Fedora CI testing against the Fedora branch of
>  the upstream repo instead of against dist-git.
>  
>  In this way, maintaining packages in Fedora would be functionally identical
>  to how an upstream project maintains their own stable branches. The only
>  Fedora difference would be that the branch would need to include the RPM
>  spec file in some well defined place. 
This will only work for a subset of upstreams though. Not everyone is
using git, and even in cases where they are it can be better to use
their released tar because what they distribute is generated from their
upstream, but cannot be used directly without duplicating their release
process and possibly introducing errors.

Another situation is gpg signed release tars, eg. parted and gnupg1 are
2 examples of that. parted being a nice example of a messy upstream
(yeah, I'm slowly working on getting that fixed) where you start with a
known release and add patches until they (me/us) do a new official
release.

Support for signed tags from upstream would help with the gpg part, but
not very many people are actually signing their tags.

I'm also not clear on where the .spec files and tests would live if you
are using a fork of the upstream. We still need dist-git to store those,
even if everything that touches them is a tool other than vim. Or maybe
I missed something :)

-- 
Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart

[Randy Barlow]

On Thu, 2019-09-26 at 08:58 -0700, Brian C. Lane wrote:
>  I'm also not clear on where the .spec files and tests would live
> if
>  you
>  are using a fork of the upstream. We still need dist-git to store
>  those,
>  even if everything that touches them is a tool other than vim. Or
>  maybe
>  I missed something :) 
I think the proposal is that your fork would contain the spec file
(i.e., it'd have a commit in it that's not in the upstream that has the
spec, and you would just rebase that every now and then.)

[Dan Čermák]

Daniel P. Berrangé <berrange(a)redhat.com&gt; writes:

>  On Thu, Sep 26, 2019 at 11:36:10AM +0200, Pierre-Yves Chibon wrote:
>
>  For packages I maintain, my preference is to touch dist-git as little
>  as possible. Ideally I would never touch dist-git at all & rather wish
>  it didn't exist at all in its current form of spec+patchfiles.
>
>  Instead I prefer a clone of the master upstream git repo and maintain a
>  branch with patches cherry-picked into it. This is used to auto-generate
>  patches for the Fedora RPM, at the same time updating the patch file list
>  in the RPM spec. The only manual step is adding the changelog entry &
>  bumping release number. 
As others have said already: this is a nice idea and I definitely would
see myself using it for one package (there I have a bunch of
non-upstream patches), but not for most of them, so having this as an
option would be pretty cool.

BUT (and that is a very big but) this is very dangerous and makes
patching just far too easy and far too tempting. If you can add a new
non-upstream patch with a simple `git commit` and be done with it, then
there's very little incentive to upstream these. Patching of packages
*should* be hard to a certain extent, it even needs to be a PITA so that
you want to get rid of that patch and upstream it. Because if you don't,
you end up with hundreds of non-upstream patches that will become a huge
burden down the road. (This is the worst case scenario and not something
against you Daniel, but I've seen it happen already.)

Thus, unless there is some kind of additional review/check/person
telling you "Stop patching downstream!", I would only allow packagers to
use the source git option with FESCO's or the packaging committee's
approval.

[Fabio Valentini]

On Thu, Sep 26, 2019 at 2:32 PM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:
>  Good Morning Everyone, 
Alright. I don't often reply to discussion threads like this, but here we go.

>  At Flock, a few of us met to discuss a future vision of the packager
> workflow.
>  This discussion was triggered by the realization that a number of initiatives
>  are happening around packaging in Fedora but there is no real shared vision on
>  what we want the packager UX/workflow to be.
>  The lack of vision on the packager workflow means we could deploy something
>  today, thinking it is the improvement over the current workflow but would
>  prevent us from (or make it harder to) doing other changes afterwords that would
>  be even more beneficial..
>
>  So once that concern was raised, we took some time during the Fedora
>  Infrastructure hackfest to gather the people interested around a white board and
>  brainstorm on what a future packager workflow could look like.
>
>  We tried not to link this process to any tool in particular as well as focus on
>  the what and why rather than any how.
>
>  Here is what the vision we came to and that we would like to discuss:
>
>  ○ Every changes to dist-git is done via pull-requests 
This seems to be overkill. Especially merge conflicts when dealing
with frequently-changing packages will make this impossible.

>  ○ Pull-requests are automatically tested 
If using Pull Requests is optional, I'm all for it.

>  ○ Every commit to dist-git (ie: PR merged) is automatically built in
> koji 
I don't think this is wise. On the one hand, it will create even more
workload in koji, and on the other hand, running "fedpkg build" for
things I *actually* want to get built is no burden at all. Other steps
in the process dominate.

>  ○ Every build in koji results automatically in an update in bodhi
This might sound like a good idea, but it breaks down when I think about:
- referencing RHBZs,
- specifying update type, severity, reboot/logout, stable/unstable
karma limits, etc.,
- mulit-package updates (this might involve buildroot overrides for
stable releases).

I don't think there's a way around manual update creation.

>  ○ Every update in bodhi is automatically tested 
Isn't this already the case?

>  ○ If the tests pass, the update is automatically pushed to the
> repository
>
>
>  For this workflow to work nicely we need to fix a few things first:
>
>  - We need to work on the change logs in the spec files, as otherwise
>    pull-requests are going to conflict more often than not 
Exactly. This is a big pain point in packaging, but it can be solved
without changing everything else (and it might be even a prerequisite
for other changes).

>  - We need to fine a place to insert the end-user information about an
> update (in
>    the PR description?) 
How do you want to do that for complex updates? Embedding YAML in the
commit message? :D
(That was a JOKE, please don't use yaml, it's terrible.)

>  - When a group of people want to collectively work on a change
> (involving one or
>    more packages), they’ll need a place to collaborate. This could be someone’s
>    fork or, preferably, a fork of the project collectively maintained by that
>    group.
>      For example: the python SIG would have a group with forks of the different
>      python packages they are working on 
Being able to use shared forks for collaboration sounds like a good
idea, but again, this is an orthogonal problem.

>  - We need to figure out how to enable opening a pull-request again
> multiple
>    branches at one. 
Again, dealing with the changelog and package Release numbers somehow
is a prerequisite to make this work for branches other than rawhide
and N+1.

>  Why should we agree now on a long term vision?
>
>  A concrete example to this, I have been playing with the idea to get ride of git
>  branches in dist-git for a while now. Basically use git as it is meant to be
>  used: have one branch for development (most often master) and branch of master
>  when you need to, not simply because a new version of Fedora happened.
>  However, if we implemented this, we would break the workflow described above as
>  suddenly there would be no way to specify that a given PR is meant to land in a
>  certain version of Fedora or another (Is this PR for rawhide? F31? F30? All of
>  them? Some of them?). 
I for one like the clean correspondence of fedora branches <=> git
branches. It makes working with packages that have different contents
in different branches of fedora easy, and intuitive. Once the main
source of conflicts between branches (mass rebuild etc. changelog
messages and differing Release numbers) is fixed, PRs can
automatically be created against multpile branches without issues.

But, I also think hiding complexity within our tooling is a *bad
idea*. There's too much *magic* hidden away already, and rpkg+fedpkg,
python-fedora, bodhi-client, koji-client, etc. are piles of spaghetti
code even *before* adding all this new stuff (I mean, things work
remarkably well, but looking at the code makes me want to cry each
time). Shoving even more functionality into these projects will make
them *less maintainable*, not more.


Just my 2¢ (or more).

Fabio

> Having this idea of where we want to go should give us a goal as well as way to
> reflect if changes we think are good in isolation would actually help achieving
> this vision or not.
>
> This proposal was based on the brainstorming session a few of us had at flock,
> but before we can see how to implement it we need more inputs and this is where
> you come in :)
>
> Do you like this vision? Would you change some pieces of it? Would you change it
> entirely?
> In an ideal world, what would packaging software look like to you?
>
> Looking forward to the discussion,
>
> Pierre
> _______________________________________________
> devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...

[Zbigniew Jędrzejewski-Szmek]

On Thu, Sep 26, 2019 at 04:16:46PM +0200, Fabio Valentini wrote:
>  > ○ Every commit to dist-git (ie: PR merged) is automatically
> built in koji
>  
>  I don't think this is wise. On the one hand, it will create even more
>  workload in koji, and on the other hand, running "fedpkg build" for
>  things I *actually* want to get built is no burden at all. Other steps
>  in the process dominate. 
The motivation for this, as I understand it, is that once a PR is merged
in dist-git, in many cases what we want is to build and create update
without any further ado.
E.g. if I get a PR titled "Update to version N", it'd be nice if the
only thing I would need to do would be to oggle the diff and press one
button.

A mechanism to opt-out would of course be necessary, but automating the
common case would be great and would free packagers to do actually meaningful
stuff.

>  > ○ Every build in koji results automatically in an update in
> bodhi
>  
>  This might sound like a good idea, but it breaks down when I think about:
>  - referencing RHBZs,
>  - specifying update type, severity, reboot/logout, stable/unstable
>  karma limits, etc.,
>  - mulit-package updates (this might involve buildroot overrides for
>  stable releases). 
That's all true. I think a package could set defaults through a config file
in dist-git. For example, needs-reboot could be easily handled by this,
since most packages need the same behaviour for every update.
The automation could be smart enough to use some basic rules for other fields,
like: major version bump → enhancement, CVE in %changelog → security update,
etc. An update could also be edited after the fact, for example to bump
severity, if appropriate. We should probably start small, but maybe at
some point we could even make bodhi look into bugzilla and bump the severity
if bugs have high severity set.

>  I don't think there's a way around manual update creation.
It definitely must stay as an option for the complex cases.

Zbyszek

[Zbigniew Jędrzejewski-Szmek]

On Thu, Sep 26, 2019 at 11:36:10AM +0200, Pierre-Yves Chibon wrote:
>  Good Morning Everyone,
>  
>  At Flock, a few of us met to discuss a future vision of the packager workflow.
>  This discussion was triggered by the realization that a number of initiatives
>  are happening around packaging in Fedora but there is no real shared vision on
>  what we want the packager UX/workflow to be.
>  The lack of vision on the packager workflow means we could deploy something
>  today, thinking it is the improvement over the current workflow but would
>  prevent us from (or make it harder to) doing other changes afterwords that would
>  be even more beneficial..
>  
>  So once that concern was raised, we took some time during the Fedora
>  Infrastructure hackfest to gather the people interested around a white board and
>  brainstorm on what a future packager workflow could look like.
>  
>  We tried not to link this process to any tool in particular as well as focus on
>  the what and why rather than any how.
>  
>  Here is what the vision we came to and that we would like to discuss:
>  
>  ○ Every changes to dist-git is done via pull-requests
>  ○ Pull-requests are automatically tested
>  ○ Every commit to dist-git (ie: PR merged) is automatically built in koji
>  ○ Every build in koji results automatically in an update in bodhi
>  ○ Every update in bodhi is automatically tested
>  ○ If the tests pass, the update is automatically pushed to the repository
>  
>  
>  For this workflow to work nicely we need to fix a few things first:
>  
>  - We need to work on the change logs in the spec files, as otherwise
>    pull-requests are going to conflict more often than not 
There was a pretty nice proposal during Flock from ngoma and ignatenkobrain
(https://flock2019.sched.com/event/SH8l/improving-packaging-experience-wit...).
In particular, the rpm release bump would be done automatically.
If we also agree to autogenerate %changelog from git commit descriptions,
we would avoid most conflicts.

>  - We need to fine a place to insert the end-user information about an
> update (in
>    the PR description?) 
Why not in the description of individual commits? PRs are a volatile thing.
Each commit is supposed to describe a single logical change, so we could
agree that the commit message automatically becomes part of the changelog
and the bodhi update description.

We'd also need a way to say that some commits don't bring not-user
visible changes. Some kind of marker in the commit message text should work
for this.

>  - When a group of people want to collectively work on a change
> (involving one or
>    more packages), they’ll need a place to collaborate. This could be someone’s
>    fork or, preferably, a fork of the project collectively maintained by that
>    group.
>      For example: the python SIG would have a group with forks of the different
>      python packages they are working on 
>  - We need to figure out how to enable opening a pull-request again multiple
>    branches at one.
>  
>  
>  Why should we agree now on a long term vision?
>  
>  A concrete example to this, I have been playing with the idea to get ride of git
>  branches in dist-git for a while now. Basically use git as it is meant to be
>  used: have one branch for development (most often master) and branch of master
>  when you need to, not simply because a new version of Fedora happened.
>  However, if we implemented this, we would break the workflow described above as
>  suddenly there would be no way to specify that a given PR is meant to land in a
>  certain version of Fedora or another (Is this PR for rawhide? F31? F30? All of
>  them? Some of them?). 
I don't think this is the right approach.
If we rearrange things so that conflicts are not a big issue (occasional
conflicts will always happen with branches, but let's say that it is the
exception, not the rule), then branching is cheap and we can still use
dist-git in current form. Git *is* meant to be used with branches.

Right now, many (most?) packages have different content in different
branches. Package versions and update guidelines and changes to
packaging make this almost unavoidable. The only option to make things
work without branches would be to require lots of conditionals (which
is not a nice option for packagers, and generally conflicts with
automatization).

Also, it should be possible for the new process to coexist with
existing practices to avoid a flag day. For some packages, possibly
indefinitely.

Also, restricting ourselves to a single branch (and a single PR) would
mean a loss of flexibility. Consider the following scenario: a new
point update comes out for a package. It is a bugfix release, and we
want to push it to all stable branches that currently have the same version.
But for one branch the test fail, because one of the dependencies is too
old. If we have multiple PRs, this is not a big issue: we might decide to
push the update to some branches, but not others.
 
>  Having this idea of where we want to go should give us a goal as well
> as way to
>  reflect if changes we think are good in isolation would actually help achieving
>  this vision or not.
>  
>  This proposal was based on the brainstorming session a few of us had at flock,
>  but before we can see how to implement it we need more inputs and this is where
>  you come in :)
>  
>  Do you like this vision? Would you change some pieces of it? Would you change it
>  entirely?
>  In an ideal world, what would packaging software look like to you? 
This proposal doesn't talk at all about the earlier part: how to automatize
the full process from upstream release to bodhi update.
The packit project tries to solve some of that, but by piling up solutions
on top of existing services. If our own infra was made more flexible and more
automatic, then I think we could build much better process to make the whole
packaging process from upstream to end user automatic.

My vision is that (for projects which are sufficiently mature and trusted to
merit this), I can click a checkbox that means:
when upstream release happens, open PR in dist-git, when that PR passes gating,
merge it, then create bodhi update, when that release passes QA requirements,
push update to stable.

anitya/upstream-release-monitoring already does the detection of the
upstream projects. Your proposal would solve the step between koji and bodhi.
Gating is slowly happening. The last step is now also automatic, and updates
are pushed based on time. The part that would open a PR for dist-git and merge
it if checks pass is still completely missing afaik.

Zbyszek

[Jeremy Cline]

On Thu, Sep 26, 2019 at 11:36:10AM +0200, Pierre-Yves Chibon wrote:
>  Good Morning Everyone,
>  
>  At Flock, a few of us met to discuss a future vision of the packager workflow.
>  This discussion was triggered by the realization that a number of initiatives
>  are happening around packaging in Fedora but there is no real shared vision on
>  what we want the packager UX/workflow to be.
>  The lack of vision on the packager workflow means we could deploy something
>  today, thinking it is the improvement over the current workflow but would
>  prevent us from (or make it harder to) doing other changes afterwords that would
>  be even more beneficial..
>  
>  So once that concern was raised, we took some time during the Fedora
>  Infrastructure hackfest to gather the people interested around a white board and
>  brainstorm on what a future packager workflow could look like.
>  
>  We tried not to link this process to any tool in particular as well as focus on
>  the what and why rather than any how.
>  
>  Here is what the vision we came to and that we would like to discuss:
>  
>  ○ Every changes to dist-git is done via pull-requests 
Allow pull requests, but don't force them upon me. There are plenty of
times I want to do things via the CLI. Also, I'd only be interested in
using pull requests more if it was in something like GitLab because the
Pagure workflow/developer experience is too painful to bother with.

>  ○ Pull-requests are automatically tested 
Sounds fine to me (GitLab's CI pipeline is pretty sweet).

>  ○ Every commit to dist-git (ie: PR merged) is automatically built in
> koji
>  ○ Every build in koji results automatically in an update in bodhi 
The combination of these two makes no sense to me. I do plenty of work
where I don't want to build it (specfile cleanup, patches, configuration
changes, etc.). I want a build that goes to users be explicit.

A better model, in my opinion, is to build every *tag*. To do a new
kernel build I could make a tag like "kernel-5.4-rc1..." and the tag
would be parsed into the specfile's NVR and built.

>  
>  Do you like this vision? Would you change some pieces of it? Would you change it
>  entirely?
>  In an ideal world, what would packaging software look like to you?
>   
Ideally? Working with specfiles and dist-git for patching and updating
is unpleasant. I want to work from the source tree of the repository and
use git to manage additional patching. Ideally I'd have a packaging
repository with the upstream as a git remote and my update process is
git-pull with git-rebase/git-cherrypick for patches. I don't ever want
to deal with dist-git in my work.

We're playing with a workflow like this for the kernel and so far I much
prefer it.

- Jeremy

[Randy Barlow]

On Thu, 2019-09-26 at 14:49 +0000, Jeremy Cline wrote:
>  The combination of these two makes no sense to me. I do plenty of
>  work
>  where I don't want to build it (specfile cleanup, patches,
>  configuration
>  changes, etc.). I want a build that goes to users be explicit.
>  
>  A better model, in my opinion, is to build every *tag*. To do a new
>  kernel build I could make a tag like "kernel-5.4-rc1..." and the tag
>  would be parsed into the specfile's NVR and built. 
I agree, and I really like the alternative suggestion here. Some people
in the thread have talked about how there are often conflicts between
branches due to the changelog, but the other common reason for
conflicts is the release field in my experience. If we use tags as an
explicit "I want this to go to users", then it solves both problems (I
consider sending all commits to end users a problem, because I often
make refactor commits that I would not want to churn users on.)

[Jeremy Cline]

On Thu, Sep 26, 2019 at 02:57:56PM -0400, Randy Barlow wrote:
>  On Thu, 2019-09-26 at 14:49 +0000, Jeremy Cline wrote:
>  > The combination of these two makes no sense to me. I do plenty of
>  > work
>  > where I don't want to build it (specfile cleanup, patches,
>  > configuration
>  > changes, etc.). I want a build that goes to users be explicit.
>  > 
>  > A better model, in my opinion, is to build every *tag*. To do a new
>  > kernel build I could make a tag like "kernel-5.4-rc1..." and the tag
>  > would be parsed into the specfile's NVR and built.
>  
>  I agree, and I really like the alternative suggestion here. Some people
>  in the thread have talked about how there are often conflicts between
>  branches due to the changelog, but the other common reason for
>  conflicts is the release field in my experience. If we use tags as an
>  explicit "I want this to go to users", then it solves both problems (I
>  consider sending all commits to end users a problem, because I often
>  make refactor commits that I would not want to churn users on.) 
The tag also provides a nice place to write release notes for the
update. I suppose you could also add support for some sort of text tag
inside commits (like when you mark a commit as fixing an issue in
Git{Lab,Hub} and look at the commits between the new tag and old one so
selective git commits could get sucked into the changelog as well.

[Randy Barlow]

On Thu, 2019-09-26 at 19:05 +0000, Jeremy Cline wrote:
>  The tag also provides a nice place to write release notes for the
>  update. I suppose you could also add support for some sort of text
>  tag
>  inside commits (like when you mark a commit as fixing an issue in
>  Git{Lab,Hub} and look at the commits between the new tag and old one
>  so
>  selective git commits could get sucked into the changelog as well. 
Oh that's beautiful. I've seen a lot of proposals to use the git
changelog to replace the spec file changelog, but I dislike those
proposals because the git changelog should be targeting an audience of
spec file authors, not end users. As cited earlier, I often make
refactor commits, and I *do* want to log those for other spec file
developers, but I wouldn't want those messages to go to end users (they
don't care or even know about the spec file).

This suggestion gives a nice clean place to write the bodhi update
description, right in git. The commit messages can remain the way they
are today: authored for the audience of spec file contributors.

We could also support special syntax in the tag message to allow people
to specify the various bodhi update fields (severity, karma
requirements, whatever else) if they want to do that that way.

[Miro Hrončok]

On 26. 09. 19 21:47, Randy Barlow wrote:
>  On Thu, 2019-09-26 at 19:05 +0000, Jeremy Cline wrote:
> > The tag also provides a nice place to write release notes for the
> > update. I suppose you could also add support for some sort of text
> > tag
> > inside commits (like when you mark a commit as fixing an issue in
> > Git{Lab,Hub} and look at the commits between the new tag and old one
> > so
> > selective git commits could get sucked into the changelog as well.
>  
>  Oh that's beautiful. I've seen a lot of proposals to use the git
>  changelog to replace the spec file changelog, but I dislike those
>  proposals because the git changelog should be targeting an audience of
>  spec file authors, not end users. As cited earlier, I often make
>  refactor commits, and I *do* want to log those for other spec file
>  developers, but I wouldn't want those messages to go to end users (they
>  don't care or even know about the spec file).
>  
>  This suggestion gives a nice clean place to write the bodhi update
>  description, right in git. The commit messages can remain the way they
>  are today: authored for the audience of spec file contributors.
>  
>  We could also support special syntax in the tag message to allow people
>  to specify the various bodhi update fields (severity, karma
>  requirements, whatever else) if they want to do that that way. 
The ability to use annotated git tags instead of changelog entries sounds 
intriguing.

The ability to use it to create updates in bodhi sounds like git porn.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Dan Čermák]

Randy Barlow <bowlofeggs(a)fedoraproject.org&gt; writes:

>  This suggestion gives a nice clean place to write the bodhi update
>  description, right in git. The commit messages can remain the way they
>  are today: authored for the audience of spec file contributors.
>
>  We could also support special syntax in the tag message to allow people
>  to specify the various bodhi update fields (severity, karma
>  requirements, whatever else) if they want to do that that way. 
I *really* like this idea, as one could still push multiple
"development" commits to git, but until one tags something, nothing is
built & released.

[Martin Kolman]

On Fri, 2019-09-27 at 00:20 +0200, Dan Čermák wrote:
>  Randy Barlow <bowlofeggs(a)fedoraproject.org&gt; writes:
>  
>  > This suggestion gives a nice clean place to write the bodhi update
>  > description, right in git. The commit messages can remain the way they
>  > are today: authored for the audience of spec file contributors.
>  > 
>  > We could also support special syntax in the tag message to allow people
>  > to specify the various bodhi update fields (severity, karma
>  > requirements, whatever else) if they want to do that that way.
>  
>  I *really* like this idea, as one could still push multiple
>  "development" commits to git, but until one tags something, nothing is
>  built & released. 
And there is even at least one precendent, where Git tags are
used to trigger
some process - the GitHub releases, that list all tags & provide archives for them.

So it seems like a sensible & already proven idea.

>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Panu Matilainen]

On 9/26/19 10:05 PM, Jeremy Cline wrote:
>  On Thu, Sep 26, 2019 at 02:57:56PM -0400, Randy Barlow wrote:
> > On Thu, 2019-09-26 at 14:49 +0000, Jeremy Cline wrote:
> >> The combination of these two makes no sense to me. I do plenty of
> >> work
> >> where I don't want to build it (specfile cleanup, patches,
> >> configuration
> >> changes, etc.). I want a build that goes to users be explicit.
> >>
> >> A better model, in my opinion, is to build every *tag*. To do a new
> >> kernel build I could make a tag like "kernel-5.4-rc1..." and the tag
> >> would be parsed into the specfile's NVR and built.
> >
> > I agree, and I really like the alternative suggestion here. Some people
> > in the thread have talked about how there are often conflicts between
> > branches due to the changelog, but the other common reason for
> > conflicts is the release field in my experience. If we use tags as an
> > explicit "I want this to go to users", then it solves both problems (I
> > consider sending all commits to end users a problem, because I often
> > make refactor commits that I would not want to churn users on.)
>  
>  The tag also provides a nice place to write release notes for the
>  update. I suppose you could also add support for some sort of text tag
>  inside commits (like when you mark a commit as fixing an issue in
>  Git{Lab,Hub} and look at the commits between the new tag and old one so
>  selective git commits could get sucked into the changelog as well. 
We've tossed around using tags for builds before in another context, but 
the idea of tag annotations for populating the user-visible changelog is 
an interesting and a totally novel idea AFAIK.

On top of using tags to, well, tag content for building (it seems so 
natural nothing could be more natural), we talked about calculating the 
release number automatically from number of commits on that branch since 
the last tag. The details seem to largely evade me, but changelog 
population was planned around picking messages out of git commit 
messages. Which has its issues. The tag annotations probably has its 
own, but it's indeed an intriguing idea.

In fact it was that discussion which prompted the development of 
automatic patch numbering and %patchlist support in spec files in rpm 
4.15, since in the planned scheme merge conflicts on release number and 
changelog would be gone, and conflicts on patch numbers was identified 
as yet another redundant piece of data that's also often prone to 
unnecessary merge conflicts.

The %changelog in specs really, really needs to die.

	- Panu -

[Neal Gompa]

On Fri, Sep 27, 2019 at 9:54 AM Panu Matilainen <pmatilai(a)redhat.com&gt; wrote:
>  On 9/26/19 10:05 PM, Jeremy Cline wrote:
>  > On Thu, Sep 26, 2019 at 02:57:56PM -0400, Randy Barlow wrote:
>  >> On Thu, 2019-09-26 at 14:49 +0000, Jeremy Cline wrote:
>  >>> The combination of these two makes no sense to me. I do plenty of
>  >>> work
>  >>> where I don't want to build it (specfile cleanup, patches,
>  >>> configuration
>  >>> changes, etc.). I want a build that goes to users be explicit.
>  >>>
>  >>> A better model, in my opinion, is to build every *tag*. To do a new
>  >>> kernel build I could make a tag like "kernel-5.4-rc1..." and the
> tag
>  >>> would be parsed into the specfile's NVR and built.
>  >>
>  >> I agree, and I really like the alternative suggestion here. Some people
>  >> in the thread have talked about how there are often conflicts between
>  >> branches due to the changelog, but the other common reason for
>  >> conflicts is the release field in my experience. If we use tags as an
>  >> explicit "I want this to go to users", then it solves both problems
> (I
>  >> consider sending all commits to end users a problem, because I often
>  >> make refactor commits that I would not want to churn users on.)
>  >
>  > The tag also provides a nice place to write release notes for the
>  > update. I suppose you could also add support for some sort of text tag
>  > inside commits (like when you mark a commit as fixing an issue in
>  > Git{Lab,Hub} and look at the commits between the new tag and old one so
>  > selective git commits could get sucked into the changelog as well.
>
>  We've tossed around using tags for builds before in another context, but
>  the idea of tag annotations for populating the user-visible changelog is
>  an interesting and a totally novel idea AFAIK.
>
>  On top of using tags to, well, tag content for building (it seems so
>  natural nothing could be more natural), we talked about calculating the
>  release number automatically from number of commits on that branch since
>  the last tag. The details seem to largely evade me, but changelog
>  population was planned around picking messages out of git commit
>  messages. Which has its issues. The tag annotations probably has its
>  own, but it's indeed an intriguing idea.
>  
This was the discussion Igor and I had at the openSUSE Summit in May.
The unanswered question I had was if we can manipulate the data
attached to a tag in Pagure UI and edit it after it was initially
pushed. If annotations are also frozen like all other things in
Dist-Git, then it fails as a usable mechanism.

>  In fact it was that discussion which prompted the development of
>  automatic patch numbering and %patchlist support in spec files in rpm
>  4.15, since in the planned scheme merge conflicts on release number and
>  changelog would be gone, and conflicts on patch numbers was identified
>  as yet another redundant piece of data that's also often prone to
>  unnecessary merge conflicts.
>
>  The %changelog in specs really, really needs to die.
>  
I would actually say that the spec in the SRPM should contain the
changelog, so that repeating the build from the exploded contents is
possible. But yes, it'd be nice if it wasn't there in the specs in
Git.


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Jeremy Cline]

On Fri, Sep 27, 2019 at 10:57:21AM -0400, Neal Gompa wrote:
>  On Fri, Sep 27, 2019 at 9:54 AM Panu Matilainen
> <pmatilai(a)redhat.com&gt; wrote:
>  >
>  > On 9/26/19 10:05 PM, Jeremy Cline wrote:
>  > > On Thu, Sep 26, 2019 at 02:57:56PM -0400, Randy Barlow wrote:
>  > >> On Thu, 2019-09-26 at 14:49 +0000, Jeremy Cline wrote:
>  > >>> The combination of these two makes no sense to me. I do plenty of
>  > >>> work
>  > >>> where I don't want to build it (specfile cleanup, patches,
>  > >>> configuration
>  > >>> changes, etc.). I want a build that goes to users be explicit.
>  > >>>
>  > >>> A better model, in my opinion, is to build every *tag*. To do a new
>  > >>> kernel build I could make a tag like "kernel-5.4-rc1..." and
> the tag
>  > >>> would be parsed into the specfile's NVR and built.
>  > >>
>  > >> I agree, and I really like the alternative suggestion here. Some people
>  > >> in the thread have talked about how there are often conflicts between
>  > >> branches due to the changelog, but the other common reason for
>  > >> conflicts is the release field in my experience. If we use tags as an
>  > >> explicit "I want this to go to users", then it solves both
> problems (I
>  > >> consider sending all commits to end users a problem, because I often
>  > >> make refactor commits that I would not want to churn users on.)
>  > >
>  > > The tag also provides a nice place to write release notes for the
>  > > update. I suppose you could also add support for some sort of text tag
>  > > inside commits (like when you mark a commit as fixing an issue in
>  > > Git{Lab,Hub} and look at the commits between the new tag and old one so
>  > > selective git commits could get sucked into the changelog as well.
>  >
>  > We've tossed around using tags for builds before in another context, but
>  > the idea of tag annotations for populating the user-visible changelog is
>  > an interesting and a totally novel idea AFAIK.
>  >
>  > On top of using tags to, well, tag content for building (it seems so
>  > natural nothing could be more natural), we talked about calculating the
>  > release number automatically from number of commits on that branch since
>  > the last tag. The details seem to largely evade me, but changelog
>  > population was planned around picking messages out of git commit
>  > messages. Which has its issues. The tag annotations probably has its
>  > own, but it's indeed an intriguing idea.
>  >
>  
>  This was the discussion Igor and I had at the openSUSE Summit in May.
>  The unanswered question I had was if we can manipulate the data
>  attached to a tag in Pagure UI and edit it after it was initially
>  pushed. If annotations are also frozen like all other things in
>  Dist-Git, then it fails as a usable mechanism.
>   
Or just make another tag with a newer release number. Of course, that
doesn't work with the auto-generated release number based on
commits-since-last-tag, but to me that's a "nice to have". You could
have it do that if a release isn't specified in the tag, though, so you
could have it all.

[Panu Matilainen]

On 9/27/19 5:57 PM, Neal Gompa wrote:
>  On Fri, Sep 27, 2019 at 9:54 AM Panu Matilainen
> <pmatilai(a)redhat.com&gt; wrote:
> >
> > On 9/26/19 10:05 PM, Jeremy Cline wrote:
> >> On Thu, Sep 26, 2019 at 02:57:56PM -0400, Randy Barlow wrote:
> >>> On Thu, 2019-09-26 at 14:49 +0000, Jeremy Cline wrote:
> >>>> The combination of these two makes no sense to me. I do plenty of
> >>>> work
> >>>> where I don't want to build it (specfile cleanup, patches,
> >>>> configuration
> >>>> changes, etc.). I want a build that goes to users be explicit.
> >>>>
> >>>> A better model, in my opinion, is to build every *tag*. To do a new
> >>>> kernel build I could make a tag like "kernel-5.4-rc1..." and
> the tag
> >>>> would be parsed into the specfile's NVR and built.
> >>>
> >>> I agree, and I really like the alternative suggestion here. Some people
> >>> in the thread have talked about how there are often conflicts between
> >>> branches due to the changelog, but the other common reason for
> >>> conflicts is the release field in my experience. If we use tags as an
> >>> explicit "I want this to go to users", then it solves both problems
> (I
> >>> consider sending all commits to end users a problem, because I often
> >>> make refactor commits that I would not want to churn users on.)
> >>
> >> The tag also provides a nice place to write release notes for the
> >> update. I suppose you could also add support for some sort of text tag
> >> inside commits (like when you mark a commit as fixing an issue in
> >> Git{Lab,Hub} and look at the commits between the new tag and old one so
> >> selective git commits could get sucked into the changelog as well.
> >
> > We've tossed around using tags for builds before in another context, but
> > the idea of tag annotations for populating the user-visible changelog is
> > an interesting and a totally novel idea AFAIK.
> >
> > On top of using tags to, well, tag content for building (it seems so
> > natural nothing could be more natural), we talked about calculating the
> > release number automatically from number of commits on that branch since
> > the last tag. The details seem to largely evade me, but changelog
> > population was planned around picking messages out of git commit
> > messages. Which has its issues. The tag annotations probably has its
> > own, but it's indeed an intriguing idea.
> >
>  
>  This was the discussion Igor and I had at the openSUSE Summit in May.
>  The unanswered question I had was if we can manipulate the data
>  attached to a tag in Pagure UI and edit it after it was initially
>  pushed. If annotations are also frozen like all other things in
>  Dist-Git, then it fails as a usable mechanism. 
That's not the discussion I was referring to as I wasn't there :)
I had this with ffesti sometime last year. Anyway...

>  
> > In fact it was that discussion which prompted the development of
> > automatic patch numbering and %patchlist support in spec files in rpm
> > 4.15, since in the planned scheme merge conflicts on release number and
> > changelog would be gone, and conflicts on patch numbers was identified
> > as yet another redundant piece of data that's also often prone to
> > unnecessary merge conflicts.
> >
> > The %changelog in specs really, really needs to die.
> >
>  
>  I would actually say that the spec in the SRPM should contain the
>  changelog, so that repeating the build from the exploded contents is
>  possible. But yes, it'd be nice if it wasn't there in the specs in
>  Git. 
Oh, sure. In the name of reproducability, the SRPM needs to have a 
physical copy of it one way or the other.

To clarify, I specifically mean %changelog in git maintained specs must 
die. It's just not as catchy ;)

	- Panu -

>  
>

[Randy Barlow]

On Thu, 2019-09-26 at 14:49 +0000, Jeremy Cline wrote:
>  > ○ Every commit to dist-git (ie: PR merged) is automatically
> built
>  > in koji
>  > ○ Every build in koji results automatically in an update in bodhi
>  
>  The combination of these two makes no sense to me. I do plenty of
>  work
>  where I don't want to build it (specfile cleanup, patches,
>  configuration
>  changes, etc.). I want a build that goes to users be explicit. 
This is also a problem for multi-package updates. I frequently do chain
builds for changes that must go out together. It would be bad if one
went out and another didn't. Also, the tests won't pass on some of
them, which will guarantee that some will go out without the others.

Sending builds to users should be explicit.

Also, I don't see value in making builds that don't go to users, and
doing that consumes resources.

[Miro Hrončok]

On 26. 09. 19 11:36, Pierre-Yves Chibon wrote:
>  ○ Every changes to dist-git is done via pull-requests 
I like this idea, however as other has pointed out, it needs to be much smoother 
experience than now.

See for example this RFE:

number 3) 
https://lists.fedoraproject.org/archives/list/[e-mail].o...

and also https://github.com/packit-service/packit/issues/131


Anyway, about our workflow, it would be awesome, if I can create 1 PR and have 
it cross tested and applied where I want. See our (Python) user story for packit:

https://github.com/packit-service/packit/issues/131



-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Vitaly Zaitsev]

On 26.09.2019 11:36, Pierre-Yves Chibon wrote:
>  ○ Every changes to dist-git is done via pull-requests 
No way! This is a terrible idea.

>  ○ Every build in koji results automatically in an update in bodhi
Not good too.

-- 
Sincerely,
  Vitaly Zaitsev (vitaly(a)easycoding.org)

[Robbie Harwood]

Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; writes:

>  Here is what the vision we came to and that we would like to
> discuss:
>
>  ○ Every changes to dist-git is done via pull-requests
>  ○ Pull-requests are automatically tested
>  ○ Every commit to dist-git (ie: PR merged) is automatically built in koji
>  ○ Every build in koji results automatically in an update in bodhi
>  ○ Every update in bodhi is automatically tested
>  ○ If the tests pass, the update is automatically pushed to the repository 
I have a *lot* of objections to this workflow, but maybe it'd be better
to take a step back.

Instead of trying to create a new workflow, why are you not starting
with defining what we have now?  And if there are problems with it that
need to be addressed, what are they and why do they motivate these
changes to you?

Thanks,
--Robbie

[Troy Dawson]

On Thu, Sep 26, 2019 at 10:48 AM Robbie Harwood <rharwood(a)redhat.com&gt; wrote:
>  Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; writes:
>
>  > Here is what the vision we came to and that we would like to discuss:
>  >
>  > ○ Every changes to dist-git is done via pull-requests
>  > ○ Pull-requests are automatically tested
>  > ○ Every commit to dist-git (ie: PR merged) is automatically built in koji
>  > ○ Every build in koji results automatically in an update in bodhi
>  > ○ Every update in bodhi is automatically tested
>  > ○ If the tests pass, the update is automatically pushed to the repository
>
>  I have a *lot* of objections to this workflow, but maybe it'd be better
>  to take a step back.
>
>  Instead of trying to create a new workflow, why are you not starting
>  with defining what we have now?  And if there are problems with it that
>  need to be addressed, what are they and why do they motivate these
>  changes to you?
>  
I'd like to second this question.
What are the problems with our current workflows?
At the beginning of this message you said there were several packaging
initiatives being worked on, but for those of us that didn't go to
Flock, we don't know what those are.
Basically, we (those not in the room with you) are starting here on
earth, while you are talking about how to properly breath while on the
moon.  There is alot of disconnect between where we are and what you
are talking about.

[Michael J Gruber]

There is the current worklflow and the current mindset. One influences the other.

For a long-time gitter, the prevailing Fedora packager mindset is still very much
"dist-cvs". dist-git is often used as merely a tool to drive
"dist-something", not so much as a vcs, and really rarely as a tool for
collaborative package management. Here are just a few examples:

- Commit messages often explain almost nothing, in particular not the "why".
(You can see the "what" from the diff...)
- Branches for different releases are often used like separate repos without any
merge-down or merge-up concept; changes cherry-picked rather than merged; release-specific
differences implented by %if's in spec rather than differences between branches.
- Discussion is on the ML or in BZ with no connection to dist-git (unless one mentions bz
# in the commit).
- Changelog in spec is completely strange from a git point of view, it even makes merging
unnecessarily hard; and no, th contained info usually is not any better than the commit
message. Oh, did I mention bodhi update notes? What a redundancy.

So, a a new worklflow which "encourages" a proper git workflow is most welcome.

[Robbie Harwood]

"Michael J Gruber" <mjg(a)fedoraproject.org&gt; writes:

>  There is the current worklflow and the current mindset. One
> influences
>  the other.
>
>  For a long-time gitter, the prevailing Fedora packager mindset is
>  still very much "dist-cvs". dist-git is often used as merely a tool to
>  drive "dist-something", not so much as a vcs, and really rarely as a
>  tool for collaborative package management. Here are just a few
>  examples: 
Thank you for the reply.

>  - Commit messages often explain almost nothing, in particular not
> the
>  "why". (You can see the "what" from the diff...)
>
>  - Branches for different releases are often used like separate repos
>  without any merge-down or merge-up concept; changes cherry-picked
>  rather than merged; release-specific differences implented by %if's in
>  spec rather than differences between branches.
>
>  - Discussion is on the ML or in BZ with no connection to dist-git
>  (unless one mentions bz # in the commit).
>
>  - Changelog in spec is completely strange from a git point of view, it
>  even makes merging unnecessarily hard; and no, th contained info
>  usually is not any better than the commit message. Oh, did I mention
>  bodhi update notes? What a redundancy.
>
>  So, a a new worklflow which "encourages" a proper git workflow is most
>  welcome. 
Leaving aside for the moment whether I agree that these are issues, none
of the changes proposed relate to the problems as far as I can see.  For
context, the changes proposed were:

>  ○ Every changes to dist-git is done via pull-requests
>  ○ Pull-requests are automatically tested
>  ○ Every commit to dist-git (ie: PR merged) is automatically built in
>  koji
>  ○ Every build in koji results automatically in an update in bodhi
>  ○ Every update in bodhi is automatically tested
>  ○ If the tests pass, the update is automatically pushed to the
>  repository 
None of these relate to branching, spec storage, or "proper" git
workflow (especially in the case of a single-maintainer package), and
nothing has been added to connect ML/BZ discussion with the commits to
dist-git, or even improve commit message quality.

Thanks,
--Robbie

[Pierre-Yves Chibon]

On Thu, Sep 26, 2019 at 11:30:28AM -0700, Troy Dawson wrote:
>  On Thu, Sep 26, 2019 at 10:48 AM Robbie Harwood
> <rharwood(a)redhat.com&gt; wrote:
>  >
>  > Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; writes:
>  >
>  > > Here is what the vision we came to and that we would like to discuss:
>  > >
>  > > ○ Every changes to dist-git is done via pull-requests
>  > > ○ Pull-requests are automatically tested
>  > > ○ Every commit to dist-git (ie: PR merged) is automatically built in koji
>  > > ○ Every build in koji results automatically in an update in bodhi
>  > > ○ Every update in bodhi is automatically tested
>  > > ○ If the tests pass, the update is automatically pushed to the repository
>  >
>  > I have a *lot* of objections to this workflow, but maybe it'd be better
>  > to take a step back.
>  >
>  > Instead of trying to create a new workflow, why are you not starting
>  > with defining what we have now?  And if there are problems with it that
>  > need to be addressed, what are they and why do they motivate these
>  > changes to you?
>  >
>  
>  I'd like to second this question.
>  What are the problems with our current workflows?
>  At the beginning of this message you said there were several packaging
>  initiatives being worked on, but for those of us that didn't go to
>  Flock, we don't know what those are. 
There was a talk from Neil that has been linked to elsewhere in this thread
which is an example of ideas impacting the packager workflow.
There is the CI work which is impacting the packager workflow.
These two are the ones that I come up from the top of my head but I'm sure there
are more.

>  Basically, we (those not in the room with you) are starting here on
>  earth, while you are talking about how to properly breath while on the
>  moon.  There is alot of disconnect between where we are and what you
>  are talking about. 
Sorry if that's the feeling I'm giving, it's really not what I was trying to
convey.
What I wanted to do is to have a discussion to discuss what we would like the
packaging experience to be in Fedora.
I thought that having a strawman proposal of what such vision could be would be
a good starting point as it would give us something to discuss, change and
adjust so we can come up with some sort of consensus ideal packaging workflow in
Fedora.

This way when we make changes to the packager workflow we have something to
measure the impact of the changes toward the long term objective (ie: is this
change a step forward the ideal packager workflow or something that is a step
back and therefore will need to be adjusted for later?).


Does that make sense?

Pierre

[Pierre-Yves Chibon]

On Wed, Oct 02, 2019 at 03:29:23AM +0200, Pierre-Yves Chibon wrote:
>  On Thu, Sep 26, 2019 at 11:30:28AM -0700, Troy Dawson wrote:
>  > On Thu, Sep 26, 2019 at 10:48 AM Robbie Harwood <rharwood(a)redhat.com&gt; wrote:
>  > >
>  > > Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; writes:
>  > >
>  > > > Here is what the vision we came to and that we would like to discuss:
>  > > >
>  > > > ○ Every changes to dist-git is done via pull-requests
>  > > > ○ Pull-requests are automatically tested
>  > > > ○ Every commit to dist-git (ie: PR merged) is automatically built in koji
>  > > > ○ Every build in koji results automatically in an update in bodhi
>  > > > ○ Every update in bodhi is automatically tested
>  > > > ○ If the tests pass, the update is automatically pushed to the repository
>  > >
>  > > I have a *lot* of objections to this workflow, but maybe it'd be better
>  > > to take a step back.
>  > >
>  > > Instead of trying to create a new workflow, why are you not starting
>  > > with defining what we have now?  And if there are problems with it that
>  > > need to be addressed, what are they and why do they motivate these
>  > > changes to you?
>  > >
>  > 
>  > I'd like to second this question.
>  > What are the problems with our current workflows?
>  > At the beginning of this message you said there were several packaging
>  > initiatives being worked on, but for those of us that didn't go to
>  > Flock, we don't know what those are.
>  
>  There was a talk from Neil that has been linked to elsewhere in this thread 
Oups, the talk was from Neal and Igor. Sorry for the typo there.


Pierre

[Christopher]

On Thu, Sep 26, 2019 at 8:33 AM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:
>  Good Morning Everyone,
>
>  At Flock, a few of us met to discuss a future vision of the packager workflow.
>  This discussion was triggered by the realization that a number of initiatives
>  are happening around packaging in Fedora but there is no real shared vision on
>  what we want the packager UX/workflow to be.
>  The lack of vision on the packager workflow means we could deploy something
>  today, thinking it is the improvement over the current workflow but would
>  prevent us from (or make it harder to) doing other changes afterwords that would
>  be even more beneficial.. 
In general, I like that people are thinking about this, but the
proposed workflow doesn't seem that great to me.

>  So once that concern was raised, we took some time during the Fedora
>  Infrastructure hackfest to gather the people interested around a white board and
>  brainstorm on what a future packager workflow could look like.
>
>  We tried not to link this process to any tool in particular as well as focus on
>  the what and why rather than any how.
>
>  Here is what the vision we came to and that we would like to discuss:
>
>  ○ Every changes to dist-git is done via pull-requests 
Yuck. This seems quite terrible to me. Pull requests are great for
collaboration, but they add extra steps to working with git that isn't
necessary during routine maintenance.

>  ○ Pull-requests are automatically tested
>  ○ Every commit to dist-git (ie: PR merged) is automatically built in koji
>  ○ Every build in koji results automatically in an update in bodhi
>  ○ Every update in bodhi is automatically tested
>  ○ If the tests pass, the update is automatically pushed to the repository 
These automated steps seem like they could be very problematic.
Updates in git can currently represent incremental progress. For
example, I might want to merge 5 PRs and then manually add a changelog
entry, before doing a build.

>  For this workflow to work nicely we need to fix a few things first:
>
>  - We need to work on the change logs in the spec files, as otherwise
>    pull-requests are going to conflict more often than not 
With version-controlled package sources, changelogs in SPEC seem so
obsolete to me. They are already problematic today when they conflict
due to changes in multiple branches.

>  - We need to fine a place to insert the end-user information about an
> update (in
>    the PR description?)
>  - When a group of people want to collectively work on a change (involving one or
>    more packages), they’ll need a place to collaborate. This could be someone’s
>    fork or, preferably, a fork of the project collectively maintained by that
>    group.
>      For example: the python SIG would have a group with forks of the different
>      python packages they are working on
>  - We need to figure out how to enable opening a pull-request again multiple
>    branches at one. 
That's not a good way to use git. This seems almost certainly likely
to result in cherry-pick'd branches. Git has excellent branch merging
features. Merging branches, rather than cherry-picking across them,
results in commits representing the same changeset being present in
multiple branches, making it easier to search git history and identify
which branches contain a specific change. Multiple PRs or
cherry-picked commits make this harder, and more likely to result in
regressions in some branches.

A better way to use git would be to patch the oldest version you
intend to patch, and merge that branch forward into newer branches,
merging any conflicts along the way. This will help avoid regressions
(if all changes are merged) and will make it easier to see which
branches contain a particular commit. Though, some people don't like
seeing merge commits.

>  Why should we agree now on a long term vision?
>
>  A concrete example to this, I have been playing with the idea to get ride of git
>  branches in dist-git for a while now. Basically use git as it is meant to be
>  used: have one branch for development (most often master) and branch of master
>  when you need to, not simply because a new version of Fedora happened.
>  However, if we implemented this, we would break the workflow described above as
>  suddenly there would be no way to specify that a given PR is meant to land in a
>  certain version of Fedora or another (Is this PR for rawhide? F31? F30? All of
>  them? Some of them?).
>
>
>  Having this idea of where we want to go should give us a goal as well as way to
>  reflect if changes we think are good in isolation would actually help achieving
>  this vision or not.
>
>  This proposal was based on the brainstorming session a few of us had at flock,
>  but before we can see how to implement it we need more inputs and this is where
>  you come in :)
>
>  Do you like this vision? Would you change some pieces of it? Would you change it
>  entirely?
>  In an ideal world, what would packaging software look like to you? 
I just don't see this proposed workflow as solving the biggest
problems that packagers face. For me, I think the biggest problem that
packagers (particularly newer packagers) face is discovery of all the
services involved in the packaging workflow, and the need to visit
separate UIs to get answers.

Rather than try to automate the workflow through these separate
services, I'd rather work be done to create a central packager
interface that helps packagers track where a package is in the
workflow, and to facilitate moving from one phase to the next. For
example, an interface could show something like a list of branches for
a package, and whether there is a corresponding build from those
branches. A button to the right of each branch could be shown which
does the equivalent of `fedpkg build` on that branch (grayed out if
there exists a build already for that commit and branch), and then
another column with a similar button to create an update.

In addition to helping the packager move a package through the
workflow, such a centralized interface would also provide information
about dependencies, their orphan/retired status, whether they've
changed since the last build. It could also show a listing of bug
reports, help maintain co-maintainer privileges, and other similar
stuff. Basically, everything the packager needs in one place.

I think such an interface would be far more useful, just to document
the existing workflow, and help packagers know what to do, than trying
to change the workflow and automate it. We shouldn't be looking at
disruptive changes to the workflow... we should be looking at lowering
the bar to packaging by making it easier to manage the current
workflow. And then, if we find that there are changes to the workflow
itself, we do that later, with corresponding changes to the
centralized interface.

>
> Looking forward to the discussion,
>
> Pierre
> _______________________________________________
> devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...

[Jason L Tibbitts III]

> >>>> "C" == Christopher 
> <ctubbsii(a)fedoraproject.org&gt; writes: 
C> With version-controlled package sources, changelogs in SPEC seem so
C> obsolete to me. They are already problematic today when they conflict
C> due to changes in multiple branches.

It's important to note that the RPM changelog is rather a different
thing from a list of commits made to the specfile.  If I went and
re-indented the spec, that would of course get a commit message but has
no relevance to the end user and should not result in a changelog
entry.  Often I make several commits to the spec, and then update the
%changelog section with a summary of user-relevant changes only when I'm
ready to bump the release.

It seems that generating %changelog from git commits would require
something like magic tags in the commit messages, which seems like
something of a pain.

 - J<

[Miro Hrončok]

On 26. 09. 19 22:06, Jason L Tibbitts III wrote:
> >>>>> "C" == Christopher 
> <ctubbsii(a)fedoraproject.org&gt; writes:
>  
>  C> With version-controlled package sources, changelogs in SPEC seem so
>  C> obsolete to me. They are already problematic today when they conflict
>  C> due to changes in multiple branches.
>  
>  It's important to note that the RPM changelog is rather a different
>  thing from a list of commits made to the specfile.  If I went and
>  re-indented the spec, that would of course get a commit message but has
>  no relevance to the end user and should not result in a changelog
>  entry.  Often I make several commits to the spec, and then update the
>  %changelog section with a summary of user-relevant changes only when I'm
>  ready to bump the release.
>  
>  It seems that generating %changelog from git commits would require
>  something like magic tags in the commit messages, which seems like
>  something of a pain. 
Or, as said elsewhere in this thread, we could use annotated tags.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Matthias Runge]

On 27/09/2019 00:04, Miro Hrončok wrote:
>  On 26. 09. 19 22:06, Jason L Tibbitts III wrote:
> >>>>>> "C" == Christopher  <ctubbsii(a)fedoraproject.org&gt;
> writes:
> >
> > C> With version-controlled package sources, changelogs in SPEC seem so
> > C> obsolete to me. They are already problematic today when they conflict
> > C> due to changes in multiple branches.
> >
> > It's important to note that the RPM changelog is rather a different
> > thing from a list of commits made to the specfile.  If I went and
> > re-indented the spec, that would of course get a commit message but has
> > no relevance to the end user and should not result in a changelog
> > entry.  Often I make several commits to the spec, and then update the
> > %changelog section with a summary of user-relevant changes only when I'm
> > ready to bump the release.
> >
> > It seems that generating %changelog from git commits would require
> > something like magic tags in the commit messages, which seems like
> > something of a pain.
>  
>  Or, as said elsewhere in this thread, we could use annotated tags.
>   
The other possibility would be to squash commits into a single one.
Especially doing PR style makes me see commit messages like "another 
fix" or "fix typo" a lot more often compared to squashed commits, where 
one is working on a feature to be submitted as one commit (unlike 50 
commits).

Matthias

[Dan Čermák]

Christopher <ctubbsii(a)fedoraproject.org&gt; writes:

>  I just don't see this proposed workflow as solving the biggest
>  problems that packagers face. For me, I think the biggest problem that
>  packagers (particularly newer packagers) face is discovery of all the
>  services involved in the packaging workflow, and the need to visit
>  separate UIs to get answers.
>
>  Rather than try to automate the workflow through these separate
>  services, I'd rather work be done to create a central packager
>  interface that helps packagers track where a package is in the
>  workflow, and to facilitate moving from one phase to the next. For
>  example, an interface could show something like a list of branches for
>  a package, and whether there is a corresponding build from those
>  branches. A button to the right of each branch could be shown which
>  does the equivalent of `fedpkg build` on that branch (grayed out if
>  there exists a build already for that commit and branch), and then
>  another column with a similar button to create an update.
>
>  In addition to helping the packager move a package through the
>  workflow, such a centralized interface would also provide information
>  about dependencies, their orphan/retired status, whether they've
>  changed since the last build. It could also show a listing of bug
>  reports, help maintain co-maintainer privileges, and other similar
>  stuff. Basically, everything the packager needs in one place.
>
>  I think such an interface would be far more useful, just to document
>  the existing workflow, and help packagers know what to do, than trying
>  to change the workflow and automate it. We shouldn't be looking at
>  disruptive changes to the workflow... we should be looking at lowering
>  the bar to packaging by making it easier to manage the current
>  workflow. And then, if we find that there are changes to the workflow
>  itself, we do that later, with corresponding changes to the
>  centralized interface. 
What you want is essentially the Open Build Service
(https://build.opensuse.org/), just for Fedora.

And yes, having something like that would be *really* good, especially
having all the overviews, build states, updates, dependencies, etc. in
one single place.

The only danger in this is, that this overview app can grow into a huge
monolith, that becomes impossible to extend at some point because it
swallowed all the functionality of our individual infra and became too
complex... But that can be avoided, if it is "just" an overview and the
rest stays the way it is.

[Miro Hrončok]

On 26. 09. 19 11:36, Pierre-Yves Chibon wrote:
>  ○ Every commit to dist-git (ie: PR merged) is automatically built in
> koji 
If we go this way, there must be a way to disable this or to dictate what side 
tag this needs to be built in.

I remember that during the Python 3.8 rebuilds in the side tag, one package had 
this automated somehow already. I was bumping the release/changelog and trying 
to build it in the side tag at least 5 times, but I was building with 
--background, so the automation always got the NEVR in Koji first.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Fabien Boucher]

On Fri, Sep 27, 2019 at 12:03 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:

>  I remember that during the Python 3.8 rebuilds in the side tag, one
>  package had
>  this automated somehow already. I was bumping the release/changelog and
>  trying
>  to build it in the side tag at least 5 times, but I was building with
>  --background, so the automation always got the NEVR in Koji first.
>
>  
I guess it is python-gear ? Where I'm experimenting some automation based
on Zuul CI.
There is a job triggered to build in Koji when master ref change. This job
can be trigger
instead when a PR is closed/merged, that way direct push will not trigger a
build.

[Miro Hrončok]

On 27. 09. 19 11:50, Fabien Boucher wrote:
>      I remember that during the Python 3.8 rebuilds in the side tag,
> one package had
>      this automated somehow already. I was bumping the release/changelog and trying
>      to build it in the side tag at least 5 times, but I was building with
>      --background, so the automation always got the NEVR in Koji first.
>  
>  I guess it is python-gear ? Where I'm experimenting some automation based on 
>  Zuul CI.
>  There is a job triggered to build in Koji when master ref change. This job can 
>  be trigger
>  instead when a PR is closed/merged, that way direct push will not trigger a build.
Yes. I've em-mailed you about the problem when it was happening, asking you to 
disable it, there was no reply and I managed to build it at the end.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Fabien Boucher]

On Fri, Sep 27, 2019 at 11:56 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:

>  Yes. I've em-mailed you about the problem when it was happening,
> asking
>  you to
>  disable it, there was no reply and I managed to build it at the end.
>
>  
So, I apologize. I missed your email.

[Richard W.M. Jones]

On Thu, Sep 26, 2019 at 11:36:10AM +0200, Pierre-Yves Chibon wrote:
>  ○ Every changes to dist-git is done via pull-requests 
Erm, no thank you.  Pull requests are a terrible workflow.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

[Matthew Miller]

On Wed, Oct 02, 2019 at 05:31:56PM +0100, Richard W.M. Jones wrote:
>  > ○ Every changes to dist-git is done via pull-requests
>  Erm, no thank you.  Pull requests are a terrible workflow. 
It's definitely the winning workflow in the open source world today,
particularly for smaller and drive-by contributions, which I think we'd
like to encourage. And there _are_ real tracking and review benefits to
having everything go through that workflow.

Do you have an alternative proposal? 


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Emmanuel Seyman]

* Matthew Miller [02/10/2019 13:33] :
>                     And there _are_ real tracking and review benefits to
>  having everything go through that workflow.
>  
>  Do you have an alternative proposal?  
I'm fine with the workflow we have now. Small and drive-by contributions
can by contributed via pull requests while core contributers (can) commit
directly to master if they so wish.

Emmanuel

[Michal Schorm]

What is IMHO crucial is the understanding that the part of Fedora
infrastructure facing maintainers *needs* to respect the fact that
each (upstream) project is different, has different workflow, version
control system, way to make releases and to ship them, ... . Thus
Fedora must allow different approaches for different package(r)s.
A lot of automation would be appreciated, but it would turn to great
pain if enforced.

So, allow to go with PR-only workflow. Allow the PRs to be tested and
allow them to be automatically merged after passed testing, if the
maintainer sets so.
Opening PRs by release monitoring would be nice. Though the release
monitoring doesn’t work for my pkgs, since upstream does a series of
changes, git submodules fetching, and patching by other projects
before they release the tarball meant to be used as the sources.
Though it takes days to weeks after the upstream mark a certain commit
as “this we will release as version X.Y.Z.”, before they prepare the
tarball and test it for every channel they support.

Also, I *strongly believe*, we - as a significant GNU/Linux distro -
should use tools as are intended, instead of contrariwise.
And I believe the Git commit messages weren’t ever supposed to contain
magic words which will be parsed by some other tooling in order to
take actions. The annotated tags, on the other hand, seems (to me)
like a brilliant idea allowing us to kill the changelog that so much
maintainers desire to.

At the same time, there is so little documentation about “the right
way” to do things, as they are meant to. And only a small amount of
packagers knows the right ways and even less are taking them.
Commit messages are often useless as they don’t explain what changed,
why, and how. Patches doesn’t have any comments. Specfiles often
doesn’t have any good comments too - what the patch does, what for is
this dependency, why is that bundled, why is that test disabled, why
the build flags override … . (not only just) Provenpackages walking
around introducing changes that doesn’t make sense and doesn’t even
have a good justifications. Some people don't even care about e.g.
mentioning bundled part of the code … .
Working with other people’s packages are really hard. But IMHO, mostly
because of the people … .

As I work with newcomers, I see that it is really hard for new faces
to start - both before becoming packagers and even after that.
A huge number of tools and places they *should ideally* work with -
dist git, bodhi, bugzilla, PRs, pagure (to reach rel-engs), FAS,
koschei, abrt, anytia, modularity, wiki pages about the packages, wiki
pages about the changes, mailing lists, IRC, rpmlint, testing, … .
Even I find difficult to explore the full toolset that Fedora offers
and after years I still have no idea what the infrastructure looks
like, or where should I look it up.

And what is IMHO missing is the easy to explore documentation, or at
least some map of tools, ways and places with a “you are standing
here” mark and easy way to understand how to get where you want to be.
A huge FAQ (which would just link to the part of the documentation
with the answer) might help too, as I constantly see people new around
me asking the same questions again and again. Example questions: “can
I delete a branch? How to set EOL date for a module - it still keeps
building. How to write this and that in RPM. what to write to
changelog, what to commit message, what to bodhi update message, … ?
What does men the “rm -rf $RPM_BUILD_ROOT” in %install section? What
does that macro means? How do I find macro for this? Never heard of
“%{?systemd_requires}”, what’s that good for? Hey, you made that
scratch build just for aarch64 - how? ...”



All other observations, based on previous 200 mails, I’d summarize as:

*Please do:*
- kill changelog, please
   - by annotated git tags, please
- Unificate the authentication methods
- Make release monitoring submit PRs instead of patches to bugzilla
- Give packagers tools for working with the distribution (e.g. show me
my dependency tree.)
- Tools to maintain the package as a part of ditro, instead of alone
   - "Give me a list of all the things my package depends on, what
hasn't been updated in X years"
   - “rebuild things that depend on my package upon this commit”
- unified web UI with all the information & build states and a unified
CLI client for doing literally everything.
- Allow to adjusts packager’s workflow as he wants (in some mantinels)
via automatization
   - here I mean mostly automated testing, reporting, bodhi updates
(templates?), PR merging

*Please don’t:*
- require {most of the ideas} (like PR-only access, single branch,
source git, every commit is built...)
   - because they just doesn't work for everybody
- Kill ancient artifact around Fedora packaging (e.g. “rm -rf
$RPM_BUILD_ROOT” in `rpmdev-newspec`, about which newcommers still
asking )

--

Michal Schorm
Software Engineer
Core Services - Databases Team
Red Hat

[Fabio Valentini]

On Wed, Oct 2, 2019 at 7:33 PM Matthew Miller <mattdm(a)fedoraproject.org&gt; wrote:
>  On Wed, Oct 02, 2019 at 05:31:56PM +0100, Richard W.M. Jones wrote:
>  > > ○ Every changes to dist-git is done via pull-requests
>  > Erm, no thank you.  Pull requests are a terrible workflow.
>
>  It's definitely the winning workflow in the open source world today,
>  particularly for smaller and drive-by contributions, which I think we'd
>  like to encourage. And there _are_ real tracking and review benefits to
>  having everything go through that workflow. 
As others in the thread have pointed out, mandatory pull requests just
make no sense for most single-maintainer projects, which most packages
probably are.

>  Do you have an alternative proposal? 
As long as PRs are opt-in, for example, if you explicitly *want* to
trigger some tests or have somebody review your changes, then I'm all
for it. The Stewardship SIG is using that exact workflow right now.

But forcing people to open and triage Pull Requests for every single
packaging change is *crazy* and will probably make it impossible for
me to continue maintaining the (pretty big) number of packages I own.

Fabio

>
> --
> Matthew Miller
> <mattdm(a)fedoraproject.org&gt;
> Fedora Project Leader
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Colin Walters]

On Wed, Oct 2, 2019, at 1:40 PM, Fabio Valentini wrote:

>  As others in the thread have pointed out, mandatory pull requests
> just
>  make no sense for most single-maintainer projects, which most packages
>  probably are. 
Well, a lot of this relates to what the *merge policy* is.  If a PR submitter can merge
their own PRs, and there's a mechanism to do "merge when tests pass" (this
is an important aspect), then submitting a PR can be just about as equally ergonomic as
`git push`.
In OpenShift we use Prow, which has the latter; I really like it.  However we also
*require* peer review (submitters can't merge their own PRs).  I'd like to require
review, but it does seem like a prerequisite is moving away from the one-repo-per-package
model.

[Robbie Harwood]

"Colin Walters" <walters(a)verbum.org&gt; writes:

>  On Wed, Oct 2, 2019, at 1:40 PM, Fabio Valentini wrote:
>
> > As others in the thread have pointed out, mandatory pull requests
> > just make no sense for most single-maintainer projects, which most
> > packages probably are.
>
>  Well, a lot of this relates to what the *merge policy* is.  If a PR
>  submitter can merge their own PRs, and there's a mechanism to do
>  "merge when tests pass" (this is an important aspect), then submitting
>  a PR can be just about as equally ergonomic as `git push`. 
With about six more emails about it, sure.  And another piece of
infrastructure that has to be up and bug-free.

Even the gating and bodhi emails today are rather a lot: I don't want to
be notified if everything worked correctly.

>  In OpenShift we use Prow, which has the latter; I really like it.
>  However we also *require* peer review (submitters can't merge their
>  own PRs).  I'd like to require review, but it does seem like a
>  prerequisite is moving away from the one-repo-per-package model. 
It also requires people to do the review, which many packages don't
have.  For many teams, this would more than double the time spent on
packaging.  That's not tenable.

Thanks,
--Robbie

[Pierre-Yves Chibon]

On Wed, Oct 02, 2019 at 03:17:33PM -0400, Robbie Harwood wrote:
>  With about six more emails about it, sure.  And another piece of
>  infrastructure that has to be up and bug-free.
>  
>  Even the gating and bodhi emails today are rather a lot: I don't want to
>  be notified if everything worked correctly. 
Sorry for side-tracking this here. We've cut down on the number of emails bodhi
sends for rawhide. If this is still too many please open a ticket so we can
discuss and see if we find a consensus on this.

Thanks,
Pierre

[Robbie Harwood]

Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; writes:

>  On Wed, Oct 02, 2019 at 03:17:33PM -0400, Robbie Harwood wrote:
>
> > With about six more emails about it, sure.  And another piece of
> > infrastructure that has to be up and bug-free.
> > 
> > Even the gating and bodhi emails today are rather a lot: I don't want to
> > be notified if everything worked correctly.
>
>  Sorry for side-tracking this here. We've cut down on the number of
>  emails bodhi sends for rawhide. If this is still too many please open
>  a ticket so we can discuss and see if we find a consensus on this. 
I appreciate that you have reduced them.  I would like that number to be
zero unless something has gone wrong - I imagine this is not something
there will be consensus around.

Thanks,
--Robbie

[Neal Gompa]

On Wed, Oct 2, 2019 at 2:45 PM Colin Walters <walters(a)verbum.org&gt; wrote:
>  On Wed, Oct 2, 2019, at 1:40 PM, Fabio Valentini wrote:
>
>  > As others in the thread have pointed out, mandatory pull requests just
>  > make no sense for most single-maintainer projects, which most packages
>  > probably are.
>
>  Well, a lot of this relates to what the *merge policy* is.  If a PR submitter can merge
> their own PRs, and there's a mechanism to do "merge when tests pass" (this
> is an important aspect), then submitting a PR can be just about as equally ergonomic as
> `git push`.
>  In OpenShift we use Prow, which has the latter; I really like it.  However we also
> *require* peer review (submitters can't merge their own PRs). 
Unfortunately, it doesn't scale for the large number of packages we
have. Pull requests would work if we had mergify[1] working on
Dist-Git, otherwise I can't see how it'd work.

[1]: https://github.com/Mergifyio/mergify-engine

>  I'd like to require review, but it does seem like a prerequisite
> is moving away from the one-repo-per-package model. 
No it isn't. A pre-requisite would be that we'd require maintainer
teams, and have to make those first-class in Fedora (rather than
barely third-class as they are now). Besides, I've worked in
distributions where you have monorepo package source control, and it's
arguably worse. Rolling through the revisions is difficult, dealing
with searching through the tree is hard, and features like git blame
basically don't work (they time out more often than they return
results). Meaning that we'd need groups and subgroups that have ACL
inheritance for projects, among other things.

I'm surprised you didn't realize these issues. You've examined Git
very deeply and you should be more than aware of how bad of idea it
would be to use a monorepo for package sources. We don't have separate
Git repos per package for no reason...


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Neal Gompa]

On Wed, Oct 2, 2019 at 3:18 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Wed, Oct 2, 2019 at 2:45 PM Colin Walters <walters(a)verbum.org&gt; wrote:
>  >
>  >
>  >
>  > On Wed, Oct 2, 2019, at 1:40 PM, Fabio Valentini wrote:
>  >
>  > > As others in the thread have pointed out, mandatory pull requests just
>  > > make no sense for most single-maintainer projects, which most packages
>  > > probably are.
>  >
>  > Well, a lot of this relates to what the *merge policy* is.  If a PR submitter can
> merge their own PRs, and there's a mechanism to do "merge when tests pass"
> (this is an important aspect), then submitting a PR can be just about as equally ergonomic
> as `git push`.
>  > In OpenShift we use Prow, which has the latter; I really like it.  However we also
> *require* peer review (submitters can't merge their own PRs).
>
>  Unfortunately, it doesn't scale for the large number of packages we
>  have. Pull requests would work if we had mergify[1] working on
>  Dist-Git, otherwise I can't see how it'd work.
>
>  [1]: https://github.com/Mergifyio/mergify-engine
>
>  > I'd like to require review, but it does seem like a prerequisite is moving away
> from the one-repo-per-package model.
>
>  No it isn't. A pre-requisite would be that we'd require maintainer
>  teams, and have to make those first-class in Fedora (rather than
>  barely third-class as they are now). Besides, I've worked in
>  distributions where you have monorepo package source control, and it's
>  arguably worse. Rolling through the revisions is difficult, dealing
>  with searching through the tree is hard, and features like git blame
>  basically don't work (they time out more often than they return
>  results). Meaning that we'd need groups and subgroups that have ACL
>  inheritance for projects, among other things.
>  
Erk, this is not worded in the right order... I mean this instead:

No it isn't. A pre-requisite would be that we'd require maintainer
teams, and have to make those first-class in Fedora (rather than
barely third-class as they are now). Meaning that we'd need groups and
subgroups that have ACL inheritance for projects, among other things.

Besides, I've worked in distributions where you have monorepo package
source control, and it's arguably worse. Rolling through the revisions
is difficult, dealing with searching through the tree is hard, and
features like git blame basically don't work (they time out more often
than they return results).

>  I'm surprised you didn't realize these issues. You've
> examined Git
>  very deeply and you should be more than aware of how bad of idea it
>  would be to use a monorepo for package sources. We don't have separate
>  Git repos per package for no reason...
>
>
>  --
>  真実はいつも一つ！/ Always, there's only one truth! 


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Colin Walters]

On Wed, Oct 2, 2019, at 3:18 PM, Neal Gompa wrote:
>  On Wed, Oct 2, 2019 at 2:45 PM Colin Walters
> <walters(a)verbum.org&gt; wrote:
>  >
>  >
>  >
>  > On Wed, Oct 2, 2019, at 1:40 PM, Fabio Valentini wrote:
>  >
>  > > As others in the thread have pointed out, mandatory pull requests just
>  > > make no sense for most single-maintainer projects, which most packages
>  > > probably are.
>  >
>  > Well, a lot of this relates to what the *merge policy* is.  If a PR submitter can
> merge their own PRs, and there's a mechanism to do "merge when tests pass"
> (this is an important aspect), then submitting a PR can be just about as equally ergonomic
> as `git push`.
>  > In OpenShift we use Prow, which has the latter; I really like it.  However we also
> *require* peer review (submitters can't merge their own PRs).
>  
>  Unfortunately, it doesn't scale for the large number of packages we
>  have. Pull requests would work if we had mergify[1] working on
>  Dist-Git, otherwise I can't see how it'd work.
>  
>  [1]: https://github.com/Mergifyio/mergify-engine 
Yes, I mentioned Prow which does something similar. 
https://github.com/kubernetes/test-infra/tree/master/prow
Which as I noted we use today in OpenShift and are moving to use in the CoreOS group as
well.


>  I'm surprised you didn't realize these issues. You've
> examined Git
>  very deeply and you should be more than aware of how bad of idea it
>  would be to use a monorepo for package sources. We don't have separate
>  Git repos per package for no reason... 
https://github.com/projectatomic/rpmdistro-gitoverlay/blob/master/doc/rew...
links to a number which do it today.  You're right that there are tradeoffs; I think
the best is probably something closer to what OpenEmbedded is doing with
"layered" repos, not one gigantic dist-git repo.

It certainly seems to me the current Fedora setup is basically just inertia from the first
dist-cvs -> dist-git conversion; no one really in the intervening time has had the
power/will to change the underlying layers, just add new layers on top.

[Simo Sorce]

On Wed, 2019-10-02 at 15:57 -0400, Colin Walters wrote:
>  
>  On Wed, Oct 2, 2019, at 3:18 PM, Neal Gompa wrote:
>  > On Wed, Oct 2, 2019 at 2:45 PM Colin Walters <walters(a)verbum.org&gt; wrote:
>  > > 
>  > > 
>  > > 
>  > > On Wed, Oct 2, 2019, at 1:40 PM, Fabio Valentini wrote:
>  > > 
>  > > > As others in the thread have pointed out, mandatory pull requests just
>  > > > make no sense for most single-maintainer projects, which most packages
>  > > > probably are.
>  > > 
>  > > Well, a lot of this relates to what the *merge policy* is.  If a PR submitter
> can merge their own PRs, and there's a mechanism to do "merge when tests
> pass" (this is an important aspect), then submitting a PR can be just about as
> equally ergonomic as `git push`.
>  > > In OpenShift we use Prow, which has the latter; I really like it.  However we
> also *require* peer review (submitters can't merge their own PRs).
>  > 
>  > Unfortunately, it doesn't scale for the large number of packages we
>  > have. Pull requests would work if we had mergify[1] working on
>  > Dist-Git, otherwise I can't see how it'd work.
>  > 
>  > [1]: https://github.com/Mergifyio/mergify-engine
>  
>  Yes, I mentioned Prow which does something similar. 
> https://github.com/kubernetes/test-infra/tree/master/prow
>  Which as I noted we use today in OpenShift and are moving to use in the CoreOS group as
> well. 
I do not know how it is working today, but when I was working on it it
was a real chore as PRs would regularly flake. Most of the time it was
openshift/kube's code fault, other times it wasn't and would cause a
lot of overhead for teams to babysit PRs that should have just merged.


>  > I'm surprised you didn't realize these issues.
> You've examined Git
>  > very deeply and you should be more than aware of how bad of idea it
>  > would be to use a monorepo for package sources. We don't have separate
>  > Git repos per package for no reason...
>  
>
> https://github.com/projectatomic/rpmdistro-gitoverlay/blob/master/doc/rew...
>  links to a number which do it today.  You're right that there are
>  tradeoffs; I think the best is probably something closer to what
>  OpenEmbedded is doing with "layered" repos, not one gigantic dist-git 
>  repo.
>  
>  It certainly seems to me the current Fedora setup is basically just
>  inertia from the first dist-cvs -> dist-git conversion; no one really
>  in the intervening time has had the power/will to change the
>  underlying layers, just add new layers on top. 
Both are true probably, there is inertia and the tradeoffs probably
were not worth the change so far.

Simo.

-- 
Simo Sorce
RHEL Crypto Team
Red Hat, Inc

[Neal Gompa]

On Wed, Oct 2, 2019 at 4:06 PM Simo Sorce <simo(a)redhat.com&gt; wrote:
>  On Wed, 2019-10-02 at 15:57 -0400, Colin Walters wrote:
>  >
>  > On Wed, Oct 2, 2019, at 3:18 PM, Neal Gompa wrote:
>  > > On Wed, Oct 2, 2019 at 2:45 PM Colin Walters <walters(a)verbum.org&gt; wrote:
>  > > >
>  > > >
>  > > >
>  > > > On Wed, Oct 2, 2019, at 1:40 PM, Fabio Valentini wrote:
>  > > >
>  > > > > As others in the thread have pointed out, mandatory pull requests
> just
>  > > > > make no sense for most single-maintainer projects, which most
> packages
>  > > > > probably are.
>  > > >
>  > > > Well, a lot of this relates to what the *merge policy* is.  If a PR
> submitter can merge their own PRs, and there's a mechanism to do "merge when
> tests pass" (this is an important aspect), then submitting a PR can be just about as
> equally ergonomic as `git push`.
>  > > > In OpenShift we use Prow, which has the latter; I really like it.  However
> we also *require* peer review (submitters can't merge their own PRs).
>  > >
>  > > Unfortunately, it doesn't scale for the large number of packages we
>  > > have. Pull requests would work if we had mergify[1] working on
>  > > Dist-Git, otherwise I can't see how it'd work.
>  > >
>  > > [1]: https://github.com/Mergifyio/mergify-engine
>  >
>  > Yes, I mentioned Prow which does something similar. 
> https://github.com/kubernetes/test-infra/tree/master/prow
>  > Which as I noted we use today in OpenShift and are moving to use in the CoreOS group
> as well.
>
>  I do not know how it is working today, but when I was working on it it
>  was a real chore as PRs would regularly flake. Most of the time it was
>  openshift/kube's code fault, other times it wasn't and would cause a
>  lot of overhead for teams to babysit PRs that should have just merged.
>  
Kubernetes' project infrastructure is probably a very bad example, as
it's primarily geared to support the CNCF's crazy process rather than
helping developers.

>  > > I'm surprised you didn't realize these issues. You've examined Git
>  > > very deeply and you should be more than aware of how bad of idea it
>  > > would be to use a monorepo for package sources. We don't have separate
>  > > Git repos per package for no reason...
>  >
>  >
> https://github.com/projectatomic/rpmdistro-gitoverlay/blob/master/doc/rew...
>  > links to a number which do it today.  You're right that there are
>  > tradeoffs; I think the best is probably something closer to what
>  > OpenEmbedded is doing with "layered" repos, not one gigantic dist-git
>  > repo.
>  >
>  > It certainly seems to me the current Fedora setup is basically just
>  > inertia from the first dist-cvs -> dist-git conversion; no one really
>  > in the intervening time has had the power/will to change the
>  > underlying layers, just add new layers on top.
>
>  Both are true probably, there is inertia and the tradeoffs probably
>  were not worth the change so far.
>  
Dist-CVS is effectively a monorepo, just as Mageia's Dist-SVN is.
Those SCMs have the advantage of supporting subtree checkouts as a
first-class operation, and don't do proper "clones" like Git and
Mercurial do. That makes it *much* simpler to deal with.

I think layered repos would not be significantly better than just
having groups and subgroups to organize the package sources git repos.
You'd get the same organizational benefits, anyway.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Fabien Boucher]

On Wed, Oct 2, 2019 at 9:20 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:

>  Unfortunately, it doesn't scale for the large number of packages we
>  have. Pull requests would work if we had mergify[1] working on
>  Dist-Git, otherwise I can't see how it'd work.
>
>  [1]: https://github.com/Mergifyio/mergify-engine
>
>  
Since some months we (the team that maintain softwarefactory-project.io and
review.rdoproject.org) are working on making Zuul [1] compatible with
Pagure in order to implement a packager workflow through Zuul's jobs around
the Pull Request model.
Zuul is a powerful gating system, scalable, that support cross-repository
jobs and artifacts sharing between jobs out of the box. Zuul's jobs are
Ansible playbooks.

We managed to do great progress that we shown at last Flock. We are
tracking the progress in this EPIC [2].
Here is how a project is configured: [3], the template is defined like so:
[4], and here the job definition of rawhide-rpm-scratch-build: [5]

Zuul's jobs configuration is flexible and we can provide templates that fit
different needs, whether the packager want a full workflow (from the
scratch build, various validation jobs such as linting/rpminspect to the
regular build) or just the scratch build.

We will soon propose this as an opt-in service for the interested
packagers. CI resources capacity will come from softwarefactory-project.io.


[1]: https://zuul-ci.org/
[2]: https://teams.fedoraproject.org/project/ci/epic/14
[3]:
https://pagure.io/fedora-zuul-jobs-config/blob/master/f/zuul.d/projects.y...
[4]: https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml
[5]: https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/jobs.yaml#_1

[Martin Kolman]

On Wed, 2019-10-02 at 14:44 -0400, Colin Walters wrote:
>  
>  On Wed, Oct 2, 2019, at 1:40 PM, Fabio Valentini wrote:
>  
>  > As others in the thread have pointed out, mandatory pull requests just
>  > make no sense for most single-maintainer projects, which most packages
>  > probably are.
>  
>  Well, a lot of this relates to what the *merge policy* is.  If a PR submitter can merge
> their own PRs, and there's a
>  mechanism to do "merge when tests pass" (this is an important aspect), then
> submitting a PR can be just about as
>  equally ergonomic as `git push`. 
Yeah, that sounds good & nice improvement to
the currently available options. In some cases I run scratchbuilds as a
kind of a smoke tests before the "real" build - this way I could just use this
and save some time. :)

All in all I think much of this discussion feels a bit redundant to me - lets just
implement the support for improved
PRs that can be easily & automatically created and that triger all sorts of tests and
builds. If the new system is good,
I'm sure many maintainers would switch to it to everyones benefit - less regressions
& less time taken maintaining
packages.

At the same time if other maintainers have their own workflow that is not compatible with
the PR workflow or makes it
redundant for their packages, they should not be forced to use it.

>  In OpenShift we use Prow, which has the latter; I really like it. 
> However we also *require* peer review (submitters
>  can't merge their own PRs).  I'd like to require review, but it does seem like a
> prerequisite is moving away from the
>  one-repo-per-package model.
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Clement Verna]

On Wed, 2 Oct 2019 at 19:34, Matthew Miller <mattdm(a)fedoraproject.org&gt;
wrote:

>  On Wed, Oct 02, 2019 at 05:31:56PM +0100, Richard W.M. Jones wrote:
>  > > ○ Every changes to dist-git is done via pull-requests
>  > Erm, no thank you.  Pull requests are a terrible workflow.
>
>  It's definitely the winning workflow in the open source world today,
>  particularly for smaller and drive-by contributions, which I think we'd
>  like to encourage. And there _are_ real tracking and review benefits to
>  having everything go through that workflow.
>  
>  Do you have an alternative proposal?
>  
Continuous Integration can be done without PRs (this is not easy in the
open source world because you cannot grant commit access to every
contributors, this is not a problem for us since only the package
maintainers have commit access), in fact eXtrem Programming [0] is all
about pushing as often and as fast as possible to the main branch in order
to get early feedback. Instead of running the tests against a PR it is
possible to run the tests against the commits in the main development
branch. I believe that in our case knowing if a particular commit broke the
branch is as valuable as knowing if the tests failed against a PR.


[0] - http://www.extremeprogramming.org/rules/integrateoften.html

>  --
>  Matthew Miller
>  <mattdm(a)fedoraproject.org&gt;
>  Fedora Project Leader
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Jeremy Cline]

On Wed, Oct 02, 2019 at 08:26:16PM +0200, Clement Verna wrote:
>  On Wed, 2 Oct 2019 at 19:34, Matthew Miller
> <mattdm(a)fedoraproject.org&gt;
>  wrote:
>  
>  > On Wed, Oct 02, 2019 at 05:31:56PM +0100, Richard W.M. Jones wrote:
>  > > > ○ Every changes to dist-git is done via pull-requests
>  > > Erm, no thank you.  Pull requests are a terrible workflow.
>  >
>  > It's definitely the winning workflow in the open source world today,
>  > particularly for smaller and drive-by contributions, which I think we'd
>  > like to encourage. And there _are_ real tracking and review benefits to
>  > having everything go through that workflow.
>  >
>  
>  > Do you have an alternative proposal?
>  >
>  
>  Continuous Integration can be done without PRs (this is not easy in the
>  open source world because you cannot grant commit access to every
>  contributors, this is not a problem for us since only the package
>  maintainers have commit access), in fact eXtrem Programming [0] is all
>  about pushing as often and as fast as possible to the main branch in order
>  to get early feedback. Instead of running the tests against a PR it is
>  possible to run the tests against the commits in the main development
>  branch. I believe that in our case knowing if a particular commit broke the
>  branch is as valuable as knowing if the tests failed against a PR.
>   
Yeah, no thanks. As someone who endlessly chasing breakages I can tell
you it is miserable. That approach sounds like an eXtremly good way for
folks to just walk away from the project.

We can have machines check for correctness with tests, why on earth
would you enforce that check *after* accepting changes?

- Jeremy

[Clement Verna]

On Thu, Oct 3, 2019, 17:43 Jeremy Cline <jeremy(a)jcline.org&gt; wrote:

>  On Wed, Oct 02, 2019 at 08:26:16PM +0200, Clement Verna wrote:
>  > On Wed, 2 Oct 2019 at 19:34, Matthew Miller <mattdm(a)fedoraproject.org&gt;
>  > wrote:
>  >
>  > > On Wed, Oct 02, 2019 at 05:31:56PM +0100, Richard W.M. Jones wrote:
>  > > > > ○ Every changes to dist-git is done via pull-requests
>  > > > Erm, no thank you.  Pull requests are a terrible workflow.
>  > >
>  > > It's definitely the winning workflow in the open source world today,
>  > > particularly for smaller and drive-by contributions, which I think we'd
>  > > like to encourage. And there _are_ real tracking and review benefits to
>  > > having everything go through that workflow.
>  > >
>  >
>  > > Do you have an alternative proposal?
>  > >
>  >
>  > Continuous Integration can be done without PRs (this is not easy in the
>  > open source world because you cannot grant commit access to every
>  > contributors, this is not a problem for us since only the package
>  > maintainers have commit access), in fact eXtrem Programming [0] is all
>  > about pushing as often and as fast as possible to the main branch in
>  order
>  > to get early feedback. Instead of running the tests against a PR it is
>  > possible to run the tests against the commits in the main development
>  > branch. I believe that in our case knowing if a particular commit broke
>  the
>  > branch is as valuable as knowing if the tests failed against a PR.
>  >
>
>  Yeah, no thanks. As someone who endlessly chasing breakages I can tell
>  you it is miserable. That approach sounds like an eXtremly good way for
>  folks to just walk away from the project.
>
>  We can have machines check for correctness with tests, why on earth
>  would you enforce that check *after* accepting changes?
>  
I think that it would be better to know which commit messed up your branch
than not (currently the case). Currently there is no concept of "accepting
a change" for packages, since packagers are just pushing commits to
branches. To me running tests after the push is a good compromise and I
don't see how that would drive people out of the project compare to
enforcing all changes to be made via PR.


>  - Jeremy
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Jeremy Cline]

On Thu, Oct 03, 2019 at 08:42:36PM +0200, Clement Verna wrote:
>  On Thu, Oct 3, 2019, 17:43 Jeremy Cline <jeremy(a)jcline.org&gt;
> wrote:
>  
>  > On Wed, Oct 02, 2019 at 08:26:16PM +0200, Clement Verna wrote:
>  > > On Wed, 2 Oct 2019 at 19:34, Matthew Miller <mattdm(a)fedoraproject.org&gt;
>  > > wrote:
>  > >
>  > > > On Wed, Oct 02, 2019 at 05:31:56PM +0100, Richard W.M. Jones wrote:
>  > > > > > ○ Every changes to dist-git is done via pull-requests
>  > > > > Erm, no thank you.  Pull requests are a terrible workflow.
>  > > >
>  > > > It's definitely the winning workflow in the open source world today,
>  > > > particularly for smaller and drive-by contributions, which I think
> we'd
>  > > > like to encourage. And there _are_ real tracking and review benefits to
>  > > > having everything go through that workflow.
>  > > >
>  > >
>  > > > Do you have an alternative proposal?
>  > > >
>  > >
>  > > Continuous Integration can be done without PRs (this is not easy in the
>  > > open source world because you cannot grant commit access to every
>  > > contributors, this is not a problem for us since only the package
>  > > maintainers have commit access), in fact eXtrem Programming [0] is all
>  > > about pushing as often and as fast as possible to the main branch in
>  > order
>  > > to get early feedback. Instead of running the tests against a PR it is
>  > > possible to run the tests against the commits in the main development
>  > > branch. I believe that in our case knowing if a particular commit broke
>  > the
>  > > branch is as valuable as knowing if the tests failed against a PR.
>  > >
>  >
>  > Yeah, no thanks. As someone who endlessly chasing breakages I can tell
>  > you it is miserable. That approach sounds like an eXtremly good way for
>  > folks to just walk away from the project.
>  >
>  > We can have machines check for correctness with tests, why on earth
>  > would you enforce that check *after* accepting changes?
>  >
>  
>  I think that it would be better to know which commit messed up your branch
>  than not (currently the case). Currently there is no concept of "accepting
>  a change" for packages, since packagers are just pushing commits to
>  branches. To me running tests after the push is a good compromise and I
>  don't see how that would drive people out of the project compare to
>  enforcing all changes to be made via PR.
>   
My reaction was mostly to the concept of pushing everything as fast as
possible to a "main branch".

And yes, running some tests after the fact is better than running no
tests ever (and has other purposes). However, finding problems *before*
they happen is orders of magnitude more valuable than just merging,
hoping, and testing afterwards to figure out where your hope strategy let
you down. It should be encouraged by making the workflow the best choice
for the developer, not by forcing them down that path.

Furthermore, it's a compromise you don't have to make. Do what ever
other CI system does: allow it to run on PRs *and* on whatever branches
you want... Perhaps even continuously* on a branch or branches to ensure
it integrates with components properly.

*for some value of continuous

[Panu Matilainen]

On 10/2/19 8:33 PM, Matthew Miller wrote:
>  On Wed, Oct 02, 2019 at 05:31:56PM +0100, Richard W.M. Jones wrote:
> >> ○ Every changes to dist-git is done via pull-requests
> > Erm, no thank you.  Pull requests are a terrible workflow.
>  
>  It's definitely the winning workflow in the open source world today,
>  particularly for smaller and drive-by contributions, which I think we'd
>  like to encourage.  
It's an awesome workflow for those cases. Not so much when you are the 
maintainer of said piece.

>  And there _are_ real tracking and review benefits to
>  having everything go through that workflow.
>   
Speaking out of experience from another project going all pull-requests: 
Yes, there are benefits. I do like somebody elses (faster) computer 
running the test-suite. In dist-git the "does it even build" would be an 
awesome gate to have at the push stage, deity knows I've done my share 
of "oops, ..." commits there that wouldn't exist if there was a gate.

My issue with all this is that the pull-request workflow (speaking 
generally) as we know it introduces a significant overhead to the 
maintainer even for the simplest changes. As a developer, I want to just 
push the damn thing already and be done with it. Instead the 
pull-request model is like a ship-in-a-bottle workflow where you first 
do the work preliminary locally and then push it someplace remote to 
fiddle it into it's place with cumbersome remote controls. And then 
spend even more time tidying up over leftover branches here and there, 
updating your local copy to what you just did and other tedious 
administrativia.

If the maintainer overhead could be eliminated (via automation) it would 
be awesome. As it is, most of the time it absolutely does not feel worth it.

	- Panu -

[Björn Persson]

Panu Matilainen wrote:
>  On 10/2/19 8:33 PM, Matthew Miller wrote:
>  > On Wed, Oct 02, 2019 at 05:31:56PM +0100, Richard W.M. Jones wrote:  
>  >>> ○ Every changes to dist-git is done via pull-requests  
>  >> Erm, no thank you.  Pull requests are a terrible workflow.  
>  > 
>  > It's definitely the winning workflow in the open source world today,
>  > particularly for smaller and drive-by contributions, which I think we'd
>  > like to encourage.   
>  
>  It's an awesome workflow for those cases. Not so much when you are the 
>  maintainer of said piece. 
In the drive-by contributor role I've always found pull requests
unwieldy. I thought they were intended for frequent contributors or
project members, for whom the added clicking might be a small burden
compared to all the work they do for the project.

Perhaps pull requests are convenient for a maintainer who receives them
in large numbers – I've only ever received one pull request so I can't
judge – but I don't see how they would encourage drive-by contributions.
If you want to encourage drive-by contributions, then you should make
it easy to submit a patch without first registering an account, forking
a project and so on.

Björn Persson

[Robbie Harwood]

Björn Persson <Bjorn(a)xn--rombobjrn-67a.se&gt; writes:

>  Panu Matilainen wrote:
> > On 10/2/19 8:33 PM, Matthew Miller wrote:
> >> On Wed, Oct 02, 2019 at 05:31:56PM +0100, Richard W.M. Jones wrote:  
> >>
> >>>> ○ Every changes to dist-git is done via pull-requests
> >>>
> >>> Erm, no thank you.  Pull requests are a terrible workflow.
> >> 
> >> It's definitely the winning workflow in the open source world today,
> >> particularly for smaller and drive-by contributions, which I think
> >> we'd like to encourage.
> > 
> > It's an awesome workflow for those cases. Not so much when you are
> > the maintainer of said piece.
>
>  In the drive-by contributor role I've always found pull requests
>  unwieldy. I thought they were intended for frequent contributors or
>  project members, for whom the added clicking might be a small burden
>  compared to all the work they do for the project.
>
>  Perhaps pull requests are convenient for a maintainer who receives
>  them in large numbers – I've only ever received one pull request so I
>  can't judge – but I don't see how they would encourage drive-by
>  contributions.  If you want to encourage drive-by contributions, then
>  you should make it easy to submit a patch without first registering an
>  account, forking a project and so on. 
I have experienced this as a maintainer as well.  The issue for drive-by
contributors is not so much pull requests as the account system itself.
For example, I had a contributor from OpenSUSE email me patches to my
pagure-hosted project (gssproxy) rather than opening a pull request
because they didn't have a Fedora account.

If you look at GitLab, for example, they support a *ton* of federated
sign-ins: GitLab supports Google, Twitter, GitHub, Bitbucket, and
Salesforce.

Thanks,
--Robbie

[Neal Gompa]

On Fri, Oct 4, 2019 at 11:03 AM Robbie Harwood <rharwood(a)redhat.com&gt; wrote:
>  Björn Persson <Bjorn(a)xn--rombobjrn-67a.se&gt; writes:
>
>  > Panu Matilainen wrote:
>  >> On 10/2/19 8:33 PM, Matthew Miller wrote:
>  >>> On Wed, Oct 02, 2019 at 05:31:56PM +0100, Richard W.M. Jones wrote:
>  >>>
>  >>>>> ○ Every changes to dist-git is done via pull-requests
>  >>>>
>  >>>> Erm, no thank you.  Pull requests are a terrible workflow.
>  >>>
>  >>> It's definitely the winning workflow in the open source world today,
>  >>> particularly for smaller and drive-by contributions, which I think
>  >>> we'd like to encourage.
>  >>
>  >> It's an awesome workflow for those cases. Not so much when you are
>  >> the maintainer of said piece.
>  >
>  > In the drive-by contributor role I've always found pull requests
>  > unwieldy. I thought they were intended for frequent contributors or
>  > project members, for whom the added clicking might be a small burden
>  > compared to all the work they do for the project.
>  >
>  > Perhaps pull requests are convenient for a maintainer who receives
>  > them in large numbers – I've only ever received one pull request so I
>  > can't judge – but I don't see how they would encourage drive-by
>  > contributions.  If you want to encourage drive-by contributions, then
>  > you should make it easy to submit a patch without first registering an
>  > account, forking a project and so on.
>
>  I have experienced this as a maintainer as well.  The issue for drive-by
>  contributors is not so much pull requests as the account system itself.
>  For example, I had a contributor from OpenSUSE email me patches to my
>  pagure-hosted project (gssproxy) rather than opening a pull request
>  because they didn't have a Fedora account.
>
>  If you look at GitLab, for example, they support a *ton* of federated
>  sign-ins: GitLab supports Google, Twitter, GitHub, Bitbucket, and
>  Salesforce.
>  
I think it'd be cool if we could support logging in via account
services of major Linux distros (Fedora, Mageia, openSUSE, etc.) and
linking them together to make it easier for cross-distro
collaboration.

I think we're a bit away from being able to do that, but it'd be *very
cool* if we could do it.





--
真実はいつも一つ！/ Always, there's only one truth!

[Björn Persson]

Robbie Harwood wrote:
> I have experienced this as a maintainer as well.  The issue for
> drive-by
> contributors is not so much pull requests as the account system itself.
> For example, I had a contributor from OpenSUSE email me patches to my
> pagure-hosted project (gssproxy) rather than opening a pull request
> because they didn't have a Fedora account. 
It's not just the account. Fork → clone → patch → commit → push →
request is a rather long series of hoops to jump through. Clone → edit
→ commit → submit should be sufficient. Or what if I could just upload
a patch and have it turned into a pull request automatically?

Björn Persson

[Panu Matilainen]

On 10/4/19 3:17 PM, Björn Persson wrote:
>  Panu Matilainen wrote:
> > On 10/2/19 8:33 PM, Matthew Miller wrote:
> >> On Wed, Oct 02, 2019 at 05:31:56PM +0100, Richard W.M. Jones wrote:
> >>>> ○ Every changes to dist-git is done via pull-requests
> >>> Erm, no thank you.  Pull requests are a terrible workflow.
> >>
> >> It's definitely the winning workflow in the open source world today,
> >> particularly for smaller and drive-by contributions, which I think we'd
> >> like to encourage.
> >
> > It's an awesome workflow for those cases. Not so much when you are the
> > maintainer of said piece.
>  
>  In the drive-by contributor role I've always found pull requests
>  unwieldy. I thought they were intended for frequent contributors or
>  project members, for whom the added clicking might be a small burden
>  compared to all the work they do for the project.
>  
>  Perhaps pull requests are convenient for a maintainer who receives them
>  in large numbers – I've only ever received one pull request so I can't
>  judge – but I don't see how they would encourage drive-by contributions.
>  If you want to encourage drive-by contributions, then you should make
>  it easy to submit a patch without first registering an account, forking
>  a project and so on. 
Oh, true. PR's are really only convenient for existing habitats of the 
ecosystem. For Fedora this means existing Fedora packagers, but as for 
GitHub, "everybody" is already there and the bar to entry is quite low. 
For the better or worse. But the "embrace" effect of these, um, services 
is such that those who do not want to join for one reason or another 
will have a difficult time contributing.

For the truly drive-by contribution from "outside", there's no beating a 
single patch sent to an email address for convenience. With the downside 
that it traditionally lacks any means to track what happens to the 
patch, if anything. OTOH it's a feature of that process, sometimes 
patches could use the right to be forgotten too.

Mind you, if I had the choice, I'd pick the good ole patches-via-email 
workflow over these "social programming" platforms any day. But it is 
what it is, and the PR workflow is not without its benefits.

	- Panu -

[Kevin Kofler]

Matthew Miller wrote:
>  Do you have an alternative proposal? 
Have you seen my reply elsewhere in this thread?

I wrote there:
>  You need a different CI approach. Maybe: 
>  * a push hook that just locks the repository and does the tests before 
>    validating the push, though I can see that becoming a PITA as well, or 
>  * an asynchronous CI push hook that just notifies the maintainer on 
>    failures, but does not otherwise touch the repository, or 
>  * an asynchronous CI push hook that automatically reverts commits that
>    fail the CI tests, or even 
>  * an asynchronous CI push hook that automatically uses a force push to 
>    revert the commits that fail the CI tests in such a way that they vanish 
>    from history entirely, as if they had never happened, 
>  or something more clever that we have not thought of yet.  
What is clear is that submitting pull requests to myself does not make any 
sense whatsoever.

        Kevin Kofler

[Matthew Miller]

On Thu, Oct 03, 2019 at 11:52:32AM +0200, Kevin Kofler wrote:
>  Have you seen my reply elsewhere in this thread? 
I did, thanks. And I can see that as a fine model too. Looking for more
ideas from Richard as well.


>  What is clear is that submitting pull requests to myself does not
> make any 
>  sense whatsoever. 
Yes, I can certainly sympathize with that.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Florian Weimer]

* Matthew Miller:

>  On Wed, Oct 02, 2019 at 05:31:56PM +0100, Richard W.M. Jones wrote:
> > > ○ Every changes to dist-git is done via pull-requests
> > Erm, no thank you.  Pull requests are a terrible workflow.
>
>  It's definitely the winning workflow in the open source world today,
>  particularly for smaller and drive-by contributions, which I think we'd
>  like to encourage. And there _are_ real tracking and review benefits to
>  having everything go through that workflow. 
Given that for dist-git, every pull request will conflict with every
other pull requests, the experience of other projects really is not
relevant here because very, very few have this kind of problem.

Even with pull requests, the Fedora contribution experience will be very
different: dist-git, CLA, neither Gitlab nor Github, and so on.

Thanks,
Florian

[Dan Čermák]

I'm late to the party, but here we go anyway.

Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; writes:

>  [snip]
>
>  Here is what the vision we came to and that we would like to discuss:
>
>  ○ Every changes to dist-git is done via pull-requests 
As long as this is not mandatory, sure.

>  ○ Pull-requests are automatically tested
>  ○ Every commit to dist-git (ie: PR merged) is automatically built in koji 
Make that every tag and we are sold.

>  ○ Every build in koji results automatically in an update in bodhi
>  ○ Every update in bodhi is automatically tested
>  ○ If the tests pass, the update is automatically pushed to the repository 
So no more user's testing being able to test updates manually? I have
packages where I'd actually prefer them to be tested manually (i3,
i3status, i3lock, sway et al), because I haven't had the time to setup
openQA for these yet.

>  For this workflow to work nicely we need to fix a few things first:
>
>  - We need to work on the change logs in the spec files, as otherwise
>    pull-requests are going to conflict more often than not 
Let's please just get rid of them and the same for the Release tag (that
should be automatically bumped by the build system on each rebuild, as
we want automatic rebuilds, don't we?).

>  - We need to fine a place to insert the end-user information about an
> update (in
>    the PR description?) 
The git tag as Randy & Jeremy suggested.

>  [snip]
>
>  Do you like this vision? Would you change some pieces of it? Would you change it
>  entirely?
>  In an ideal world, what would packaging software look like to you? 
Like a combination of our current way of doing things combined with
openSUSE's Open Build Service (not only the automatic rebuild part,
although that is nice too).

What imho OBS did right is that it provides a single entry point for
packaging: a unified web UI with all the information & build states and
a unified CLI client for doing literally everything.
You want to update package `foo`? `osc branch devel:FooProj:foo && osc
checkout home:MyUserName:branches:devel:FooProj:foo`, make your changes
there, commit them and send a submitrequest (something like a pull
request). You want to become maintainer of a package? `osc
reqmaintainership`. Who maintains `bar`? `osc maintainer bar`. And so
on…

So what I'd love to see would be the single entry point to packaging
that Christopher described, not only as a web application, but also as a
CLI client.

To be more specific:
- My package's spec file is tracked in dist-git or git + git-lsf,
  depending on a setting I'll either have one branch for all the Fedora
  & EPEL versions or one branch for each version.

- Commits to the repository result in CI run (a default one, like
  rpmlint + additional custom tests). I can push -f all commits since
  the last tag, but nothing before that. Once I tag a commit, it is
  build "for real" (and tested) and automatically submitted as an update
  to bodhi. The changelog gets generated from the tag message and
  included into bodhi.
  On the other hand it must be possible to submit multiple packages as
  an update. I could imagine something like a `fedpkg update-multiple
  package1:$commit1 package2:$commit2 -m $message` that would tag the
  specified commits and push all builds into a single update to bodhi.

- the-new-hotness will send pull requests once a package is updated
  upstream. The pull request gets built & tested as an ordinary commit
  would. As a packager I have the option to "merge + tag" (also being
  able to modify the tag), which automatically submits this as an
  update.
  This would tremendously simplify the update process for most of my
  packages.

- If I would have a package that requires a lot of patches, then I can
  be granted the option to use source-git and keep my patches as commits
  on top of an upstream branch. This should imho require some form of
  approval though, as this can easily escalate into hundreds of patches.

- Package updates cause a rebuild of all dependencies (+ a test run of
  each of these). Only a successful rebuild is allowed to be submitted
  to bodhi.

- Allow the creation of package "namespaces" or "projects" (stolen
from
  OBS' development projects): a packager or a group can create a
  namespace on the git forge, into which they can fork/link arbitrary
  packages from the distribution. The forked packages become a part of
  the buildroot of this namespace, all others are inherited from the
  distribution itself and are unaffected by changes in this namespace.

  This should allow packagers to run experiments which do not affect the
  distributions build root, but still be able to test the impact of
  their changes on a larger set of packages. Packagers should then be
  able to send their changes back via a mass-pull-request/mass-commit.

- Extend the fedpkg CLI to support more actions on pagure, e.g. finding
  out who is the current package maintainer of Foo or forking the
  package Bar and sending a PR back.


Hope all of this makes sense and would be something that is worth
pursuing.


Cheers,

Dan

[Kevin Fenzi]

On Thu, Sep 26, 2019 at 11:36:10AM +0200, Pierre-Yves Chibon wrote:
>  Good Morning Everyone, 
...snip...

So, a few thoughts in general on the thread and ideas... 

I think it might be helpfull for us to come up with some personas?
I know that kind of thing seems silly a lot of the time, but I think we
have some pretty clear ones and it would help us come up with a workflow
that could work for much of our maintainers instead of just one group. 

ie, for example I think we have a group of folks who maintain a very
small number of packages. These folks might be upstream maintaining
their software as a fedora package or just someone who uses / cares for
it. They want something simple where they can update packages/modules
from upstream releases on an optional number of releases using just a
tarball or other release artifacts. 

Then there's people who maintain 100's or many hundreds of packages,
usually in a stack (perl, ruby, node, etc). They want the ability to
update things but they also need to make big changes accross their
collection of packages for improvments or changes in the ecosystem. 

There's people who maintain a application and it's deps who may
need/want to update/test things in a bundle, etc. 

There's people who care about EPEL and want to follow the Fedora
branches, but only if they don't make major changes. 

And likely other cases I haven't thought of in a few seconds of
pondering, but we should definitely consider in our design.

I think it's pretty clear that the "Everything is a PR" work flow
doesn't work for everyone. As well as the "Sources are a git repo".
So, I think a ideal setup would allow people to make choices and switch
between them. Of course I don't think we should just allow everything,
but we should have path's that cover a large group and choices only
where we have to to accomodate the needs of the different groups. 

I like the idea of leveraging tags as a way to provide information to
our new system what to build and test. Allowing commits without
builds/testing would help the ecosystem user make a bunch of changes,
then they could tag that what they have is ready to build/test/make an
update out of, while a user doing just small updates could tag on every
commit if they wanted to. Perhaps we could also determine all the
information we want in tags and design it such that a web application, a
cli application or even a user who understood the format could just git
tag and enter the needed information. This would allow groups using each
of those to all work the same way as they choose. If we went crazy far
down this road: what if all our information is in tags? who can commit,
whois watching, etc... but then we would have to hate ourselves enough
to do gpg signed commits to make sure who was who.

I think we all agree we need to handle release and changelogs in an
automated way that lets commits/PR's be more generally useful. 

I also think we have not really any good tools for maintainers to know
important things like: "what is the full list of everything my package
depends on" or "Of all the things my package depends on, what hasn't
been updated in X years" or "What media is my package on?" which we have
wanted for ages. Not sure if that fits in here, but it would sure be
nice to have, and it would be required for any of the 'rebuild things
that depend on my package when I change it'.

I'm not sure how to handle the dychomoty between having different spec
files for each release and wanting to maintain just one spec that has a
bunch of crazy conditionals in it. Even thought I do this too, I think
we need a workflow that discourages this somehow. 

Anyhow, I think it might be worth making some personas and trying to map
out how tags could contain the information we might want and how that
would look?

kevin

[Miro Hrončok]

On 03. 10. 19 1:32, Kevin Fenzi wrote:
>  I'm not sure how to handle the dychomoty between having different
> spec
>  files for each release and wanting to maintain just one spec that has a
>  bunch of crazy conditionals in it. Even thought I do this too, I think
>  we need a workflow that discourages this somehow. 
I believe that moving release tag and changelog entry to annotated tags solves 
this problem (or makes it insignificant).

It means you can just cleanly cherry pick a fix anywhere it is relevant (most of 
the time) instead of fighting the changelog conflicts all the time.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Kevin Kofler]

Miro Hrončok wrote:
>  On 03. 10. 19 1:32, Kevin Fenzi wrote:
> > I'm not sure how to handle the dychomoty between having different spec
> > files for each release and wanting to maintain just one spec that has a
> > bunch of crazy conditionals in it. Even thought I do this too, I think
> > we need a workflow that discourages this somehow.
>  
>  I believe that moving release tag and changelog entry to annotated tags
>  solves this problem (or makes it insignificant).
>  
>  It means you can just cleanly cherry pick a fix anywhere it is relevant
>  (most of the time) instead of fighting the changelog conflicts all the
>  time. 
I don't understand the people actually maintaining different changelogs for 
the releases. I just merge master into the release branches when I push an 
update, and if that includes some changelog entries for Rawhide-only mass 
rebuilds, so be it. Removing them is not worth breaking fast-forwardability 
of the branches.

I even go as far as reverting branch-only commits and then doing the 
bidirectional merge trick to restore fast forwardability. That of course 
clobbers the branch-only changelog section and replaces it with the one from 
master, but that's just how things are. Again, I think fast-forwardability 
is more useful than per-branch changelogs.

        Kevin Kofler

[Miro Hrončok]

On 03. 10. 19 11:58, Kevin Kofler wrote:
>  I don't understand the people actually maintaining different
> changelogs for
>  the releases. I just merge master into the release branches when I push an
>  update, and if that includes some changelog entries for Rawhide-only mass
>  rebuilds, so be it. Removing them is not worth breaking fast-forwardability
>  of the branches. 

It goes like this:

  - master and f31 are at the same commit "aaaaaa"
  - I push a change only possible in rawhide, commit "bbbbbb" to master
    (it includes release bump and changelog entry)
  - a commit relevant for both, "cccccc" is pushed to master
    (it includes release bump and changelog entry)
  - on f31, I run `git cherry-pick cccccc` => conflict

I don't worry about having "Fedora 31 mass rebuild" or "Rebuilt for
python 3.8" 
changelong entries in Fedora 29 (it gives me a little flinch, but nothing 
serious). i worry about the bbbbbb commit I cannot merge into f31 (e.g. if it 
implements some Fedora 32 change).

Then obviously, people start inventing %if spaghetti.



-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Miroslav Suchý]

Dne 03. 10. 19 v 12:18 Miro Hrončok napsal(a):
>  Then obviously, people start inventing %if spaghetti. 
Nope,
   %if spaghetti
comes from people who are upstream author of some project (usually layered application)
and have to support the packages 
(usually cli tools for their project) for all Fedoras, Epels. Also for Debians, Suses,
Mageias... And they are actually 
also package maintainers in those distributions. And they do not want to maintain various
versions of spec files in 
various distributions.

In all projects, I have been working so far, the push to Fedora from upstream was fully
automated and I did not need to 
know about dist-git branches at all.

-- 
Miroslav Suchy, RHCA
Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys

[Kevin Kofler]

Miro Hrončok wrote:
>  It goes like this:
>  
>    - master and f31 are at the same commit "aaaaaa"
>    - I push a change only possible in rawhide, commit "bbbbbb" to master
>      (it includes release bump and changelog entry)
>    - a commit relevant for both, "cccccc" is pushed to master
>      (it includes release bump and changelog entry)
>    - on f31, I run `git cherry-pick cccccc` => conflict
>  
>  I don't worry about having "Fedora 31 mass rebuild" or "Rebuilt for
> python
>  3.8" changelong entries in Fedora 29 (it gives me a little flinch, but
>  nothing serious). i worry about the bbbbbb commit I cannot merge into f31
>  (e.g. if it implements some Fedora 32 change).
>  
>  Then obviously, people start inventing %if spaghetti. 
And %if is actually the correct fix for this issue.

See, e.g., the one I had to add to qt5-qtwebengine after you broke it for
F29 with your mass change:
https://src.fedoraproject.org/rpms/qt5-qtwebengine/c/05a52d121d49972989ae...

IMHO, mass changes are only acceptable if the result builds on all supported
Fedora releases. Breaking the build for F29 is only acceptable after it
reaches EOL. So your mass change should have included this %if, or ideally
an update pushed to F29 to make the conditional unnecessary.

        Kevin Kofler

[Fabio Valentini]

On Sat, Oct 5, 2019, 02:17 Kevin Kofler <kevin.kofler(a)chello.at&gt; wrote:

>  Miro Hrončok wrote:
>  > It goes like this:
>  >
>  >   - master and f31 are at the same commit "aaaaaa"
>  >   - I push a change only possible in rawhide, commit "bbbbbb" to master
>  >     (it includes release bump and changelog entry)
>  >   - a commit relevant for both, "cccccc" is pushed to master
>  >     (it includes release bump and changelog entry)
>  >   - on f31, I run `git cherry-pick cccccc` => conflict
>  >
>  > I don't worry about having "Fedora 31 mass rebuild" or "Rebuilt
> for
>  python
>  > 3.8" changelong entries in Fedora 29 (it gives me a little flinch, but
>  > nothing serious). i worry about the bbbbbb commit I cannot merge into f31
>  > (e.g. if it implements some Fedora 32 change).
>  >
>  > Then obviously, people start inventing %if spaghetti.
>
>  And %if is actually the correct fix for this issue.
>
>  See, e.g., the one I had to add to qt5-qtwebengine after you broke it for
>  F29 with your mass change:
>
>
> https://src.fedoraproject.org/rpms/qt5-qtwebengine/c/05a52d121d49972989ae...
>
>  IMHO, mass changes are only acceptable if the result builds on all
>  supported
>  Fedora releases. Breaking the build for F29 is only acceptable after it
>  reaches EOL. So your mass change should have included this %if, or ideally
>  an update pushed to F29 to make the conditional unnecessary.
>  
That's your opinion. Some packagers (like me) actually maintain the
branches for each fedora release separately.

Doing mass changes that way would also place a pretty big burden upon
anyone who actually wants to work on fedora-wide improvements.

Additionally, if-guarding every non-backwards compatible change will result
in unmaintainable, brittle and broken .spec files pretty fast. Nobody
should be expected to work through if-else-endif spaghetti (and I'm not
even talking about automated tools here, which almost never will handle
conditionals entirely correctly, and probably never can). And never mind
that people don't actually remove conditionals for EOL fedora releases ...

Fabio


>          Kevin Kofler
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Kevin Kofler]

Fabio Valentini wrote:
>  Additionally, if-guarding every non-backwards compatible change will
>  result in unmaintainable, brittle and broken .spec files pretty fast.
>  Nobody should be expected to work through if-else-endif spaghetti (and I'm
>  not even talking about automated tools here, which almost never will
>  handle conditionals entirely correctly, and probably never can). And never
>  mind that people don't actually remove conditionals for EOL fedora
>  releases ... 
I do, for most packages. At least when I need to work on the package anyway. 
I wouldn't submit a build just for that.

Though for some packages, I get told to leave ancient conditionals in for 
EPEL. (You can usually recognize them because the conditionals also include 
%{?rhel}.)

        Kevin Kofler

[Miro Hrončok]

On 05. 10. 19 2:16, Kevin Kofler wrote:
>  Miro Hrončok wrote:
> > It goes like this:
> >
> >    - master and f31 are at the same commit "aaaaaa"
> >    - I push a change only possible in rawhide, commit "bbbbbb" to master
> >      (it includes release bump and changelog entry)
> >    - a commit relevant for both, "cccccc" is pushed to master
> >      (it includes release bump and changelog entry)
> >    - on f31, I run `git cherry-pick cccccc` => conflict
> >
> > I don't worry about having "Fedora 31 mass rebuild" or "Rebuilt
> for python
> > 3.8" changelong entries in Fedora 29 (it gives me a little flinch, but
> > nothing serious). i worry about the bbbbbb commit I cannot merge into f31
> > (e.g. if it implements some Fedora 32 change).
> >
> > Then obviously, people start inventing %if spaghetti.
>  And %if is actually the correct fix for this issue.
>  
>  See, e.g., the one I had to add to qt5-qtwebengine after you broke it for
>  F29 with your mass change:
>
> https://src.fedoraproject.org/rpms/qt5-qtwebengine/c/05a52d121d49972989ae...
Your example is not valid. This is not a mass change, this was an individual 
change presented to the package maintainers via a PR that was not merged by me:

https://src.fedoraproject.org/rpms/qt5-qtwebengine/pull-request/3

Had there been a "please, make it build on F29" comment, I would have adapted
it.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Kevin Kofler]

Miro Hrončok wrote:
>  Your example is not valid. This is not a mass change, this was an
>  individual change presented to the package maintainers via a PR that was
>  not merged by me:
>  
>  https://src.fedoraproject.org/rpms/qt5-qtwebengine/pull-request/3
>  
>  Had there been a "please, make it build on F29" comment, I would have
>  adapted it. 
Well, your pull request did not state anywhere that it introduces a 
construct that only works on F30 and newer, so I don't think it is fair to 
shift the blame to the person who approved it (Rex Dieter). Your description 
only stated "make it work both before and after python2 retirement", which 
sounds as if it were fully backwards compatible.

It should be up to the person making this type of changes to test it on all 
supported releases. Sure, this makes it harder to make packaging changes 
across many packages, but IMHO, that is actually a beneficial side effect. 
The goal should really be to not require this kind of changes at all, 
because it also means that third-party packages have to adjust to them. 
Instead, the packaging should remain backwards-compatible.

And in this case, I wonder whether the conditional is even needed or whether 
we could just use the F29 version everywhere. As long as python27 Provides: 
python2 (and it should!), I don't see why BuildRequires: python2 would not 
be good enough and a file BuildRequires is needed. And python2-rpm-macros 
are needed as BuildRequires because the specfile also uses %{__python2} in 
the snippets. (Previously, the package had BuildRequires: python2-devel, 
which drags in both python2 and python2-rpm-macros.) Something changed in 
F30 to make python2-rpm-macros dragged in by default, this was not the case 
in F29.

        Kevin Kofler

[Vít Ondruch]

Dne 03. 10. 19 v 11:58 Kevin Kofler napsal(a):
>  Miro Hrončok wrote:
> > On 03. 10. 19 1:32, Kevin Fenzi wrote:
> >> I'm not sure how to handle the dychomoty between having different spec
> >> files for each release and wanting to maintain just one spec that has a
> >> bunch of crazy conditionals in it. Even thought I do this too, I think
> >> we need a workflow that discourages this somehow.
> > I believe that moving release tag and changelog entry to annotated tags
> > solves this problem (or makes it insignificant).
> >
> > It means you can just cleanly cherry pick a fix anywhere it is relevant
> > (most of the time) instead of fighting the changelog conflicts all the
> > time.
>  I don't understand the people actually maintaining different changelogs for 
>  the releases. I just merge master into the release branches when I push an 
>  update, and if that includes some changelog entries for Rawhide-only mass 
>  rebuilds, so be it. Removing them is not worth breaking fast-forwardability 
>  of the branches. 

It depends how you maintain your packages. My guess is (and I am sorry
if I am mistaken) that you don't follow the

https://fedoraproject.org/wiki/Updates_Policy#Philosophy

If you followed this policy, then you would touch the stable branches
just rarely and therefore keeping them fast forwardable would be just
waste of time.


Vít


>
> I even go as far as reverting branch-only commits and then doing the 
> bidirectional merge trick to restore fast forwardability. That of course 
> clobbers the branch-only changelog section and replaces it with the one from 
> master, but that's just how things are. Again, I think fast-forwardability 
> is more useful than per-branch changelogs.
>
>         Kevin Kofler
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Miro Hrončok]

On 03. 10. 19 15:23, Vít Ondruch wrote:
>  
>  Dne 03. 10. 19 v 11:58 Kevin Kofler napsal(a):
> > Miro Hrončok wrote:
> >> On 03. 10. 19 1:32, Kevin Fenzi wrote:
> >>> I'm not sure how to handle the dychomoty between having different spec
> >>> files for each release and wanting to maintain just one spec that has a
> >>> bunch of crazy conditionals in it. Even thought I do this too, I think
> >>> we need a workflow that discourages this somehow.
> >> I believe that moving release tag and changelog entry to annotated tags
> >> solves this problem (or makes it insignificant).
> >>
> >> It means you can just cleanly cherry pick a fix anywhere it is relevant
> >> (most of the time) instead of fighting the changelog conflicts all the
> >> time.
> > I don't understand the people actually maintaining different changelogs for
> > the releases. I just merge master into the release branches when I push an
> > update, and if that includes some changelog entries for Rawhide-only mass
> > rebuilds, so be it. Removing them is not worth breaking fast-forwardability
> > of the branches.
>  
>  
>  It depends how you maintain your packages. My guess is (and I am sorry
>  if I am mistaken) that you don't follow the
>  
>  https://fedoraproject.org/wiki/Updates_Policy#Philosophy 
I do.

>  If you followed this policy, then you would touch the stable
> branches
>  just rarely and therefore keeping them fast forwardable would be just
>  waste of time. 
I touch stable branches for bugfix upgrades.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Vít Ondruch]

Dne 03. 10. 19 v 15:56 Miro Hrončok napsal(a):
>  On 03. 10. 19 15:23, Vít Ondruch wrote:
> >
> > Dne 03. 10. 19 v 11:58 Kevin Kofler napsal(a):
> >> Miro Hrončok wrote:
> >>> On 03. 10. 19 1:32, Kevin Fenzi wrote:
> >>>> I'm not sure how to handle the dychomoty between having different
> >>>> spec
> >>>> files for each release and wanting to maintain just one spec that
> >>>> has a
> >>>> bunch of crazy conditionals in it. Even thought I do this too, I
> >>>> think
> >>>> we need a workflow that discourages this somehow.
> >>> I believe that moving release tag and changelog entry to annotated
> >>> tags
> >>> solves this problem (or makes it insignificant).
> >>>
> >>> It means you can just cleanly cherry pick a fix anywhere it is
> >>> relevant
> >>> (most of the time) instead of fighting the changelog conflicts all the
> >>> time.
> >> I don't understand the people actually maintaining different
> >> changelogs for
> >> the releases. I just merge master into the release branches when I
> >> push an
> >> update, and if that includes some changelog entries for Rawhide-only
> >> mass
> >> rebuilds, so be it. Removing them is not worth breaking
> >> fast-forwardability
> >> of the branches.
> >
> >
> > It depends how you maintain your packages. My guess is (and I am sorry
> > if I am mistaken) that you don't follow the
> >
> > https://fedoraproject.org/wiki/Updates_Policy#Philosophy
>
>  I do.
>
> > If you followed this policy, then you would touch the stable branches
> > just rarely and therefore keeping them fast forwardable would be just
> > waste of time.
>
>  I touch stable branches for bugfix upgrades.
>  
But this sooner or later means rebase in Rawhide and just applying
patches in the stable branch. So my point still stands.


Vít

[Stephen John Smoogen]

On Thu, 3 Oct 2019 at 11:20, Vít Ondruch <vondruch(a)redhat.com&gt; wrote:
>  Dne 03. 10. 19 v 15:56 Miro Hrončok napsal(a):
>  > On 03. 10. 19 15:23, Vít Ondruch wrote:
>  >>
>  >> Dne 03. 10. 19 v 11:58 Kevin Kofler napsal(a):
>  >>> Miro Hrončok wrote:
>  >>>> On 03. 10. 19 1:32, Kevin Fenzi wrote:
>  >>>>> I'm not sure how to handle the dychomoty between having
> different
>  >>>>> spec
>  >>>>> files for each release and wanting to maintain just one spec that
>  >>>>> has a
>  >>>>> bunch of crazy conditionals in it. Even thought I do this too, I
>  >>>>> think
>  >>>>> we need a workflow that discourages this somehow.
>  >>>> I believe that moving release tag and changelog entry to annotated
>  >>>> tags
>  >>>> solves this problem (or makes it insignificant).
>  >>>>
>  >>>> It means you can just cleanly cherry pick a fix anywhere it is
>  >>>> relevant
>  >>>> (most of the time) instead of fighting the changelog conflicts all the
>  >>>> time.
>  >>> I don't understand the people actually maintaining different
>  >>> changelogs for
>  >>> the releases. I just merge master into the release branches when I
>  >>> push an
>  >>> update, and if that includes some changelog entries for Rawhide-only
>  >>> mass
>  >>> rebuilds, so be it. Removing them is not worth breaking
>  >>> fast-forwardability
>  >>> of the branches.
>  >>
>  >>
>  >> It depends how you maintain your packages. My guess is (and I am sorry
>  >> if I am mistaken) that you don't follow the
>  >>
>  >> https://fedoraproject.org/wiki/Updates_Policy#Philosophy
>  >
>  > I do.
>  >
>  >> If you followed this policy, then you would touch the stable branches
>  >> just rarely and therefore keeping them fast forwardable would be just
>  >> waste of time.
>  >
>  > I touch stable branches for bugfix upgrades.
>  >
>
>  But this sooner or later means rebase in Rawhide and just applying
>  patches in the stable branch. So my point still stands. 
Which point? You have had multiple. The one where you say doesn't
follow the procedure does not stand.


-- 
Stephen J Smoogen.

[Kevin Kofler]

Vít Ondruch wrote:
>  It depends how you maintain your packages. My guess is (and I am
> sorry
>  if I am mistaken) that you don't follow the
>  
>  https://fedoraproject.org/wiki/Updates_Policy#Philosophy
>  
>  If you followed this policy, then you would touch the stable branches
>  just rarely and therefore keeping them fast forwardable would be just
>  waste of time. 
It depends on what, how, and when upstream releases.

If we have different upstream releases in different Fedora releases, then of 
course those releases should have different specfiles. But if we are 
shipping the same upstream release, I don't see why I should clone the 
specfile n times.

There are many possible reasons for shipping the same upstream release 
across different Fedora releases:
* There might just not have been a newer upstream release in this time.
* Heck, there might even not be any new upstream releases at all anymore,
  which is typical for compatibility libraries such as qt3.
* The newer upstream releases might be bugfix-only releases applicable to
  stable Fedora releases just fine. (In other words, there might just not
  have been a newer upstream feature release in this time.)
* It can be necessary for security reasons to upgrade to a newer feature
  release. (E.g., this is the case for browser packages such as firefox or
  chromium. Also for qt5-qtwebengine to some extent, though at least Qt
  maintains LTS branches including QtWebEngine security backports these
  days.)
* It can be necessary to upgrade a library to a backwards-compatible feature
  release because applications are depending on a newer version. (The main
  example there is KF5 (KDE Frameworks 5), which does not maintain bugfix
  branches at all. But, e.g., Qt also sometimes needs to be upgraded to meet
  application demands.)
* There can be other reasons why backporting the newer feature release is a
  good idea. (E.g., if it only adds features and does not remove or break
  anything, and if it has been thouroughly tested for regressions without
  any being found, why would it not be OK to push?)

That said, I am not a fan of the Updates Policy as written because it is 
written in the form "do not push new feature releases as updates unless X, Y 
or Z", whereas I think it should be "do push new feature releases as updates 
unless X', Y' or Z'". Still, I think we all agree that some feature
releases 
just need to go out (e.g., because they include security fixes that are 
impractical to backport) whereas some definitely must never go out as 
updates (e.g., anything that breaks user configuration).

        Kevin Kofler

[Kevin Fenzi]

On Sat, Oct 05, 2019 at 02:35:59AM +0200, Kevin Kofler wrote:
>  Vít Ondruch wrote:
>  > It depends how you maintain your packages. My guess is (and I am sorry
>  > if I am mistaken) that you don't follow the
>  > 
>  > https://fedoraproject.org/wiki/Updates_Policy#Philosophy
>  > 
>  > If you followed this policy, then you would touch the stable branches
>  > just rarely and therefore keeping them fast forwardable would be just
>  > waste of time.
>  
>  It depends on what, how, and when upstream releases.
>  
>  If we have different upstream releases in different Fedora releases, then of 
>  course those releases should have different specfiles. But if we are 
>  shipping the same upstream release, I don't see why I should clone the 
>  specfile n times. 
In order to reduce complexity and allow for easier automated changes?

You could make a spec file for any package that "works" on every
version/release of every rpm based distro. It would be full of if this
then that and duplicated sections with subtle differences, it could even
do all versions. 

Right now what we have is maintainer(s) preference on how complex they
want their spec files to get before they just seperate our another copy
for the complex case. I think thats bad for a number of reasons...

...snip section about why you would want to ship the same upstream
release in multiple fedora releases...

yes, thats all fine, but in a lot of cases you run into issues doing
this, which means you have to drive up the complexity of your one spec
file or just ship different ones per fedora release. I think this
increasing complexity is bad for the project, and often bad for
maintainers as well. 

On the one hand you only have to edit one spec file, but you have to
figure out the logic and have no way to know if it's right without
rebuilding it in all the places you want to use it and carefully
checking each build. 

On the other you have to edit more spec files and you have to swap in
context as to why say fedora 29 is doing X and master is doing Y, but
it's a lot more clear what it's doing and why. 

kevin

[Matthew Miller]

On Sat, Oct 05, 2019 at 02:35:59AM +0200, Kevin Kofler wrote:
>  There are many possible reasons for shipping the same upstream
> release 
>  across different Fedora releases: 
[...snip good list of reasons...]


>  That said, I am not a fan of the Updates Policy as written because it
> is
>  written in the form "do not push new feature releases as updates unless X,
>  Y or Z", whereas I think it should be "do push new feature releases as
>  updates unless X', Y' or Z'". Still, I think we all agree that some
>  feature releases just need to go out (e.g., because they include security
>  fixes that are impractical to backport) whereas some definitely must never
>  go out as updates (e.g., anything that breaks user configuration). 
I agree on this too.

A key goal of the modularity project is to allow some of the cases to be
better addressed by allowing packagers to think in terms of upstream changes
which affect user experience separate from the Fedora release cycle. The
default stream for a package shouldn't be updated in disruptive ways in
shipped releases, but a new-version stream could _also_ be available. And at
the same time, if you upgrade to a new Fedora OS release but still need the
old behavior and the packager is still maintaining it, you should be able to
opt in to it.

Now, whether it's delivered that perfectly yet is a matter of ... well, we
know it hasn't and there's more work to get there, especially in terms of
packager experience. But I think it's a good idea and something ultimately
we will get to.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Kevin Kofler]

Matthew Miller wrote:
>  A key goal of the modularity project is to allow some of the cases to
> be
>  better addressed by allowing packagers to think in terms of upstream
>  changes which affect user experience separate from the Fedora release
>  cycle. The default stream for a package shouldn't be updated in disruptive
>  ways in shipped releases, but a new-version stream could _also_ be
>  available. And at the same time, if you upgrade to a new Fedora OS release
>  but still need the old behavior and the packager is still maintaining it,
>  you should be able to opt in to it. 
Sure, I fully understand the theoretical benefits to be had from Modularity 
(though I still think that this is much more useful for LTS distributions 
such as RHEL or CentOS than for Fedora). The issue is that it all breaks 
down when modules depend on each other (and they already do), because of the 
unavoidable versioning conflicts (Module A requires Module C version 1, 
Module B requires Module C version 2, and only one version of Module C can 
be installed) that bring us Module Hell, a.k.a., RPM Hell 2.0. And this 
follows directly from the specification of the Modularity feature. And it 
has already happened in practice (see the libgit2 chaos).

        Kevin Kofler

[Dridi Boukelmoune]

On Tue, Oct 8, 2019 at 12:06 AM Kevin Kofler <kevin.kofler(a)chello.at&gt; wrote:
>  Matthew Miller wrote:
>  > A key goal of the modularity project is to allow some of the cases to be
>  > better addressed by allowing packagers to think in terms of upstream
>  > changes which affect user experience separate from the Fedora release
>  > cycle. The default stream for a package shouldn't be updated in disruptive
>  > ways in shipped releases, but a new-version stream could _also_ be
>  > available. And at the same time, if you upgrade to a new Fedora OS release
>  > but still need the old behavior and the packager is still maintaining it,
>  > you should be able to opt in to it.
>
>  Sure, I fully understand the theoretical benefits to be had from Modularity
>  (though I still think that this is much more useful for LTS distributions
>  such as RHEL or CentOS than for Fedora). The issue is that it all breaks
>  down when modules depend on each other (and they already do), because of the
>  unavoidable versioning conflicts (Module A requires Module C version 1,
>  Module B requires Module C version 2, and only one version of Module C can
>  be installed) that bring us Module Hell, a.k.a., RPM Hell 2.0. And this
>  follows directly from the specification of the Modularity feature. And it
>  has already happened in practice (see the libgit2 chaos). 
Modularity should have been opt-in until major bugs are ironed out,
and maybe we would have realized in time that either it's great or it
doesn't solve anything the problem it's addressing.

Dridi

[Kevin Kofler]

Dridi Boukelmoune wrote:
>  Modularity should have been opt-in until major bugs are ironed out,
>  and maybe we would have realized in time that either it's great or it
>  doesn't solve anything the problem it's addressing. 
+1

        Kevin Kofler

[Matthew Miller]

On Tue, Oct 08, 2019 at 02:06:06AM +0200, Kevin Kofler wrote:
>  Sure, I fully understand the theoretical benefits to be had from
> Modularity 
>  (though I still think that this is much more useful for LTS distributions 
>  such as RHEL or CentOS than for Fedora). The issue is that it all breaks 
>  down when modules depend on each other (and they already do), because of the 
>  unavoidable versioning conflicts (Module A requires Module C version 1, 
>  Module B requires Module C version 2, and only one version of Module C can 
>  be installed) that bring us Module Hell, a.k.a., RPM Hell 2.0. And this 
>  follows directly from the specification of the Modularity feature. And it 
>  has already happened in practice (see the libgit2 chaos). 
Yeah, I agree that there's a problem with non-parallel-installable modules
that aren't effectively leaves.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Kevin Kofler]

Matthew Miller wrote:
>  Yeah, I agree that there's a problem with
> non-parallel-installable modules
>  that aren't effectively leaves. 
The problem does not only happen if the module is a non-leaf at module 
level, but there can also be conflicts at package level, if the modules 
bundle non-leaf packages that then conflict between the 2 modules. 
(Actually, there could even be package-level conflicts from leaf packages in 
2 different modules, but usually, the conflicting packages are bundled for a 
reason, so they are typically not leaves.) So banning non-leaf modules would 
not by itself be enough to solve the problem (because the modules would then 
resort to bundling the non-leaf packages they need and running into the 
package-level conflicts).

        Kevin Kofler