Pagure on dist-git, the state of things

[Pierre-Yves Chibon]

Good Morning Everyone,

I figured it has been a while since I reported progress on making pagure a
front-end for dist-git.

So here is a small status update.

What works:
- Well currently pagure is working as a front-end for dist-git in stg:
  https://src.stg.fedoraproject.org/pagure/
- Hosting repos, browsing them, creating PR works all fine (w/ one , see
  below)
- Features we want to exclude have been turned off (no user management on the
  pagure side)

What does not work:
- Syncing the ACLs from pkgdb to gitolite takes ~3 minutes in prod, in stg, the
  updated script in stg (which sync the ACLs from pkgdb to gitolite and pagure)
  runs in ~30/35 minutes (if it doesn't crash, which my last run just did)
  This is a blocker since it means it takes 30 to 35 minutes for someone to get
  access to their git repo or (worse) their fork
- This pagure instance has the same issue as the main one, including a heisenbug
  we're trying to track but have had no luck reproducing so far :(

What needs to be done:
- Fix the sync script
  - Make it *way* faster than it is
  - Make it creates the project on pagure using the releng user rather than
    relying on the first contributor it finds in the list of maintainers
  - Make fedmsg-genacls be triggered on pagure's fork fedmsg message so that we
    re-generate the gitolite configuration file when someone forks a project
    (and thus give them access to their fork)
- Once above is done: call for more testers

In the future:
- I think we will want to deprecate pkgdb entirely, so while the work above is
  important, ideally it won't be there for too long.
- With pkgdb out of the loop, we'll need to figure some things out:
  - Where/How to store the contact info for bugzilla
    - Not sure relying on pagure's ACLs there is the way to go since we would
      loose a level of granularity in the ACLs that I know people like and ask
      for (having commit w/o being on the CC list in bugzilla or being on the CC
      list w/o being a packager)
  - How/when to require people be part of the packager group in FAS?
    - Since one of the idea of pagure is to make it easier for "drive-by"
      contribution to spec files, requiring to be a packager should only be
      there for maintainers, but pagure doesn't have this level of
      information/requirement, so we would need to find something or some place
      to add this requirement or see if that requirement still stands

This is all I can think of for now, I'll update this thread if I come up with
more ideas/challenges.


Have a nice day,
Pierre

[Dennis Gilmore]

Can you please report this to releng also?

Dennis


El jue, 20-04-2017 a las 15:45 +0200, Pierre-Yves Chibon escribió:
>  Good Morning Everyone,
>  
>  I figured it has been a while since I reported progress on making
>  pagure a
>  front-end for dist-git.
>  
>  So here is a small status update.
>  
>  What works:
>  - Well currently pagure is working as a front-end for dist-git in
>  stg:
>    https://src.stg.fedoraproject.org/pagure/
>  - Hosting repos, browsing them, creating PR works all fine (w/ one ,
>  see
>    below)
>  - Features we want to exclude have been turned off (no user
>  management on the
>    pagure side)
>  
>  What does not work:
>  - Syncing the ACLs from pkgdb to gitolite takes ~3 minutes in prod,
>  in stg, the
>    updated script in stg (which sync the ACLs from pkgdb to gitolite
>  and pagure)
>    runs in ~30/35 minutes (if it doesn't crash, which my last run just
>  did)
>    This is a blocker since it means it takes 30 to 35 minutes for
>  someone to get
>    access to their git repo or (worse) their fork
>  - This pagure instance has the same issue as the main one, including
>  a heisenbug
>    we're trying to track but have had no luck reproducing so far :(
>  
>  What needs to be done:
>  - Fix the sync script
>    - Make it *way* faster than it is
>    - Make it creates the project on pagure using the releng user
>  rather than
>      relying on the first contributor it finds in the list of
>  maintainers
>    - Make fedmsg-genacls be triggered on pagure's fork fedmsg message
>  so that we
>      re-generate the gitolite configuration file when someone forks a
>  project
>      (and thus give them access to their fork)
>  - Once above is done: call for more testers
>  
>  In the future:
>  - I think we will want to deprecate pkgdb entirely, so while the work
>  above is
>    important, ideally it won't be there for too long.
>  - With pkgdb out of the loop, we'll need to figure some things out:
>    - Where/How to store the contact info for bugzilla
>      - Not sure relying on pagure's ACLs there is the way to go since
>  we would
>        loose a level of granularity in the ACLs that I know people
>  like and ask
>        for (having commit w/o being on the CC list in bugzilla or
>  being on the CC
>        list w/o being a packager)
>    - How/when to require people be part of the packager group in FAS?
>      - Since one of the idea of pagure is to make it easier for
>  "drive-by"
>        contribution to spec files, requiring to be a packager should
>  only be
>        there for maintainers, but pagure doesn't have this level of
>        information/requirement, so we would need to find something or
>  some place
>        to add this requirement or see if that requirement still stands
>  
>  This is all I can think of for now, I'll update this thread if I come
>  up with
>  more ideas/challenges.
>  
>  
>  Have a nice day,
>  Pierre
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproj
>  ect.org

[Till Maas]

Hi,

On Thu, Apr 20, 2017 at 03:45:16PM +0200, Pierre-Yves Chibon wrote:

>  - With pkgdb out of the loop, we'll need to figure some things
> out:
>    - Where/How to store the contact info for bugzilla
>      - Not sure relying on pagure's ACLs there is the way to go since we would
>        loose a level of granularity in the ACLs that I know people like and ask
>        for (having commit w/o being on the CC list in bugzilla or being on the CC
>        list w/o being a packager)
>    - How/when to require people be part of the packager group in FAS?
>      - Since one of the idea of pagure is to make it easier for "drive-by"
>        contribution to spec files, requiring to be a packager should only be
>        there for maintainers, but pagure doesn't have this level of
>        information/requirement, so we would need to find something or some place
>        to add this requirement or see if that requirement still stands 
sorry if I missed it, but pkgdb manages some more things where I not yet
understand how this maps to the pagure dist-git workflow:

- Request a new package to be added to pagure dist-git/a package to be
  unretired
- Requests for new (EPEL) branches
- Orphan/Retire a package
- Manage development status of collections (In development, live, EOL)
- Configuring upstream release monitoring
- Manage koshei integration

Kind regards
Till

[Randy Barlow]

On Thu, 2017-04-20 at 15:45 +0200, Pierre-Yves Chibon wrote:
>  - I think we will want to deprecate pkgdb entirely, so while the
> work
>  above is
>    important, ideally it won't be there for too long.
>  - With pkgdb out of the loop, we'll need to figure some things out: 
Just to note - Bodhi uses pkgdb to figure out ACLs and who to e-mail
about updates (possibly more, but that's what comes to mind) so we'd
need to coordinate the EOL of pkgdb with a Bodhi update too.

[Clint Savage]

On Thu, Apr 20, 2017 at 7:45 AM, Pierre-Yves Chibon <pingou(a)pingoured.fr&gt;
wrote:

>  Good Morning Everyone,
>
>  I figured it has been a while since I reported progress on making pagure a
>  front-end for dist-git.
>
>  .. snip .. 

>  What needs to be done:
>  - Fix the sync script
>    - Make it *way* faster than it is
>    - Make it creates the project on pagure using the releng user rather than
>      relying on the first contributor it finds in the list of maintainers
>    - Make fedmsg-genacls be triggered on pagure's fork fedmsg message so
>  that we
>      re-generate the gitolite configuration file when someone forks a
>  project
>      (and thus give them access to their fork)
>  - Once above is done: call for more testers
>  
Would git mirroring work well here? If so, it might be worth checking out
grokmirror. (https://github.com/lfit/grokmirror)

.. snip ..

>  Have a nice day,
>  Pierre
>
>  
Hope this helps,

herlo

[Dennis Gilmore]

El jue, 20-04-2017 a las 15:45 +0200, Pierre-Yves Chibon escribió:
>  Good Morning Everyone,
>  
>  I figured it has been a while since I reported progress on making
>  pagure a
>  front-end for dist-git.
>  
>  So here is a small status update.
>  
>  What works:
>  - Well currently pagure is working as a front-end for dist-git in
>  stg:
>    https://src.stg.fedoraproject.org/pagure/
>  - Hosting repos, browsing them, creating PR works all fine (w/ one ,
>  see
>    below)
>  - Features we want to exclude have been turned off (no user
>  management on the
>    pagure side)
>  
>  What does not work:
>  - Syncing the ACLs from pkgdb to gitolite takes ~3 minutes in prod,
>  in stg, the
>    updated script in stg (which sync the ACLs from pkgdb to gitolite
>  and pagure)
>    runs in ~30/35 minutes (if it doesn't crash, which my last run just
>  did)
>    This is a blocker since it means it takes 30 to 35 minutes for
>  someone to get
>    access to their git repo or (worse) their fork
>  - This pagure instance has the same issue as the main one, including
>  a heisenbug
>    we're trying to track but have had no luck reproducing so far :(
>  
>  What needs to be done:
>  - Fix the sync script
>    - Make it *way* faster than it is
>    - Make it creates the project on pagure using the releng user
>  rather than
>      relying on the first contributor it finds in the list of
>  maintainers
>    - Make fedmsg-genacls be triggered on pagure's fork fedmsg message
>  so that we
>      re-generate the gitolite configuration file when someone forks a
>  project
>      (and thus give them access to their fork)
>  - Once above is done: call for more testers
>  
>  In the future:
>  - I think we will want to deprecate pkgdb entirely, so while the work
>  above is
>    important, ideally it won't be there for too long.
>  - With pkgdb out of the loop, we'll need to figure some things out:
>    - Where/How to store the contact info for bugzilla
>      - Not sure relying on pagure's ACLs there is the way to go since
>  we would
>        loose a level of granularity in the ACLs that I know people
>  like and ask
>        for (having commit w/o being on the CC list in bugzilla or
>  being on the CC
>        list w/o being a packager)
>    - How/when to require people be part of the packager group in FAS?
>      - Since one of the idea of pagure is to make it easier for
>  "drive-by"
>        contribution to spec files, requiring to be a packager should
>  only be
>        there for maintainers, but pagure doesn't have this level of
>        information/requirement, so we would need to find something or
>  some place
>        to add this requirement or see if that requirement still stands
>  
>  This is all I can think of for now, I'll update this thread if I come
>  up with
>  more ideas/challenges. 
critpath is stored in pkgdb and pulled into bodhi and other
places. Was
just updating critpath and it stuck out to me.

Dennis

[Pierre-Yves Chibon]

On Thu, Apr 20, 2017 at 08:33:23PM +0200, Till Maas wrote:
>  Hi,
>  
>  On Thu, Apr 20, 2017 at 03:45:16PM +0200, Pierre-Yves Chibon wrote:
>  
>  > - With pkgdb out of the loop, we'll need to figure some things out:
>  >   - Where/How to store the contact info for bugzilla
>  >     - Not sure relying on pagure's ACLs there is the way to go since we would
>  >       loose a level of granularity in the ACLs that I know people like and ask
>  >       for (having commit w/o being on the CC list in bugzilla or being on the CC
>  >       list w/o being a packager)
>  >   - How/when to require people be part of the packager group in FAS?
>  >     - Since one of the idea of pagure is to make it easier for "drive-by"
>  >       contribution to spec files, requiring to be a packager should only be
>  >       there for maintainers, but pagure doesn't have this level of
>  >       information/requirement, so we would need to find something or some place
>  >       to add this requirement or see if that requirement still stands
>  
>  sorry if I missed it, but pkgdb manages some more things where I not yet
>  understand how this maps to the pagure dist-git workflow:
>  
>  - Request a new package to be added to pagure dist-git/a package to be
>    unretired 
New package to add == request for a new PoC of a new component in bugzilla

>  - Requests for new (EPEL) branches 
== request for a new PoC in Fedora EPEL in bugzilla

>  - Orphan/Retire a package 
Orphan == request to change the PoC of the component in bugzilla

>  - Manage development status of collections (In development, live,
> EOL) 
This is going away with the modularity effort and could be ported to PDC in the
mean time.

>  - Configuring upstream release monitoring
>  - Manage koshei integration 
We could include these two in the substitute of our current pkgdb.

The only remaining item which I'm not entirely sure how to handle it is retiring
a package.


Do note that this is all currently WIP or rather thoughts in progress :)

Pierre

[Pierre-Yves Chibon]

On Thu, Apr 20, 2017 at 03:19:49PM -0400, Randy Barlow wrote:
>  On Thu, 2017-04-20 at 15:45 +0200, Pierre-Yves Chibon wrote:
>  > - I think we will want to deprecate pkgdb entirely, so while the work
>  > above is
>  >   important, ideally it won't be there for too long.
>  > - With pkgdb out of the loop, we'll need to figure some things out:
>  
>  Just to note - Bodhi uses pkgdb to figure out ACLs and who to e-mail
>  about updates (possibly more, but that's what comes to mind) so we'd
>  need to coordinate the EOL of pkgdb with a Bodhi update too. 
Bodhi still sends email? Doesn't it rely on FMN now?

But yes that change will require coordination anyway indeed.


Pierre

[Pierre-Yves Chibon]

On Thu, Apr 20, 2017 at 09:35:17PM -0500, Dennis Gilmore wrote:
>  critpath is stored in pkgdb and pulled into bodhi and other places.
> Was
>  just updating critpath and it stuck out to me. 
We could move this pretty easily to PDC I think.


Pierre

[Pierre-Yves Chibon]

On Thu, Apr 20, 2017 at 01:51:14PM -0600, Clint Savage wrote:
>     On Thu, Apr 20, 2017 at 7:45 AM, Pierre-Yves Chibon
> <pingou(a)pingoured.fr&gt;
>     wrote:
>  
>       Good Morning Everyone,
>  
>       I figured it has been a while since I reported progress on making pagure
>       a
>       front-end for dist-git.
>  
>     .. snip ..
>  
>       What needs to be done:
>       - Fix the sync script
>       Â  - Make it *way* faster than it is
>       Â  - Make it creates the project on pagure using the releng user rather
>       than
>       Â  Â  relying on the first contributor it finds in the list of
>       maintainers
>       Â  - Make fedmsg-genacls be triggered on pagure's fork fedmsg message so
>       that we
>       Â  Â  re-generate the gitolite configuration file when someone forks a
>       project
>       Â  Â  (and thus give them access to their fork)
>       - Once above is done: call for more testers
>  
>     Would git mirroring work well here? If so, it might be worth checking out
>     grokmirror. (https://github.com/lfit/grokmirror) 
Not entirely sure to see how that would work. This seems to mirror git
repositories, but we are not mirroring them here, what we want to mirror is the
ACLs from pkgdb to gitolite (that we have already) and to pagure (that we have
but need to improve/optimize).

Could you expand on what your thoughts were? Maybe I'm miss-understanding them :)


Thanks,
Pierre

[Clint Savage]

On Fri, Apr 21, 2017 at 2:26 AM, Pierre-Yves Chibon <pingou(a)pingoured.fr&gt;
wrote:

>  On Thu, Apr 20, 2017 at 01:51:14PM -0600, Clint Savage wrote:
>  >    On Thu, Apr 20, 2017 at 7:45 AM, Pierre-Yves Chibon <
>  pingou(a)pingoured.fr&gt;
>  >    wrote:
>  >
>  >      Good Morning Everyone,
>  >
>  >      I figured it has been a while since I reported progress on making
>  pagure
>  >      a
>  >      front-end for dist-git.
>  >
>  >    .. snip ..
>  >
>  >      What needs to be done:
>  >      - Fix the sync script
>  >      Â  - Make it *way* faster than it is
>  >      Â  - Make it creates the project on pagure using the releng user
>  rather
>  >      than
>  >      Â  Â  relying on the first contributor it finds in the list of
>  >      maintainers
>  >      Â  - Make fedmsg-genacls be triggered on pagure's fork fedmsg
>  message so
>  >      that we
>  >      Â  Â  re-generate the gitolite configuration file when someone
>  forks a
>  >      project
>  >      Â  Â  (and thus give them access to their fork)
>  >      - Once above is done: call for more testers
>  >
>  >    Would git mirroring work well here? If so, it might be worth checking
>  out
>  >    grokmirror. (https://github.com/lfit/grokmirror)
>
>  Not entirely sure to see how that would work. This seems to mirror git
>  repositories, but we are not mirroring them here, what we want to mirror
>  is the
>  ACLs from pkgdb to gitolite (that we have already) and to pagure (that we
>  have
>  but need to improve/optimize).
>
>  Could you expand on what your thoughts were? Maybe I'm miss-understanding
>  them :)
>
>  
I am not completely sure how pagure worked under the covers, but I know it
uses gitolite, at least that's what I'd read above. At The Linux
Foundation, when I worked there, we mirrored bare repositories using
grokmirror, and fast, too, with hundreds, sometimes thousands, of remotes
as well.

Maybe the crux of the problem is around compiling the gitolite.conf (and
its plethora of sub-configuration files), so this might not be a solution
for you. But if it's about syncing between two git resources, grokmirror
would provide that functionality. It's been a while since I've worked with
it, but we could chat on IRC next week and maybe setup an example. Only if
that sort of thing would help though.

herlo




>  Thanks,
>  Pierre
>
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to infrastructure-leave@lists.
>  fedoraproject.org
>
>

[Randy Barlow]

On Fri, 2017-04-21 at 10:23 +0200, Pierre-Yves Chibon wrote:
>  Bodhi still sends email? Doesn't it rely on FMN now? 
Yeah Bodhi sends e-mails directly still. In fact, I've got a rather
vexing issue about it:

https://github.com/fedora-infra/bodhi/issues/1403

>  But yes that change will require coordination anyway indeed. 
Cool.

[Paul W. Frields]

Reply-to is set to infra@ so we can keep discussion in one place.

On Thu, Apr 20, 2017 at 12:08:35PM -0500, Dennis Gilmore wrote:
>  Can you please report this to releng also?
>  
>  Dennis
>  
>  
>  El jue, 20-04-2017 a las 15:45 +0200, Pierre-Yves Chibon escribió:
>  > Good Morning Everyone,
>  > 
>  > I figured it has been a while since I reported progress on making
>  > pagure a
>  > front-end for dist-git.
>  > 
>  > So here is a small status update.
>  > 
>  > What works:
>  > - Well currently pagure is working as a front-end for dist-git in
>  > stg:
>  >   https://src.stg.fedoraproject.org/pagure/
>  > - Hosting repos, browsing them, creating PR works all fine (w/ one ,
>  > see
>  >   below)
>  > - Features we want to exclude have been turned off (no user
>  > management on the
>  >   pagure side)
>  > 
>  > What does not work:
>  > - Syncing the ACLs from pkgdb to gitolite takes ~3 minutes in prod,
>  > in stg, the
>  >   updated script in stg (which sync the ACLs from pkgdb to gitolite
>  > and pagure)
>  >   runs in ~30/35 minutes (if it doesn't crash, which my last run just
>  > did)
>  >   This is a blocker since it means it takes 30 to 35 minutes for
>  > someone to get
>  >   access to their git repo or (worse) their fork
>  > - This pagure instance has the same issue as the main one, including
>  > a heisenbug
>  >   we're trying to track but have had no luck reproducing so far :(
>  > 
>  > What needs to be done:
>  > - Fix the sync script
>  >   - Make it *way* faster than it is
>  >   - Make it creates the project on pagure using the releng user
>  > rather than
>  >     relying on the first contributor it finds in the list of
>  > maintainers
>  >   - Make fedmsg-genacls be triggered on pagure's fork fedmsg message
>  > so that we
>  >     re-generate the gitolite configuration file when someone forks a
>  > project
>  >     (and thus give them access to their fork)
>  > - Once above is done: call for more testers
>  > 
>  > In the future:
>  > - I think we will want to deprecate pkgdb entirely, so while the work
>  > above is
>  >   important, ideally it won't be there for too long.
>  > - With pkgdb out of the loop, we'll need to figure some things out:
>  >   - Where/How to store the contact info for bugzilla
>  >     - Not sure relying on pagure's ACLs there is the way to go since
>  > we would
>  >       loose a level of granularity in the ACLs that I know people
>  > like and ask
>  >       for (having commit w/o being on the CC list in bugzilla or
>  > being on the CC
>  >       list w/o being a packager)
>  >   - How/when to require people be part of the packager group in FAS?
>  >     - Since one of the idea of pagure is to make it easier for
>  > "drive-by"
>  >       contribution to spec files, requiring to be a packager should
>  > only be
>  >       there for maintainers, but pagure doesn't have this level of
>  >       information/requirement, so we would need to find something or
>  > some place
>  >       to add this requirement or see if that requirement still stands
>  > 
>  > This is all I can think of for now, I'll update this thread if I come
>  > up with
>  > more ideas/challenges.
>  > 
>  > 
>  > Have a nice day,
>  > Pierre
>  > _______________________________________________
>  > infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  > To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproj
>  > ect.org 


>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org 

-- 
Paul W. Frields                                http://paul.frields.org/
  gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233  5906 ACDB C937 BD11 3717
  http://redhat.com/   -  -  -  -   http://pfrields.fedorapeople.org/
    The open source story continues to grow: http://opensource.com

[Paul W. Frields]

Just making sure the whole original message made it to rel-eng list.
Reply-to is set to infra@ so we can keep discussion in one place.

On Thu, Apr 20, 2017 at 12:08:35PM -0500, Dennis Gilmore wrote:
> Can you please report this to releng also?
> 
> Dennis
> 
> 
> El jue, 20-04-2017 a las 15:45 +0200, Pierre-Yves Chibon escribió:
> > Good Morning Everyone,
> > 
> > I figured it has been a while since I reported progress on making pagure a
> > front-end for dist-git.
> > 
> > So here is a small status update.
> > 
> > What works:
> > - Well currently pagure is working as a front-end for dist-git in stg:
> >   https://src.stg.fedoraproject.org/pagure/
> > - Hosting repos, browsing them, creating PR works all fine (w/ one , see
> >   below)
> > - Features we want to exclude have been turned off (no user management on the
> >   pagure side)
> > 
> > What does not work:
> > - Syncing the ACLs from pkgdb to gitolite takes ~3 minutes in prod, in stg, the
> >   updated script in stg (which sync the ACLs from pkgdb to gitolite and pagure)
> >   runs in ~30/35 minutes (if it doesn't crash, which my last run just did)
> >   This is a blocker since it means it takes 30 to 35 minutes for someone to get
> >   access to their git repo or (worse) their fork
> > - This pagure instance has the same issue as the main one, including a
heisenbug
> >   we're trying to track but have had no luck reproducing so far :(
> > 
> > What needs to be done:
> > - Fix the sync script
> >   - Make it *way* faster than it is
> >   - Make it creates the project on pagure using the releng user rather than
> >     relying on the first contributor it finds in the list of maintainers
> >   - Make fedmsg-genacls be triggered on pagure's fork fedmsg message so that
we
> >     re-generate the gitolite configuration file when someone forks a project
> >     (and thus give them access to their fork)
> > - Once above is done: call for more testers
> > 
> > In the future:
> > - I think we will want to deprecate pkgdb entirely, so while the work above is
> >   important, ideally it won't be there for too long.
> > - With pkgdb out of the loop, we'll need to figure some things out:
> >   - Where/How to store the contact info for bugzilla
> >     - Not sure relying on pagure's ACLs there is the way to go since we
would
> >       loose a level of granularity in the ACLs that I know people like and ask
> >       for (having commit w/o being on the CC list in bugzilla or being on the
CC
> >       list w/o being a packager)
> >   - How/when to require people be part of the packager group in FAS?
> >     - Since one of the idea of pagure is to make it easier for
"drive-by"
> >       contribution to spec files, requiring to be a packager should only be
> >       there for maintainers, but pagure doesn't have this level of
> >       information/requirement, so we would need to find something or some place
> >       to add this requirement or see if that requirement still stands
> > 
> > This is all I can think of for now, I'll update this thread if I come up
with
> > more ideas/challenges.
> > 
> > 
> > Have a nice day,
> > Pierre