On Tue, 31 Mar 2020 at 14:57, Neal Gompa <ngompa13@gmail.com> wrote:

On Tue, Mar 31, 2020 at 7:10 AM Clement Verna <cverna@fedoraproject.org> wrote:
>
> I just want to give a bit of insight from someone who is working day to day on Fedora's infrastructure, since I believe that might help give a bit more empathy towards the Why of this decision.
>
> For me the Fedora's Infrastructure is in a very bad shape there is a fair load of technical debt and trying to change or improve anything results in a long list of reason why we can do it because services X depends on service Y which depends on Z. Since I joined the CPE team (little bit more than 2 years ago) we have not been able to make any kind of significant progress towards fighting this technical debt. Every year we fill a white board with what needs to be done :
>
> Python 3 apps migration,
> FAS replacement,
> fedmsg retirement,
> FMN replacement,
> Fedora-packages replacement,
> PDC replacement,
> Porting application to OIDC,
> Improve Releng automation,
> Improving Anitya and the-new-hotness,
> .....
>
> Every single year the same items are coming back because we spend most of our time firefighting services to keep users happy and keep Fedora release schedule. This has a very demoralizing effect on the people working in the team, it seems like we will never be able to make any significant improvement, and our day to day job is to close couple tickets and you keep watching the pile of tickets growing. There is no feeling of accomplishment and a general sentiment that whatever we do, it will suck.
>
> A little over a year ago we have expressed our need to drop applications, this is something we have to do to be able to stay sane and keep a sustainable life-work balance. From that effort to handover applications (Elections, Nuancier, Fedocal, Badges) to another group of people in the community, not much happened mainly because of GDPR and the legal responsibility of owning such applications, but as far as I know we don't do much maintenance work on these applications any more since we now have a few volunteers that are looking after them or helping with finding an alternative solution.
>
> Now on the list of application we develop and run, I think Pagure is logical candidate to try and find an alternative to it, but before doing this it was important to make sure we captured all the use case and feature needed from a git forge and see if any of these justified spending cycles in development and maintenance work. My understanding of the decision is that Pagure does not provide any significant advantage over GitLab. I know that for many, the fact that Pagure is developed by Fedora is an advantage, but from my perspective as someone that as to deal with all the other services in Fedora's Infra this is a major disadvantage.
>
> Overall I think it is important to keep in mind that there is a lot of work happening behind the scene to provide the people in the Fedora community a good experience contributing to Fedora. I think we are doing a good job at it, but that takes us an enormous effort and over the long term this is not sustainable, also keeping in mind that we keep adding and want to keep adding new things to Fedora.
>
> I hope that my perspective helps a little.
>

Clement,

I want to say thank you for all the hard work you do as a member of
the Fedora community and as a member of the CPE team. You've done
fantastic work for the community and it's always a pleasure to work
with you. And that goes for all the members of the CPE team. I totally
understand where you are coming from. And it *is* very demoralizing to
see the same things over and over again, looking as if you've made no
progress on these things. I've been there with my work at $DAYJOB
before, many times. And as you and others are aware, I've been poking
around throughout infrastructure projects to help with some of these
initiatives over the years, so I'm keenly aware of the size and scope
of many of these.

However, I think some of this is self-inflicted. I don't want to
entirely rehash my original email with my thoughts on this, so please
read that for more detail[1], but I think we *really* should consider
that the lack of community exposure to to the codebases themselves
(especially as an avenue for contributing to the Fedora Project!) is
an underlying problem here. This has created a persistent cycle of
"community member makes cool project to support Fedora" → "it gets
adopted silently and no one really talks about it or advertises it" →
"nobody knows about it and the community member is the only one
working on it" → "the community member burns out and it gets piled
onto Fedora Engineering/CPE to maintain". This is basically how CPE
team got >100 codebases.

I agree with that, but for me there also a bit of a chicken and egg problem. I think we don't make more promotion and help people to onboard with the code base because we already don't have the time to maintain properly all these services, half of them if not more (I might be exaggerating a little :-) ) don't have an up to date Readme file with instruction on how to setup the development environment.

Another aspect that does not help a lot of our code bases are legacy application using technology stack from more than 10 years ago, try to find anyone interested to learn about Turbogears2 and mako templates (tech stack for fedora-packages).

This is a cycle that I've been *trying* to break. I'll be the first to
admit that I'm not the greatest programmer in the world, but I try to
contribute in terms of code, code reviews, testing, etc. But one thing
I've been doing for the past couple of years is *community project
advocacy*. I've been examining all the projects we have in Fedora and
seeing which ones I can help seed communities out of. My hope is that
by nurturing communities that are all interested in the success of
these projects, we can make them more independently sustainable.

Thank you for all your work and efforts, this is really appreciated.

As for Pagure itself, I think this is where we fundamentally disagree.
I think it behooves us to own and provide an experience tailored for
our community from beginning to end. That's why we have Koji, Bodhi,
Dist-Git, and many other tools in that part of the lifecycle. The
packager experience is literally the lifeblood of the project, and our
contributors are the core of what makes Fedora successful. Pagure
gives us an opportunity to do right by them that I *really* don't
think we can do with any alternatives.

I am not convinced that having a custom git forge is mandatory to provide an great experience to the community. I wasn't really around the community before Pagure, but I as far as I understand it the experience was better before Pagure and people were able to do more self servicing. I believe that there is an alternative to having the packager workflow tightly coupled to the git forge, this is also maybe a good opportunity to rethink some of that workflow and explore different solutions.

That does *not* mean that CPE team should be the sole owner of the
Pagure *codebase*. On that point, I agree. And that's why I've spent a
lot of time and energy since late 2018 working on building up that
community. It's finally starting to bear fruit too: there's at least
one entity interested in building a product around it and contributing
to help support that product, there's the FSF preparing to launch a
new forge using Pagure, there's the Trisquel GNU+Linux distribution
working on a Pagure deployment to host their code and packaging, and
there's a few other things I've got up my sleeve to help broaden the
community with not just users, but also contributors.

Are there deficiencies in Pagure? Of course there are. I'm not
claiming that Pagure is perfect. But I think that keeping on with
Pagure and giving that community an opportunity to close the gap on
needed features for Fedora/CentOS/Red Hat is the right way to go.
Right now, I don't *know* what the important gaps are. I can make some
guesses, but it'd be a lot better if we had a list of missing features
and their relative important and why. That would help focus
development to meet those needs.

The Fedora community itself has indicated that they want to keep on
with Pagure, and many Fedorans are Pythonistas, which means that
everyone can easily contribute to help make it better for everyone.

Genuine question, from the people that have participated in this thread, How many could dedicated couple hours a week to work on Pagure ? I think many community member are already struggling to keep up with their current level of contribution, I am not sure than everyone can easily contribute to Pagure. But I am happy to wrong :-).