Fedora Council December 2018 Hackfest Report

[Matthew Miller]

https://communityblog.fedoraproject.org/fedora-council-december-2018-hackfes...

In December, the Fedora Council met in Minneapolis, Minnesota for 
several days of meetings. With the holidays now behind us, here's our
summary of what happened.

Strategic direction update
==========================

The most important part of the meeting was our update to Fedora's
strategic direction. You may have read the [Community Blog post about
this] already. While the wording — or at least the fact that we've
written it down — may be new, the ideas aren't. This document
represents the next step from the efforts that began with [Fedora.next]
back in 2013. Our goal is to allow members of the Fedora community to
build solutions that focus on the specific requirements of their target
users. This means, among other things, a more decomposed and
self-service build process.

The earlier post generated a lot of discussion and feedback. I'm
planning a series of Community Blog articles summarizing and responding
to that feedback — stay tuned!

Objective updates
=================

We also heard from three of the four current Objective leads. (Our
Internet of Things lead, Peter Robinson couldn't make the meeting,
unfortunately.) The CI/CI Objective is wrapping up and Dominik Perpeet
will be publishing a summary soon. The Modularity Objective is also
ending its current phase. [Modules for Everyone] was a key feature of
Fedora 29, and the Modularity team is continuing to improve the
technical implementation as well as make it more approachable for
community contributors. Both Objective leads will be proposing next
steps to the Council in early 2019.\

The Lifecycle Objective is just getting started. It has two goals: make
the release process scale and allow & encourage more community ownership
for deliverables. This ties in tightly to the strategy update — this
work is required to get us to where we want to be. Paul Frields is
working to gauge what's needed to implement this now, including the
proposed long cycle for Fedora 31 ([discussed on the devel mailing
list]). The Council in general would prefer to keep to the regular
six-month cycle.

Mindshare
=========

The Mindshare team previously adopted a policy of spending up to $100
for release parties with minimal approval required. This has been
successful, and the Council would like Mindshare to build on this with
further investment. We agreed that Mindshare should adopt a policy of
allowing up to $150 for activities that promote the use of Fedora
solutions in communities. This could include release parties, web
hosting, or other relevant activities. We want to encourage
experimentation, so we're not requiring the activities to be successful
to get reimbursed — they just need to be successful to get future
funding. The Council's goal is to have 100 of these proposals funded in
FY2020 (which starts in March of 2019). In order to encourage funding
these proposals throughout the year, unallocated funds from this budget
will be pulled into the Council budget at the end of each fiscal
quarter. Look for guidance from Mindshare on how to make these requests
soon.

Communication
=============

We've started experimenting with using [Discourse] as the asynchronous
communication tool for some teams. For synchronous communication, some
teams are using Telegram in addition to — or instead of — IRC. Each
platform has strengths and weaknesses. After discussion, the Council
came to the conclusion that communication fragmentation is unavoidable.
In a project as big as Fedora, people work in different ways and have
different preferences. We leave it to each team to decide what
communication methods work best for their team.

To help connect everyone together despite this, we have requested a
central project management service from the Infrastructure team —
probably [Taiga], although we're asking the team to also look at GitLab
for this purpose. We'll have a dedicated instance, likely hosted, and we
ask each team to have a minimum presence on that tool, whether they use
it otherwise or not. The presence should, at a minimum, indicate the
team's communication methods for synchronous and asynchronous
communication and where project information may be found if not in the
shared tool. That way, there will be an automatically-curated list of
active teams. (See https://tree.taiga.io/discover for an idea of what
this would look like — imagine each project there is a Fedora team.)

Infrastructure
==============

The success of our strategy depends on improvements to our
infrastructure. The Infrastructure team has limited resources, so we
need to ensure they're able to work on areas that add the most value to
the project. This means a shift away from running all layers of the
stack and focusing more on application management. The goal is to have
the Infra team administering applications, not low-level infrastructure.
(Even if that makes the team name confusing — sorry!) We want agility
in our applications and deployment. We want drive-by contributors to be
able to realistically contribute to the infrastructure team.

We also talked about GitHub. Ideally, we want everything to be on open
source services (e.g. Taiga, Pagure, or GitLab). But, as a pragmatic
matter, we recognize that GitHub has a huge network effect — there are
millions of users and developers there, and millions of open source and
free software projects hosted there, including software that's
fundamental to the Fedora operating system. We'd like better integration
and syncing with tools like Pagure to give access to that network effect
on all-free software, but we also know that there isn't a lot of
developer time to make and maintain those kind of features. Therefore,
we're willing to accept people in Fedora hosting their subprojects on
GitHub. We've got to focus on what we do that's unique (and only do
things which are unique when we have a special need to meet our project
goals). Git hosting is not one of those things.

General Council business
========================

The Council wants to make it clear that community input is welcome on
Council matters. Members of the Fedora community may provide non-binding
votes on Council tickets and participate in meetings. Speaking of
meetings, we're replacing the Subproject report meetings with regular
updates from the FESCo, Mindshare, and Diversity & Inclusion
representatives. This should help provide better visibility into those
organizations for the Council and the community at large.

We will also move all Council policies out of the wiki to
[docs.fedoraproject.org]. The Council will use the wiki only as a
scratch space for works in progress. Durable documentation will live on
the docs site. We encourage other teams to consider doing the same.

Meeting Minutes
===============

We didn't actually conduct the meeting in IRC, but we took minutes in
the same way. Here's our detailed record.

#startmeeting Fedora Council Hackfest 2018

#chair mattdm bcotton tyll contyk dperpeet langdon dgilmore bex
jonatoni sumantro stickster

#topic Mission overview + Strategic framework

#info Community Members are encouraged to contribute to the decision
process with non-binding votes

#link
https://qz.com/work/1468580/the-four-layers-of-communication-in-a-functional-team/

#info mattdm shows his favorite graph

#topic Fedora Project Strategy: "How do we make our mission real?"

#action Dominik will speak more loudly

#agreed Council will replace subproject report meetings with updates
from FESCo, Mindshare, and D&I rep

#agreed Council adopts the strategy proposal (+9, 0, -0)

#action mattdm to post the strategy proposal to commblog

#agreed Council will make a final vote on the strategy proposal on 9
January 2019 (+8,0,0)

#topic Moving Council Policies to Docs

#agreed Council will move all Council policies and other durable
Council documents from the Wiki to docs.fedoraproject.org and remove old
wiki pages. (8,0,-0)

#info policies should be kept in a repo separate from other documents
for ease of watching

#topic Objective Update: CI/CD

#action

dperpeet to write up final objective report for publication on Community
Blog by January 1

#action dperpeet to propose a new Objective for future CI/CD work

#info the new Objective should include working with other stakeholder
groups including IoT and SilverBlue

#topic Objective Update: Modularity

#action langdon to propose a new Objective for future Modularity work

#topic Objective Update: Lifecycle

#topic $150 release parties

#agreed Mindshare should move the easy process to $150 and encourage
more non-RP uses (+9,0,-0)

#agreed: Mindshare should start providing $150 base support for
solutions to help them grow (+9,0,-0)

#agreed: Mindshare should start attracting larger requests and develop
a process. These requests are judged using Council provided Fedora
Project strategy as guidance. (+9,0,-0)

#agreed: Mindshare should target at least 100 $150 events in FY20
(+9,0,-0)

#agreed Unspent budget allocated to the $150 event program will be
pulled into the Council budget at the end of each fiscal quarter
beginning with FY20 (+10,0,-0)

#topic Localization

#action bex to begin a conversation in the translation community about
a platform that meets the needs of the translation workflow

#topic Logo

#info Adam Miller will not get a tattoo with our current logo because
people will ask him why he has a Facebook tattoo

#info The design team is working on getting a proposal to the community
to vote soon so we can select the new one by January 31 in final form
for production

#topic Infrastructure

#agreed The Council supports greater efficiency in the infrastructure
to allow more to be done, even when this means that we move away from
self-hosted or self-maintained infrastructure. (+9,0,-0)

#agreed The Fedora Project wants to advance free and open source
software and as a pragmatic matter we recognize that some infrastructure
needs may be best served by using closed source or non-free tools today.
Therefore the Council is willing to accept closed source or non-free
tools in Fedora's infrastructure where free and open source tools are
not viable or not available. (+9,0,-0)

#action contyk, FESCo to work with Infra to examine current
applications and determine: 1. which applications can be moved out of
the datacenter immediately or in the short term, 2. Which applications
have industry-standard open source or proprietary alternatives that we
could move to.

#topic Communication

#agreed Fedora will offer a central place for teams and SIGs to be
discoverable, do project management, etc. Having a landing page will be
a requirement for all teams and SIGs in Fedora. (+10, 0, -0)

#agreed The Council will ask the Infrastructure team to evaluate
providing the central place as Taiga versus Gitlab based on requirements
provided by Council. (+10, 0, -0)

#action mattdm to write requirements doc for these two things above.

#agreed The Fedora Council embraces fragmentation in our communication
platforms — this is a reality we can't fight. The Central Place will
provide a way for anyone to find the communication tools used by any
group. (+10, 0, -0)

#topic [Ticket #198]

#agreed Document proposed in ticket #198 is accepted for delivery to
Legal for drafting updates to the Trademark policy. (+10, 0, -0)

#topic Code of Conduct enforcement

#agreed The FCAIC is empowered to take action on Code of Conduct
reports with an additional +1 from another core Council member or the
Diversity & Inclusion Advisor and report back to Council. (+10, 0, -0)

#topic Ask Fedora and getting help

#agreed Council authorizes the hosting of a separate Discourse instance
to replace ask.fedoraproject.org to be funded out of Fedora community
budget. (+9, 0, 0)

#endmeeting

[Community Blog post about this]: https://communityblog.fedoraproject.org/?p=6802
  [Fedora.next]: https://fedoraproject.org/wiki/Fedora.next
  [Modules for Everyone]: https://fedoraproject.org/wiki/Changes/ModulesForEveryone
  [discussed on the devel mailing list]: https://lists.fedoraproject.org/archives/list/[e-mail]/...
  [Discourse]: https://discussion.fedoraproject.org/
  [Taiga]: https://taiga.io/
  [docs.fedoraproject.org]: https://docs.fedoraproject.org/
  [Ticket #198]: https://pagure.io/Fedora-Council/tickets/issue/198

[Kevin Fenzi]

On 1/4/19 7:49 AM, Matthew Miller wrote:
...snip...
> To help connect everyone together despite this, we have requested a
> central project management service from the Infrastructure team —
> probably [Taiga], although we're asking the team to also look at GitLab
> for this purpose. We'll have a dedicated instance, likely hosted, and we
> ask each team to have a minimum presence on that tool, whether they use
> it otherwise or not. The presence should, at a minimum, indicate the
> team's communication methods for synchronous and asynchronous
> communication and where project information may be found if not in the
> shared tool. That way, there will be an automatically-curated list of
> active teams. (See https://tree.taiga.io/discover for an idea of what
> this would look like — imagine each project there is a Fedora team.)

Is this request public anywhere? I know we talked about it in the past,
but it would be nice to have someplace people to see the requirements
and chime in with their thoughts. I'm not sure if this would best be a
list thing or a ticket or what...
> Infrastructure
>
> The success of our strategy depends on improvements to our
> infrastructure. The Infrastructure team has limited resources, so we
> need to ensure they're able to work on areas that add the most value to
> the project.

ok.
> This means a shift away from running all layers of the
> stack and focusing more on application management. The goal is to have
> the Infra team administering applications, not low-level infrastructure.

I am not sure if this really follows from the goal above.
Sure, low level infrastructure takes some time, but I don't think it's
all that much, and also some of it's unavoidable.
> (Even if that makes the team name confusing — sorry!) We want agility
> in our applications and deployment. We want drive-by contributors to be
> able to realistically contribute to the infrastructure team.

Sure, those are all good things, but I don't understand what it is that
you want us to do here. (more below)
> We also talked about GitHub. Ideally, we want everything to be on open
> source services (e.g. Taiga, Pagure, or GitLab). But, as a pragmatic
> matter, we recognize that GitHub has a huge network effect — there are
> millions of users and developers there, and millions of open source and
> free software projects hosted there, including software that's
> fundamental to the Fedora operating system. We'd like better integration
> and syncing with tools like Pagure to give access to that network effect
> on all-free software, but we also know that there isn't a lot of
> developer time to make and maintain those kind of features. Therefore,
> we're willing to accept people in Fedora hosting their subprojects on
> GitHub. We've got to focus on what we do that's unique (and only do
> things which are unique when we have a special need to meet our project
> goals). Git hosting is not one of those things.

Sure, in infrastructure for applications we have left hosting decisions
up to the people doing the work. Thats why there are some of our infra
apps on github.

...snip...
> #topic Infrastructure
>
> #agreed The Council supports greater efficiency in the infrastructure
> to allow more to be done, even when this means that we move away from
> self-hosted or self-maintained infrastructure. (+9,0,-0)
>
> #agreed The Fedora Project wants to advance free and open source
> software and as a pragmatic matter we recognize that some infrastructure
> needs may be best served by using closed source or non-free tools today.
> Therefore the Council is willing to accept closed source or non-free
> tools in Fedora's infrastructure where free and open source tools are
> not viable or not available. (+9,0,-0)

So, what do you mean running non free tools "in Fedora's infrastructure"
here? As noted above we always said upstreams for applications or the
like can do as the people doing the work decide, but does this mean we
should (for example) consider running oracle db if we feel it would be
better for our database loads? Can you clarify here? We don't currently
run any non free software actually in infrastructure.
> #action contyk, FESCo to work with Infra to examine current
> applications and determine: 1. which applications can be moved out of
> the datacenter immediately or in the short term,

Moved out of the datacenter to... where? Cloud instances?
Hosted solutions? Based on what critera?

This is going to cause a lot of short term work for... well, everyone.
Especially in the case of things we don't spend much time on now, since
everyone will have to adjust their workflow and we will have to spend
time migrating things.
> Which applications
>
> have industry-standard open source or proprietary alternatives that we
> could move to.

I'm not coming up with too long a list here. Most of our apps are highly
specialised for distro making, but:

* fedocal (which we pretty much spend no time on)
* pagure (which we want to keep? or no?)
* nagios (pagerduty?)
* magazine /communityblog ( wordpress.org? but there's very little time
spent on these too)
* mailing lists (discourse?)
* openstack (The big consumer here is copr... if it could move to the
cloud or somewhere and we didn't need a openstack I would be very happy).

I think things like koji are particularly ill suited to move anywhere
since we need to run Fedora on the builders, they have to run virt, and
we have lots of arches that aren't commonly available in cloud.

Thanks for the update and discussion on this!

kevin

[Matthew Miller]

On Mon, Jan 07, 2019 at 09:26:25AM -0800, Kevin Fenzi wrote:
> > central project management service from the Infrastructure team —
> > probably [Taiga], although we're asking the team to also look at GitLab
> > for this purpose. We'll have a dedicated instance, likely hosted, and we
> > ask each team to have a minimum presence on that tool, whether they use
> > it otherwise or not. The presence should, at a minimum, indicate the
> > team's communication methods for synchronous and asynchronous
> > communication and where project information may be found if not in the
> > shared tool. That way, there will be an automatically-curated list of
> > active teams. (See https://tree.taiga.io/discover for an idea of what
> > this would look like — imagine each project there is a Fedora team.)
>
> Is this request public anywhere? I know we talked about it in the past,
> but it would be nice to have someplace people to see the requirements
> and chime in with their thoughts. I'm not sure if this would best be a
> list thing or a ticket or what...

Maybe tracked in Taiga? :) I've been talking with Jim about it —  I'll check
with him about what he thinks would be best.
> > This means a shift away from running all layers of the
> > stack and focusing more on application management. The goal is to have
> > the Infra team administering applications, not low-level infrastructure.
>
> I am not sure if this really follows from the goal above.
> Sure, low level infrastructure takes some time, but I don't think it's
> all that much, and also some of it's unavoidable.

As I understand it, there was just a multi-month, multi-person project to
update OpenStack. That's the kind of thing I'm talking about.
> > #agreed The Fedora Project wants to advance free and open source
> > software and as a pragmatic matter we recognize that some infrastructure
> > needs may be best served by using closed source or non-free tools today.
> > Therefore the Council is willing to accept closed source or non-free
> > tools in Fedora's infrastructure where free and open source tools are
> > not viable or not available. (+9,0,-0)
>
> So, what do you mean running non free tools "in Fedora's infrastructure"
> here? As noted above we always said upstreams for applications or the
> like can do as the people doing the work decide, but does this mean we
> should (for example) consider running oracle db if we feel it would be
> better for our database loads? Can you clarify here? We don't currently
> run any non free software actually in infrastructure.

Hmmm; I'm not super sure about that exact wording in re-reading it. I think
"infrastructure" there is to be read very broadly. We meant things like:
hosting VMs in a cloud service where the cloud service stack might not be
open source all the way to bare metal (GCE, Azure, AWS). We'd want to make
sure we aren't locked in (for example, by using OpenShift on top of that).
This certainly wasn't meant to be an instruction to go look for proprietary
replacements for infrastructure. And "we feel it would be better" is
certainly not the same as "not viable or not available".

Although we aren't in this situation now, one thing I can imagine is back
before Amazon released open-licensed versions of their tool to upload
images. I think it probably would have been okay for Fedora to use that
proprietary tool in order to make Fedora Cloud available to people.

n
> > #action contyk, FESCo to work with Infra to examine current
> > applications and determine: 1. which applications can be moved out of
> > the datacenter immediately or in the short term,
>
> Moved out of the datacenter to... where? Cloud instances?
> Hosted solutions? Based on what critera?

Any of those things. I think in general, if there's an open source hosted
version of something (like Taiga) that should absolutely be the default.
Otherwise, yes, cloud instances —  or better, running containers in a
provided-to-us-as-a-service OpenShift.
> This is going to cause a lot of short term work for... well, everyone.
> Especially in the case of things we don't spend much time on now, since
> everyone will have to adjust their workflow and we will have to spend
> time migrating things.

I recognize that this is non-trivial, and it's possible that it's easier to
just let some things sit where they are. But a lot of those normally
low-maintenance things are tech-debt time bombs. Or, they're really
languishing.
> > Which applications
> >
> > have industry-standard open source or proprietary alternatives that we
> > could move to.
>
> I'm not coming up with too long a list here. Most of our apps are highly
> specialised for distro making, but:
> fedocal (which we pretty much spend no time on)

Yeah, this'd be a good candidate. It was a great, useful tool for its time
but is rather dated. It could use a *lot* of development work, and
realistically, we all know it's not going to get it.
> pagure (which we want to keep? or no?)

I think we're pretty committed to pagure for src.fpo. But for general git
hosting... I think it's a great project — but I also see that Debian, GNOME,
Linux Kernel, are evaluating it and picking GitLab. Meanwhile GitLab (and
GitHub) are growing new useful features rapidly and there's no way we can
invest development to keep up. I don't think anything like "kill pagure!"
but I also don't think that Fedora *as a project* can afford to invest
significant time.
> nagios (pagerduty?)

I had a great conversation with Jef Spatela who is now at Sensu at
AnsibleFest. Maybe Sensu. :)
> magazine /communityblog ( wordpress.org? but there's very little time
>
> spent on these too)

All of the "very little times" do add up.
> mailing lists (discourse?)

Or a more traditional mailing list provider — maybe ponee.io?
https://fedora.fsn.ponee.io/
> openstack (The big consumer here is copr... if it could move to the
>
> cloud or somewhere and we didn't need a openstack I would be very happy).

Yes. I think this is something we can make happen. Possibly we can work
something out with Mass Open Cloud.
> I think things like koji are particularly ill suited to move anywhere
> since we need to run Fedora on the builders, they have to run virt, and
> we have lots of arches that aren't commonly available in cloud.

Yeah, multi-arch is going to be a sticking point for a lot of things.

[Neal Gompa]

On Mon, Jan 7, 2019 at 3:41 PM Matthew Miller  wrote:
> On Mon, Jan 07, 2019 at 09:26:25AM -0800, Kevin Fenzi wrote:
> > > central project management service from the Infrastructure team —
> > > probably [Taiga], although we're asking the team to also look at GitLab
> > > for this purpose. We'll have a dedicated instance, likely hosted, and we
> > > ask each team to have a minimum presence on that tool, whether they use
> > > it otherwise or not. The presence should, at a minimum, indicate the
> > > team's communication methods for synchronous and asynchronous
> > > communication and where project information may be found if not in the
> > > shared tool. That way, there will be an automatically-curated list of
> > > active teams. (See https://tree.taiga.io/discover for an idea of what
> > > this would look like — imagine each project there is a Fedora team.)
> >
> > Is this request public anywhere? I know we talked about it in the past,
> > but it would be nice to have someplace people to see the requirements
> > and chime in with their thoughts. I'm not sure if this would best be a
> > list thing or a ticket or what...
>
> Maybe tracked in Taiga? :) I've been talking with Jim about it —  I'll check
> with him about what he thinks would be best.
> > > This means a shift away from running all layers of the
> > > stack and focusing more on application management. The goal is to have
> > > the Infra team administering applications, not low-level infrastructure.
> >
> > I am not sure if this really follows from the goal above.
> > Sure, low level infrastructure takes some time, but I don't think it's
> > all that much, and also some of it's unavoidable.
>
> As I understand it, there was just a multi-month, multi-person project to
> update OpenStack. That's the kind of thing I'm talking about.
> > > #agreed The Fedora Project wants to advance free and open source
> > > software and as a pragmatic matter we recognize that some infrastructure
> > > needs may be best served by using closed source or non-free tools today.
> > > Therefore the Council is willing to accept closed source or non-free
> > > tools in Fedora's infrastructure where free and open source tools are
> > > not viable or not available. (+9,0,-0)
> >
> > So, what do you mean running non free tools "in Fedora's infrastructure"
> > here? As noted above we always said upstreams for applications or the
> > like can do as the people doing the work decide, but does this mean we
> > should (for example) consider running oracle db if we feel it would be
> > better for our database loads? Can you clarify here? We don't currently
> > run any non free software actually in infrastructure.
>
> Hmmm; I'm not super sure about that exact wording in re-reading it. I think
> "infrastructure" there is to be read very broadly. We meant things like:
> hosting VMs in a cloud service where the cloud service stack might not be
> open source all the way to bare metal (GCE, Azure, AWS). We'd want to make
> sure we aren't locked in (for example, by using OpenShift on top of that).
> This certainly wasn't meant to be an instruction to go look for proprietary
> replacements for infrastructure. And "we feel it would be better" is
> certainly not the same as "not viable or not available".
>
> Although we aren't in this situation now, one thing I can imagine is back
> before Amazon released open-licensed versions of their tool to upload
> images. I think it probably would have been okay for Fedora to use that
> proprietary tool in order to make Fedora Cloud available to people.
>
> n
> > > #action contyk, FESCo to work with Infra to examine current
> > > applications and determine: 1. which applications can be moved out of
> > > the datacenter immediately or in the short term,
> >
> > Moved out of the datacenter to... where? Cloud instances?
> > Hosted solutions? Based on what critera?
>
> Any of those things. I think in general, if there's an open source hosted
> version of something (like Taiga) that should absolutely be the default.
> Otherwise, yes, cloud instances —  or better, running containers in a
> provided-to-us-as-a-service OpenShift.

We should be careful about this. We got screwed over with Transifex
years ago, and it'd be bad if that was repeated.

If we can make self-hosting our services simpler, easier, and let more
people do it, that'd be the holy grail.
> > This is going to cause a lot of short term work for... well, everyone.
> > Especially in the case of things we don't spend much time on now, since
> > everyone will have to adjust their workflow and we will have to spend
> > time migrating things.
>
> I recognize that this is non-trivial, and it's possible that it's easier to
> just let some things sit where they are. But a lot of those normally
> low-maintenance things are tech-debt time bombs. Or, they're really
> languishing.

It'd be better if it was easier for more people to participate. We
have a lot of infra-types. It's kind of amazing in a bad way that
despite all that, we don't really have an easy way for more people to
help on that side.
> > > Which applications
> > >
> > > have industry-standard open source or proprietary alternatives that we
> > > could move to.
> >
> > I'm not coming up with too long a list here. Most of our apps are highly
> > specialised for distro making, but:
> > fedocal (which we pretty much spend no time on)
>
> Yeah, this'd be a good candidate. It was a great, useful tool for its time
> but is rather dated. It could use a *lot* of development work, and
> realistically, we all know it's not going to get it.

Are there any nice open source calendar/meeting assignment web apps
out there we could deploy? I feel like this is a common enough venture
that someone has done something...
> > pagure (which we want to keep? or no?)
>
> I think we're pretty committed to pagure for src.fpo. But for general git
> hosting... I think it's a great project — but I also see that Debian, GNOME,
> Linux Kernel, are evaluating it and picking GitLab. Meanwhile GitLab (and
> GitHub) are growing new useful features rapidly and there's no way we can
> invest development to keep up. I don't think anything like "kill pagure!"
> but I also don't think that Fedora *as a project* can afford to invest
> significant time.

Pagure as a project is starting to pick up steam in its own right. And
users outside of Fedora (both public and private) are also turning
into contributors. If anything, this is one of the better successes
we've had in a while.

As for the feature development, Pagure is keeping pace with solutions
like GitHub. We're now at the point where Pagure meets the features
that Debian was requesting for their Alioth migration, and other
groups are considering rolling out Pagure instances as well.

As for the pagure.io forge, a large part of its lack of success is that
people don't know how to move to it. Heck, even I don't really know
how to effectively migrate a project from GitHub.com or GitLab.com or
some other site to Pagure.io and migrate things like issues, PRs, etc.
It is a good solution and it is developing rapidly into being a good
competitor to the others.

It helps that Pagure is drastically simpler than most other solutions
to deploy and extend.
> > nagios (pagerduty?)
>
> I had a great conversation with Jef Spatela who is now at Sensu at
> AnsibleFest. Maybe Sensu. :)

I've had poor experiences with PagerDuty... Never used Sesu, but it
looks interesting...
> > magazine /communityblog ( wordpress.org? but there's very little time
> >
> > spent on these too)
>
> All of the "very little times" do add up.
> > mailing lists (discourse?)
>
> Or a more traditional mailing list provider — maybe ponee.io?
> https://fedora.fsn.ponee.io/

Please no. I *like* our mailing list system as it is.
> > openstack (The big consumer here is copr... if it could move to the
> >
> > cloud or somewhere and we didn't need a openstack I would be very happy).
>
> Yes. I think this is something we can make happen. Possibly we can work
> something out with Mass Open Cloud.

I think it's pretty obvious our Copr host platform (RHOSP 5) has been
a disappointment. It's unfortunate that there's no formal avenue for
people to sponsor and offer resources to improve services like Copr
and assist with improving the quality of the service. People want to
depend on it, and it's still not getting the love it needs.

Supposedly it was going to be upgraded and improved, but I dunno
what's going on there. Beyond that, we should consider making it
possible for people and companies to be able to sponsor effort and
resources in Fedora, similar to what openSUSE does. That's how their
OBS has so much more resources than our Koji and Copr systems
combined.
> > I think things like koji are particularly ill suited to move anywhere
> > since we need to run Fedora on the builders, they have to run virt, and
> > we have lots of arches that aren't commonly available in cloud.
>
> Yeah, multi-arch is going to be a sticking point for a lot of things.

Losing the ability for people to be able to trivially self-host
variants of our infrastructure would be a huge loss, in my mind.

[Kevin Fenzi]

On 1/7/19 12:41 PM, Matthew Miller wrote:
> On Mon, Jan 07, 2019 at 09:26:25AM -0800, Kevin Fenzi wrote:

...snip...
> > I am not sure if this really follows from the goal above.
> > Sure, low level infrastructure takes some time, but I don't think it's
> > all that much, and also some of it's unavoidable.
>
> As I understand it, there was just a multi-month, multi-person project to
> update OpenStack. That's the kind of thing I'm talking about.

Yes. I'd be 1100% fine with never dealing with openstack again, but I
think it's largely the exception/outlyer here.
> > So, what do you mean running non free tools "in Fedora's infrastructure"
> > here? As noted above we always said upstreams for applications or the
> > like can do as the people doing the work decide, but does this mean we
> > should (for example) consider running oracle db if we feel it would be
> > better for our database loads? Can you clarify here? We don't currently
> > run any non free software actually in infrastructure.
>
> Hmmm; I'm not super sure about that exact wording in re-reading it. I think
> "infrastructure" there is to be read very broadly. We meant things like:
> hosting VMs in a cloud service where the cloud service stack might not be
> open source all the way to bare metal (GCE, Azure, AWS). We'd want to make
> sure we aren't locked in (for example, by using OpenShift on top of that).
> This certainly wasn't meant to be an instruction to go look for proprietary
> replacements for infrastructure. And "we feel it would be better" is
> certainly not the same as "not viable or not available".

ok. That makes me feel better. :)

You really can't be 100% open source and provide network services (look
at all those non free routers handling your packets, etc). I just think
we need to make sure and use as much open source as we can and avoid
building on something we can't get out of.
> Although we aren't in this situation now, one thing I can imagine is back
> before Amazon released open-licensed versions of their tool to upload
> images. I think it probably would have been okay for Fedora to use that
> proprietary tool in order to make Fedora Cloud available to people.

ok.
> n
> > > #action contyk, FESCo to work with Infra to examine current
> > > applications and determine: 1. which applications can be moved out of
> > > the datacenter immediately or in the short term,
> >
> > Moved out of the datacenter to... where? Cloud instances?
> > Hosted solutions? Based on what critera?
>
> Any of those things. I think in general, if there's an open source hosted
> version of something (like Taiga) that should absolutely be the default.
> Otherwise, yes, cloud instances —  or better, running containers in a
> provided-to-us-as-a-service OpenShift.

I'm not sure how much control that gives up to do things that way, but I
guess we can investigate it.
> > This is going to cause a lot of short term work for... well, everyone.
> > Especially in the case of things we don't spend much time on now, since
> > everyone will have to adjust their workflow and we will have to spend
> > time migrating things.
>
> I recognize that this is non-trivial, and it's possible that it's easier to
> just let some things sit where they are. But a lot of those normally
> low-maintenance things are tech-debt time bombs. Or, they're really
> languishing.

Sure, but I am objecting to the "move it all now!"
How about "move it all when it makes sense?"
> > > Which applications
> > >
> > > have industry-standard open source or proprietary alternatives that we
> > > could move to.
> >
> > I'm not coming up with too long a list here. Most of our apps are highly
> > specialised for distro making, but:
> > fedocal (which we pretty much spend no time on)
>
> Yeah, this'd be a good candidate. It was a great, useful tool for its time
> but is rather dated. It could use a *lot* of development work, and
> realistically, we all know it's not going to get it.

True.
> > pagure (which we want to keep? or no?)
>
> I think we're pretty committed to pagure for src.fpo. But for general git
> hosting... I think it's a great project — but I also see that Debian, GNOME,
> Linux Kernel, are evaluating it and picking GitLab. Meanwhile GitLab (and
> GitHub) are growing new useful features rapidly and there's no way we can
> invest development to keep up. I don't think anything like "kill pagure!"
> but I also don't think that Fedora *as a project* can afford to invest
> significant time.

ok.
> > nagios (pagerduty?)
>
> I had a great conversation with Jef Spatela who is now at Sensu at
> AnsibleFest. Maybe Sensu. :)

well, my tenative plan was to just look at configuring the Prometheus
that comes with openshift to handle those things in openshift and then
move as much as makes sense there... but we could look at it.
> > magazine /communityblog ( wordpress.org? but there's very little time
> >
> > spent on these too)
>
> All of the "very little times" do add up.

I suppose, but the migration of them is much more time.
> > mailing lists (discourse?)
>
> Or a more traditional mailing list provider — maybe ponee.io?
> https://fedora.fsn.ponee.io/

Never heard of it, but ok.
I think this is a place where moving will also be pretty disruptive,
especially if people need to change filters, but perhaps we could avoid
that.
> > openstack (The big consumer here is copr... if it could move to the
> >
> > cloud or somewhere and we didn't need a openstack I would be very happy).
>
> Yes. I think this is something we can make happen. Possibly we can work
> something out with Mass Open Cloud.

Well, the sticking point is the storage. It needs a lot.
> > I think things like koji are particularly ill suited to move anywhere
> > since we need to run Fedora on the builders, they have to run virt, and
> > we have lots of arches that aren't commonly available in cloud.
>
> Yeah, multi-arch is going to be a sticking point for a lot of things.

Yep. Also things that are specialized to our needs and not generic
enough to have anyone who deals with them.

kevin

[Kevin Fenzi]

On 1/7/19 1:04 PM, Neal Gompa wrote:
...snip...
> It'd be better if it was easier for more people to participate. We
> have a lot of infra-types. It's kind of amazing in a bad way that
> despite all that, we don't really have an easy way for more people to
> help on that side.

There definitely is...become an appentice, send patches for things, get
added to other groups and help out with that area.

True more access takes more time, but we definitely have had community
folks helping out.

...snip...
> Pagure as a project is starting to pick up steam in its own right. And
> users outside of Fedora (both public and private) are also turning
> into contributors. If anything, this is one of the better successes
> we've had in a while.
>
> As for the feature development, Pagure is keeping pace with solutions
> like GitHub. We're now at the point where Pagure meets the features
> that Debian was requesting for their Alioth migration, and other
> groups are considering rolling out Pagure instances as well.
>
> As for the pagure.io forge, a large part of its lack of success is that
> people don't know how to move to it. Heck, even I don't really know
> how to effectively migrate a project from GitHub.com or GitLab.com or
> some other site to Pagure.io and migrate things like issues, PRs, etc.
> It is a good solution and it is developing rapidly into being a good
> competitor to the others.

https://pagure.io/pagure-importer
was used to migrate things from trac, and it has a github mode, but I am
not sure what state it's in.
> > > openstack (The big consumer here is copr... if it could move to the
> > >
> > > cloud or somewhere and we didn't need a openstack I would be very happy).
> >
> > Yes. I think this is something we can make happen. Possibly we can work
> > something out with Mass Open Cloud.
>
> I think it's pretty obvious our Copr host platform (RHOSP 5) has been
> a disappointment. It's unfortunate that there's no formal avenue for
> people to sponsor and offer resources to improve services like Copr
> and assist with improving the quality of the service. People want to
> depend on it, and it's still not getting the love it needs.

I'm not sure how donated resources could help here.
How could we trust that they were not tampering with builds?
How about someone donating instances that are all slow, slowing down
builds.
What happens if the network is down between copr and the donated stuff?
Its a can of worms. ;( I'm sure there are solutions to all these, but
copr doesn't have any of those in place currently.
> Supposedly it was going to be upgraded and improved, but I dunno
> what's going on there. Beyond that, we should consider making it
> possible for people and companies to be able to sponsor effort and
> resources in Fedora, similar to what openSUSE does. That's how their
> OBS has so much more resources than our Koji and Copr systems
> combined.

I have been working on a new openstack for the last N months.
It's pretty close to ready finally, but it sounds like perhaps we want
to drop it and move stuff elsewhere now.
> > > I think things like koji are particularly ill suited to move anywhere
> > > since we need to run Fedora on the builders, they have to run virt, and
> > > we have lots of arches that aren't commonly available in cloud.
> >
> > Yeah, multi-arch is going to be a sticking point for a lot of things.
>
> Losing the ability for people to be able to trivially self-host
> variants of our infrastructure would be a huge loss, in my mind.

Yeah.

kevin

[Neal Gompa]

On Mon, Jan 7, 2019 at 4:38 PM Kevin Fenzi  wrote:
> On 1/7/19 1:04 PM, Neal Gompa wrote:
> ...snip...
> > It'd be better if it was easier for more people to participate. We
> > have a lot of infra-types. It's kind of amazing in a bad way that
> > despite all that, we don't really have an easy way for more people to
> > help on that side.
>
> There definitely is...become an appentice, send patches for things, get
> added to other groups and help out with that area.
>
> True more access takes more time, but we definitely have had community
> folks helping out.
>
> ...snip...

Well, I guess I've been doing the "sending patches" thing for
different infrastructure things, but I don't know what this
apprenticeship thing is... Any info about that anywhere?
> > Pagure as a project is starting to pick up steam in its own right. And
> > users outside of Fedora (both public and private) are also turning
> > into contributors. If anything, this is one of the better successes
> > we've had in a while.
> >
> > As for the feature development, Pagure is keeping pace with solutions
> > like GitHub. We're now at the point where Pagure meets the features
> > that Debian was requesting for their Alioth migration, and other
> > groups are considering rolling out Pagure instances as well.
> >
> > As for the pagure.io forge, a large part of its lack of success is that
> > people don't know how to move to it. Heck, even I don't really know
> > how to effectively migrate a project from GitHub.com or GitLab.com or
> > some other site to Pagure.io and migrate things like issues, PRs, etc.
> > It is a good solution and it is developing rapidly into being a good
> > competitor to the others.
>
> https://pagure.io/pagure-importer
> was used to migrate things from trac, and it has a github mode, but I am
> not sure what state it's in.

Hmm, we really should collect these things in one place so it's easier
for people to find...
> > > > openstack (The big consumer here is copr... if it could move to the
> > > >
> > > > cloud or somewhere and we didn't need a openstack I would be very happy).
> > >
> > > Yes. I think this is something we can make happen. Possibly we can work
> > > something out with Mass Open Cloud.
> >
> > I think it's pretty obvious our Copr host platform (RHOSP 5) has been
> > a disappointment. It's unfortunate that there's no formal avenue for
> > people to sponsor and offer resources to improve services like Copr
> > and assist with improving the quality of the service. People want to
> > depend on it, and it's still not getting the love it needs.
>
> I'm not sure how donated resources could help here.
> How could we trust that they were not tampering with builds?
> How about someone donating instances that are all slow, slowing down
> builds.
> What happens if the network is down between copr and the donated stuff?
> Its a can of worms. ;( I'm sure there are solutions to all these, but
> copr doesn't have any of those in place currently.

openSUSE's donation program has the people usually ship the hardware
to them to be set up by the openSUSE Heroes (their equivalent of the
Fedora admin team). Alternatively, the route CentOS goes where control
is handed over is an option, too. But I think we'd be better served by
people giving us equipment (and maybe even money) and allowing people
to work on their paid time to contribute to leveraging that infra.
> > Supposedly it was going to be upgraded and improved, but I dunno
> > what's going on there. Beyond that, we should consider making it
> > possible for people and companies to be able to sponsor effort and
> > resources in Fedora, similar to what openSUSE does. That's how their
> > OBS has so much more resources than our Koji and Copr systems
> > combined.
>
> I have been working on a new openstack for the last N months.
> It's pretty close to ready finally, but it sounds like perhaps we want
> to drop it and move stuff elsewhere now.

If there's not much holding it back, we should transition to it
anyway. If it improves the management experience and performance of
the Copr system, it'd be a huge boon.

[Stephen John Smoogen]

On Mon, 7 Jan 2019 at 16:05, Neal Gompa  wrote:
> On Mon, Jan 7, 2019 at 3:41 PM Matthew Miller  wrote:
> >
> > I recognize that this is non-trivial, and it's possible that it's easier to
> > just let some things sit where they are. But a lot of those normally
> > low-maintenance things are tech-debt time bombs. Or, they're really
> > languishing.
>
> It'd be better if it was easier for more people to participate. We
> have a lot of infra-types. It's kind of amazing in a bad way that
> despite all that, we don't really have an easy way for more people to
> help on that side.

This is where we run into the 'screwed before' setting in
infrastructure. Most people coming in expect that they will have root
on all systems right away. The problem is that multiple instances
these people have also uploaded their private ssh keys to public
places, had passwords like 'youcantguessthis1', reverse ssh shells for
easy debugging in production, and similar things which they had done
for years and don't see why we should be such sticklers on not letting
them do it here. And when it blows up.. they are gone and other people
spend a couple of 96 hour days fixing things. All of which usually
gets followed by "we should never let this happen again", various
processes and gates to slow down a future problem... years later
people forget the 96 hour days thing and just feel those processes are
a problem.

I am not saying our processes aren't in need of improvement, but they
need to be redesigned from a point of view of being sustainable,
reliable and a limited resource budget. [Several of the ones that
would be more agile usually require us to have 10x the number of
systems we currently have as you do multi-site A-B testing and such.]
> > > > Which applications
> > > >
> > > > have industry-standard open source or proprietary alternatives that we
> > > > could move to.
> > >
> > > I'm not coming up with too long a list here. Most of our apps are highly
> > > specialised for distro making, but:
> > > fedocal (which we pretty much spend no time on)
> >
> > Yeah, this'd be a good candidate. It was a great, useful tool for its time
> > but is rather dated. It could use a *lot* of development work, and
> > realistically, we all know it's not going to get it.
>
> Are there any nice open source calendar/meeting assignment web apps
> out there we could deploy? I feel like this is a common enough venture
> that someone has done something...

The reason fedocal was written was pretty much every other
calendar/meeting is written just well enough to work for whatever
project the writer wanted. After that calendaring gets 'hard' and you
end up with no one putting in that '10% (aka 90%)' that is needed
because it costs a lot of time and effort and little reason because
you fixed the itch already. I last looked in 2016 but fedocal at that
time met our itches better while all the others would require lots of
add-on work.

[Matthew Miller]

On Mon, Jan 07, 2019 at 01:26:26PM -0800, Kevin Fenzi wrote:
> ok. That makes me feel better. :)
>
> You really can't be 100% open source and provide network services (look
> at all those non free routers handling your packets, etc). I just think
> we need to make sure and use as much open source as we can and avoid
> building on something we can't get out of.

Absolutely! Freedom (in the sense of software freedom!) remains fundamental.
> Sure, but I am objecting to the "move it all now!"
> How about "move it all when it makes sense?"

I guess we are saying "we think it makes sense as a priority". :)

[Kevin Fenzi]

On 1/7/19 1:59 PM, Neal Gompa wrote:
> On Mon, Jan 7, 2019 at 4:38 PM Kevin Fenzi  wrote:
> > On 1/7/19 1:04 PM, Neal Gompa wrote:
> > ...snip...
> > > It'd be better if it was easier for more people to participate. We
> > > have a lot of infra-types. It's kind of amazing in a bad way that
> > > despite all that, we don't really have an easy way for more people to
> > > help on that side.
> >
> > There definitely is...become an appentice, send patches for things, get
> > added to other groups and help out with that area.
> >
> > True more access takes more time, but we definitely have had community
> > folks helping out.
> >
> > ...snip...
>
> Well, I guess I've been doing the "sending patches" thing for
> different infrastructure things, but I don't know what this
> apprenticeship thing is... Any info about that anywhere?

it's pointed to from our getting started doc:

https://fedoraproject.org/wiki/Infrastructure/GettingStarted

https://fedoraproject.org/wiki/Infrastructure_Apprentice

...snip...
> openSUSE's donation program has the people usually ship the hardware
> to them to be set up by the openSUSE Heroes (their equivalent of the
> Fedora admin team). Alternatively, the route CentOS goes where control
> is handed over is an option, too. But I think we'd be better served by
> people giving us equipment (and maybe even money) and allowing people
> to work on their paid time to contribute to leveraging that infra.

I'm not sure the legal logistics of this. Is that a "donation"?
Fedora doesn't exist as a legal entity to accept it as far as I know.

Also, we have been offered donated hardware in the past, but it's
usually more trouble than it's worth. (ie, "I was about to toss this
PentiumIII box out, but I thought I would donate it to you instead").
All these are things we could work out however.
...snip...
> If there's not much holding it back, we should transition to it
> anyway. If it improves the management experience and performance of
> the Copr system, it'd be a huge boon.

I agree the hard part is hopefuly over, but there is still ongoing
support (upgrades, updates, adding new machines, etc).

kevin