It’s time to transform the Fedora devel list into something new

[Matthew Miller]

It’s time to transform the Fedora devel list into something new
===============================================================

My first post on this list was over 19 years ago. (It was about
Bugzilla. I was a fan!) Ever since those early days, devel list has
been the heart and center of Fedora activity. Now, I hope to convince
all of us here that it’s time for something different.

As it is, devel list is too much for many people to follow — people
we’d like to have around. It covers many different things at once, yet
also drives us towards more scattered communications. Our infamous
mega-threads are not really effective for getting to community
consensus, and tend to bring out the worst in us.

I propose that we transition devel list, and eventually most of our
mailing lists, to Fedora Discussion (our Discourse-powered forum).

I know this is a big change, but, hear me out…


We’re missing people
--------------------

A Mastodon post from long-time Fedora contributor Major Hayden got me
thinking:

>  How do people make so much time available for mailing list
>  discourse?
>
>  Once I ensure my team has the technical guidance they need
>  and I work through the tasks of work that I owe other
>  people, I take a look at the mailing list and say: "Oh my
>  gosh, what the heck happened here?" Then the discussion
>  goes further off the rails while I'm typing out a reply and
>  my reply is no longer relevant.
>
>  — https://tootchute.com/@major/109666036733834421 
I know many Fedora folks, old-school and new, for whom devel list is
just too much. Some of it is the sheer volume, but this “off the rails”
tendency is real — threads drift, get into back-and-forth debates about
particular details, etc. And… some people aren’t here because — in
contrast with our “Friends” foundation, it isn’t always a nice place to
be (and mailing lists don’t provide many tools for moderation, except
the big hammer of outright bans).

Ben Cotton recently did some basic analysis on devel list traffic over
time, and there’s a clear trend: fewer people are participating, even
though the number of different threads goes up. I don’t think this is
because of any decline in Fedora contributors overall — I think it’s
that conversations are happening elsewhere.


Big threads are … bad, actually
-------------------------------

When we have something to talk about, it tends to explode into a big
thread. The  thing in January with FESCo’s frame pointers decision
(https://lists.fedoraproject.org/archives/list/[e-mail].o...)
is a good example of things going badly.

Most of the conversation was under the subject “Schedule for Tuesday's
FESCo Meeting (2023-01-03)”, because everything started as a reply to
that. That’s pretty easy to overlook. It’s possible for replies to
change  the subject when replying, but that can’t be done
retroactively, and then isn’t consistent (and it breaks threading in
Gmail, too).

Then, things got rather hostile, making it hard to have a reasonable
conversation about the issues (both technical and procedural). And
then, things went in circles without adding anything new.

This could have all gone a lot better.

And that’s just one example. Take a look back at any mega-thread, and
you’ll find similar — and worse. When things get heated, the only way
to intervene is by adding more.  There are often long subthreads of two
people going back and forth on tangents. Then, other conversation
branches duplicate that, or refer across. Classical email tools don’t
actually handle this kind of thing very well at all. In my experience,
it only really works if you keep up with the conversation in almost
real time, which has its own problems even when that’s possible.


We’re scattered in actual practice
----------------------------------

Devel list may be the center, but we have _hundreds_ of Fedora mailing
lists. A dozen or so are reasonably active (Test, Legal, ARM…) but most
are inactive or dead. Some are just meeting reminders over and over —
for meetings that aren’t even active anymore. It’s easy to make but
hard to _unmake_ a mailing list.

For lists that are active, the split is confusing — when should
something be on the packaging list rather than devel? What happens when
something is related to both Cloud and Server, or Workstation and KDE?
One can post to both lists, but if someone replies and isn’t subscribed
to both, the conversation gets split.

With “devel” as the main list, conversations about marketing, design,
events, and so on don’t really have a central place. (The Mindshare
list never really caught on.) That makes these important activities
feel even more disconnected and secondary in status — and they
shouldn’t be.

Many groups have actually moved away from lists to using tickets for
team conversations — both those non-engineering functions
and development. Design Team has a mailing list, but mostly for
announcements: the work happens in tickets. Workstation largely uses
their Pagure tracker. And CoreOS conversations happen almost entirely
in tickets on Github.

Tickets are made for tracking specific, actionable tasks, and that kind
of tracking is part of why teams use them over mailing lists — but
Fedora teams use them for open development conversations too. I think
that’s largely a symptom of mailing lists not being enough for what we
need. The trackers have media support, editing for typos or updates,
reactions for simple agreement, tagging people, and granular
subscriptions. They are effectively “off-label use” mini-forums that
teams can quietly move to using without the sort of conversation I
expect this message to generate.



Airplane diagram, survivorship bias
-----------------------------------

Since I’m posting this to devel list, I do expect a lot of push-back.
Maybe I’ll be surprised and more of y’all are already with me on this
and just waiting for something to happen. But overall I expect a tough
crowd.

You’ve probably heard the story about bullet holes in airplanes
returning from missions and the accompanying diagram — find it athere
https://en.wikipedia.org/wiki/Survivorship_bias if that’s new to you.

The set of remaining regular participants on this list is naturally
biased towards those for whom it is working just fine. But, core Fedora
development discussion can’t be limited to that ever-shrinking group.
Consider who isn’t here. The problems are real, and the trend isn’t in
a good direction.



Devel List is too many different things!
----------------------------------------

We use it for Change discussion (resulting in those
not-actually-so-great big threads).

We use it for introductions and onboarding — we’re usually pretty good
at that, actually (but it adds to the overall load of following the
list). We’re not very consistent, though.

We discuss packaging: guidelines, help on different topics,
coordination on specific work. There’s unclear overlap with the
packaging list, though, which is a bit confusing.

We talk about higher-level Fedora OS development topics that don’t fit
anywhere else. For example, this on installer environment size:
https://lists.fedoraproject.org/archives/list/[e-mail].o...
Sometimes, these are FESCo topics (which FESCo heroically struggles to
keep here rather than on Pagure tickets…).

Sometimes, people talk about a particular interest in the hopes of
forming a Special Interest Group. Often that does result in a new SIG
launching — sometimes with a brand-new mailing list which then ends up
getting only a few posts ( and a whole lot of spam years later). But
this really suffers from the survivorship problem: many people who
might be interested will never notice.

We have release process announcements — mostly to devel-announce, but
then occasionally replies and discussion here. Blocker bugs and other
test-list and QA topics are cross-posted, as part of the release cycle.

And then there are a lot of robo-messages: reports, reminders, etc.
These are really valuable to a few people, but add a lot more to wade
through for everyone else trying to keep up.

There are certainly others, as well.

In short… there probably shouldn’t just be one thing. But, the
cross-posting problem makes it hard to split up as a mailing list.



Enter Discourse
---------------

If you’ve talked with me about anything related to any of this in the
past ten years, you probably already know that I like Discourse. It’s
good software made by cool people. And, it’s entirely open source, with
a SaaS business model but with real, usable releases. (No open core, no
“open source theoretically but good luck”.) And, we have it in
production in Fedora already, at https://discussion.fedoraproject.org,
with categories for announcements, user help, project discussion, and
social conversation — as well as special categories for dedicated
workflows.

In Project Discussion, each different Fedora team can have its own tag,
and you can subscribe to those that you’re interested in. Cross-posting
is easy: tag a post with multiple teams.

Topics can be renamed, if needed, or split out into side-conversations.
The long tangents from these conversations can actually be interesting
on their own without distracting from the main topic. Moderation tools
allow us to handle posts that are outside of expected Fedora
contributor behavior, with varying levels of action as appropriate.

You can use markdown formatting, including images (with easy addition
of alt text for people for whom images are a barrier). You can edit
your posts to fix typos or correct mistakes. There are polls and lots
of other useful features.

And, you can interact with it all by email. That interface isn’t
perfect, but it’s way better than any other forum software I’ve seen.
(I’ve written a guide: https://discussion.fedoraproject.org/t/25960)
At the same time, your individual email address is hidden, so it won’t
be a spam magnet (or a target for off-list harassment, a problem we
unfortunately have with devel list).

That said, it is web-first software. (Or mobile, if that’s your thing.)
That requires some adjustment, I know. I hope opening up a Fedora
Discussion tab – or keeping one open — becomes an easy habit.


Not just Fedora
---------------

There’s a big trend towards Discourse in open source projects overall.
Python and Gnome have both migrated entirely from their mailing lists.
Ansible is working on it. Plus, there’s Rust, Kubernetes, Nextcloud,
Flathub, Grafana, Home Assistant, KDE, and I’m sure many others.


Concrete proposal
-----------------

I’m not suggesting we shut down devel list next week. And I think we’ll
have some mailing lists for quite a long time. But, I think it’s time
to start moving some specific things, with the eventual goal of closing
every mailing list we can.


First, I’d like to move the Changes discussion. They will still be
posted to devel-announce, but responses directed to Project Discussion
in a new #changes tag. Ben tells me that this is a FESCo decision,
which seems reasonable.

Second, I think other FESCo-related conversations should move. I hope
this will reduce the urge to have back-and-forth exchanges in the
tickets. For the Fedora Council, I set up a bot which automatically
creates a discussion topic when a ticket is filed, leaving the ticket
just for votes and recording of outcome. FESCo could use something
similar.

Third, I’ll add a tag for general Fedora OS development conversation.
Maybe “#devel”, but if someone has more narrow suggestions, I’ll take
them.

Fourth, I’d like to update our documentation, process, and expectations
for newcomers — say hello on Discussion (and Fedora Chat, if you like)
rather than a mailing list. (I’d like to close the Fedora Join list at
this point.)

Fifth, all packaging-related discussions (including the separate
packaging mailing list). We already have a #package-maintainers tag
with some existing discussion.

Sixth, automated posts, as much as we can. These should go to dedicated
Workflow categories, where people who want can watch them but where
they won’t overwhelm human interactions. People who want can watch
them, and it’s easy to quote-reply into a new linked topic in the
Project Discussions category.

And finally… shut down the devel list itself. Perhaps at the end of
2023?

We should also shut down all of the little lists that haven’t had
anything but spam in the last two years. We could maybe do that sooner.
We should stop creating new lists now — we can create new Discussion
tags instead.

I expect the announcement lists to stay for the foreseeable future
(although we might feed them from Discourse rather than the other way
around). Other lists which are patches, commit messages, and other
automations might stick around for a while — but really might be better
served by a log aggregation and analysis system or something else.

Other teams who want to keep mailing lists can, but I’d like to move
those too, and eventually I think we’ll want to shut them down too — or
perhaps convert them to announcement lists.


Next steps
----------

I know this is a big change. I’ve been thinking of writing this message
for a long time. I’d really like to convince everyone that it’s the
right thing — or at least, an acceptable one.


What about specific decisions related to my proposal? For each:

Because altering the Changes process is a FESCo decision, I’ll file a
ticket about that shortly.

FESCo moving their own other conversations is, of course, also a FESCo
decision.

Assuming the first moves forward, I will create the general #devel tag
(or other name we come up with) when I create the #change-proposal tag.

Moving the packaging list is a Packaging Committee decision.

Automated posts can be moved at any time. I can work with the people
who own the generation of those reports to figure out a good answer for
each.

The outcome for other team lists is up to each team.

And, I think shutting down devel overall is ultimately a Fedora Council
decision.

For right now, though: let’s discuss — on the list!

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader