It’s time to transform the Fedora devel list into something new =============================================================== My first post on this list was over 19 years ago. (It was about Bugzilla. I was a fan!) Ever since those early days, devel list has been the heart and center of Fedora activity. Now, I hope to convince all of us here that it’s time for something different. As it is, devel list is too much for many people to follow — people we’d like to have around. It covers many different things at once, yet also drives us towards more scattered communications. Our infamous mega-threads are not really effective for getting to community consensus, and tend to bring out the worst in us. I propose that we transition devel list, and eventually most of our mailing lists, to Fedora Discussion (our Discourse-powered forum). I know this is a big change, but, hear me out… We’re missing people -------------------- A Mastodon post from long-time Fedora contributor Major Hayden got me thinking: I know many Fedora folks, old-school and new, for whom devel list is just too much. Some of it is the sheer volume, but this “off the rails” tendency is real — threads drift, get into back-and-forth debates about particular details, etc. And… some people aren’t here because — in contrast with our “Friends” foundation, it isn’t always a nice place to be (and mailing lists don’t provide many tools for moderation, except the big hammer of outright bans). Ben Cotton recently did some basic analysis on devel list traffic over time, and there’s a clear trend: fewer people are participating, even though the number of different threads goes up. I don’t think this is because of any decline in Fedora contributors overall — I think it’s that conversations are happening elsewhere. Big threads are … bad, actually ------------------------------- When we have something to talk about, it tends to explode into a big thread. The thing in January with FESCo’s frame pointers decision (https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...) is a good example of things going badly. Most of the conversation was under the subject “Schedule for Tuesday's FESCo Meeting (2023-01-03)”, because everything started as a reply to that. That’s pretty easy to overlook. It’s possible for replies to change  the subject when replying, but that can’t be done retroactively, and then isn’t consistent (and it breaks threading in Gmail, too). Then, things got rather hostile, making it hard to have a reasonable conversation about the issues (both technical and procedural). And then, things went in circles without adding anything new. This could have all gone a lot better. And that’s just one example. Take a look back at any mega-thread, and you’ll find similar — and worse. When things get heated, the only way to intervene is by adding more.  There are often long subthreads of two people going back and forth on tangents. Then, other conversation branches duplicate that, or refer across. Classical email tools don’t actually handle this kind of thing very well at all. In my experience, it only really works if you keep up with the conversation in almost real time, which has its own problems even when that’s possible. We’re scattered in actual practice ---------------------------------- Devel list may be the center, but we have _hundreds_ of Fedora mailing lists. A dozen or so are reasonably active (Test, Legal, ARM…) but most are inactive or dead. Some are just meeting reminders over and over — for meetings that aren’t even active anymore. It’s easy to make but hard to _unmake_ a mailing list. For lists that are active, the split is confusing — when should something be on the packaging list rather than devel? What happens when something is related to both Cloud and Server, or Workstation and KDE? One can post to both lists, but if someone replies and isn’t subscribed to both, the conversation gets split. With “devel” as the main list, conversations about marketing, design, events, and so on don’t really have a central place. (The Mindshare list never really caught on.) That makes these important activities feel even more disconnected and secondary in status — and they shouldn’t be. Many groups have actually moved away from lists to using tickets for team conversations — both those non-engineering functions and development. Design Team has a mailing list, but mostly for announcements: the work happens in tickets. Workstation largely uses their Pagure tracker. And CoreOS conversations happen almost entirely in tickets on Github. Tickets are made for tracking specific, actionable tasks, and that kind of tracking is part of why teams use them over mailing lists — but Fedora teams use them for open development conversations too. I think that’s largely a symptom of mailing lists not being enough for what we need. The trackers have media support, editing for typos or updates, reactions for simple agreement, tagging people, and granular subscriptions. They are effectively “off-label use” mini-forums that teams can quietly move to using without the sort of conversation I expect this message to generate. Airplane diagram, survivorship bias ----------------------------------- Since I’m posting this to devel list, I do expect a lot of push-back. Maybe I’ll be surprised and more of y’all are already with me on this and just waiting for something to happen. But overall I expect a tough crowd. You’ve probably heard the story about bullet holes in airplanes returning from missions and the accompanying diagram — find it athere https://en.wikipedia.org/wiki/Survivorship_bias if that’s new to you. The set of remaining regular participants on this list is naturally biased towards those for whom it is working just fine. But, core Fedora development discussion can’t be limited to that ever-shrinking group. Consider who isn’t here. The problems are real, and the trend isn’t in a good direction. Devel List is too many different things! ---------------------------------------- We use it for Change discussion (resulting in those not-actually-so-great big threads). We use it for introductions and onboarding — we’re usually pretty good at that, actually (but it adds to the overall load of following the list). We’re not very consistent, though. We discuss packaging: guidelines, help on different topics, coordination on specific work. There’s unclear overlap with the packaging list, though, which is a bit confusing. We talk about higher-level Fedora OS development topics that don’t fit anywhere else. For example, this on installer environment size: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o... Sometimes, these are FESCo topics (which FESCo heroically struggles to keep here rather than on Pagure tickets…). Sometimes, people talk about a particular interest in the hopes of forming a Special Interest Group. Often that does result in a new SIG launching — sometimes with a brand-new mailing list which then ends up getting only a few posts ( and a whole lot of spam years later). But this really suffers from the survivorship problem: many people who might be interested will never notice. We have release process announcements — mostly to devel-announce, but then occasionally replies and discussion here. Blocker bugs and other test-list and QA topics are cross-posted, as part of the release cycle. And then there are a lot of robo-messages: reports, reminders, etc. These are really valuable to a few people, but add a lot more to wade through for everyone else trying to keep up. There are certainly others, as well. In short… there probably shouldn’t just be one thing. But, the cross-posting problem makes it hard to split up as a mailing list. Enter Discourse --------------- If you’ve talked with me about anything related to any of this in the past ten years, you probably already know that I like Discourse. It’s good software made by cool people. And, it’s entirely open source, with a SaaS business model but with real, usable releases. (No open core, no “open source theoretically but good luck”.) And, we have it in production in Fedora already, at https://discussion.fedoraproject.org, with categories for announcements, user help, project discussion, and social conversation — as well as special categories for dedicated workflows. In Project Discussion, each different Fedora team can have its own tag, and you can subscribe to those that you’re interested in. Cross-posting is easy: tag a post with multiple teams. Topics can be renamed, if needed, or split out into side-conversations. The long tangents from these conversations can actually be interesting on their own without distracting from the main topic. Moderation tools allow us to handle posts that are outside of expected Fedora contributor behavior, with varying levels of action as appropriate. You can use markdown formatting, including images (with easy addition of alt text for people for whom images are a barrier). You can edit your posts to fix typos or correct mistakes. There are polls and lots of other useful features. And, you can interact with it all by email. That interface isn’t perfect, but it’s way better than any other forum software I’ve seen. (I’ve written a guide: https://discussion.fedoraproject.org/t/25960) At the same time, your individual email address is hidden, so it won’t be a spam magnet (or a target for off-list harassment, a problem we unfortunately have with devel list). That said, it is web-first software. (Or mobile, if that’s your thing.) That requires some adjustment, I know. I hope opening up a Fedora Discussion tab – or keeping one open — becomes an easy habit. Not just Fedora --------------- There’s a big trend towards Discourse in open source projects overall. Python and Gnome have both migrated entirely from their mailing lists. Ansible is working on it. Plus, there’s Rust, Kubernetes, Nextcloud, Flathub, Grafana, Home Assistant, KDE, and I’m sure many others. Concrete proposal ----------------- I’m not suggesting we shut down devel list next week. And I think we’ll have some mailing lists for quite a long time. But, I think it’s time to start moving some specific things, with the eventual goal of closing every mailing list we can. First, I’d like to move the Changes discussion. They will still be posted to devel-announce, but responses directed to Project Discussion in a new #changes tag. Ben tells me that this is a FESCo decision, which seems reasonable. Second, I think other FESCo-related conversations should move. I hope this will reduce the urge to have back-and-forth exchanges in the tickets. For the Fedora Council, I set up a bot which automatically creates a discussion topic when a ticket is filed, leaving the ticket just for votes and recording of outcome. FESCo could use something similar. Third, I’ll add a tag for general Fedora OS development conversation. Maybe “#devel”, but if someone has more narrow suggestions, I’ll take them. Fourth, I’d like to update our documentation, process, and expectations for newcomers — say hello on Discussion (and Fedora Chat, if you like) rather than a mailing list. (I’d like to close the Fedora Join list at this point.) Fifth, all packaging-related discussions (including the separate packaging mailing list). We already have a #package-maintainers tag with some existing discussion. Sixth, automated posts, as much as we can. These should go to dedicated Workflow categories, where people who want can watch them but where they won’t overwhelm human interactions. People who want can watch them, and it’s easy to quote-reply into a new linked topic in the Project Discussions category. And finally… shut down the devel list itself. Perhaps at the end of 2023? We should also shut down all of the little lists that haven’t had anything but spam in the last two years. We could maybe do that sooner. We should stop creating new lists now — we can create new Discussion tags instead. I expect the announcement lists to stay for the foreseeable future (although we might feed them from Discourse rather than the other way around). Other lists which are patches, commit messages, and other automations might stick around for a while — but really might be better served by a log aggregation and analysis system or something else. Other teams who want to keep mailing lists can, but I’d like to move those too, and eventually I think we’ll want to shut them down too — or perhaps convert them to announcement lists. Next steps ---------- I know this is a big change. I’ve been thinking of writing this message for a long time. I’d really like to convince everyone that it’s the right thing — or at least, an acceptable one. What about specific decisions related to my proposal? For each: Because altering the Changes process is a FESCo decision, I’ll file a ticket about that shortly. FESCo moving their own other conversations is, of course, also a FESCo decision. Assuming the first moves forward, I will create the general #devel tag (or other name we come up with) when I create the #change-proposal tag. Moving the packaging list is a Packaging Committee decision. Automated posts can be moved at any time. I can work with the people who own the generation of those reports to figure out a good answer for each. The outcome for other team lists is up to each team. And, I think shutting down devel overall is ultimately a Fedora Council decision. For right now, though: let’s discuss — on the list! -- Matthew Miller <mattdm(a)fedoraproject.org&gt; Fedora Project Leader