FYI
The server
https://retrace.fedoraproject.org/
should be maintained by ABRT team, but that team no longer exists. It is not updated and maintained for more than year. I am sponsor of this this service and promised to take of Ansible playbooks. With current level of maintenance I am no longer willing to take care of it.
I will terminate this server next week. I already notified remaining of ABRT team and got an approval from them.
On Tue, Aug 11, 2026 at 03:53:46PM +0200, Miroslav Suchý via infrastructure wrote:
FYI
The server
https://retrace.fedoraproject.org/
should be maintained by ABRT team, but that team no longer exists. It is not updated and maintained for more than year. I am sponsor of this this service and promised to take of Ansible playbooks. With current level of maintenance I am no longer willing to take care of it.
I will terminate this server next week. I already notified remaining of ABRT team and got an approval from them.
This was on my mind last week actually, as I was looking at what servers were still on rhel8/9.
Let me know if I can help any, and do you have any thoughts on what we should repurpose the hardware to do? Or should we look at some need in infrastructure?
Thanks,
kevin
Dne 12. 08. 26 v 3:44 dop. Kevin Fenzi via infrastructure napsal(a):
This was on my mind last week actually, as I was looking at what servers were still on rhel8/9.
Let me know if I can help any, and do you have any thoughts on what we should repurpose the hardware to do? Or should we look at some need in infrastructure?
I thought it is in AWS and I would just take snapshot and terminate it. But it is not!
And I even cannot ssh-login there :(
I checked the records and the server is from 2020. So it is more likely e-waste now. Or we can re-use it as Copr builder.
Since this is not AWS and I cannot snapshot it, I will rather switch off the http service. Wait a week and unless somebody objects we can throw it away or reprovision and reuse.
On Wed, Aug 12, 2026 at 01:31:18PM +0200, Miroslav Suchý via infrastructure wrote:
Dne 12. 08. 26 v 3:44 dop. Kevin Fenzi via infrastructure napsal(a):
This was on my mind last week actually, as I was looking at what servers were still on rhel8/9.
Let me know if I can help any, and do you have any thoughts on what we should repurpose the hardware to do? Or should we look at some need in infrastructure?
I thought it is in AWS and I would just take snapshot and terminate it. But it is not!
And I even cannot ssh-login there :(
I checked the records and the server is from 2020. So it is more likely e-waste now. Or we can re-use it as Copr builder.
Since this is not AWS and I cannot snapshot it, I will rather switch off the http service. Wait a week and unless somebody objects we can throw it away or reprovision and reuse.
yeah, it's an old machine. I'll ponder on what if anything we could still use it for.
kevin
I can think of lots of good uses in data land!
I would need some sort of access though. If that's a possibility, let's talk.
Michael Winters
On August 12, 2026 12:51:35 PM CDT, Kevin Fenzi via infrastructure infrastructure@lists.fedoraproject.org wrote:
On Wed, Aug 12, 2026 at 01:31:18PM +0200, Miroslav Suchý via infrastructure wrote:
Dne 12. 08. 26 v 3:44 dop. Kevin Fenzi via infrastructure napsal(a):
This was on my mind last week actually, as I was looking at what servers were still on rhel8/9.
Let me know if I can help any, and do you have any thoughts on what we should repurpose the hardware to do? Or should we look at some need in infrastructure?
I thought it is in AWS and I would just take snapshot and terminate it. But it is not!
And I even cannot ssh-login there :(
I checked the records and the server is from 2020. So it is more likely e-waste now. Or we can re-use it as Copr builder.
Since this is not AWS and I cannot snapshot it, I will rather switch off the http service. Wait a week and unless somebody objects we can throw it away or reprovision and reuse.
yeah, it's an old machine. I'll ponder on what if anything we could still use it for.
kevin
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
Dne 13. 08. 26 v 2:14 dop. Michael Winters via infrastructure napsal(a):
I can think of lots of good uses in data land!
I would need some sort of access though. If that's a possibility, let's talk.
??? Can you be more specific?
On Wed, 2026-08-12 at 13:31 +0200, Miroslav Suchý via infrastructure wrote:
Dne 12. 08. 26 v 3:44 dop. Kevin Fenzi via infrastructure napsal(a):
This was on my mind last week actually, as I was looking at what servers were still on rhel8/9.
Let me know if I can help any, and do you have any thoughts on what we should repurpose the hardware to do? Or should we look at some need in infrastructure?
I thought it is in AWS and I would just take snapshot and terminate it. But it is not!
And I even cannot ssh-login there :(
I checked the records and the server is from 2020. So it is more likely e-waste now. Or we can re-use it as Copr builder.
Since this is not AWS and I cannot snapshot it, I will rather switch off the http service. Wait a week and unless somebody objects we can throw it away or reprovision and reuse.
FAF, which is part of this, is/was a very useful service (though it seems to have never been updated for F43 and F44). We don't have anything else to answer questions like "what are the most commonly- encountered crashes?" and "how often is this specific crash being hit?" and "does it look like the intended fix for this crash actually worked?"
I know there's a big resource issue with the maintenance of the underlying tools, but in theory, we (QA team) definitely "object" to losing these capabilities.
Hi -
I thought it is in AWS and I would just take snapshot and terminate it. But it is not!
And I even cannot ssh-login there :(
[...] I know there's a big resource issue with the maintenance of the underlying tools, but in theory, we (QA team) definitely "object" to losing these capabilities.
Could this be an opportunity to sic an AI on the task of maintaining a server/service? Give a good one root on the VM, and let it cook. It's on death-row so to speak, so the stakes are not large, and it might just work.
- FChE
Dne 18. 08. 26 v 11:31 odp. Frank Ch. Eigler via infrastructure napsal(a):
Could this be an opportunity to sic an AI on the task of maintaining a server/service? Give a good one root on the VM, and let it cook. It's on death-row so to speak, so the stakes are not large, and it might just work.
Do **you** want to try it?
Hi -
Could this be an opportunity to sic an AI on the task of maintaining a server/service? Give a good one root on the VM, and let it cook. It's on death-row so to speak, so the stakes are not large, and it might just work.
Do **you** want to try it?
Given access, and a todo list, I can certainly try it as an experiment. And if I and the AI fail, we're probably no worse off.
- FChE
On Wed, Aug 19, 2026 at 05:37:59PM -0400, Frank Ch. Eigler via infrastructure wrote:
Hi -
Could this be an opportunity to sic an AI on the task of maintaining a server/service? Give a good one root on the VM, and let it cook. It's on death-row so to speak, so the stakes are not large, and it might just work.
Do **you** want to try it?
Given access, and a todo list, I can certainly try it as an experiment. And if I and the AI fail, we're probably no worse off.
Well, perhaps I am just a paranoid sysadmin, but I can think of lots of bad things it could do. Causing problems for other machines on it's vlan, spewing emails or other traffic, causing undue load on other systems (koji, src, download), etc.
The other problem is... would it be able to maintain this long term? I mean even if it gets things back working, IMHO it still needs someone to spend time keeping it working or fixing things.
kevin
Dne 18. 08. 26 v 11:25 odp. Adam Williamson napsal(a):
I know there's a big resource issue with the maintenance of the underlying tools, but in theory, we (QA team) definitely "object" to losing these capabilities.
I do not want it too.
But unless you back your objections with claim that you (or somebody) is willing to work in this, then .... I am sorry. :(
FTR: From my past experience, it would require at least one engineer to work on it full time. In the peak, ABRT had 4 engineers working full time on the project.
On Wed, Aug 19, 2026, at 04:34, Miroslav Suchý via infrastructure wrote:
Dne 18. 08. 26 v 11:25 odp. Adam Williamson napsal(a):
I know there's a big resource issue with the maintenance of the underlying tools, but in theory, we (QA team) definitely "object" to losing these capabilities.
I do not want it too.
But unless you back your objections with claim that you (or somebody) is willing to work in this, then .... I am sorry. :(
FTR: From my past experience, it would require at least one engineer to work on it full time. In the peak, ABRT had 4 engineers working full time on the project.
And going from the infrastructure side, we needed those many engineers to deal with breakages, data problems and general cleanup. When they were cut down, the amount of need to fix this under the radar went up for a while. The fix was to stop monitoring more parts and just let silent turning off bits. The server needs at least 1 person and probably 2 people to keep up maintenance and general feeding. It needs storage, and more CPU for processing, and it needs to get reworked from being another Proof of Concept pushed into Production like so many other parts of Fedora.
-- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys
-- _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
On Wed, Aug 19, 2026, at 10:32, Frank Ch. Eigler wrote:
Hi -
[...] It needs storage, and more CPU for processing, and it needs to get reworked from being another Proof of Concept pushed into Production like so many other parts of Fedora.
Are you aware of a todo list for this server?
The last time I dealt with a todo list for this box was 2019-2020 during the PHX2 to IAD2 move, and I think that was when most of the team supporting it were reassigned. The todo list was what I stated above..
Honestly I thought it had been retired already.
- FChE
On Wed, 2026-08-19 at 10:34 +0200, Miroslav Suchý wrote:
Dne 18. 08. 26 v 11:25 odp. Adam Williamson napsal(a):
I know there's a big resource issue with the maintenance of the underlying tools, but in theory, we (QA team) definitely "object" to losing these capabilities.
I do not want it too.
But unless you back your objections with claim that you (or somebody) is willing to work in this, then .... I am sorry. :(
I'm not, but you didn't ask for *volunteers*, you asked for *objections*. So I objected.
I know there is a resource issue here. It's been discussed in various channels for a long time. But this is a public record, and I don't want this public record to show that everyone was fine with the service going away, when someone asked whether "somebody objects".
On Wed, Aug 19, 2026, at 14:57, Adam Williamson via infrastructure wrote:
On Wed, 2026-08-19 at 10:34 +0200, Miroslav Suchý wrote:
Dne 18. 08. 26 v 11:25 odp. Adam Williamson napsal(a):
I know there's a big resource issue with the maintenance of the underlying tools, but in theory, we (QA team) definitely "object" to losing these capabilities.
I do not want it too.
But unless you back your objections with claim that you (or somebody) is willing to work in this, then .... I am sorry. :(
I'm not, but you didn't ask for *volunteers*, you asked for *objections*. So I objected.
I don't see where Miroslav asked for any objections in the following:
``` The server
https://retrace.fedoraproject.org/
should be maintained by ABRT team, but that team no longer exists. It is not updated and maintained for more than year. I am sponsor of this this service and promised to take of Ansible playbooks. With current level of maintenance I am no longer willing to take care of it.
I will terminate this server next week. I already notified remaining of ABRT team and got an approval from them. ```
Did I miss some other email where he asked for objections?
I know there is a resource issue here. It's been discussed in various channels for a long time. But this is a public record, and I don't want this public record to show that everyone was fine with the service going away, when someone asked whether "somebody objects". --
On Wed, 2026-08-19 at 15:04 -0400, Stephen J Smoogen via infrastructure wrote:
On Wed, Aug 19, 2026, at 14:57, Adam Williamson via infrastructure wrote:
On Wed, 2026-08-19 at 10:34 +0200, Miroslav Suchý wrote:
Dne 18. 08. 26 v 11:25 odp. Adam Williamson napsal(a):
I know there's a big resource issue with the maintenance of the underlying tools, but in theory, we (QA team) definitely "object" to losing these capabilities.
I do not want it too.
But unless you back your objections with claim that you (or somebody) is willing to work in this, then .... I am sorry. :(
I'm not, but you didn't ask for *volunteers*, you asked for *objections*. So I objected.
I don't see where Miroslav asked for any objections in the following:
The server https://retrace.fedoraproject.org/ should be maintained by ABRT team, but that team no longer exists. It is not updated and maintained for more than year. I am sponsor of this this service and promised to take of Ansible playbooks. With current level of maintenance I am no longer willing to take care of it. I will terminate this server next week. I already notified remaining of ABRT team and got an approval from them.Did I miss some other email where he asked for objections?
You missed the mail I replied to, and quoted, which says:
"Since this is not AWS and I cannot snapshot it, I will rather switch off the http service. Wait a week and unless somebody objects we can throw it away or reprovision and reuse."
On Wed, Aug 19, 2026, at 17:58, Adam Williamson via infrastructure wrote:
On Wed, 2026-08-19 at 15:04 -0400, Stephen J Smoogen via infrastructure wrote:
On Wed, Aug 19, 2026, at 14:57, Adam Williamson via infrastructure wrote:
On Wed, 2026-08-19 at 10:34 +0200, Miroslav Suchý wrote:
Dne 18. 08. 26 v 11:25 odp. Adam Williamson napsal(a):
I know there's a big resource issue with the maintenance of the underlying tools, but in theory, we (QA team) definitely "object" to losing these capabilities.
I do not want it too.
But unless you back your objections with claim that you (or somebody) is willing to work in this, then .... I am sorry. :(
I'm not, but you didn't ask for *volunteers*, you asked for *objections*. So I objected.
I don't see where Miroslav asked for any objections in the following:
The server https://retrace.fedoraproject.org/ should be maintained by ABRT team, but that team no longer exists. It is not updated and maintained for more than year. I am sponsor of this this service and promised to take of Ansible playbooks. With current level of maintenance I am no longer willing to take care of it. I will terminate this server next week. I already notified remaining of ABRT team and got an approval from them.Did I miss some other email where he asked for objections?
You missed the mail I replied to, and quoted, which says:
"Since this is not AWS and I cannot snapshot it, I will rather switch off the http service. Wait a week and unless somebody objects we can throw it away or reprovision and reuse."
Yes I missed that part. My fault completely.
-- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@fosstodon.org https://www.happyassassin.net
-- _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
I noticed Packager Dashboard gets its ABRT data from FAF, so I put together an issue to remove the integration after seeing the decommissioning notice. With Adam mentioning that FAF is still useful for QA, I think it would be better to put the removal on hold for now
Limon
infrastructure@lists.fedoraproject.org
-
Adam Williamson -
Frank Ch. Eigler -
Kevin Fenzi -
Michael Winters -
Miroslav Suchý -
Sheikh Limon -
Stephen J Smoogen