This is something that has been coming for quite a while now and while it was discussed among QA and Infra, there hasn't been enough communication with the rest of Fedora. For that, I apologize.
Taskotron has been in maintenance-only mode for a couple of years now pending a replacement of some kind. That replacement has taken longer to materialize than originally planned but the servers which run Taskotron are almost out of warranty and will not be moving to the new datacenter with the rest of Fedora's infrastructure.
Given all of this, we have decided that 2020-04-30 will be the day that Taskotron will be turned off so that we're not trying to turn stuff off in the middle of the infra move later this summer.
--------------------------- What does this mean for me? ---------------------------
If you're not a packager, you will probably not notice the shutdown.
If you are a packager, this will affect some of the results that you currently see in Bodhi under the "Automated Tests" tab. At the moment, most of those results come from Taskotron. If those checks are run in the future, they will likely show up with slightly different names.
-------------------------------- Is anything replacing Taskotron? --------------------------------
In the short term, there is a Jenkins instance which is running rpminspect. Rpminspect includes tests which are effectively the same as rpmlint and rpmgrill which removes the need to run them.
We already have rpminspect results going into production resultsdb and showing up in Bodhi.
In the longer term, there will be a new solution to run rpminspect, rpmdeplint and other static-analysis style checks coming from the Fedora CI folks which they will be deploying before Flock.
--------------------- What about ResultsDB? ---------------------
ResultsDB is part of Taskotron but it is used by other bits like Bodhi and Greenwave. The discussion around where ResultsDB will live going forward and who will be responsible for the deployment is still ongoing but ResultsDB will not be turned off as part of this EOL process.
If you have any questions about this, feel free to reply to this thread.
Thanks,
Tim
_______________________________________________ test-announce mailing list -- test-announce@lists.fedoraproject.org To unsubscribe send an email to test-announce-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test-announce@lists.fedoraproj...
On Fri, Mar 6, 2020 at 4:31 PM Tim Flink tflink@redhat.com wrote:
In the short term, there is a Jenkins instance which is running rpminspect. Rpminspect includes tests which are effectively the same as rpmlint and rpmgrill which removes the need to run them.
For packages with a <package-name>.rpmlintrc file in the git repo, does rpminspect take that file into account?
On Sat, Mar 7, 2020 at 12:38 AM Jerry James loganjerry@gmail.com wrote:
On Fri, Mar 6, 2020 at 4:31 PM Tim Flink tflink@redhat.com wrote:
In the short term, there is a Jenkins instance which is running rpminspect. Rpminspect includes tests which are effectively the same as rpmlint and rpmgrill which removes the need to run them.
For packages with a <package-name>.rpmlintrc file in the git repo, does rpminspect take that file into account?
This is best answered by David Cantrell (in CC). If he fails to respond, please file a RFE in: https://github.com/rpminspect/rpminspect
On Fri, 6 Mar 2020 16:37:33 -0700 Jerry James loganjerry@gmail.com wrote:
On Fri, Mar 6, 2020 at 4:31 PM Tim Flink tflink@redhat.com wrote:
In the short term, there is a Jenkins instance which is running rpminspect. Rpminspect includes tests which are effectively the same as rpmlint and rpmgrill which removes the need to run them.
For packages with a <package-name>.rpmlintrc file in the git repo, does rpminspect take that file into account?
The short version is no, not at the moment.
rpminspect is mostly capable of using custom configurations - that's something that David Cantrell (the primary author of rpminspect) has been talking about since he started promoting rpminspect.
It's a pretty simple matter to look for a config file in dist-git but I don't believe that the exact format or mechanism for translating something user generated into a partial config that rpminspect understands has been figured out yet.
We were planning to do that in the Jenkins instance that is running rpminspect but now that there is a new to-be-announced system that will replace that setup later this year, I'm not sure it's worth the effort to get that working in two places.
If this is something that you feel needs a higher priority, I'd suggest starting a conversation with the Fedora CI folks on ci@
Thanks,
Tim
On 07. 03. 20 0:29, Tim Flink wrote:
If you have any questions about this, feel free to reply to this thread.
Hello Tim.
I have 3 questions.
1) On what level of Fedora leadership was this decision made (and where can I read about it)?
2) Why are we removing something that arguably has a lot of value with something that doesn't exist yet?
3) And to get a bit more specific: How do we rewrite our general Python versions checks [1] to the Jenkins thing and how do we deploy them? (Our point of contact for this was previously Kamil Páral -- but I suppose that changes now, correct?)
Thanks for running Taskotron so far. It helped us a lot in the past with the Python 2 to 3 transition.
[1] https://github.com/fedora-python/taskotron-python-versions
On Sat, Mar 7, 2020 at 1:05 AM Miro Hrončok mhroncok@redhat.com wrote:
On 07. 03. 20 0:29, Tim Flink wrote:
If you have any questions about this, feel free to reply to this thread.
Hello Tim.
I have 3 questions.
I'll selectively pick just the questions I want to reply to and leave the rest to Tim :-)
- On what level of Fedora leadership was this decision made (and where
can I read about it)?
- Why are we removing something that arguably has a lot of value with
something that doesn't exist yet?
It's not a direct response to your question, but one important fact is that our beefy machines in Fedora Infrastructure are out of warranty now (and replacing them would cost a lot of money, I assume). That's one of the major reasons why they won't be migrated to the new datacenter location, I believe.
- And to get a bit more specific: How do we rewrite our general Python
versions checks [1] to the Jenkins thing and how do we deploy them? (Our point of contact for this was previously Kamil Páral -- but I suppose that changes now, correct?)
AFAIK, here's an example of a generic test support in tmt: https://github.com/psss/tmt/pull/96/files
This just defines the environment and workflow. The actual command is inside the "--script 'some command'" part (in your case something like "--script 'python3 python_versions_check.py foobar-1.2-1.fc31'") , which is provided by the dispatcher (something like taskotron-trigger). The dispatcher part is not ready yet, but will be ready (reportedly) before Flock. So then you'll write some dispatcher integration code that will generate the correct command for your check. You can run it already without the dispatcher integration (by providing the --script command locally, i.e. running it on your own machine), but that's just for testing purposes, of course.
This needs to get finished, polished, documentation created, examples shown, etc. It's not ready yet (and if I'm talking nonsense, someone please correct me).
This should all fall under Fedora CI, as far as I know, so they should be the point of contact: https://docs.fedoraproject.org/en-US/ci/
Thanks for running Taskotron so far. It helped us a lot in the past with the Python 2 to 3 transition.
I'm glad it had a positive impact in some areas. 👋
On Mon, Mar 09, 2020 at 03:04:20PM +0100, Kamil Paral wrote:
On Sat, Mar 7, 2020 at 1:05 AM Miro Hrončok mhroncok@redhat.com wrote:
On 07. 03. 20 0:29, Tim Flink wrote:
If you have any questions about this, feel free to reply to this thread.
Hello Tim.
I have 3 questions.
I'll selectively pick just the questions I want to reply to and leave the rest to Tim :-)
- On what level of Fedora leadership was this decision made (and where
can I read about it)?
- Why are we removing something that arguably has a lot of value with
something that doesn't exist yet?
It's not a direct response to your question, but one important fact is that our beefy machines in Fedora Infrastructure are out of warranty now (and replacing them would cost a lot of money, I assume). That's one of the major reasons why they won't be migrated to the new datacenter location, I believe.
Maybe that shows my ignorance, but has the approach of using machines w/o warranty been considered? Most servers seem to work just fine for years and years, and the warranty is mostly useful to catch factory defects.
Zbyszek
It's not a direct response to your question, but one important fact is that our beefy machines in Fedora Infrastructure are out of warranty now (and replacing them would cost a lot of money, I assume). That's one of the major reasons why they won't be migrated to the new datacenter location, I believe.
Maybe that shows my ignorance, but has the approach of using machines w/o warranty been considered? Most servers seem to work just fine for years and years, and the warranty is mostly useful to catch factory defects.
We've often used machines out of warranty for more capacity where they're non critical services, I suspect part of the issue here is there's a DC move where all of the Fedora infra is being shipped across the US and I suspect (I'm not working on this project) that the cost of shipping EOL servers from PHX to RDU is probably a lot more than the value of the equipment so it likely doesn't make financial sense to do so in this case.
On Mon, 9 Mar 2020 at 11:00, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Mon, Mar 09, 2020 at 03:04:20PM +0100, Kamil Paral wrote:
On Sat, Mar 7, 2020 at 1:05 AM Miro Hrončok mhroncok@redhat.com wrote:
On 07. 03. 20 0:29, Tim Flink wrote:
If you have any questions about this, feel free to reply to this
thread.
Hello Tim.
I have 3 questions.
I'll selectively pick just the questions I want to reply to and leave the rest to Tim :-)
- On what level of Fedora leadership was this decision made (and where
can I read about it)?
- Why are we removing something that arguably has a lot of value with
something that doesn't exist yet?
It's not a direct response to your question, but one important fact is
that
our beefy machines in Fedora Infrastructure are out of warranty now (and replacing them would cost a lot of money, I assume). That's one of the major reasons why they won't be migrated to the new datacenter location,
I
believe.
Maybe that shows my ignorance, but has the approach of using machines w/o warranty been considered? Most servers seem to work just fine for years and years, and the warranty is mostly useful to catch factory defects.
Warranty covers disk drives.. which the server just reported a drive dropped within the last hour. QA boxes seem to lose drives more than any other hardware we have.. we have had to rebuild QA boxes regularly from scratch multiple times from 3 drives dieing on raid-6.
Second the new datacentre is size constrained. We currently have 13 racks of equipment and the new data centre is 5 racks. While the new racks are 10U bigger that really only means we have '6' U of old rack space. The move is several thousand miles of roads/trains/etc. While bundled carefully, you can expect anywhere from 5 to 20% of the equipment to need warranty repairs for broken drives/motherboards/etc after the move. The more stuff you move the larger the percentage because the more likely things cascade. So we are shipping the bare minimum of hardware which has no warranty with the knowledge that things like PPC/Arm may not work and will need focus from the 2 sysadmins in Fedoraproject.
Third new work on taskotron ended internally a while ago. Most of the fixes I know of have been the sort of 'well huh who would have coded it that way.. oh git blame it would have to be me.' for a while. This end of life was going to happen either now or any day ending in Y when those fixes weren't possible anymore.
On Sat, 7 Mar 2020 01:03:53 +0100 Miro Hrončok mhroncok@redhat.com wrote:
On 07. 03. 20 0:29, Tim Flink wrote:
If you have any questions about this, feel free to reply to this thread.
Hello Tim.
I have 3 questions.
- On what level of Fedora leadership was this decision made (and
where can I read about it)?
Much of it was an informal discussion between the Taskotron developers where we came to the conclusion that Fedora CI was in a better place to run these checks for Fedora.
- Why are we removing something that arguably has a lot of value
with something that doesn't exist yet?
I don't really have much to add to the conversation that has already happened elsewhere in the thread. This particular EOL date was chosen due to the infra move and the space constraints they have going forward.
Even if we wanted to revive Taskotron at this point, a lot of work would be needed to change things to fit into current CPE paradigms. We'd also need to address the bit-rot from being in "minimal effort maintenance mode" for years.
- And to get a bit more specific: How do we rewrite our general
Python versions checks [1] to the Jenkins thing and how do we deploy them? (Our point of contact for this was previously Kamil Páral -- but I suppose that changes now, correct?)
I was under the impression that the python-versions check wasn't needed going forward. Did I misunderstand something?
For the short term, I can hack those to run in Jenkins for now so that it (and any others needed) continue to run and aren't disrupted.
They would eventually need to be ported to the new setup which I'm told will be based around tmt. Some of those details have come up already and the Fedora CI folks will have more information once more of the required bits are in place.
Thanks for running Taskotron so far. It helped us a lot in the past with the Python 2 to 3 transition.
I'm glad to hear that it's been helpful. Thank you for your support over the years.
Tim
[1] https://github.com/fedora-python/taskotron-python-versions
On 10. 03. 20 19:16, Tim Flink wrote:
I was under the impression that the python-versions check wasn't needed going forward. Did I misunderstand something?
Technically, it is not needed for Fedora 32+ unless people go and push serious regressions. However we monitor rawhide pretty closely and report bugs immediately when something like that happens.
For Fedora 30 and 31 the checks still make sense until those releases go EOL. Not sure however if it's worth investing time in.
For the short term, I can hack those to run in Jenkins for now so that it (and any others needed) continue to run and aren't disrupted.
I'd be more interested in how's that supposed to be done rather than the result. From you offer, it seems that you actually know that. I would be interested in hacking something up together with you to see how this works. But honestly, rather than "your package requires both Pythons" I'd hack "your package breaks dependencies for 80 other packages". Do you think that's possible?
They would eventually need to be ported to the new setup which I'm told will be based around tmt. Some of those details have come up already and the Fedora CI folks will have more information once more of the required bits are in place.
I am honestly afraid that the Fedora CI folks are overwhelmed already. At least that's the impression I get form the tasks that remain unanswered for months :(
https://pagure.io/fedora-ci/general/issues?status=Open&author=churchyard
On Sat, Mar 7, 2020 at 12:31 AM Tim Flink tflink@redhat.com wrote:
What does this mean for me?
If you're not a packager, you will probably not notice the shutdown.
If you are a packager, this will affect some of the results that you currently see in Bodhi under the "Automated Tests" tab. At the moment, most of those results come from Taskotron.
To clarify, Taskotron results are those prefixed with "dist.". Depending on the nature of the update, there can be a number of "update.", "fedora-ci." or "org.centos." results [1] which are not provided by Taskotron (and will not be affected by Taskotron shutdown).
[1] As an example, see https://bodhi.fedoraproject.org/updates/FEDORA-2020-8b563bc5f4
-
Jerry James -
Kamil Paral -
Miro Hrončok -
Peter Robinson -
Stephen John Smoogen -
Tim Flink -
Zbigniew Jędrzejewski-Szmek