8:14 a.m.
churchyard reported a new issue against the project: `releng` that you are following:
``
The https://fedoraproject.org/wiki/Fails_to_build_from_source policy says:
If a FTBFS bug remains in NEW state, a weekly reminder will be sent.
This never happened.
(Opened also as https://pagure.io/fesco/issue/1973 - feel free to close the less
appropriate one.)
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
12:44 p.m.
The issue: `The FTBFS cleanup policy is not happening` of project: `releng` has been
assigned to `mohanboddu` by mohanboddu.
https://pagure.io/releng/issue/7718
12:44 p.m.
mohanboddu added a new comment to an issue you are following:
``
We dont have the tooling ready for it, I will work on it asap.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
1:22 p.m.
till added a new comment to an issue you are following:
``
@zbyszek is also working on this and currently uses the Bugzilla NEEDINFO reminders for
this.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
6:25 a.m.
churchyard added a new comment to an issue you are following:
``
The package will be orphaned after the FTBFS bug is in NEW state for
8 weeks.
This should happen this week. No remainders have been sent so far.
Is the orphaning still happening or is it blocked by the reminders?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
1:31 p.m.
churchyard added a new comment to an issue you are following:
``
I am sorry that I poke at this again and again, but is there anything I can do to move
this forward?
**I can send the reminders myself**, if this would make the policy happen. Let me know.
There are 1042 FTBFS bugzillas in NEW state. Those packages should have been orphaned
already by this policy, yet nothing happened yet.
Getting unmaintained packages orphaned (and later retired) is important to get rid of old
cruft, for me personally this is relevant for the "get rid of python2" effort,
where plenty of packages that use python2 (yet nobody updates them) would go away. I also
believe this is needed for the general health of the project.
This appears to be the consensus of FESCo anyway https://pagure.io/fesco/issue/1890
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
3:31 p.m.
mohanboddu added a new comment to an issue you are following:
``
@churchyard Can you create a script to do this and send it as PR. As I dont have time for
this, I would appreciate your help.
Thanks
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
10:12 a.m.
churchyard added a new comment to an issue you are following:
``
I'll have a look in couple days.
Just to make sure, the script will:
* get all NEW bugs blocking the FTBFS tracker
* post a comment (to all fo them) from https://pagure.io/fesco/issue/1890#comment-520503
(except it says F29)
* set a needinfo flag (assignee)
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
4:29 a.m.
churchyard added a new comment to an issue you are following:
``
Here's a script:
https://github.com/hroncok/ftbfs_bugzilla/blob/master/ftbfs_weekly_remind... - it is
public domain.
It updates bugz one by one and maintains a list of already updated bugz (at filesystem
level) = it can be interrupted and run again.
I guess updating in bulk would be faster, but:
* I found no way to set needinfo for assignee in bulk
* it tends to error and there is no (easy) way to tell which bugs got updated
So this is slow but quite reliable.
There is also bugzilla.log that gives you all the debug information needed.
The script has a TESTING boolean, set to True if you really want to run this, keep at
False to use partner-bugzilla and fake bugz. Uses click and python-bugzilla (available
from PyPI or RPMs). Tested on Python 3.7 only, should work with 3.6 as well.
Let me know if this is not what you wanted.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
6:42 a.m.
churchyard added a new comment to an issue you are following:
``
Is the script OK for what you needed? How can I help? Shall I run it myself?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
8:21 a.m.
churchyard added a new comment to an issue you are following:
``
https://pagure.io/releng/pull-request/7881
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
5:15 a.m.
churchyard added a new comment to an issue you are following:
``
A week before the mass branching any packages which still have open
FTBFS bugs from the previous release will be retired.
That shall be 2019-02-12.
We have 1043 open F29 FTBFS bugs. Are we going to retire 1043 packages in February?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
9:45 a.m.
syeghiay added a new comment to an issue you are following:
``
@humaton will talk to @mohanboddu about this ticket.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
12:38 p.m.
churchyard added a new comment to an issue you are following:
``
We have a new FTBFS tracker for a new release. Can we get this started please? How can I
help?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
1:13 p.m.
kevin added a new comment to an issue you are following:
``
So, we need to set this up in infrastructure.
It looks like it uses an interactive login? Can we have it take a user/token for running
in cron?
I guess we also need to determine what user we want to use here... releng? bodhi?
something new for just this use?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
2:55 p.m.
churchyard added a new comment to an issue you are following:
``
It looks like it uses an interactive login? Can we have it take a
user/token for running in cron?
Do you have an example of such script so I don't need to reinvent the wheel?
I guess we also need to determine what user we want to use here...
Releng files those bugs, so I guess we can reuse that user as well.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
12:10 p.m.
kevin added a new comment to an issue you are following:
``
There is this one:
https://github.com/fedora-infra/fas/blob/develop/tools/export-bugzilla.py
Yeah, releng user makes the most sense I guess...
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
8:51 a.m.
churchyard added a new comment to an issue you are following:
``
`from turbogears import config`, yummy :(
Do you have an example config I can see to rewrite this without turbogears? I guess I can
use plain ConfigParser.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
8:54 a.m.
churchyard added a new comment to an issue you are following:
``
I'm going to assume this for now:
```
[bugzilla]
username = releng
password = hunter2
```
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
9:07 a.m.
churchyard added a new comment to an issue you are following:
``
Adjusted https://pagure.io/releng/pull-request/7881 to use a config file. Adjust the
config file paths as desired.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
2:14 p.m.
churchyard added a new comment to an issue you are following:
``
Can we get the cron set up please? I'll try to talk to @kevin on IRC on Monday, so we
can work on it together.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
2:26 p.m.
smooge added a new comment to an issue you are following:
``
Kevin is out on vacation until the 17th. I will try and look at what needs to be done on
Monday.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
8:46 a.m.
The status of the issue: `The FTBFS cleanup policy is not happening` of project: `releng`
has been updated to: Closed as Fixed by smooge.
https://pagure.io/releng/issue/7718
8:46 a.m.
smooge added a new comment to an issue you are following:
``
Mohan and I deployed this this morning. It should start bugging people on the weekly
cron.weekly run
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
8:49 a.m.
churchyard added a new comment to an issue you are following:
``
Thanks.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
6:38 a.m.
The status of the issue: `The FTBFS cleanup policy is not happening` of project: `releng`
has been updated to: Open by churchyard.
https://pagure.io/releng/issue/7718
6:38 a.m.
churchyard added a new comment to an issue you are following:
``
While I have seen some of the comments, last time they have happened at 2019-06-17. This
is apparently not happening weekly.
Take for example this [randomly selected F30FTBFS
bug](https://bugzilla.redhat.com/show_bug.cgi?id=1675170):
The comments happened:
* 2019-04-27 (possibly a test run)
* 2019-06-03
* 2019-06-17 (2 weeks after the previous, not 1 week)
If more comments appear today, we probably just run it bi-weekly instead of weekly.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
8:34 a.m.
smooge added a new comment to an issue you are following:
``
Cron job script was hosed. I have fixed the script and I am now running the script by
hand.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
8:41 a.m.
smooge added a new comment to an issue you are following:
``
OK running the script, it gave the following output:
```
[root@compose-x86-01 ~][PROD]# su ftbfs /etc/cron.weekly/ftbfs.cron
Cloning into 'releng'...
remote: Counting objects: 7869, done.
remote: Compressing objects: 100% (3487/3487), done.
remote: Total 7869 (delta 4422), reused 7712 (delta 4286)
Receiving objects: 100% (7869/7869), 1.36 MiB | 1.89 MiB/s, done.
Resolving deltas: 100% (4422/4422), done.
Loading bug IDs from /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED. Will not fill
those. Remove /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED to stop this from
happening.
Gathering bugz, this can take a while...
There are 755 NEW bugz, will send a reminder
Will ignore 758 bugz from /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED
Will update 25 bugz
[####################################] 100% Finished!
Updated 0 bugz
Failed to update 25 bugz
https://bugzilla.redhat.com/show_bug.cgi?id=1674669
https://bugzilla.redhat.com/show_bug.cgi?id=1674680
https://bugzilla.redhat.com/show_bug.cgi?id=1674753
https://bugzilla.redhat.com/show_bug.cgi?id=1674783
https://bugzilla.redhat.com/show_bug.cgi?id=1674817
https://bugzilla.redhat.com/show_bug.cgi?id=1675022
https://bugzilla.redhat.com/show_bug.cgi?id=1675036
https://bugzilla.redhat.com/show_bug.cgi?id=1675097
https://bugzilla.redhat.com/show_bug.cgi?id=1675108
https://bugzilla.redhat.com/show_bug.cgi?id=1675364
https://bugzilla.redhat.com/show_bug.cgi?id=1675407
https://bugzilla.redhat.com/show_bug.cgi?id=1675415
https://bugzilla.redhat.com/show_bug.cgi?id=1675454
https://bugzilla.redhat.com/show_bug.cgi?id=1675586
https://bugzilla.redhat.com/show_bug.cgi?id=1675593
https://bugzilla.redhat.com/show_bug.cgi?id=1675684
https://bugzilla.redhat.com/show_bug.cgi?id=1675784
https://bugzilla.redhat.com/show_bug.cgi?id=1675892
https://bugzilla.redhat.com/show_bug.cgi?id=1675899
https://bugzilla.redhat.com/show_bug.cgi?id=1675918
https://bugzilla.redhat.com/show_bug.cgi?id=1676017
https://bugzilla.redhat.com/show_bug.cgi?id=1676164
https://bugzilla.redhat.com/show_bug.cgi?id=1676181
https://bugzilla.redhat.com/show_bug.cgi?id=1676205
https://bugzilla.redhat.com/show_bug.cgi?id=1679955
```
I am not sure what is causing bzapi.update_bugs([bug.id], update) to give an exception.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
8:42 a.m.
churchyard added a new comment to an issue you are following:
``
rerunning the script on failure should do.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
9:33 a.m.
smooge added a new comment to an issue you are following:
``
It seemed to fail identically on rerun
```
[root@compose-x86-01 ~][PROD]# su ftbfs /etc/cron.weekly/ftbfs.cron
Cloning into 'releng'...
remote: Counting objects: 7869, done.
remote: Compressing objects: 100% (3487/3487), done.
remote: Total 7869 (delta 4422), reused 7713 (delta 4286)
Receiving objects: 100% (7869/7869), 1.36 MiB | 1.63 MiB/s, done.
Resolving deltas: 100% (4422/4422), done.
Loading bug IDs from /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED. Will not fill
those. Remove /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED to stop this from
happening.
Gathering bugz, this can take a while...
There are 755 NEW bugz, will send a reminder
Will ignore 758 bugz from /home/ftbfs/.cache/FTBFS_weekly_reminder/ALREADY_FILED
Will update 25 bugz
[####################################] 100% Finished!
Updated 0 bugz
Failed to update 25 bugz
https://bugzilla.redhat.com/show_bug.cgi?id=1674669
https://bugzilla.redhat.com/show_bug.cgi?id=1674680
https://bugzilla.redhat.com/show_bug.cgi?id=1674753
https://bugzilla.redhat.com/show_bug.cgi?id=1674783
https://bugzilla.redhat.com/show_bug.cgi?id=1674817
https://bugzilla.redhat.com/show_bug.cgi?id=1675022
https://bugzilla.redhat.com/show_bug.cgi?id=1675036
https://bugzilla.redhat.com/show_bug.cgi?id=1675097
https://bugzilla.redhat.com/show_bug.cgi?id=1675108
https://bugzilla.redhat.com/show_bug.cgi?id=1675364
https://bugzilla.redhat.com/show_bug.cgi?id=1675407
https://bugzilla.redhat.com/show_bug.cgi?id=1675415
https://bugzilla.redhat.com/show_bug.cgi?id=1675454
https://bugzilla.redhat.com/show_bug.cgi?id=1675586
https://bugzilla.redhat.com/show_bug.cgi?id=1675593
https://bugzilla.redhat.com/show_bug.cgi?id=1675684
https://bugzilla.redhat.com/show_bug.cgi?id=1675784
https://bugzilla.redhat.com/show_bug.cgi?id=1675892
https://bugzilla.redhat.com/show_bug.cgi?id=1675899
https://bugzilla.redhat.com/show_bug.cgi?id=1675918
https://bugzilla.redhat.com/show_bug.cgi?id=1676017
https://bugzilla.redhat.com/show_bug.cgi?id=1676164
https://bugzilla.redhat.com/show_bug.cgi?id=1676181
https://bugzilla.redhat.com/show_bug.cgi?id=1676205
https://bugzilla.redhat.com/show_bug.cgi?id=1679955
ftbfs_weekly_reminder.py had an error condition
Look in /tmp/ftbfs_reminder.8AGrhM for more info
```
Here is the script
```
#!/bin/bash
# A weekly cron job which actually does the items needed.
TMPDIR=`mktemp -d /tmp/ftbfs_reminder.XXXXXX`
GITREPO=https://pagure.io/releng.git
SCRIPT=ftbfs_weekly_reminder.py
if [ $? -eq 0 ]; then
cd ${TMPDIR}
git clone ${GITREPO}
if [ $? -eq 0 ]; then
cd releng/scripts
./${SCRIPT}
if [ $? -ne 0 ]; then
echo "${SCRIPT} had an error condition"
echo "Look in ${TMPDIR} for more info"
# Do not clean up trash
exit 1
fi
else
echo "Unable to clone ${GITREPO}"
echo "Look in ${TMPDIR} for more info"
# Do not clean up trash
exit 1
fi
cd /tmp/
rm -rf $TMPDIR
else
echo "Unable to create ${TMPDIR}"
exit 1
fi
```
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
9:56 a.m.
churchyard added a new comment to an issue you are following:
``
You can't ask Marek Mahut <mmahut(a)redhat.com> because that
account is disabled.
It seems that the script needs to distinguish temporary BZ nonsense failures against
permanent ones :(
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
5:01 a.m.
churchyard added a new comment to an issue you are following:
``
do you happen to have /tmp/ftbfs_reminder.8AGrhM ? the script loggs the exceptions.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
5:09 a.m.
churchyard added a new comment to an issue you are following:
``
correction, the exception is logged in bugzilla.log
```
Traceback (most recent call last):
File "ftbfs_weekly_reminder.py", line 90, in send_reminder
bzapi.update_bugs([bug.id], update)
File "/usr/lib/python3.7/site-packages/bugzilla/base.py", line 1321, in
update_bugs
return self._proxy.Bug.update(tmp)
File "/usr/lib64/python3.7/xmlrpc/client.py", line 1112, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 104, in
_ServerProxy__request
self)._ServerProxy__request(methodname, params)
File "/usr/lib64/python3.7/xmlrpc/client.py", line 1452, in __request
verbose=self.__verbose
File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 201, in
request
return self._request_helper(url, request_body)
File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 179, in
_request_helper
return self.parse_response(response)
File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 154, in
parse_response
return unmarshaller.close()
File "/usr/lib64/python3.7/xmlrpc/client.py", line 656, in close
raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 32000: "You can't ask Martin Preisler
<mpreisle(a)redhat.com> because that account is disabled.">
```
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
5:19 a.m.
churchyard added a new comment to an issue you are following:
``
See https://pagure.io/releng/pull-request/8480 - it now attempts not to add needinfo if it
breaks. The script output is also useful in case we want to see what redhat bugzilla
accounts are gone.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
8:57 a.m.
The status of the issue: `The FTBFS cleanup policy is not happening` of project: `releng`
has been updated to: Closed as Fixed by churchyard.
https://pagure.io/releng/issue/7718
8:57 a.m.
churchyard added a new comment to an issue you are following:
``
Thanks @smooge @mohanboddu
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
6:19 a.m.
churchyard added a new comment to an issue you are following:
``
https://pagure.io/releng/issue/8522 is the request to retire FTBFS packages according to
the policy
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
2:42 a.m.
The status of the issue: `The FTBFS cleanup policy is not happening` of project: `releng`
has been updated to: Open by churchyard.
https://pagure.io/releng/issue/7718
2:42 a.m.
churchyard added a new comment to an issue you are following:
``
I have a feeling that this stopped happening again.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
2:44 a.m.
churchyard added a new comment to an issue you are following:
``
To clarify, the comments happened on 2019-07-02 and than stopped. No comments a week
later, no comments 2 weeks later either.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
7:27 a.m.
smooge added a new comment to an issue you are following:
``
I made a spalling mistake in the cron job so it was not able to run. I have corrected the
cron job and run the script by hand.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
7:30 a.m.
The status of the issue: `The FTBFS cleanup policy is not happening` of project: `releng`
has been updated to: Closed as Fixed by smooge.
https://pagure.io/releng/issue/7718
7:32 a.m.
The status of the issue: `The FTBFS cleanup policy is not happening` of project: `releng`
has been updated to: Open by smooge.
https://pagure.io/releng/issue/7718
7:33 a.m.
smooge added a new comment to an issue you are following:
``
Script failed to run:
```
[root@compose-x86-01 ~][PROD]# su -s /bin/sh ftbfs
sh-5.0$ sh -x /usr/local/bin/ftbfs.sh
++ mktemp -d /tmp/ftbfs_reminder.XXXXXX
+ TMPDIR=/tmp/ftbfs_reminder.jutVAp
+ GITREPO=https://pagure.io/releng.git
+ SCRIPT=ftbfs_weekly_reminder.py
+ '[' 0 -eq 0 ']'
+ cd /tmp/ftbfs_reminder.jutVAp
+ git clone https://pagure.io/releng.git
Cloning into 'releng'...
remote: Counting objects: 7902, done.
remote: Compressing objects: 100% (3401/3401), done.
remote: Total 7902 (delta 4449), reused 7835 (delta 4405)
Receiving objects: 100% (7902/7902), 1.36 MiB | 741.00 KiB/s, done.
Resolving deltas: 100% (4449/4449), done.
+ '[' 0 -eq 0 ']'
+ cd releng/scripts
+ ./ftbfs_weekly_reminder.py
Gathering bugz, this can take a while...
There are 681 NEW bugz, will send a reminder
Traceback (most recent call last):
File "./ftbfs_weekly_reminder.py", line 126, in <module>
if ignore:
NameError: name 'ignore' is not defined
+ '[' 1 -ne 0 ']'
+ echo 'ftbfs_weekly_reminder.py had an error condition'
ftbfs_weekly_reminder.py had an error condition
+ echo 'Look in /tmp/ftbfs_reminder.jutVAp for more info'
Look in /tmp/ftbfs_reminder.jutVAp for more info
+ exit 1
```
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
7:37 a.m.
churchyard added a new comment to an issue you are following:
``
https://pagure.io/releng/pull-request/8560
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7718
1735
days inactive
2065
days old
rel-eng@lists.fedoraproject.org
45 comments
6 participants
participants (6)
-
Kevin Fenzi -
Miro Hrončok
-
Mohan Boddu
-
Stephen J Smoogen
-
Suzanne Yeghiayan
-
Till Maas