[releng] PR #6844: Add a script that will get the packages that have all the specified branches but are retired
1:56 p.m.
mprahl opened a new pull-request against the project: `releng` that you are following:
``
Add a script that will get the packages that have all the specified branches but are
retired
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
2:12 p.m.
ralph commented on the pull-request: `Add a script that will get the packages that have
all the specified branches but are retired` that you are following:
``
Should specify in the docstring:
- have *any* of the specified branches marked as...
or
- have *all* of the specified branches marked as inactive...
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
2:17 p.m.
ralph commented on the pull-request: `Add a script that will get the packages that have
all the specified branches but are retired` that you are following:
``
I got the following error when trying to run this:
scripts/pdc❯ python get-retired-packages.py master
Traceback (most recent call last):
File "get-retired-packages.py", line 70, in <module>
pkg_sets = pool.map(get_pkg_branch_status, args.branches)
File "/usr/lib64/python2.7/multiprocessing/pool.py", line 251, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib64/python2.7/multiprocessing/pool.py", line 567, in get
raise self._value
TypeError: descriptor 'union' of 'set' object needs an argument
Can we get some better error reporting?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
2:18 p.m.
ralph commented on the pull-request: `Add a script that will get the packages that have
all the specified branches but are retired` that you are following:
``
To explain more: I think `multiprocessing.pool` is masking the *real* traceback inside
one of the subprocesses. Can you catch the error in `get_pkg_branch_status` and log the
details before crashing?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
2:22 p.m.
ralph commented on the pull-request: `Add a script that will get the packages that have
all the specified branches but are retired` that you are following:
``
This would do it:
diff --git a/scripts/pdc/get-retired-packages.py
b/scripts/pdc/get-retired-packages.py
index 0d160a9..5e45355 100644
--- a/scripts/pdc/get-retired-packages.py
+++ b/scripts/pdc/get-retired-packages.py
@@ -7,6 +7,7 @@ except ImportError:
from urllib.parse import urlencode
import multiprocessing.pool
import argparse
+import traceback
from math import ceil
from functools import partial
import requests
@@ -47,6 +48,20 @@ def get_pkgs_from_page(branch_name, page):
return pkgs_set
+def handle_errors(f):
+ def _wrapper(*args, **kwargs):
+ """ A decorator for `f` that prints tracebacks.
"""
+ try:
+ return f(*args, **kwargs)
+ except:
+ traceback.print_exc()
+ raise
+ _wrapper.__name__ = f.__name__
+ _wrapper.__doc__ = f.__doc__
+ return _wrapper
+
+
+@handle_errors
def get_pkg_branch_status(branch_name):
# Get total number of branches that fit the query
component_branch_page_one = \
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
2:29 p.m.
ralph commented on the pull-request: `Add a script that will get the packages that have
all the specified branches but are retired` that you are following:
``
It might also be nice to print a warning to `stderr` if there are no retired packages
found, like this:
diff --git a/scripts/pdc/get-retired-packages.py
b/scripts/pdc/get-retired-packages.py
index 0d160a9..05f702c 100644
--- a/scripts/pdc/get-retired-packages.py
+++ b/scripts/pdc/get-retired-packages.py
@@ -1,12 +1,16 @@
""" get-retired-packages.py - Gets all the packages that have the
specified
branches marked as inactive in PDC
"""
+from __future__ import print_function
+
try:
from urllib import urlencode
except ImportError:
from urllib.parse import urlencode
+
import multiprocessing.pool
import argparse
+import sys
from math import ceil
from functools import partial
import requests
@@ -28,13 +32,13 @@ def get_component_branch_page(branch_name, page, page_size=100):
rv = requests.get(pdc_api_query_url, timeout=30)
except (requests.ConnectionError, requests.ConnectTimeout):
print('The connection to PDC failed')
- exit(1)
+ sys.exit(1)
try:
return rv.json()
except ValueError:
print('The data returned from PDC was not JSON')
- exit(1)
+ sys.exit(1)
def get_pkgs_from_page(branch_name, page):
@@ -61,7 +65,10 @@ def get_pkg_branch_status(branch_name):
pkg_sets = pool.map(partial_get_pkgs_from_page, range(1, num_pages + 1))
pool.close()
# Return a set of all the packages from the pages queried
- return set.union(*pkg_sets)
+ if pkg_sets:
+ return set.union(*pkg_sets)
+ else:
+ return set()
if __name__ == '__main__':
@@ -72,5 +79,9 @@ if __name__ == '__main__':
# Return only the packages that have all the specified branches and are
# retired
- for pkg in sorted(list(set.intersection(*pkg_sets))):
- print(pkg)
+ pkgs = list(set.intersection(*pkg_sets))
+ if pkgs:
+ for pkg in sorted(pkgs):
+ print(pkg)
+ else:
+ print("No retired packages on %r" % args.branches,
file=sys.stderr)
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
3:33 p.m.
mprahl commented on the pull-request: `Add a script that will get the packages that have
all the specified branches but are retired` that you are following:
``
Thanks for the review. I addressed the feedback in the latest commit.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
4:32 p.m.
till commented on the pull-request: `Add a script that will get the packages that have all
the specified branches but are retired` that you are following:
``
IMHO this should be printed to stderr because it is an error message.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
4:32 p.m.
till commented on the pull-request: `Add a script that will get the packages that have all
the specified branches but are retired` that you are following:
``
IMHO this should be printed to stderr because it is an error message.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
4:35 p.m.
till commented on the pull-request: `Add a script that will get the packages that have all
the specified branches but are retired` that you are following:
``
IMHO it would be good to explain why 8 is a good choice here. It is not obvious to me I
must confess.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
4:36 p.m.
till commented on the pull-request: `Add a script that will get the packages that have all
the specified branches but are retired` that you are following:
``
Is it safe to assume that pages will always contain 100 entries? It does not seem to be
set in the actual queries.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
4:39 p.m.
till commented on the pull-request: `Add a script that will get the packages that have all
the specified branches but are retired` that you are following:
``
The lines above this comments after assigning the help variable should IMHO be moved into
the main conditional below.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
4:41 p.m.
till commented on the pull-request: `Add a script that will get the packages that have all
the specified branches but are retired` that you are following:
``
Why exactly four branches? Is it because of el6, epel7, branched and rawhide are the usual
suspects?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
7:43 a.m.
mprahl commented on the pull-request: `Add a script that will get the packages that have
all the specified branches but are retired` that you are following:
``
Yes, the I'm using a `page_size` of `100` in `get_component_branch_page` by default.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
7:55 a.m.
mprahl commented on the pull-request: `Add a script that will get the packages that have
all the specified branches but are retired` that you are following:
``
It is just an arbitrary amount to improve performance without overloading PDC. I can make
this a configuration option.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
8:05 a.m.
mprahl commented on the pull-request: `Add a script that will get the packages that have
all the specified branches but are retired` that you are following:
``
I'll set a configuration option called MAX_NUM_PROCESSES in the file which will
dictate the maximum number of processes to create. If the number of CPUs is lower than
that value, the number of CPUs will be used for the number of processes to create.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
8:15 a.m.
mprahl commented on the pull-request: `Add a script that will get the packages that have
all the specified branches but are retired` that you are following:
``
@till thanks for the review. I addressed your comments.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/pull-request/6844
1:24 p.m.
ralph merged a pull-request against the project: `releng` that you are following.
Merged pull-request:
``
Add a script that will get the packages that have all the specified branches but are
retired
``
https://pagure.io/releng/pull-request/6844
2455
days inactive
2504
days old
rel-eng@lists.fedoraproject.org
17 comments
3 participants
participants (3)
-
Matt Prahl -
Ralph Bean
-
Till Maas