Mass Python 2 packages Removal

[Miro Hrončok]

Hi, this is about:

https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal

We have ~300 bugz open and blocking

https://bugzilla.redhat.com/show_bug.cgi?id=PY2REMOVAL

Some of them are > 1 week old and hence anyone can go and remove the 
mentioned (sub)packages. I've retired the packages that were for 
retirement and I will retire more once they are unblocked (needs to go 
to compose -> portingdb before the automation recognizes it).

I open more bugs as they are closed (I don't wan to deal with 1000 bugs 
at once).

Retirement is easy. The problematic part is removing subpackages. I go 
and manually choose the packages that appear not to be just simple "py 
spec generates py2 and py3 subpackages" combo, but a bit more 
complicated, and I leave the "simple" ones alone for now.

I'm theoretically able to remove a simple py2 subpackage in about a 
minute, yet it is tedious and after ~20 packages I need to stop (before 
I go crazy).

We have something in between 900 and 2000 packages to remove (there are 
900 leaves to remove and we may make another 1100 into leaves as we go).

Is anyone willing to look into automating this? Write a spec depython2izer?

This usually means:

- removing python2- and python- BRs
  - removing %package -n python2 and it's metadata, description & %files
  - removing all %py2_build, %py2_install or their older manual variants
  - removing python2 tests from %check
  - (sometimes) removing %py3dir or python3 dir pushd/popd dance
  - (sometimes) removing with_python3 conditionals as they make no sense

IIRC Zbyszek said at Flock that he might work on that. Is that still 
true? If not, shall I draft something?

[Zbigniew Jędrzejewski-Szmek]

On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
> Is anyone willing to look into automating this? Write a spec depython2izer?
>
> This usually means:
> removing python2- and python- BRs
> removing %package -n python2 and it's metadata, description & %files
> removing all %py2_build, %py2_install or their older manual variants
> removing python2 tests from %check
> (sometimes) removing %py3dir or python3 dir pushd/popd dance
> (sometimes) removing with_python3 conditionals as they make no sense
>
> IIRC Zbyszek said at Flock that he might work on that. Is that still
> true? If not, shall I draft something?

Yeah, I wanted to work on this. When do you need it?

Zbyszek

[Miro Hrončok]

On 18.9.2018 21:43, Zbigniew Jędrzejewski-Szmek wrote:
> On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
> > Is anyone willing to look into automating this? Write a spec depython2izer?
> >
> > This usually means:
> > removing python2- and python- BRs
> > removing %package -n python2 and it's metadata, description & %files
> > removing all %py2_build, %py2_install or their older manual variants
> > removing python2 tests from %check
> > (sometimes) removing %py3dir or python3 dir pushd/popd dance
> > (sometimes) removing with_python3 conditionals as they make no sense
> >
> > IIRC Zbyszek said at Flock that he might work on that. Is that still
> > true? If not, shall I draft something?
>
> Yeah, I wanted to work on this. When do you need it?

Thank You!

I could use it right now. Realistically, we should get this done by the 
end of 2018. Given how we start with leaves, and that there is a week 
waiting period at each point, I think we would really need this working 
by the end of October. Getting a somehow testable version of it in 2-3 
weeks would be great, if it's doable.

[Miro Hrončok]

On 18.9.2018 21:57, Miro Hrončok wrote:
> On 18.9.2018 21:43, Zbigniew Jędrzejewski-Szmek wrote:
> > On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
> > > Is anyone willing to look into automating this? Write a spec 
> > > depython2izer?
> > >
> > > This usually means:
> > >
> > > - removing python2- and python- BRs
> > >   - removing %package -n python2 and it's metadata, description & %files
> > >   - removing all %py2_build, %py2_install or their older manual variants
> > >   - removing python2 tests from %check
> > >   - (sometimes) removing %py3dir or python3 dir pushd/popd dance
> > >   - (sometimes) removing with_python3 conditionals as they make no sense
> > >
> > > IIRC Zbyszek said at Flock that he might work on that. Is that still
> > > true? If not, shall I draft something?
> >
> > Yeah, I wanted to work on this. When do you need it?
>
> Thank You!
>
> I could use it right now. Realistically, we should get this done by the 
> end of 2018. Given how we start with leaves, and that there is a week 
> waiting period at each point, I think we would really need this working 
> by the end of October. Getting a somehow testable version of it in 2-3 
> weeks would be great, if it's doable.

Just a note that all bugs for packages that can be dropped now are filled.

[Zbigniew Jędrzejewski-Szmek]

> > > On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
> > > > Is anyone willing to look into automating this? Write a spec
> > > > depython2izer?

Hi,

I wrote something that mostly seems to work:
https://pagure.io/pyrenamer/blob/master/f/depython2ize.py

../pyrenamer/depython2ize.py -b -d python-alchimia python-gzipstream \
  python-ivi python-pdfkit python-simpleparse python-walkdir \
  python-XStatic-Jasmine python-yourls ...

If does:
> > > > - removing python2- and python- BRs

yes
> > > > - removing %package -n python2 and it's metadata, description & %files

yes
> > > > - removing all %py2_build, %py2_install or their older manual variants

yes
> > > > - removing python2 tests from %check

yes
> > > > - (sometimes) removing %py3dir or python3 dir pushd/popd dance

yes (by default, can be disabled with a switch)
> > > > - (sometimes) removing with_python3 conditionals as they make no sense

no

The workflow is to first calls it with just -bd for review, and then with -bwn
to actually commit.

I tested it for packages from $(bugzilla query --blocked=1625773 -s NEW).
For the ones that are called python-* it seems to work, and quite a lot
of them even build fine. Sometimes manual fixups will be necessary (e.g.
when the py2 directories or files are referenced from %build in non-trivial
ways).

For packages with generic names it doesn't work because python2 subpackage
detection is broken. Should be relatively easy to fix. I'll work on that 
tomorrow, so please just ignore them for now.

PTAL. Let me know what bits are missing or buggy.

I'd also be happy to help with pushing some of those patches to
dist-git and doing the builds, but for this we need to coordinate.

Example patches: http://in.waw.pl/~zbyszek/fedora/depy2/.

Zbyszek

[Miro Hrončok]

On 9.10.2018 22:51, Zbigniew Jędrzejewski-Szmek wrote:
> > > > On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
> > > > > Is anyone willing to look into automating this? Write a spec
> > > > > depython2izer?
>
> Hi,
>
> I wrote something that mostly seems to work:
> https://pagure.io/pyrenamer/blob/master/f/depython2ize.py
>
> ../pyrenamer/depython2ize.py -b -d python-alchimia python-gzipstream \
>    python-ivi python-pdfkit python-simpleparse python-walkdir \
>    python-XStatic-Jasmine python-yourls ...
>
> If does:
> > > > > - removing python2- and python- BRs
>
> yes
> > > > > - removing %package -n python2 and it's metadata, description & %files
>
> yes
> > > > > - removing all %py2_build, %py2_install or their older manual variants
>
> yes
> > > > > - removing python2 tests from %check
>
> yes
> > > > > - (sometimes) removing %py3dir or python3 dir pushd/popd dance
>
> yes (by default, can be disabled with a switch)
> > > > > - (sometimes) removing with_python3 conditionals as they make no sense
>
> no
>
> The workflow is to first calls it with just -bd for review, and then with -bwn
> to actually commit.
>
> I tested it for packages from $(bugzilla query --blocked=1625773 -s NEW).
> For the ones that are called python-* it seems to work, and quite a lot
> of them even build fine. Sometimes manual fixups will be necessary (e.g.
> when the py2 directories or files are referenced from %build in non-trivial
> ways).
>
> For packages with generic names it doesn't work because python2 subpackage
> detection is broken. Should be relatively easy to fix. I'll work on that
> tomorrow, so please just ignore them for now.
>
> PTAL. Let me know what bits are missing or buggy.
>
> I'd also be happy to help with pushing some of those patches to
> dist-git and doing the builds, but for this we need to coordinate.
>
> Example patches: http://in.waw.pl/~zbyszek/fedora/depy2/.

Thank You, Zbyszek,
this looks very good.

Before we start going wild and use this, there are 2 things we should 
consider:

1) We should query bugzilla for NEW bugs without comments. Those with 
comments can be hand-triaged later (it isn't be many). Some comments may 
say OK, some may say STOP.

2) There is a list of packages to remove. We should use the list and 
check if the script doesn't remove anything else (e.g. it may be that we 
want to remove python2-foo-coolfeature, but we need to keep 
python2-foo). The results are obtainable via [1] and I've copied my last 
results to [2]. For every package the script wants to remove a check 
should be made that pkgs[name]["verdict"] == "drop_now".

I'll try the script myself for a couple packages tomorrow.

For coordination: I was doing it in a way that I've assigned the bug to 
me and changed to ASSIGNED, built, CLOSED RAWHIDE. I guess we can 
continue doing this (individually or in bulks) and any number of 
provenpackagers can proceed as long as [1]/[2] is checked.

[1] https://github.com/fedora-python/portingdb#check-drops
[2] https://churchyard.fedorapeople.org/results.json

[Miro Hrončok]

On 9.10.2018 23:42, Miro Hrončok wrote:
> On 9.10.2018 22:51, Zbigniew Jędrzejewski-Szmek wrote:
> > > > > On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
> > > > > > Is anyone willing to look into automating this? Write a spec
> > > > > > depython2izer?
> >
> > Hi,
> >
> > I wrote something that mostly seems to work:
> > https://pagure.io/pyrenamer/blob/master/f/depython2ize.py
> >
> > ../pyrenamer/depython2ize.py -b -d python-alchimia python-gzipstream \
> >    python-ivi python-pdfkit python-simpleparse python-walkdir \
> >    python-XStatic-Jasmine python-yourls ...
> >
> > If does:
> > > > > > - removing python2- and python- BRs
> >
> > yes
> > > > > > - removing %package -n python2 and it's metadata, description & 
> > > > > > %files
> >
> > yes
> > > > > > - removing all %py2_build, %py2_install or their older manual 
> > > > > > variants
> >
> > yes
> > > > > > - removing python2 tests from %check
> >
> > yes
> > > > > > - (sometimes) removing %py3dir or python3 dir pushd/popd dance
> >
> > yes (by default, can be disabled with a switch)
> > > > > > - (sometimes) removing with_python3 conditionals as they make 
> > > > > > no sense
> >
> > no
> >
> > The workflow is to first calls it with just -bd for review, and then 
> > with -bwn
> > to actually commit.
> >
> > I tested it for packages from $(bugzilla query --blocked=1625773 -s NEW).
> > For the ones that are called python-* it seems to work, and quite a lot
> > of them even build fine. Sometimes manual fixups will be necessary (e.g.
> > when the py2 directories or files are referenced from %build in 
> > non-trivial
> > ways).
> >
> > For packages with generic names it doesn't work because python2 
> > subpackage
> > detection is broken. Should be relatively easy to fix. I'll work on that
> > tomorrow, so please just ignore them for now.
> >
> > PTAL. Let me know what bits are missing or buggy.
> >
> > I'd also be happy to help with pushing some of those patches to
> > dist-git and doing the builds, but for this we need to coordinate.
> >
> > Example patches: http://in.waw.pl/~zbyszek/fedora/depy2/.
>
> Thank You, Zbyszek,
> this looks very good.
>
> Before we start going wild and use this, there are 2 things we should 
> consider:
> We should query bugzilla for NEW bugs without comments. Those with
>
> comments can be hand-triaged later (it isn't be many). Some comments may 
> say OK, some may say STOP.
> There is a list of packages to remove. We should use the list and
>
> check if the script doesn't remove anything else (e.g. it may be that we 
> want to remove python2-foo-coolfeature, but we need to keep 
> python2-foo). The results are obtainable via [1] and I've copied my last 
> results to [2]. For every package the script wants to remove a check 
> should be made that pkgs[name]["verdict"] == "drop_now".

I'm writing this part now.
> I'll try the script myself for a couple packages tomorrow.
>
> For coordination: I was doing it in a way that I've assigned the bug to 
> me and changed to ASSIGNED, built, CLOSED RAWHIDE. I guess we can 
> continue doing this (individually or in bulks) and any number of 
> provenpackagers can proceed as long as [1]/[2] is checked.
>
> [1] https://github.com/fedora-python/portingdb#check-drops
> [2] https://churchyard.fedorapeople.org/results.json

[Miro Hrončok]

On 9.10.2018 22:51, Zbigniew Jędrzejewski-Szmek wrote:
> > > > On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
> > > > > Is anyone willing to look into automating this? Write a spec
> > > > > depython2izer?
>
> Hi,
>
> I wrote something that mostly seems to work:
> https://pagure.io/pyrenamer/blob/master/f/depython2ize.py
>
> ../pyrenamer/depython2ize.py -b -d python-alchimia python-gzipstream \
>    python-ivi python-pdfkit python-simpleparse python-walkdir \
>    python-XStatic-Jasmine python-yourls ...
>
> If does:
> > > > > - removing python2- and python- BRs
>
> yes
> > > > > - removing %package -n python2 and it's metadata, description & %files
>
> yes
> > > > > - removing all %py2_build, %py2_install or their older manual variants
>
> yes
> > > > > - removing python2 tests from %check
>
> yes
> > > > > - (sometimes) removing %py3dir or python3 dir pushd/popd dance
>
> yes (by default, can be disabled with a switch)
> > > > > - (sometimes) removing with_python3 conditionals as they make no sense
>
> no
>
> The workflow is to first calls it with just -bd for review, and then with -bwn
> to actually commit.
>
> I tested it for packages from $(bugzilla query --blocked=1625773 -s NEW).
> For the ones that are called python-* it seems to work, and quite a lot
> of them even build fine. Sometimes manual fixups will be necessary (e.g.
> when the py2 directories or files are referenced from %build in non-trivial
> ways).
>
> For packages with generic names it doesn't work because python2 subpackage
> detection is broken. Should be relatively easy to fix. I'll work on that
> tomorrow, so please just ignore them for now.
>
> PTAL. Let me know what bits are missing or buggy.

python-XStatic-* packages have a lot of remaining mvs with 
python2_sitelib involved. e.g. python-XStatic-Jasmine, 
python-XStatic-Font-Awesome.

[Miro Hrončok]

On 9.10.2018 22:51, Zbigniew Jędrzejewski-Szmek wrote:
> > > > On Tue, Sep 18, 2018 at 03:14:41PM +0200, Miro Hrončok wrote:
> > > > > Is anyone willing to look into automating this? Write a spec
> > > > > depython2izer?
>
> Hi,
>
> I wrote something that mostly seems to work:
> https://pagure.io/pyrenamer/blob/master/f/depython2ize.py
>
> ../pyrenamer/depython2ize.py -b -d python-alchimia python-gzipstream \
>    python-ivi python-pdfkit python-simpleparse python-walkdir \
>    python-XStatic-Jasmine python-yourls ...
> ...

Hi, I've open a fresh batch of bugs. Please make sure you only remove 
the subpackages after a week has passed.

Thanks,