15 thousands python3-* packages for Fedora

[Miroslav Suchý]

Hi,
I just finished packaging of 15 634 python3-* packages for Fedora.
  https://copr.fedorainfracloud.org/coprs/g/copr/PyPI3/
  You can click on Builds and Monitoring tab. But be aware that those
pages are HUGE (40MB) and they are loading and rendering several
minutes. Tab "Packages" is timeouting and this will be fixed in next
release.

I am now building python2-* packages too:
  https://copr.fedorainfracloud.org/coprs/g/copr/PyPI2/

This is only for rawhide and for statistic purposes, but there will be
more. Read on.

Once the build of python2-* packages will finish, I gather the data and
build those packages once again in other project. Packages which
succeeded in both PyPI2 and PyPI3 projects will be build with both
python2-* and python3-* subpackages. Everything else will be build with
only one subpackage.

Why I am doing this?
Because I can. :) While I can imagine several obvious scenarios how this
can help Fedora users, I anticipate there are some use cases which I can
not imagine now, and which will surprise me for sure.

Why I'm writing this?
This projects is not yet finished. But this is first result, which can
be used/tested/remixed. Once this rebuilds for all Fedoras and Epels
will be finished I will announce it to Fedora users. But right now I'm
focused on you - developers.
These packages were automatically generated by pyp2rpm. They may work
(or not). So you can create some (automated) tests which can test
functionality of all those packages. You can look why your favorite
module failed (there is still 50k packages which are failing to build
now) and suggest improvements to pyp2rpm:
  https://github.com/fedora-python/pyp2rpm
Or come with something else which can raise the quality of those
packages. The future is open and waiting for your ideas.

If you have any question or idea (anything missing in our API of CLI
tools?), do not hesitate to contact me.

Mirek

[Vít Ondruch]

Do I understand correctly that you build just the latest version of each
package?


Vít


Dne 19.5.2016 v 08:51 Miroslav Suchy napsal(a):
>  Hi,
>  I just finished packaging of 15 634 python3-* packages for Fedora.
>    https://copr.fedorainfracloud.org/coprs/g/copr/PyPI3/
>    You can click on Builds and Monitoring tab. But be aware that those
>  pages are HUGE (40MB) and they are loading and rendering several
>  minutes. Tab "Packages" is timeouting and this will be fixed in next
>  release.
>
>  I am now building python2-* packages too:
>    https://copr.fedorainfracloud.org/coprs/g/copr/PyPI2/
>
>  This is only for rawhide and for statistic purposes, but there will be
>  more. Read on.
>
>  Once the build of python2-* packages will finish, I gather the data and
>  build those packages once again in other project. Packages which
>  succeeded in both PyPI2 and PyPI3 projects will be build with both
>  python2-* and python3-* subpackages. Everything else will be build with
>  only one subpackage.
>
>  Why I am doing this?
>  Because I can. :) While I can imagine several obvious scenarios how this
>  can help Fedora users, I anticipate there are some use cases which I can
>  not imagine now, and which will surprise me for sure.
>
>  Why I'm writing this?
>  This projects is not yet finished. But this is first result, which can
>  be used/tested/remixed. Once this rebuilds for all Fedoras and Epels
>  will be finished I will announce it to Fedora users. But right now I'm
>  focused on you - developers.
>  These packages were automatically generated by pyp2rpm. They may work
>  (or not). So you can create some (automated) tests which can test
>  functionality of all those packages. You can look why your favorite
>  module failed (there is still 50k packages which are failing to build
>  now) and suggest improvements to pyp2rpm:
>    https://github.com/fedora-python/pyp2rpm
>  Or come with something else which can raise the quality of those
>  packages. The future is open and waiting for your ideas.
>
>  If you have any question or idea (anything missing in our API of CLI
>  tools?), do not hesitate to contact me.
>
>  Mirek
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  http://lists.fedoraproject.org/admin/lists/[e-mail]

[Miroslav Suchý]

Dne 19.5.2016 v 10:35 Vít Ondruch napsal(a):
>  Do I understand correctly that you build just the latest version of
> each
>  package? 
Yes.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

[Avram Lubkin]

I'd suggest running pylint against the packages and skipping anything below
a certain threshold. There are currently over 80,000 packages on PyPi and
the vast majority are poorly written.

I think a more worthwhile effort would be to use the PyPi package rankings (
http://pypi-ranking.info/alltime) and identify any packages in say the top
500 that don't currently have Fedora packages or aren't included in the
standard library.


On Thu, May 19, 2016 at 2:51 AM, Miroslav Suchy <msuchy(a)redhat.com&gt; wrote:

>  Hi,
>  I just finished packaging of 15 634 python3-* packages for Fedora.
>    https://copr.fedorainfracloud.org/coprs/g/copr/PyPI3/
>    You can click on Builds and Monitoring tab. But be aware that those
>  pages are HUGE (40MB) and they are loading and rendering several
>  minutes. Tab "Packages" is timeouting and this will be fixed in next
>  release.
>
>  I am now building python2-* packages too:
>    https://copr.fedorainfracloud.org/coprs/g/copr/PyPI2/
>
>  This is only for rawhide and for statistic purposes, but there will be
>  more. Read on.
>
>  Once the build of python2-* packages will finish, I gather the data and
>  build those packages once again in other project. Packages which
>  succeeded in both PyPI2 and PyPI3 projects will be build with both
>  python2-* and python3-* subpackages. Everything else will be build with
>  only one subpackage.
>
>  Why I am doing this?
>  Because I can. :) While I can imagine several obvious scenarios how this
>  can help Fedora users, I anticipate there are some use cases which I can
>  not imagine now, and which will surprise me for sure.
>
>  Why I'm writing this?
>  This projects is not yet finished. But this is first result, which can
>  be used/tested/remixed. Once this rebuilds for all Fedoras and Epels
>  will be finished I will announce it to Fedora users. But right now I'm
>  focused on you - developers.
>  These packages were automatically generated by pyp2rpm. They may work
>  (or not). So you can create some (automated) tests which can test
>  functionality of all those packages. You can look why your favorite
>  module failed (there is still 50k packages which are failing to build
>  now) and suggest improvements to pyp2rpm:
>    https://github.com/fedora-python/pyp2rpm
>  Or come with something else which can raise the quality of those
>  packages. The future is open and waiting for your ideas.
>
>  If you have any question or idea (anything missing in our API of CLI
>  tools?), do not hesitate to contact me.
>
>  Mirek
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  http://lists.fedoraproject.org/admin/lists/[e-mail]
>

[Stephen John Smoogen]

On 19 May 2016 at 08:47, Avram Lubkin <aviso(a)rockhopper.net&gt; wrote:
>  I'd suggest running pylint against the packages and skipping
> anything below
>  a certain threshold. There are currently over 80,000 packages on PyPi and
>  the vast majority are poorly written.
>  
That isn't why he is doing this. I don't think he is looking to get
15,000 packages added to Fedora. He is just making sure that if you
need that poorly written python software... it is still in an rpm
somewhere. The bigger thing he is working on is the tool to do this
and probably eventually for other mega-repos.. the outcome is a side
effect which may be useful to people or not.. yay if it is.. no skin
off his nose if it isn't.

>  I think a more worthwhile effort would be to use the PyPi package
> rankings
>  (http://pypi-ranking.info/alltime) and identify any packages in say the top
>  500 that don't currently have Fedora packages or aren't included in the
>  standard library.
>
>
>  On Thu, May 19, 2016 at 2:51 AM, Miroslav Suchy <msuchy(a)redhat.com&gt; wrote:
> >
> > Hi,
> > I just finished packaging of 15 634 python3-* packages for Fedora.
> >   https://copr.fedorainfracloud.org/coprs/g/copr/PyPI3/
> >   You can click on Builds and Monitoring tab. But be aware that those
> > pages are HUGE (40MB) and they are loading and rendering several
> > minutes. Tab "Packages" is timeouting and this will be fixed in next
> > release.
> >
> > I am now building python2-* packages too:
> >   https://copr.fedorainfracloud.org/coprs/g/copr/PyPI2/
> >
> > This is only for rawhide and for statistic purposes, but there will be
> > more. Read on.
> >
> > Once the build of python2-* packages will finish, I gather the data and
> > build those packages once again in other project. Packages which
> > succeeded in both PyPI2 and PyPI3 projects will be build with both
> > python2-* and python3-* subpackages. Everything else will be build with
> > only one subpackage.
> >
> > Why I am doing this?
> > Because I can. :) While I can imagine several obvious scenarios how this
> > can help Fedora users, I anticipate there are some use cases which I can
> > not imagine now, and which will surprise me for sure.
> >
> > Why I'm writing this?
> > This projects is not yet finished. But this is first result, which can
> > be used/tested/remixed. Once this rebuilds for all Fedoras and Epels
> > will be finished I will announce it to Fedora users. But right now I'm
> > focused on you - developers.
> > These packages were automatically generated by pyp2rpm. They may work
> > (or not). So you can create some (automated) tests which can test
> > functionality of all those packages. You can look why your favorite
> > module failed (there is still 50k packages which are failing to build
> > now) and suggest improvements to pyp2rpm:
> >   https://github.com/fedora-python/pyp2rpm
> > Or come with something else which can raise the quality of those
> > packages. The future is open and waiting for your ideas.
> >
> > If you have any question or idea (anything missing in our API of CLI
> > tools?), do not hesitate to contact me.
> >
> > Mirek
> > --
> > devel mailing list
> > devel(a)lists.fedoraproject.org
> > http://lists.fedoraproject.org/admin/lists/[e-mail]
>
>
>
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  http://lists.fedoraproject.org/admin/lists/[e-mail]
>  


-- 
Stephen J Smoogen.

[Miroslav Suchý]

Dne 19.5.2016 v 14:59 Stephen John Smoogen napsal(a):
>  That isn't why he is doing this. I don't think he is looking
> to get
>  15,000 packages added to Fedora. He is just making sure that if you
>  need that poorly written python software... it is still in an rpm
>  somewhere.  
Yes. This is not intended for inclusion in Fedora. Remember the Fedora Rings? This is the
very outer ring.
Albeit it may help maintainers. For example there is several python3 packages in this
project, which are already
included in Fedora, but only in python2 version and the python3 submodule is missing. So
the maintainers can copy the
spec code and use it.

And talking about the crap modules. Yes, there are plenty of them. Some of them even do
not have tar.gz included so they
will never succeed. But some of them succeeded and ... yeah they are crap ... but the
world outside ... the world where
people try to get earn money is sometime really weird. And sometimes you *need* the crap.
For whatever reason. I'm not
asking. I'm not judging. If you need it, it is there.
I expect that there will pop up some other tools, which will help you to make better
decision and which will tell you
that there is plenty of unfixed CVE on that module or something similar. So you can you
can choose some alternative. Or
if you insist - then here it is - crap module packaged as very shiny rpm package.

> > > I think a more worthwhile effort would be to use the PyPi
> package rankings
> > > (http://pypi-ranking.info/alltime) and identify any packages in say the top
> > > 500 that don't currently have Fedora packages or aren't included in the
> > > standard library. 
This is very interesting idea. I'm not sure about the inclusion in Fedora, but I will
discuss it with Michal Cyprian
(pyp2rpm maintainer) and we will try to improve pyp2rpm so it will for those high ranked
modules.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys