Changes in packages workflow vs. modular Fedora

[Miro Hrončok]

Hi,

I was thinking about this for a while and I got the impression that this 
is something I don't know the answer for. The question is a bit harder 
to formulate simply, so let put it in examples:


a) A need packaging guideline is about to be discussed. A member of FPC 
wants to know how many packages would be affected so they run a quick 
grep query over all the rawhide specfiles at [0].

b) A CVE is found in a web framework, so bugz are filled for the 
"framewrok" package to be fixed in Fedora 27, because newer Fedoras have 
newer version of the framework where the CVE is not applicable.

c) A new version of interpreted language lands in rawhide. All packages 
depending on the interpeter need mass rebuild in a side tag.

d) A packager decides to retire a library and they check nothing in 
Fedora requires it.

I wonder how are such situations meant to be handled with modules?
Do we build modules for rawhide? If so, should Fedora changes (such as, 
but not limited to, "the Changes") somehow handle all the modules, or is 
it the modular maintainer responsibility to make sure their module works 
on the next Fedora version?


For the above examples, here are some more specific situations with more 
specific questions:


a) I was planning to propose a more strict "No more automagic Python 
bytecompilation" [1] change for Fedora 30 where we would query all 
packages that depend on the old behavior and mass add "%global 
_python_bytecompile_extra 1" to all of them, so we could switch the 
default to be 0. Normally, we would do it in Rawhide only. But how do I 
handle modules? How do I find out what modular packages rely on the old 
behavior?

b) A CVE was filled for Django [2]. How does the security team 
responsible for tracking CVEs figure out we have Django 1.6 in modular 
Fedora? How is a bugzilla filled against a modular package anyway?

c) When we recently mass rebuilt Python 3 packages for Python 3.7, 
should we go and seek for modules that use Python 3 (how do I do that?) 
and rebuild the packages in the modules (or even the modules?)?

d) (this example is not real (yet)) We decide to retire (remove) 
python2-sphinx because upstream Sphinx no longer supports Python 2. We 
make sure that nothing in Fedora requires or buildrequires it, as all 
Fedora's Python packages use python3-sphinx or no Sphinx at all. However 
there is a Django 1.6 module where python-django uses python2-sphinx to 
build the docs, while python2-sphinx is not part of the module itself. 
How do we find out about this and is it our reprehensibility to keep it 
in rawhide or add it to the module?


Sorry if those things are obvious, yet I don't know the answers. There 
might be even more examples where traditionally we would only think 
rawhide, but now with modules the problem seems more complex.


[0] http://pkgs.fedoraproject.org/repo/rpm-specs-latest.tar.xz
[1] 
https://fedoraproject.org/wiki/Changes/No_more_automagic_Python_bytecompi...
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1609031
-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 3.8.2018 16:43, Miro Hrončok wrote:
>  Hi,
>  
>  I was thinking about this for a while and I got the impression that this 
>  is something I don't know the answer for. The question is a bit harder 
>  to formulate simply, so let put it in examples:
>  
>  
>  a) A need packaging guideline is about to be discussed. A member of FPC 
>  wants to know how many packages would be affected so they run a quick 
>  grep query over all the rawhide specfiles at [0].
>  
>  b) A CVE is found in a web framework, so bugz are filled for the 
>  "framewrok" package to be fixed in Fedora 27, because newer Fedoras have 
>  newer version of the framework where the CVE is not applicable.
>  
>  c) A new version of interpreted language lands in rawhide. All packages 
>  depending on the interpeter need mass rebuild in a side tag.
>  
>  d) A packager decides to retire a library and they check nothing in 
>  Fedora requires it.
>  
>  I wonder how are such situations meant to be handled with modules?
>  Do we build modules for rawhide? If so, should Fedora changes (such as, 
>  but not limited to, "the Changes") somehow handle all the modules, or is 
>  it the modular maintainer responsibility to make sure their module works 
>  on the next Fedora version?
>  
>  
>  For the above examples, here are some more specific situations with more 
>  specific questions:
>  
>  
>  a) I was planning to propose a more strict "No more automagic Python 
>  bytecompilation" [1] change for Fedora 30 where we would query all 
>  packages that depend on the old behavior and mass add "%global 
>  _python_bytecompile_extra 1" to all of them, so we could switch the 
>  default to be 0. Normally, we would do it in Rawhide only. But how do I 
>  handle modules? How do I find out what modular packages rely on the old 
>  behavior?
>  
>  b) A CVE was filled for Django [2]. How does the security team 
>  responsible for tracking CVEs figure out we have Django 1.6 in modular 
>  Fedora? How is a bugzilla filled against a modular package anyway?
>  
>  c) When we recently mass rebuilt Python 3 packages for Python 3.7, 
>  should we go and seek for modules that use Python 3 (how do I do that?) 
>  and rebuild the packages in the modules (or even the modules?)?
>  
>  d) (this example is not real (yet)) We decide to retire (remove) 
>  python2-sphinx because upstream Sphinx no longer supports Python 2. We 
>  make sure that nothing in Fedora requires or buildrequires it, as all 
>  Fedora's Python packages use python3-sphinx or no Sphinx at all. However 
>  there is a Django 1.6 module where python-django uses python2-sphinx to 
>  build the docs, while python2-sphinx is not part of the module itself. 
>  How do we find out about this and is it our reprehensibility to keep it 
>  in rawhide or add it to the module?
>  
>  
>  Sorry if those things are obvious, yet I don't know the answers. There 
>  might be even more examples where traditionally we would only think 
>  rawhide, but now with modules the problem seems more complex.
>  
>  
>  [0] http://pkgs.fedoraproject.org/repo/rpm-specs-latest.tar.xz
>  [1] 
>  https://fedoraproject.org/wiki/Changes/No_more_automagic_Python_bytecompi... 
>  
>  [2] https://bugzilla.redhat.com/show_bug.cgi?id=1609031 
This has received no reply in ~2 weeks. Am I the only one who ask those 
questions?

Sorry for the bump (if nobody replies anyway, I'll stop bumping (and 
probably also caring about the answers)).

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Vít Ondruch]

You triggered my interest, so I tried to get some information how many
modules there are in Fedora, but

1) mbs.fedoraproject.org spits on me just some JSON instead of some user
friendly web UI,

2) pagure cannot show content of modules namespace nor can reliably
search for modules,

3) opening the MBS change [1], says that the documentation is at [2] and
this immediately redirects me to [3], where is just some marketing stuff
without any useful information. Oh, there is "Docs X" link which points
to some documentation URL which has fragments such as
"subprojects/fesco", it can't be useful at all, right?

Frankly, I cannot imagine that somebody would try it even by accident.


V.


[1] https://fedoraproject.org/wiki/Changes/ModuleBuildService

[2] https://fedoraproject.org/wiki/Modularity/Infra

[3] https://docs.pagure.org/modularity/



Dne 3.8.2018 v 16:43 Miro Hrončok napsal(a):
>  Hi,
>
>  I was thinking about this for a while and I got the impression that
>  this is something I don't know the answer for. The question is a bit
>  harder to formulate simply, so let put it in examples:
>
>
>  a) A need packaging guideline is about to be discussed. A member of
>  FPC wants to know how many packages would be affected so they run a
>  quick grep query over all the rawhide specfiles at [0].
>
>  b) A CVE is found in a web framework, so bugz are filled for the
>  "framewrok" package to be fixed in Fedora 27, because newer Fedoras
>  have newer version of the framework where the CVE is not applicable.
>
>  c) A new version of interpreted language lands in rawhide. All
>  packages depending on the interpeter need mass rebuild in a side tag.
>
>  d) A packager decides to retire a library and they check nothing in
>  Fedora requires it.
>
>  I wonder how are such situations meant to be handled with modules?
>  Do we build modules for rawhide? If so, should Fedora changes (such
>  as, but not limited to, "the Changes") somehow handle all the modules,
>  or is it the modular maintainer responsibility to make sure their
>  module works on the next Fedora version?
>
>
>  For the above examples, here are some more specific situations with
>  more specific questions:
>
>
>  a) I was planning to propose a more strict "No more automagic Python
>  bytecompilation" [1] change for Fedora 30 where we would query all
>  packages that depend on the old behavior and mass add "%global
>  _python_bytecompile_extra 1" to all of them, so we could switch the
>  default to be 0. Normally, we would do it in Rawhide only. But how do
>  I handle modules? How do I find out what modular packages rely on the
>  old behavior?
>
>  b) A CVE was filled for Django [2]. How does the security team
>  responsible for tracking CVEs figure out we have Django 1.6 in modular
>  Fedora? How is a bugzilla filled against a modular package anyway?
>
>  c) When we recently mass rebuilt Python 3 packages for Python 3.7,
>  should we go and seek for modules that use Python 3 (how do I do
>  that?) and rebuild the packages in the modules (or even the modules?)?
>
>  d) (this example is not real (yet)) We decide to retire (remove)
>  python2-sphinx because upstream Sphinx no longer supports Python 2. We
>  make sure that nothing in Fedora requires or buildrequires it, as all
>  Fedora's Python packages use python3-sphinx or no Sphinx at all.
>  However there is a Django 1.6 module where python-django uses
>  python2-sphinx to build the docs, while python2-sphinx is not part of
>  the module itself. How do we find out about this and is it our
>  reprehensibility to keep it in rawhide or add it to the module?
>
>
>  Sorry if those things are obvious, yet I don't know the answers. There
>  might be even more examples where traditionally we would only think
>  rawhide, but now with modules the problem seems more complex.
>
>
>  [0] http://pkgs.fedoraproject.org/repo/rpm-specs-latest.tar.xz
>  [1]
>  https://fedoraproject.org/wiki/Changes/No_more_automagic_Python_bytecompi...
>  [2] https://bugzilla.redhat.com/show_bug.cgi?id=1609031

[Ken Dreyer]

On Thu, Aug 16, 2018 at 9:28 AM, Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  This has received no reply in ~2 weeks. Am I the only one who ask those
>  questions? 
I'm interested as well, I just have no idea about the answers.

- Ken

[Kevin Fenzi]

On 08/17/2018 03:21 AM, Vít Ondruch wrote:
>  You triggered my interest, so I tried to get some information how
> many
>  modules there are in Fedora, but
>  
>  1) mbs.fedoraproject.org spits on me just some JSON instead of some user
>  friendly web UI,
>  
>  2) pagure cannot show content of modules namespace nor can reliably
>  search for modules, 
https://src.fedoraproject.org/projects/modules/*

shows 126 modules

kevin

[Mikolaj Izdebski]

On 08/17/2018 12:21 PM, Vít Ondruch wrote:
>  You triggered my interest, so I tried to get some information how
> many
>  modules there are in Fedora, but
>  
>  1) mbs.fedoraproject.org spits on me just some JSON instead of some user
>  friendly web UI, 
MBS UI is available at:
https://release-engineering.github.io/mbs-ui/
(not hosted by Fedora Infrastructure)

-- 
Mikolaj Izdebski
Senior Software Engineer, Red Hat
IRC: mizdebsk

[Vít Ondruch]

Dne 17.8.2018 v 21:48 Kevin Fenzi napsal(a):
>  On 08/17/2018 03:21 AM, Vít Ondruch wrote:
> > You triggered my interest, so I tried to get some information how many
> > modules there are in Fedora, but
> >
> > 1) mbs.fedoraproject.org spits on me just some JSON instead of some user
> > friendly web UI,
> >
> > 2) pagure cannot show content of modules namespace nor can reliably
> > search for modules,
>  https://src.fedoraproject.org/projects/modules/* 
Ah, thx for the tip. I wish this was somehow doable from UI.

V.

[Petr Šabata]

On Fri, Aug 03, 2018 at 04:43:15PM +0200, Miro Hrončok wrote:
>  Hi,
>  
>  I was thinking about this for a while and I got the impression that this is
>  something I don't know the answer for. The question is a bit harder to
>  formulate simply, so let put it in examples: 
I see no one's really responded to this yet (ack!), so let me try...

>  a) A need packaging guideline is about to be discussed. A member of
> FPC
>  wants to know how many packages would be affected so they run a quick grep
>  query over all the rawhide specfiles at [0]. 
Obviously this wouldn't be enough.  We would need to find all
modules shipped with Rawhide, then all the RPM stream branches
they build from, and grep those as well.  Perhaps we could
do something to generate archives that include all of this
automatically.

Currently supported modules are tagged in the f*-modular* tags.

>  b) A CVE is found in a web framework, so bugz are filled for the
> "framewrok"
>  package to be fixed in Fedora 27, because newer Fedoras have newer version
>  of the framework where the CVE is not applicable. 
Also valid.  Rather than just relying on the non-modular package
upgrade path, all currently supported modules and their RPMs
would need to be checked.

>  c) A new version of interpreted language lands in rawhide. All
> packages
>  depending on the interpeter need mass rebuild in a side tag. 
This would be a new stream of the interpreter module.
Supposedly those packages would be also modularized?  In that
case everything that builds against the interpreter module with
stream expansion would need to be "rebuilt".

This also affects platform branching and needs to be solved
before f30.  It should be easy to automate this.  And there's
no point in limiting that to platform branching.

>  d) A packager decides to retire a library and they check nothing in
> Fedora
>  requires it. 
Similar to a) and b).  They need to check the currently supported
modules as well.

>  I wonder how are such situations meant to be handled with modules?
>  Do we build modules for rawhide? If so, should Fedora changes (such as, but
>  not limited to, "the Changes") somehow handle all the modules, or is it the
>  modular maintainer responsibility to make sure their module works on the
>  next Fedora version? 
We're aware there are many gaps here.  I consider identifying &
resolving them a priority for the Modularity WG at the moment,
so thank you for contributing.

>  For the above examples, here are some more specific situations with
> more
>  specific questions:
>  
>  
>  a) I was planning to propose a more strict "No more automagic Python
>  bytecompilation" [1] change for Fedora 30 where we would query all packages
>  that depend on the old behavior and mass add "%global
>  _python_bytecompile_extra 1" to all of them, so we could switch the default
>  to be 0. Normally, we would do it in Rawhide only. But how do I handle
>  modules? How do I find out what modular packages rely on the old behavior? 
Modules typically build the same content in all contexts
(buildroots).  If that's a problem for this particular change
of yours, I think the proper way would be adding %{fedora}
conditionals around that macro definition.

>  b) A CVE was filled for Django [2]. How does the security team
> responsible
>  for tracking CVEs figure out we have Django 1.6 in modular Fedora? How is a
>  bugzilla filled against a modular package anyway? 
They should file a report against Fedora Modules rather than
Fedora, although in practice I suspect people will report bugs
as usual.  Then it's about the non-modular package maintainer
re-assigning the bug properly, if necessary.

>  c) When we recently mass rebuilt Python 3 packages for Python 3.7,
> should we
>  go and seek for modules that use Python 3 (how do I do that?) and rebuild
>  the packages in the modules (or even the modules?)? 
Python is part of the "platform" module, so you really need
to check all the RPM stream branches used by the currently
supported modules.

You do not rebuild these packages, you rebuild the modules.
If there's no change to the SPEC files, you'll have to instruct
MBS to do a full rebuild (fedpkg module-build --optional
rebuild_strategy=all).  We also plan to add more options to
limit these rebuilds to specific platforms so you wouldn't be
rebuilding for all three or more releases if not required.

>  d) (this example is not real (yet)) We decide to retire (remove)
>  python2-sphinx because upstream Sphinx no longer supports Python 2. We make
>  sure that nothing in Fedora requires or buildrequires it, as all Fedora's
>  Python packages use python3-sphinx or no Sphinx at all. However there is a
>  Django 1.6 module where python-django uses python2-sphinx to build the docs,
>  while python2-sphinx is not part of the module itself. How do we find out
>  about this and is it our reprehensibility to keep it in rawhide or add it to
>  the module? 
I wouldn't say it's your responsibility to resolve the issue
but it is your responsibility to file a bug for the module.

>  Sorry if those things are obvious, yet I don't know the answers.
> There might
>  be even more examples where traditionally we would only think rawhide, but
>  now with modules the problem seems more complex. 
Keep them coming.

I hope this helps at least a little bit, even this late.

P

>  
>  
>  [0] http://pkgs.fedoraproject.org/repo/rpm-specs-latest.tar.xz
>  [1] https://fedoraproject.org/wiki/Changes/No_more_automagic_Python_bytecompi...
>  [2] https://bugzilla.redhat.com/show_bug.cgi?id=1609031
>  -- 
>  Miro Hrončok
>  --
>  Phone: [phone]
>  IRC: mhroncok
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail].o...

[Miro Hrončok]

On 13.9.2018 22:44, Petr Šabata wrote:
>  On Fri, Aug 03, 2018 at 04:43:15PM +0200, Miro Hrončok wrote:
> > Hi,
> >
> > I was thinking about this for a while and I got the impression that this is
> > something I don't know the answer for. The question is a bit harder to
> > formulate simply, so let put it in examples:
>  
>  I see no one's really responded to this yet (ack!), so let me try... 
Thank You! I was starting to feel like nobody cares.

> > a) A need packaging guideline is about to be discussed. A member
> of FPC
> > wants to know how many packages would be affected so they run a quick grep
> > query over all the rawhide specfiles at [0].
>  
>  Obviously this wouldn't be enough.  We would need to find all
>  modules shipped with Rawhide, then all the RPM stream branches
>  they build from, and grep those as well.  Perhaps we could
>  do something to generate archives that include all of this
>  automatically. 
Please do, otherwise this would be tedious. There are archives with 
rawhide specs.

> > b) A CVE is found in a web framework, so bugz are filled for the
> "framewrok"
> > package to be fixed in Fedora 27, because newer Fedoras have newer version
> > of the framework where the CVE is not applicable.
>  
>  Also valid.  Rather than just relying on the non-modular package
>  upgrade path, all currently supported modules and their RPMs
>  would need to be checked. 
Is the Red Hat security response team aware of that? Do they file bugs 
for modules?

> > c) A new version of interpreted language lands in rawhide. All
> packages
> > depending on the interpeter need mass rebuild in a side tag.
>  
>  This would be a new stream of the interpreter module. 
No interpreter module here. An interpreter that lives in non-modular 
Fedora, but various modules possibly use it. I have no idea what modules 
use it (if any) and I have no idea how to easily find out.

I know there is at least one module that uses non-modular Python 2. At 
least we won't rebase that Python to a newer version.

> > d) A packager decides to retire a library and they check nothing
> in Fedora
> > requires it.
>  
>  Similar to a) and b).  They need to check the currently supported
>  modules as well. 
How? We are mass removing python2 packages from rawhide that nothing in 
rawhide depend on. Our automation has no clue about modules and if there 
are modules that use python2-... packages from nonmodular Fedora, we 
need to know.

> > I wonder how are such situations meant to be handled with
> modules?
> > Do we build modules for rawhide? If so, should Fedora changes (such as, but
> > not limited to, "the Changes") somehow handle all the modules, or is it
> the
> > modular maintainer responsibility to make sure their module works on the
> > next Fedora version?
>  
>  We're aware there are many gaps here.  I consider identifying &
>  resolving them a priority for the Modularity WG at the moment,
>  so thank you for contributing. 
Thanks for looking into this.

> > a) I was planning to propose a more strict "No more
> automagic Python
> > bytecompilation" [1] change for Fedora 30 where we would query all packages
> > that depend on the old behavior and mass add "%global
> > _python_bytecompile_extra 1" to all of them, so we could switch the default
> > to be 0. Normally, we would do it in Rawhide only. But how do I handle
> > modules? How do I find out what modular packages rely on the old behavior?
>  
>  Modules typically build the same content in all contexts
>  (buildroots).  If that's a problem for this particular change
>  of yours, I think the proper way would be adding %{fedora}
>  conditionals around that macro definition. 
No, I don't want to only add this thing if %fedora. Quite the opposite, 
I need to gather a list of packages that rely on the old behavior, so I 
can hotfix/workaround them with that line. See 
https://bugzilla.redhat.com/show_bug.cgi?id=1626685#c2 for what I did in 
the non-modular world to better understand what I mean.

> > b) A CVE was filled for Django [2]. How does the security team
> responsible
> > for tracking CVEs figure out we have Django 1.6 in modular Fedora? How is a
> > bugzilla filled against a modular package anyway?
>  
>  They should file a report against Fedora Modules rather than
>  Fedora, although in practice I suspect people will report bugs
>  as usual.  Then it's about the non-modular package maintainer
>  re-assigning the bug properly, if necessary. 
I've added a note to that bug about the modular build. Got no response IIRC.

> > c) When we recently mass rebuilt Python 3 packages for Python
> 3.7, should we
> > go and seek for modules that use Python 3 (how do I do that?) and rebuild
> > the packages in the modules (or even the modules?)?
>  
>  Python is part of the "platform" module, so you really need
>  to check all the RPM stream branches used by the currently
>  supported modules. 
The change is done now. We've checked no modules. What happens now?

How do I see what modules depend on (non modular) Python 3?

>  You do not rebuild these packages, you rebuild the modules.
>  If there's no change to the SPEC files, you'll have to instruct
>  MBS to do a full rebuild (fedpkg module-build --optional
>  rebuild_strategy=all).  We also plan to add more options to
>  limit these rebuilds to specific platforms so you wouldn't be
>  rebuilding for all three or more releases if not required. 
Is there a mass rebuild of modules like with packages?

> > d) (this example is not real (yet)) We decide to retire (remove)
> > python2-sphinx because upstream Sphinx no longer supports Python 2. We make
> > sure that nothing in Fedora requires or buildrequires it, as all Fedora's
> > Python packages use python3-sphinx or no Sphinx at all. However there is a
> > Django 1.6 module where python-django uses python2-sphinx to build the docs,
> > while python2-sphinx is not part of the module itself. How do we find out
> > about this and is it our reprehensibility to keep it in rawhide or add it to
> > the module?
>  
>  I wouldn't say it's your responsibility to resolve the issue
>  but it is your responsibility to file a bug for the module. 
How would I know about that in the first place? How do I query it?

> > Sorry if those things are obvious, yet I don't know the
> answers. There might
> > be even more examples where traditionally we would only think rawhide, but
> > now with modules the problem seems more complex.
>  
>  Keep them coming.
>  
>  I hope this helps at least a little bit, even this late. 
If you would ignore the questions entirely, I guess it would mean less 
work for me :D

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Petr Šabata]

On Thu, Sep 13, 2018 at 11:07:50PM +0200, Miro Hrončok wrote:
>  On 13.9.2018 22:44, Petr Šabata wrote:
>  > On Fri, Aug 03, 2018 at 04:43:15PM +0200, Miro Hrončok wrote:
>  > > Hi,
>  > > 
>  > > I was thinking about this for a while and I got the impression that this is
>  > > something I don't know the answer for. The question is a bit harder to
>  > > formulate simply, so let put it in examples:
>  > 
>  > I see no one's really responded to this yet (ack!), so let me try...
>  
>  Thank You! I was starting to feel like nobody cares. 
I really hope that's not the case :)

>  > > a) A need packaging guideline is about to be discussed. A
> member of FPC
>  > > wants to know how many packages would be affected so they run a quick grep
>  > > query over all the rawhide specfiles at [0].
>  > 
>  > Obviously this wouldn't be enough.  We would need to find all
>  > modules shipped with Rawhide, then all the RPM stream branches
>  > they build from, and grep those as well.  Perhaps we could
>  > do something to generate archives that include all of this
>  > automatically.
>  
>  Please do, otherwise this would be tedious. There are archives with rawhide
>  specs. 
Noted.  I don't have any ETA at this point, though.

>  > > b) A CVE is found in a web framework, so bugz are filled
> for the "framewrok"
>  > > package to be fixed in Fedora 27, because newer Fedoras have newer version
>  > > of the framework where the CVE is not applicable.
>  > 
>  > Also valid.  Rather than just relying on the non-modular package
>  > upgrade path, all currently supported modules and their RPMs
>  > would need to be checked.
>  
>  Is the Red Hat security response team aware of that? Do they file bugs for
>  modules? 
Not yet.  I think this is tied to the previous point.  They will
also need that archive or something similar.  Also noted.

>  > > c) A new version of interpreted language lands in rawhide.
> All packages
>  > > depending on the interpeter need mass rebuild in a side tag.
>  > 
>  > This would be a new stream of the interpreter module.
>  
>  No interpreter module here. An interpreter that lives in non-modular Fedora,
>  but various modules possibly use it. I have no idea what modules use it (if
>  any) and I have no idea how to easily find out.
>  
>  I know there is at least one module that uses non-modular Python 2. At least
>  we won't rebase that Python to a newer version. 
Again the archive ;) We should be including stream-branched
SPEC files as well as currently included modulemds.

>  > > d) A packager decides to retire a library and they check
> nothing in Fedora
>  > > requires it.
>  > 
>  > Similar to a) and b).  They need to check the currently supported
>  > modules as well.
>  
>  How? We are mass removing python2 packages from rawhide that nothing in
>  rawhide depend on. Our automation has no clue about modules and if there are
>  modules that use python2-... packages from nonmodular Fedora, we need to
>  know. 
Same as above.

>  > > I wonder how are such situations meant to be handled with
> modules?
>  > > Do we build modules for rawhide? If so, should Fedora changes (such as, but
>  > > not limited to, "the Changes") somehow handle all the modules, or is
> it the
>  > > modular maintainer responsibility to make sure their module works on the
>  > > next Fedora version?
>  > 
>  > We're aware there are many gaps here.  I consider identifying &
>  > resolving them a priority for the Modularity WG at the moment,
>  > so thank you for contributing.
>  
>  Thanks for looking into this. 
We (Modularity WG) are thinking about new way of organizing &
tracking all these things (currently it's mostly just a shared
document).  We'll announce it once ready.

>  > > a) I was planning to propose a more strict "No more
> automagic Python
>  > > bytecompilation" [1] change for Fedora 30 where we would query all
> packages
>  > > that depend on the old behavior and mass add "%global
>  > > _python_bytecompile_extra 1" to all of them, so we could switch the
> default
>  > > to be 0. Normally, we would do it in Rawhide only. But how do I handle
>  > > modules? How do I find out what modular packages rely on the old behavior?
>  > 
>  > Modules typically build the same content in all contexts
>  > (buildroots).  If that's a problem for this particular change
>  > of yours, I think the proper way would be adding %{fedora}
>  > conditionals around that macro definition.
>  
>  No, I don't want to only add this thing if %fedora. Quite the opposite, I
>  need to gather a list of packages that rely on the old behavior, so I can
>  hotfix/workaround them with that line. See
>  https://bugzilla.redhat.com/show_bug.cgi?id=1626685#c2 for what I did in the
>  non-modular world to better understand what I mean. 
If you're okay with implementing the change in all supported
releases, that just makes it easier.  My point was that if you
change is only applicable to Rawhide, for instance, you should
conditionalize your edits, as modules are typically built for
all of them.

>  > > b) A CVE was filled for Django [2]. How does the security
> team responsible
>  > > for tracking CVEs figure out we have Django 1.6 in modular Fedora? How is a
>  > > bugzilla filled against a modular package anyway?
>  > 
>  > They should file a report against Fedora Modules rather than
>  > Fedora, although in practice I suspect people will report bugs
>  > as usual.  Then it's about the non-modular package maintainer
>  > re-assigning the bug properly, if necessary.
>  
>  I've added a note to that bug about the modular build. Got no response IIRC.
>  
>  > > c) When we recently mass rebuilt Python 3 packages for Python 3.7, should we
>  > > go and seek for modules that use Python 3 (how do I do that?) and rebuild
>  > > the packages in the modules (or even the modules?)?
>  > 
>  > Python is part of the "platform" module, so you really need
>  > to check all the RPM stream branches used by the currently
>  > supported modules.
>  
>  The change is done now. We've checked no modules. What happens now?
>  
>  How do I see what modules depend on (non modular) Python 3? 
It's not easy to do at the moment.

I suggest you see what modules are tagged into
f{29|30}-modular[-updates[-candidate]].  Then, depending on what
you find out, either scan their binaries or see their modulemds
to see what packages they include and from what branches.

>  > You do not rebuild these packages, you rebuild the modules.
>  > If there's no change to the SPEC files, you'll have to instruct
>  > MBS to do a full rebuild (fedpkg module-build --optional
>  > rebuild_strategy=all).  We also plan to add more options to
>  > limit these rebuilds to specific platforms so you wouldn't be
>  > rebuilding for all three or more releases if not required.
>  
>  Is there a mass rebuild of modules like with packages? 
Not yet, unfortunately.  We did it manually for F30 but not
for F29.  The process needs to be updated; it's on our agenda.

>  > > d) (this example is not real (yet)) We decide to retire
> (remove)
>  > > python2-sphinx because upstream Sphinx no longer supports Python 2. We make
>  > > sure that nothing in Fedora requires or buildrequires it, as all Fedora's
>  > > Python packages use python3-sphinx or no Sphinx at all. However there is a
>  > > Django 1.6 module where python-django uses python2-sphinx to build the docs,
>  > > while python2-sphinx is not part of the module itself. How do we find out
>  > > about this and is it our reprehensibility to keep it in rawhide or add it to
>  > > the module?
>  > 
>  > I wouldn't say it's your responsibility to resolve the issue
>  > but it is your responsibility to file a bug for the module.
>  
>  How would I know about that in the first place? How do I query it? 
It will either be in the archive :) or you check the tags and
their binaries & modulemds.

>  > > Sorry if those things are obvious, yet I don't know the
> answers. There might
>  > > be even more examples where traditionally we would only think rawhide, but
>  > > now with modules the problem seems more complex.
>  > 
>  > Keep them coming.
>  > 
>  > I hope this helps at least a little bit, even this late.
>  
>  If you would ignore the questions entirely, I guess it would mean less work
>  for me :D 
Apologies ;)

P

>  
>  -- 
>  Miro Hrončok
>  --
>  Phone: [phone]
>  IRC: mhroncok
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Miro Hrončok]

On 17.9.2018 12:36, Petr Šabata wrote:
>  On Thu, Sep 13, 2018 at 11:07:50PM +0200, Miro Hrončok wrote:
> > On 13.9.2018 22:44, Petr Šabata wrote:
> >> On Fri, Aug 03, 2018 at 04:43:15PM +0200, Miro Hrončok wrote:
> >>> Hi,
> >>>
> >>> I was thinking about this for a while and I got the impression that this is
> >>> something I don't know the answer for. The question is a bit harder to
> >>> formulate simply, so let put it in examples:
> >>
> >> I see no one's really responded to this yet (ack!), so let me try...
> >
> > Thank You! I was starting to feel like nobody cares.
>  
>  I really hope that's not the case :)
>  
> >>> a) A need packaging guideline is about to be discussed. A member of FPC
> >>> wants to know how many packages would be affected so they run a quick grep
> >>> query over all the rawhide specfiles at [0].
> >>
> >> Obviously this wouldn't be enough.  We would need to find all
> >> modules shipped with Rawhide, then all the RPM stream branches
> >> they build from, and grep those as well.  Perhaps we could
> >> do something to generate archives that include all of this
> >> automatically.
> >
> > Please do, otherwise this would be tedious. There are archives with rawhide
> > specs.
>  
>  Noted.  I don't have any ETA at this point, though.
>  
> >>> b) A CVE is found in a web framework, so bugz are filled for the
> "framewrok"
> >>> package to be fixed in Fedora 27, because newer Fedoras have newer version
> >>> of the framework where the CVE is not applicable.
> >>
> >> Also valid.  Rather than just relying on the non-modular package
> >> upgrade path, all currently supported modules and their RPMs
> >> would need to be checked.
> >
> > Is the Red Hat security response team aware of that? Do they file bugs for
> > modules?
>  
>  Not yet.  I think this is tied to the previous point.  They will
>  also need that archive or something similar.  Also noted.
>  
> >>> c) A new version of interpreted language lands in rawhide. All packages
> >>> depending on the interpeter need mass rebuild in a side tag.
> >>
> >> This would be a new stream of the interpreter module.
> >
> > No interpreter module here. An interpreter that lives in non-modular Fedora,
> > but various modules possibly use it. I have no idea what modules use it (if
> > any) and I have no idea how to easily find out.
> >
> > I know there is at least one module that uses non-modular Python 2. At least
> > we won't rebase that Python to a newer version.
>  
>  Again the archive ;) We should be including stream-branched
>  SPEC files as well as currently included modulemds. 
Not really here. Parsing specs is hard. We actually need a proper 
repoquery setup.

This is what we have now:

https://fedoraproject.org/wiki/SIGs/Python/UpgradingPython#After_a_side_t...

> >>> d) A packager decides to retire a library and they check
> nothing in Fedora
> >>> requires it.
> >>
> >> Similar to a) and b).  They need to check the currently supported
> >> modules as well.
> >
> > How? We are mass removing python2 packages from rawhide that nothing in
> > rawhide depend on. Our automation has no clue about modules and if there are
> > modules that use python2-... packages from nonmodular Fedora, we need to
> > know.
>  
>  Same as above. 
Same as above. Repoquery, not archive.

>  
> >>> I wonder how are such situations meant to be handled with modules?
> >>> Do we build modules for rawhide? If so, should Fedora changes (such as, but
> >>> not limited to, "the Changes") somehow handle all the modules, or
> is it the
> >>> modular maintainer responsibility to make sure their module works on the
> >>> next Fedora version?
> >>
> >> We're aware there are many gaps here.  I consider identifying &
> >> resolving them a priority for the Modularity WG at the moment,
> >> so thank you for contributing.
> >
> > Thanks for looking into this.
>  
>  We (Modularity WG) are thinking about new way of organizing &
>  tracking all these things (currently it's mostly just a shared
>  document).  We'll announce it once ready.
>  
> >>> a) I was planning to propose a more strict "No more automagic Python
> >>> bytecompilation" [1] change for Fedora 30 where we would query all
> packages
> >>> that depend on the old behavior and mass add "%global
> >>> _python_bytecompile_extra 1" to all of them, so we could switch the
> default
> >>> to be 0. Normally, we would do it in Rawhide only. But how do I handle
> >>> modules? How do I find out what modular packages rely on the old behavior?
> >>
> >> Modules typically build the same content in all contexts
> >> (buildroots).  If that's a problem for this particular change
> >> of yours, I think the proper way would be adding %{fedora}
> >> conditionals around that macro definition.
> >
> > No, I don't want to only add this thing if %fedora. Quite the opposite, I
> > need to gather a list of packages that rely on the old behavior, so I can
> > hotfix/workaround them with that line. See
> > https://bugzilla.redhat.com/show_bug.cgi?id=1626685#c2 for what I did in the
> > non-modular world to better understand what I mean.
>  
>  If you're okay with implementing the change in all supported
>  releases, that just makes it easier.  My point was that if you
>  change is only applicable to Rawhide, for instance, you should
>  conditionalize your edits, as modules are typically built for
>  all of them.
>  
> >>> b) A CVE was filled for Django [2]. How does the security team responsible
> >>> for tracking CVEs figure out we have Django 1.6 in modular Fedora? How is a
> >>> bugzilla filled against a modular package anyway?
> >>
> >> They should file a report against Fedora Modules rather than
> >> Fedora, although in practice I suspect people will report bugs
> >> as usual.  Then it's about the non-modular package maintainer
> >> re-assigning the bug properly, if necessary.
> >
> > I've added a note to that bug about the modular build. Got no response IIRC.
> >
> >>> c) When we recently mass rebuilt Python 3 packages for Python 3.7, should we
> >>> go and seek for modules that use Python 3 (how do I do that?) and rebuild
> >>> the packages in the modules (or even the modules?)?
> >>
> >> Python is part of the "platform" module, so you really need
> >> to check all the RPM stream branches used by the currently
> >> supported modules.
> >
> > The change is done now. We've checked no modules. What happens now?
> >
> > How do I see what modules depend on (non modular) Python 3?
>  
>  It's not easy to do at the moment.
>  
>  I suggest you see what modules are tagged into
>  f{29|30}-modular[-updates[-candidate]].  Then, depending on what
>  you find out, either scan their binaries or see their modulemds
>  to see what packages they include and from what branches.
>  
> >> You do not rebuild these packages, you rebuild the modules.
> >> If there's no change to the SPEC files, you'll have to instruct
> >> MBS to do a full rebuild (fedpkg module-build --optional
> >> rebuild_strategy=all).  We also plan to add more options to
> >> limit these rebuilds to specific platforms so you wouldn't be
> >> rebuilding for all three or more releases if not required.
> >
> > Is there a mass rebuild of modules like with packages?
>  
>  Not yet, unfortunately.  We did it manually for F30 but not
>  for F29.  The process needs to be updated; it's on our agenda.
>  
> >>> d) (this example is not real (yet)) We decide to retire (remove)
> >>> python2-sphinx because upstream Sphinx no longer supports Python 2. We make
> >>> sure that nothing in Fedora requires or buildrequires it, as all
> Fedora's
> >>> Python packages use python3-sphinx or no Sphinx at all. However there is a
> >>> Django 1.6 module where python-django uses python2-sphinx to build the docs,
> >>> while python2-sphinx is not part of the module itself. How do we find out
> >>> about this and is it our reprehensibility to keep it in rawhide or add it to
> >>> the module?
> >>
> >> I wouldn't say it's your responsibility to resolve the issue
> >> but it is your responsibility to file a bug for the module.
> >
> > How would I know about that in the first place? How do I query it?
>  
>  It will either be in the archive :) or you check the tags and
>  their binaries & modulemds. 

This is not going to be trivial. If you could provide examples, it would 
be great.
> >>> Sorry if those things are obvious, yet I don't know
> the answers. There might
> >>> be even more examples where traditionally we would only think rawhide, but
> >>> now with modules the problem seems more complex.
> >>
> >> Keep them coming.
> >>
> >> I hope this helps at least a little bit, even this late.
> >
> > If you would ignore the questions entirely, I guess it would mean less work
> > for me :D
>  
>  Apologies ;) 
Don't worry. I guess my questions mean more work for you ;)

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Petr Šabata]

On Mon, Sep 17, 2018 at 12:45:59PM +0200, Miro Hrončok wrote:
>  On 17.9.2018 12:36, Petr Šabata wrote:
>  > On Thu, Sep 13, 2018 at 11:07:50PM +0200, Miro Hrončok wrote:
>  > > > > d) (this example is not real (yet)) We decide to retire (remove)
>  > > > > python2-sphinx because upstream Sphinx no longer supports Python 2.
> We make
>  > > > > sure that nothing in Fedora requires or buildrequires it, as all
> Fedora's
>  > > > > Python packages use python3-sphinx or no Sphinx at all. However there
> is a
>  > > > > Django 1.6 module where python-django uses python2-sphinx to build
> the docs,
>  > > > > while python2-sphinx is not part of the module itself. How do we find
> out
>  > > > > about this and is it our reprehensibility to keep it in rawhide or
> add it to
>  > > > > the module?
>  > > > 
>  > > > I wouldn't say it's your responsibility to resolve the issue
>  > > > but it is your responsibility to file a bug for the module.
>  > > 
>  > > How would I know about that in the first place? How do I query it?
>  > 
>  > It will either be in the archive :) or you check the tags and
>  > their binaries & modulemds.
>  
>  
>  This is not going to be trivial. If you could provide examples, it would be
>  great. 
It indeed isn't.

To get the list of F29 modular tags, you could run:
% koji list-tags | grep -P '^f29-modular(-.*)?$'

To get the list of modules in those tags, e.g. F29 "GA":
% koji list-tagged f29-modular

But now it gets complicated.  Even if you fetch the modulemd from
koji or MBS, you won't have the list of RPM artifacts in these
(yet, that will change in the near future).  For that you'll need
the composed modulemd.  So ignoring the above, you could just
check the latest F29 modular repos, get the modules.yaml.gz
in their repodata and see the data.artifacts.rpms sections
which list all their RPMs in the repo.  Once you have those,
you can query them individually.  However, since DNF hides
disabled streams, you'd either need to download them or enable
each stream before running the queries.  I know, it's bad.
We need to enhance repoquery quite a bit here.

I guess you could also fetch primary.xml and work with that.

If you're interested in SPEC files, the modulemds in koji and MBS
tell you what refs were used to build the components, both the
original input (usually branch names) and the expanded variant
(commit hashes).

Example for testmodule-master-20180405123256.c2c572ec:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1065618

You can then fetch these from dist-git.

Feel free to ask on our channel so that we can find a workable
solution for you now... and something for the future.

P