[ELN] Missing /usr/bin/bzr in ELN (solved, but open questions)

[Miro Hrončok]

Hello,

I've realized recently that the Python 3.9.0 final build failed in ELN:

python3.9-3.9.0-1.eln104:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1620970

The build failure was:
   File 
"/tmp/tmpip_uvs06/pip-20.1.1-py2.py3-none-any.whl/pip/_vendor/toml/decoder.py",

line 47
     TIME_RE = re.compile("([0-9]{2}):([0-9]{2}):([0-9]{2})(\.([0-9]{3,6}))?")
                          ^
SyntaxError: invalid escape sequence \.

I wondered why did this happen in ELN only and not in Rawhide.
The reason is: in Rawhide we have pip 20.2.2 which contains the fix of this problem.

So I investigated further and realized pip in ELN was not updated because the 
build is failing. The failure is:

     No matching package to install: '/usr/bin/bzr'

In Fedora, the executable is provided by breezy:

$ repoquery --repo=rawhide -l breezy | grep /usr/bin
/usr/bin/brz
/usr/bin/bzr
/usr/bin/bzr-receive-pack
/usr/bin/bzr-upload-pack
/usr/bin/git-remote-brz
/usr/bin/git-remote-bzr

However, breezy is not included in ELN at all:

$ repoquery --repo=eln-\* breezy
(nothing)

Despite the fact that it is in:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-43fde95e1f

I can however see it in the ELN Koji buildroot:

$ repoquery --repo=kojieln breezy
breezy-0:3.0.2-4.eln103.x86_64

Yet it doesn't have the proper file:

$ repoquery --repo=kojieln -l breezy | grep /usr/bin
/usr/bin/brz
/usr/bin/bzr-receive-pack
/usr/bin/bzr-upload-pack
/usr/bin/git-remote-brz


I have discovered this is caused by a Fedora-only %if conditional in the spec file:

	
     # https://fedoraproject.org/wiki/Changes/ReplaceBazaarWithBreezy
     %if 0%{?fedora} >= 32
     %bcond_without replace_bzr
     %else
     %bcond_with replace_bzr
     %endif

I'll fix the conditional to make it RHEL-aware. However I have several questions:

1) Why is breezy not in the ELN compose, but is in the Koji buildroot?
2) As pip's maintainer, should I've been notified about the missing dependency?
3) When I fix breezy, should I resubmit the eln rebuilds of pip and Python?

Thanks,
-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Neal Gompa]

On Wed, Oct 21, 2020 at 3:00 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  Hello,
>
>  I've realized recently that the Python 3.9.0 final build failed in ELN:
>
>  python3.9-3.9.0-1.eln104:
>  https://koji.fedoraproject.org/koji/buildinfo?buildID=1620970
>
>  The build failure was:
>     File
>
> "/tmp/tmpip_uvs06/pip-20.1.1-py2.py3-none-any.whl/pip/_vendor/toml/decoder.py",
>  line 47
>       TIME_RE = re.compile("([0-9]{2}):([0-9]{2}):([0-9]{2})(\.([0-9]{3,6}))?")
>                            ^
>  SyntaxError: invalid escape sequence \.
>
>  I wondered why did this happen in ELN only and not in Rawhide.
>  The reason is: in Rawhide we have pip 20.2.2 which contains the fix of this problem.
>
>  So I investigated further and realized pip in ELN was not updated because the
>  build is failing. The failure is:
>
>       No matching package to install: '/usr/bin/bzr'
>
>  In Fedora, the executable is provided by breezy:
>
>  $ repoquery --repo=rawhide -l breezy | grep /usr/bin
>  /usr/bin/brz
>  /usr/bin/bzr
>  /usr/bin/bzr-receive-pack
>  /usr/bin/bzr-upload-pack
>  /usr/bin/git-remote-brz
>  /usr/bin/git-remote-bzr
>
>  However, breezy is not included in ELN at all:
>
>  $ repoquery --repo=eln-\* breezy
>  (nothing)
>
>  Despite the fact that it is in:
>  https://bodhi.fedoraproject.org/updates/FEDORA-2020-43fde95e1f
>
>  I can however see it in the ELN Koji buildroot:
>
>  $ repoquery --repo=kojieln breezy
>  breezy-0:3.0.2-4.eln103.x86_64
>
>  Yet it doesn't have the proper file:
>
>  $ repoquery --repo=kojieln -l breezy | grep /usr/bin
>  /usr/bin/brz
>  /usr/bin/bzr-receive-pack
>  /usr/bin/bzr-upload-pack
>  /usr/bin/git-remote-brz
>
>
>  I have discovered this is caused by a Fedora-only %if conditional in the spec file:
>
>
>       # https://fedoraproject.org/wiki/Changes/ReplaceBazaarWithBreezy
>       %if 0%{?fedora} >= 32
>       %bcond_without replace_bzr
>       %else
>       %bcond_with replace_bzr
>       %endif
>
>  I'll fix the conditional to make it RHEL-aware. However I have several questions:
>
>  1) Why is breezy not in the ELN compose, but is in the Koji buildroot? 
It's defined as a buildroot only package:
https://github.com/minimization/content-resolver-input/blob/master/config...

>  2) As pip's maintainer, should I've been notified about the
> missing dependency? 
Probably not, I think the idea was that ELN SIG would manage that exclusively.

>  3) When I fix breezy, should I resubmit the eln rebuilds of pip and
> Python?
>  
No. Just submit to Rawhide and the Jenkins job for ELN will take care
of it for you after it succeeds in Rawhide.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Miro Hrončok]

On 10/21/20 10:10 PM, Neal Gompa wrote:
>  On Wed, Oct 21, 2020 at 3:00 PM Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
> > 1) Why is breezy not in the ELN compose, but is in the Koji buildroot?
>  
>  It's defined as a buildroot only package:
>
> https://github.com/minimization/content-resolver-input/blob/master/config...
I wonder why. git blame suggests this is automated somehow.

> > 2) As pip's maintainer, should I've been notified about
> the missing dependency?
>  
>  Probably not, I think the idea was that ELN SIG would manage that exclusively. 
So I should stop caring?

> > 3) When I fix breezy, should I resubmit the eln rebuilds of pip
> and Python?
> >
>  
>  No. Just submit to Rawhide and the Jenkins job for ELN will take care
>  of it for you after it succeeds in Rawhide. 
Python and pip succeeded in rawhide already, this is an ELN-only fix.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Neal Gompa]

On Wed, Oct 21, 2020 at 7:45 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 10/21/20 10:10 PM, Neal Gompa wrote:
>  > On Wed, Oct 21, 2020 at 3:00 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  >> 1) Why is breezy not in the ELN compose, but is in the Koji buildroot?
>  >
>  > It's defined as a buildroot only package:
>  >
> https://github.com/minimization/content-resolver-input/blob/master/config...
>
>  I wonder why. git blame suggests this is automated somehow.
>  
Hmm, that's a good question...

>  >> 2) As pip's maintainer, should I've been notified
> about the missing dependency?
>  >
>  > Probably not, I think the idea was that ELN SIG would manage that exclusively.
>
>  So I should stop caring?
>  
I think you should still care, because in practice I don't think it's
working out that way.

>  >> 3) When I fix breezy, should I resubmit the eln rebuilds of
> pip and Python?
>  >>
>  >
>  > No. Just submit to Rawhide and the Jenkins job for ELN will take care
>  > of it for you after it succeeds in Rawhide.
>
>  Python and pip succeeded in rawhide already, this is an ELN-only fix.
>  
I mean that their bot listens for packages built in Rawhide and
triggers a rebuild automatically for ELN.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Miro Hrončok]

On 10/22/20 1:59 AM, Neal Gompa wrote:
> >>> 3) When I fix breezy, should I resubmit the eln rebuilds
> of pip and Python?
> >>>
> >> No. Just submit to Rawhide and the Jenkins job for ELN will take care
> >> of it for you after it succeeds in Rawhide.
> > Python and pip succeeded in rawhide already, this is an ELN-only fix.
> >
>  I mean that their bot listens for packages built in Rawhide and
>  triggers a rebuild automatically for ELN. 
I've just built breezy in rawhide and it an automatic build was triggered in 
ELN, as expected.

My question however is, whether I shall now build python-pip and later python3.9 
in ELN myself or wait for some retry builds (which could take up to <unknown 
time> to happen in the correct order).

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Aleksandra Fedorova]

Hi, Miro,

On Thu, Oct 22, 2020 at 6:54 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 10/22/20 1:59 AM, Neal Gompa wrote:
>  >>>> 3) When I fix breezy, should I resubmit the eln rebuilds of pip and
> Python?
>  >>>>
>  >>> No. Just submit to Rawhide and the Jenkins job for ELN will take care
>  >>> of it for you after it succeeds in Rawhide.
>  >> Python and pip succeeded in rawhide already, this is an ELN-only fix.
>  >>
>  > I mean that their bot listens for packages built in Rawhide and
>  > triggers a rebuild automatically for ELN.
>
>  I've just built breezy in rawhide and it an automatic build was triggered in
>  ELN, as expected.
>
>  My question however is, whether I shall now build python-pip and later python3.9
>  in ELN myself or wait for some retry builds (which could take up to <unknown
>  time> to happen in the correct order). 
It definitely will help if you submit the builds on your own.

We have a rebuild loop for the missing updates, and we have ELN SIG
reviewing the status of the repository [2]. But submitting the builds
will indeed be faster.

[1] https://docs.fedoraproject.org/en-US/eln/status/

-- 
Aleksandra Fedorova
bookwar

[Miro Hrončok]

On 10/22/20 7:03 PM, Aleksandra Fedorova wrote:
>  Hi, Miro,
>  
>  On Thu, Oct 22, 2020 at 6:54 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
> >
> > On 10/22/20 1:59 AM, Neal Gompa wrote:
> >>>>> 3) When I fix breezy, should I resubmit the eln rebuilds of pip and
> Python?
> >>>>>
> >>>> No. Just submit to Rawhide and the Jenkins job for ELN will take care
> >>>> of it for you after it succeeds in Rawhide.
> >>> Python and pip succeeded in rawhide already, this is an ELN-only fix.
> >>>
> >> I mean that their bot listens for packages built in Rawhide and
> >> triggers a rebuild automatically for ELN.
> >
> > I've just built breezy in rawhide and it an automatic build was triggered in
> > ELN, as expected.
> >
> > My question however is, whether I shall now build python-pip and later python3.9
> > in ELN myself or wait for some retry builds (which could take up to <unknown
> > time> to happen in the correct order).
>  
>  It definitely will help if you submit the builds on your own.
>  
>  We have a rebuild loop for the missing updates, and we have ELN SIG
>  reviewing the status of the repository [2]. But submitting the builds
>  will indeed be faster. 
Thanks. What is the correct way?

    $ fedpkg build --target=eln

?
-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Aleksandra Fedorova]

On Thu, Oct 22, 2020 at 7:41 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 10/22/20 7:03 PM, Aleksandra Fedorova wrote:
>  > Hi, Miro,
>  >
>  > On Thu, Oct 22, 2020 at 6:54 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  >>
>  >> On 10/22/20 1:59 AM, Neal Gompa wrote:
>  >>>>>> 3) When I fix breezy, should I resubmit the eln rebuilds of pip
> and Python?
>  >>>>>>
>  >>>>> No. Just submit to Rawhide and the Jenkins job for ELN will take
> care
>  >>>>> of it for you after it succeeds in Rawhide.
>  >>>> Python and pip succeeded in rawhide already, this is an ELN-only fix.
>  >>>>
>  >>> I mean that their bot listens for packages built in Rawhide and
>  >>> triggers a rebuild automatically for ELN.
>  >>
>  >> I've just built breezy in rawhide and it an automatic build was triggered
> in
>  >> ELN, as expected.
>  >>
>  >> My question however is, whether I shall now build python-pip and later
> python3.9
>  >> in ELN myself or wait for some retry builds (which could take up to <unknown
>  >> time> to happen in the correct order).
>  >
>  > It definitely will help if you submit the builds on your own.
>  >
>  > We have a rebuild loop for the missing updates, and we have ELN SIG
>  > reviewing the status of the repository [2]. But submitting the builds
>  > will indeed be faster.
>
>  Thanks. What is the correct way?
>
>      $ fedpkg build --target=eln
>
>  ? 
Yes. Thank you.

I was sure it has been documented already, but apparently we
documented everything about buildroot, except this part. I'll add it.

>  --
>  Miro Hrončok
>  --
>  Phone: [phone]
>  IRC: mhroncok
>  
-- 
Aleksandra Fedorova
bookwar

[Miro Hrončok]

On 10/22/20 7:50 PM, Aleksandra Fedorova wrote:
>  On Thu, Oct 22, 2020 at 7:41 PM Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
> > What is the correct way?
> >
> >      $ fedpkg build --target=eln
> >
> > ?
>  
>  Yes. Thank you.
>  
>  I was sure it has been documented already, but apparently we
>  documented everything about buildroot, except this part. I'll add it. 
Funny. I wanted to use chain-build:


[python3.9 (master)]$ fedpkg chain-build --target=eln python-pip
Could not execute chainbuild: Package python3.9-3.9.0-1.fc34 has already been built
Note: You can skip this check with --skip-nvr-check. See help for more info.

[python3.9 (master)]$ fedpkg chain-build --target=eln python-pip --skip-nvr-check
fedpkg: error: unrecognized arguments: --skip-nvr-check


So I proceeded to build the packages sequentially:


[python-pip (master)]$ fedpkg build --target=eln
Building python-pip-20.2.2-1.fc34 for eln
...
(this worked fine, because the rawhide build happened with .fc33)

[python3.9 (master)]$ fedpkg build --target=eln
Could not execute build: Package python3.9-3.9.0-1.fc34 has already been built
Note: You can skip this check with --skip-nvr-check. See help for more info.

[python3.9 (master)]$ fedpkg build --target=eln --skip-nvr-check
Building python3.9-3.9.0-1.fc34 for eln
...


Aleksandra, please document the need for --skip-nvr-check when you document this.

I'll open a bug for fedpkg about the chain-build problem.


-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 10/22/20 8:10 PM, Miro Hrončok wrote:
>  I wanted to use chain-build:
>  
>  
>  [python3.9 (master)]$ fedpkg chain-build --target=eln python-pip
>  Could not execute chainbuild: Package python3.9-3.9.0-1.fc34 has already been built
>  Note: You can skip this check with --skip-nvr-check. See help for more info.
>  
>  [python3.9 (master)]$ fedpkg chain-build --target=eln python-pip --skip-nvr-check
>  fedpkg: error: unrecognized arguments: --skip-nvr-check
>  
>  ...
>  
>  I'll open a bug for fedpkg about the chain-build problem. 
https://bugzilla.redhat.com/show_bug.cgi?id=1890701

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Adam Samalik]

On Thu, 22 Oct 2020 at 01:46, Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:

>  On 10/21/20 10:10 PM, Neal Gompa wrote:
>  > On Wed, Oct 21, 2020 at 3:00 PM Miro Hrončok <mhroncok(a)redhat.com&gt;
>  wrote:
>  >> 1) Why is breezy not in the ELN compose, but is in the Koji buildroot?
>  >
>  > It's defined as a buildroot only package:
>  >
>
> https://github.com/minimization/content-resolver-input/blob/master/config...
>
>  I wonder why. git blame suggests this is automated somehow.
>  
Content Resolver doesn't resolve the buildroot itself, it's done by a
different service [1]. That's because
it needs to rebuild some SRPMs for each architecture, which is beyond what
Content Resolver
should do.

That yaml file was the initial, hacky way to get buildroot packages in from
that service. But we don't
use it anymore.

The best way to get buildroot info is through the UI:
https://tiny.distro.builders/view-buildroot--view-eln--x86_64.html
I found breezy there, and opening it, it says it's required to build the
python-pip SRPM:
https://tiny.distro.builders/view-rpm--view-eln--breezy.html

(Please ignore the SRPM section on that page, there was a bug [2])

[1] https://github.com/minimization/dep-tracker
[2] https://github.com/minimization/content-resolver-input/issues/354



>  >> 2) As pip's maintainer, should I've been notified
> about the missing
>  dependency?
>  >
>  > Probably not, I think the idea was that ELN SIG would manage that
>  exclusively.
>
>  So I should stop caring?
>
>  >> 3) When I fix breezy, should I resubmit the eln rebuilds of pip and
>  Python?
>  >>
>  >
>  > No. Just submit to Rawhide and the Jenkins job for ELN will take care
>  > of it for you after it succeeds in Rawhide.
>
>  Python and pip succeeded in rawhide already, this is an ELN-only fix.
>
>  --
>  Miro Hrončok
>  --
>  Phone: [phone]
>  IRC: mhroncok
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  

-- 

Adam Šamalík
---------------------------
Senior Software Engineer
Red Hat