experiencing dnf Error: Failed to synchronize cache for repo 'fedora' or ''updates'

[Michal Novotny]

Hello,

I am occasionally experiencing the following error in my day-to-day dnf use:

    Failed to synchronize cache for repo 'fedora'

or

    Failed to synchronize cache for repo 'updates'

I've had that happened even in local mock builds.

Do you also experience this error upon dnf operations like `dnf install` or
`dnf refresh` or in local mock builds?

Thank you
clime

[Miro Hrončok]

On 20.7.2018 06:25, Michal Novotny wrote:
>  Hello,
>  
>  I am occasionally experiencing the following error in my day-to-day dnf use:
>  
>       Failed to synchronize cache for repo 'fedora'
>  
>  or
>  
>       Failed to synchronize cache for repo 'updates'
>  
>  I've had that happened even in local mock builds.
>  
>  Do you also experience this error upon dnf operations like `dnf install` 
>  or `dnf refresh` or in local mock builds?
>   
Happened to me yesterday. And immediate rerun "fixed" it. Maybe we can 
add some retrying to mock?

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Kamil Paral]

On Fri, Jul 20, 2018 at 9:37 AM, Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:

>  On 20.7.2018 06:25, Michal Novotny wrote:
>
> > Hello,
> >
> > I am occasionally experiencing the following error in my day-to-day dnf
> > use:
> >
> >      Failed to synchronize cache for repo 'fedora'
> >
> > or
> >
> >      Failed to synchronize cache for repo 'updates'
> >
> > I've had that happened even in local mock builds.
> >
> > Do you also experience this error upon dnf operations like `dnf install`
> > or `dnf refresh` or in local mock builds?
> >
> >
>  Happened to me yesterday. And immediate rerun "fixed" it. Maybe we can add
>  some retrying to mock?
>  
We see this quite frequently in Taskotron, because we execute thousands
jobs a day. I had to add "retry" keyword to all ansible tasks that use the
dnf module. The reason is that MirrorManager sometimes returns an HTTP 5xx
error (but when you try again, it works fine).

It would help if DNF retried failed network requests (5xx codes), that's
the best place to fix this for everybody. Anybody cares enough to file a
bug and link it here?

[Michal Novotny]

On Fri, Jul 20, 2018 at 10:45 AM Kamil Paral <kparal(a)redhat.com&gt; wrote:

>  On Fri, Jul 20, 2018 at 9:37 AM, Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
>
> > On 20.7.2018 06:25, Michal Novotny wrote:
> >
> >> Hello,
> >>
> >> I am occasionally experiencing the following error in my day-to-day dnf
> >> use:
> >>
> >>      Failed to synchronize cache for repo 'fedora'
> >>
> >> or
> >>
> >>      Failed to synchronize cache for repo 'updates'
> >>
> >> I've had that happened even in local mock builds.
> >>
> >> Do you also experience this error upon dnf operations like `dnf install`
> >> or `dnf refresh` or in local mock builds?
> >>
> >>
> > Happened to me yesterday. And immediate rerun "fixed" it. Maybe we can
> > add some retrying to mock?
> >
>
>  We see this quite frequently in Taskotron, because we execute thousands
>  jobs a day. I had to add "retry" keyword to all ansible tasks that use the
>  dnf module. The reason is that MirrorManager sometimes returns an HTTP 5xx
>  error (but when you try again, it works fine).
>
>  It would help if DNF retried failed network requests (5xx codes), that's
>  the best place to fix this for everybody. Anybody cares enough to file a
>  bug and link it here?
>  
I actually already filed the respective bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1591225

There is even PR:

https://github.com/rpm-software-management/dnf/pull/1109

that got blocked for an unknown reason and mainly unknown period.

I wanted to confirm that more people are experiencing this issue, that it's
not just some "local" network problem. The problem is that dnf does not
respect max_retries option for a repo when mirror manager is being
contacted. I have no idea why it hasn't been fixed yet given that it
affects basically all dnf users and it also quite significantly affects
building in Copr.

clime

[Kevin Fenzi]

On 07/20/2018 03:55 AM, Michal Novotny wrote:

>  
>  I actually already filed the respective bugs:
>  
>  https://bugzilla.redhat.com/show_bug.cgi?id=1591225
>  
>  There is even PR:
>  
>  https://github.com/rpm-software-management/dnf/pull/1109
>  
>  that got blocked for an unknown reason and mainly unknown period.
>  
>  I wanted to confirm that more people are experiencing this issue, that it's
>  not just some "local" network problem. The problem is that dnf does not
>  respect max_retries option for a repo when mirror manager is being
>  contacted. I have no idea why it hasn't been fixed yet given that it
>  affects basically all dnf users and it also quite significantly affects
>  building in Copr. 
Thats a good workaround and we should do it, but I sure wish we could
track down when and why those sporadic 503's come from our mirrorlist
containers. ;(

Perhaps we could add further debugging and track it down. I'll see what
I can do...

kevin

[Miro Hrončok]

On 20.7.2018 19:56, Kevin Fenzi wrote:
>  On 07/20/2018 03:55 AM, Michal Novotny wrote:
>  
> >
> > I actually already filed the respective bugs:
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=1591225
> >
> > There is even PR:
> >
> > https://github.com/rpm-software-management/dnf/pull/1109
> >
> > that got blocked for an unknown reason and mainly unknown period.
> >
> > I wanted to confirm that more people are experiencing this issue, that it's
> > not just some "local" network problem. The problem is that dnf does not
> > respect max_retries option for a repo when mirror manager is being
> > contacted. I have no idea why it hasn't been fixed yet given that it
> > affects basically all dnf users and it also quite significantly affects
> > building in Copr.
>  
>  Thats a good workaround and we should do it, but I sure wish we could
>  track down when and why those sporadic 503's come from our mirrorlist
>  containers. ;(
>  
>  Perhaps we could add further debugging and track it down. I'll see what
>  I can do... 
Can we use something like Sentry? https://sentry.io/

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Kevin Fenzi]

On 07/20/2018 11:15 PM, Miro Hrončok wrote:
>  On 20.7.2018 19:56, Kevin Fenzi wrote:
> > On 07/20/2018 03:55 AM, Michal Novotny wrote:
> >
> >>
> >> I actually already filed the respective bugs:
> >>
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1591225
> >>
> >> There is even PR:
> >>
> >> https://github.com/rpm-software-management/dnf/pull/1109
> >>
> >> that got blocked for an unknown reason and mainly unknown period.
> >>
> >> I wanted to confirm that more people are experiencing this issue,
> >> that it's
> >> not just some "local" network problem. The problem is that dnf does
> not
> >> respect max_retries option for a repo when mirror manager is being
> >> contacted. I have no idea why it hasn't been fixed yet given that it
> >> affects basically all dnf users and it also quite significantly affects
> >> building in Copr.
> >
> > Thats a good workaround and we should do it, but I sure wish we could
> > track down when and why those sporadic 503's come from our mirrorlist
> > containers. ;(
> >
> > Perhaps we could add further debugging and track it down. I'll see what
> > I can do...
>  
>  Can we use something like Sentry? https://sentry.io/ 
Thats a non free 3rd party service? no thanks. :)

However, smooge spent a bunch of time this weekend tracking a lot of
this down. It's not solved, but it should be much better now and he's
going to try and quash the last bits next week.

kevin

[Michal Novotny]

On Sun, Jul 22, 2018 at 8:03 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:

>  On 07/20/2018 11:15 PM, Miro Hrončok wrote:
>  > On 20.7.2018 19:56, Kevin Fenzi wrote:
>  >> On 07/20/2018 03:55 AM, Michal Novotny wrote:
>  >>
>  >>>
>  >>> I actually already filed the respective bugs:
>  >>>
>  >>> https://bugzilla.redhat.com/show_bug.cgi?id=1591225
>  >>>
>  >>> There is even PR:
>  >>>
>  >>> https://github.com/rpm-software-management/dnf/pull/1109
>  >>>
>  >>> that got blocked for an unknown reason and mainly unknown period.
>  >>>
>  >>> I wanted to confirm that more people are experiencing this issue,
>  >>> that it's
>  >>> not just some "local" network problem. The problem is that dnf
> does not
>  >>> respect max_retries option for a repo when mirror manager is being
>  >>> contacted. I have no idea why it hasn't been fixed yet given that it
>  >>> affects basically all dnf users and it also quite significantly affects
>  >>> building in Copr.
>  >>
>  >> Thats a good workaround and we should do it, but I sure wish we could
>  >> track down when and why those sporadic 503's come from our mirrorlist
>  >> containers. ;(
>  >>
>  >> Perhaps we could add further debugging and track it down. I'll see what
>  >> I can do...
>  >
>  > Can we use something like Sentry? https://sentry.io/
>
>  Thats a non free 3rd party service? no thanks. :)
>
>  However, smooge spent a bunch of time this weekend tracking a lot of
>  this down. It's not solved, but it should be much better now and he's
>  going to try and quash the last bits next week.
>  
Thank you a lot for this. I think that our infrastructure does really an
admirable job. I greatly
appreciate the effort to fix this.

It would still be good if dnf implemented the retry feature for mirror
manager.
I don't quite understand why block
https://github.com/rpm-software-management/dnf/pull/1109.

clime


>  kevin
>
>
>
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>

[Kevin Fenzi]

On 07/23/2018 12:14 AM, Michal Novotny wrote:
>  
>  Thank you a lot for this. I think that our infrastructure does really an
>  admirable job. I greatly
>  appreciate the effort to fix this. 
Yeah, thank smooge for tracking down a logic error in our container
restart script. :)
>  
>  It would still be good if dnf implemented the retry feature for mirror
>  manager. 
I agree.

>  I don't quite understand why block
>  https://github.com/rpm-software-management/dnf/pull/1109. 
Likely they are just busy with other things, hopefully they will get to it.

kevin

[Miro Hrončok]

On 22.7.2018 20:02, Kevin Fenzi wrote:
>  On 07/20/2018 11:15 PM, Miro Hrončok wrote:
> > On 20.7.2018 19:56, Kevin Fenzi wrote:
> >> On 07/20/2018 03:55 AM, Michal Novotny wrote:
> >>
> >>>
> >>> I actually already filed the respective bugs:
> >>>
> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1591225
> >>>
> >>> There is even PR:
> >>>
> >>> https://github.com/rpm-software-management/dnf/pull/1109
> >>>
> >>> that got blocked for an unknown reason and mainly unknown period.
> >>>
> >>> I wanted to confirm that more people are experiencing this issue,
> >>> that it's
> >>> not just some "local" network problem. The problem is that dnf does
> not
> >>> respect max_retries option for a repo when mirror manager is being
> >>> contacted. I have no idea why it hasn't been fixed yet given that it
> >>> affects basically all dnf users and it also quite significantly affects
> >>> building in Copr.
> >>
> >> Thats a good workaround and we should do it, but I sure wish we could
> >> track down when and why those sporadic 503's come from our mirrorlist
> >> containers. ;(
> >>
> >> Perhaps we could add further debugging and track it down. I'll see what
> >> I can do...
> >
> > Can we use something like Sentry? https://sentry.io/
>  
>  Thats a non free 3rd party service? no thanks. :) 
It's an libre service we can run by ourselves if desired.

https://github.com/getsentry/sentry


-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Kevin Fenzi]

On 07/23/2018 02:22 AM, Miro Hrončok wrote:

>  It's an libre service we can run by ourselves if desired.
>  
>  https://github.com/getsentry/sentry 
Ah, I didn't see that from the main website. ;)

Will take a look, thanks for the suggestion!

kevin

[Fabio Valentini]

On Fri, Jul 20, 2018 at 6:27 AM Michal Novotny <clime(a)redhat.com&gt; wrote:
>  Hello,
>
>  I am occasionally experiencing the following error in my day-to-day dnf use:
>
>      Failed to synchronize cache for repo 'fedora'
>
>  or
>
>      Failed to synchronize cache for repo 'updates'
>
>  I've had that happened even in local mock builds. 
These errors are the primary reason for why copr builds fail for me.
It also sometimes happens locally in mock, and dnf, too.
So it probably really is a connection issue when getting the
mirrorlists or something?

Fabio

> Do you also experience this error upon dnf operations like `dnf install` or `dnf
refresh` or in local mock builds?
>
> Thank you
> clime
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail].o...