MirrorManager documentation and links

[Brian Long (brilong)]

I started getting output from report_mirror saying exception ProtocolError not defined.  I
had to dig around to find the latest client code and compare to what I had installed.  I
wonder if some links need to be cleaned up on the fedoraproject.org website.

 

In my mind, the following site is the source of truth wrt. MirrorManager:

https://fedoraproject.org/wiki/Infrastructure/Mirroring

 

I think some of the links on this page might need to be updated.  Specifically, where it
says “The MirrorManager software keeps track of all the mirrors”,
http://mirrormanager.org/ takes me to https://apps.fedoraproject.org/ and I have to
manually select MirrorManager from the graph.  I finally get directed to
https://admin.fedoraproject.org/mirrormanager/.  It would be nice if
http://mirrormanager.org/ took me straight to the right URL. 

 

At the bottom of the https://admin.fedoraproject.org/mirrormanager/ page, it has a link to
https://fedorahosted.org/mirrormanager/ which is broken since fedorahosted is retired.  I
then manually click on https://pagure.io/ and have to search for mirrormanager.  I get
directed here:  https://pagure.io/mirrormanager.  The Overview tab needs to be reformatted
and it’s specific to MirrorManager server.

 

I ended up downloading https://pagure.io/mirrormanager/blob/master/f/client/report_mirror
and comparing to the version I had installed (mirrormanager-client-1.4.4-6.el7.noarch). 
It turns out the only difference is as follows:

 

--- /usr/bin/report_mirror 2016-06-23 09:55:20.000000000 -0400

+++ /usr/local/bin/report_mirror   2017-04-13 07:45:26.059404424 -0400

@@ -290,7 +292,7 @@

                 print server.checkin(data)

             except socket.error, m:

                 print "Error checking in: %s.  Please try again later." %
(m[1])

-            except ProtocolError:

+            except xmlrpclib.ProtocolError:

                 print "Error checking in: Service Temporarily Unavailable.  Please
try again later."

                 sys.exit(1)

             except xmlrpclib.Fault:

 

 

Also, what’s the difference between the code on pagure.io and
https://github.com/fedora-infra/mirrormanager2/?  This github link has README containing
old mailing list information:

Mailing list for announcements:http://www.redhat.com/mailman/listinfo/mirror-list
Mailing list for discussions:http://www.redhat.com/mailman/listinfo/mirror-list-d

 

Could someone update the mirrormanager-client to include the new code?  Could someone
consider changing some of the links I mentioned above?

 

Thank you.

 

/Brian/

-- 

Brian Long

Cisco Systems, Inc.

Government, Trust and Technology Services

Security & Trust Organization

Research Triangle Park, NC

[Adrian Reber]

On Thu, Apr 13, 2017 at 12:12:52PM +0000, Brian Long (brilong) wrote:
>  I started getting output from report_mirror saying exception
> ProtocolError not defined.  I had to dig around to find the latest client code and compare
> to what I had installed.  I wonder if some links need to be cleaned up on the
> fedoraproject.org website. 
I also get sometimes errors like this from report_mirror:

xmlrpclib.ProtocolError: <ProtocolError for
admin.fedoraproject.org/mirrormanager/xmlrpc: 500 Internal Server Error>

I am not sure why those errors happen, but this is kind of 'normal'.
Sometimes it happens more often sometimes not but I never looked closer
at it as most of the times the next check-in works and the crawler will
also detect the right status.

>  In my mind, the following site is the source of truth wrt.
> MirrorManager:
>  
>  https://fedoraproject.org/wiki/Infrastructure/Mirroring 
[...]
>  Also, what’s the difference between the code on pagure.io and
> https://github.com/fedora-infra/mirrormanager2/?  This github link has README containing
> old mailing list information:
>  
>  Mailing list for announcements:http://www.redhat.com/mailman/listinfo/mirror-list
>  Mailing list for discussions:http://www.redhat.com/mailman/listinfo/mirror-list-d 
A few days ago I also saw that and thought I should update it. I will
try to correct all the links to point at the right locations. Thanks for
collecting all the locations of outdated information.

>  Could someone update the mirrormanager-client to include the new
> code?  Could someone consider changing some of the links I mentioned above? 
Which package do you have installed. I think we already provide the
latest code in the mirrormanager2-client package. We probably need to
remove the old mirrormanager-client package so that everybody gets the
latest mirrormanager2-client package.

		Adrian

[Brian Long (brilong)]

Adrian,

On 4/14/17, 6:04 AM, "Adrian Reber" <adrian(a)lisas.de&gt; wrote:

>  Could someone update the mirrormanager-client to include the new
> code?  Could someone consider changing some of the links I mentioned above? 
    
    Which package do you have installed. I think we already provide the
    latest code in the mirrormanager2-client package. We probably need to
    remove the old mirrormanager-client package so that everybody gets the
    latest mirrormanager2-client package.
    

I’m running CentOS 7.3 as my mirror host.  It has the following:
$ rpm -q mirrormanager-client
mirrormanager-client-1.4.4-6.el7.noarch

I do not see mirrormanager2-client in standard repos nor EPEL.  Which repo contains
mirrormanager2-client for RHEL/CentOS systems?
$ sudo yum list mirrormanager2-client

/Brian/

[Jason L Tibbitts III]

> >>>> "AR" == Adrian Reber
> <adrian(a)lisas.de&gt; writes: 
AR> I also get sometimes errors like this from report_mirror:

quick-fedora-mirror (which does not use report_mirror but instead makes
the necessary xmlrpc call itself) will also log these events when it
does its checkins:

journalctl -t quick-fedora-mirror --since -1week |grep 'Internal Server Error'|wc
-l
21

Apr 13 11:21:05 pubmirror1.math.uh.edu quick-fedora-mirror[26110]: mirrormanager checkin
start
Apr 13 11:21:05 pubmirror1.math.uh.edu quick-fedora-mirror[26110]: Processing fedora-alt
(in alt) as pubmirror1-alt
Apr 13 11:21:06 pubmirror1.math.uh.edu quick-fedora-mirror[26110]: Making xmlrpc call for
fedora-alt
Apr 13 11:22:06 pubmirror1.math.uh.edu quick-fedora-mirror[26110]: Doesn't look like
we got a good return from mirrormanager.
Apr 13 11:22:06 pubmirror1.math.uh.edu quick-fedora-mirror[26110]: 500 Internal Server
Error Internal Server Error The server encountered an internal error or misconfiguration
and was unable to complete your request. Please contact the server administrator at  
root@localhost to inform them of the time this error occurred,  and the actions you
performed just before this error. More information about this error may be available in
the server error log.
Apr 13 11:22:06 pubmirror1.math.uh.edu quick-fedora-mirror[26110]: Processing fedora-alt:
end
Apr 13 11:22:06 pubmirror1.math.uh.edu quick-fedora-mirror[26110]: mirrormanager checkin
end
Apr 13 11:22:06 pubmirror1.math.uh.edu quick-fedora-mirror[26110]: Run end.

The frequency appears to have gone up lately, but I'm not sure why.
q-f-m now (since April 6th) checks in each module individually instead
of doing one huge checkin and only checks in the modules which actually
changed.  So there should be quite a bit less work to do on
mirrormanager's end of things.

In any case, I have this issue open to implement a basic retry mechanism
for checkins: https://pagure.io/quick-fedora-mirror/issue/46

Hopefully I'll get to that soon.

 - J<

[Chuck Anderson]

On Fri, Apr 14, 2017 at 12:31:20PM -0500, Jason L Tibbitts III wrote:
>  >>>>> "AR" == Adrian Reber
> <adrian(a)lisas.de&gt; writes:
>  
>  AR> I also get sometimes errors like this from report_mirror:
>  
>  quick-fedora-mirror (which does not use report_mirror but instead makes
>  the necessary xmlrpc call itself) 
Has that part been fixed now?  I couldn't get it to work when I
deployed q-f-m a month ago, and I thought I read something about it
being known to not work correctly yet.

[Kevin Fenzi]

On 04/13/2017 06:12 AM, Brian Long (brilong) wrote:
>  I started getting output from report_mirror saying exception
> ProtocolError not defined.  I had to dig around to find the latest client code and compare
> to what I had installed.  I wonder if some links need to be cleaned up on the
> fedoraproject.org website. 
Yep. Finally getting to doing so. ;)

>  In my mind, the following site is the source of truth wrt.
> MirrorManager:
>  
>  https://fedoraproject.org/wiki/Infrastructure/Mirroring 
yep. Thats what we point everyone to.
>  I think some of the links on this page might need to be updated. 
> Specifically, where it says “The MirrorManager software keeps track of all the mirrors”,
> http://mirrormanager.org/ takes me to https://apps.fedoraproject.org/ and I have to
> manually select MirrorManager from the graph.  I finally get directed to
> https://admin.fedoraproject.org/mirrormanager/.  It would be nice if
> http://mirrormanager.org/ took me straight to the right URL.  
Well, that link was supposed to take you to the source for
mirrormanager, but no longer does. I have added a correct link and also
clarified what it's pointing to there.
>  
>   
>  
>  At the bottom of the https://admin.fedoraproject.org/mirrormanager/ page, it has a link
> to https://fedorahosted.org/mirrormanager/ which is broken since fedorahosted is retired. 
> I then manually click on https://pagure.io/ and have to search for mirrormanager.  I get
> directed here:  https://pagure.io/mirrormanager.  The Overview tab needs to be reformatted
> and it’s specific to MirrorManager server. 
Yeah, thats not the active upstream either, it's github.

Made a PR to correct upstream, and we should get it in the next app update:

https://github.com/fedora-infra/mirrormanager2/pull/209

>  
>   
>  
>  I ended up downloading https://pagure.io/mirrormanager/blob/master/f/client/report_mirror
> and comparing to the version I had installed (mirrormanager-client-1.4.4-6.el7.noarch). 
> It turns out the only difference is as follows: 
...snip....

>  Also, what’s the difference between the code on pagure.io and
> https://github.com/fedora-infra/mirrormanager2/?  This github link has README containing
> old mailing list information
>  
>  Mailing list for announcements:http://www.redhat.com/mailman/listinfo/mirror-list
>  Mailing list for discussions:http://www.redhat.com/mailman/listinfo/mirror-list-d 
Yes, see above PR with fixes.
>  
>  Could someone update the mirrormanager-client to include the new code?  Could someone
> consider changing some of the links I mentioned above? 
Yes, we need to push out new versions.

kevin