Proposal to mirror Docker images

[Randy Barlow]

During Flock 2016, I had the opportunity to talk with Adam Miller,                        
                         
Dennis Gilmore, and Pierre-Yves Chibon about the technical challenges                     
                         
with distributing Docker images with Fedora's extensive mirror network.               
                             
These conversations helped me to solidify a proposal for how Fedora                       
                         
could solve this problem, outlined below.                                                 
                         
                                                                                          
                         
                                                                                          
                         
High level view                                                                           
                         
===============                                                                           
                         
                                                                                          
                         
In summary, the proposal is to write a patch for the docker client that                   
                         
will give it the capability to accept metalink responses upon docker                      
                         
pull operations. We would also need to add support for Docker images to                   
                         
mirror list and mirror manager. Additionally, we will need a small tool                   
                         
to pull the content to be mirrored out of a docker registry and write                     
                         
them to disk in a format that can be mirrored, as well as some Ansible                    
                         
code to run the tool when there is new content to be mirrored.                            
                         
                                                                                          
                         
                                                                                          
                         
Background                                                                                
                         
==========                                                                                
                         
                                                                                          
                         
The Fedora project wishes to begin distributing new types of content                      
                         
than it has in the past. One of the types that has been identified as a                   
                         
goal is the Docker image. Adam Miller has already done the work that                      
                         
will allow packagers to build Docker images, but we still need a way to                   
                         
distribute those builds to Fedora's users. Adam Miller's implementation           
                                 
helpfully drops the builds we want into a Docker registry.                                
                         
                                                                                          
                         
                                                                                          
                         
Proposed Changes                                                                          
                         
================                                                                          
                         
                                                                                          
                         
Mirror List                                                                               
                         
-----------                                                                               
                         
                                                                                          
                         
Users will be pointing their docker clients at Mirror List when they                      
                         
docker pull Fedora's Docker images. In order for this to work, we will                
                             
need to make two changes to Mirror List so that it can respond to the                     
                         
docker client properly. The first change is that Mirror List will need                    
                         
to respond with a special header and a body of "{}" when the docker             
                                   
client sends a GET request for /v2/. The second change is that it will                    
                         
need to return a metalink document when the client makes additional                       
                         
requests so that the clients can be redirected to a list of mirrors                       
                         
near their locations, just as it does with the dnf client today.                          
                         
                                                                                          
                         
The docker client typically connects to port 5000. We could run a                         
                         
second instance of Mirror List on port 5000 if we wanted to isolate it                    
                         
from the current instance. We can also have the docker client pull from                   
                         
443 as dnf does if we want to keep the deployment simpler.                                
                         


Mirror Manager
--------------

We will need to make a few changes to Mirror Manager as well. We will
need to provide an interface to allow mirror admins to opt in/out of
mirroring Docker content. We will also need to modify the curler to
detect whether a given mirror is up to date or not. We will need to make
sure that UMDL is updated when content changes.


docker
------

The most significant work required will likely be modifying the docker
client to enable it to properly handle the metalink responses it will be
receiving from Mirror List. When requesting the manifest, it will
receive a metalink document that will give it a priority ordered list of
mirrors. It will need to work through the list in order until it reaches
a mirror that has the correct checksum for the requested manifest. It
will then use that same mirror for the subsequent blob requests.

There is some concern that such a feature would not be accepted by the
upstream docker project. If we were to proceed with this proposal, we
would propose this patch to the upstream Docker project. If upstream
were not willing to accept the feature, we would need to have the Fedora
docker packager carry this patch as a downstream add on.


New Tool
--------

The last piece that is needed is a tool that can create the filesystem
tree that we want to synchronize out to the mirrors. The mirrors only
need to carry manifests and blobs, so the tool needs only to pull these
documents out of the registry that Adam Miller has set up and write them
to disk in a particular structure. For optimization, we could use
hardlinks for blobs that are common across the various images (for
example, the Fedora base blob will be the same in all images) to save
rsync time and mirror disk space.

Additionally, we will need a playbook to run this new tool in response
to fedmsgs. We may be able to use Adam Miller's loopabull project to run
such a playbook at the right times.


Conclusion
==========

Thanks for reading, and please respond with any comments or questions
you have about this proposal. I'm happy to clarify any points further,
and if you have any alternative proposals I'd love to hear those as
well.