[RFC] DistGit Container Image namespacing for Layered Image Build Service

[Adam Miller]

Hello all,
    In the Fedora 24 timeframe the Fedora Release Engineering group is
aiming to deliver the Layered Image Build Service[0] to allow Fedora
contributors to build containers images. In the first iteration of
this we're targeting Docker layered image support. Part of this will
be to allow Fedora contributors to maintain Dockerfiles much like we
maintain rpm spec files via DistGit.

However we need to make a couple of decisions about naming conventions
such that we can distinguish between rpms and container images as they
are stored in Fedora DistGit. Also, alternatively we could setup a
completely separate DistGit for container images that is disjoint from
the one used for rpms. I would love feedback on everyone's preferences
between those two options.

If we were to go with the former rather than the latter, we would need
to find a way to "namespace" container images so they can be
determined as different. I've thought about this a lot and I worry
about defining a namespace by some alphanumberic sequence because I
just know that at some point there will end up being a piece of
software in the ecosystem that we want to package as a rpm that will
share this pattern and result in problematic filtering. We could
accept that risk and simply say "this sequence is a reserved word" or
use a special character as the leading character in a DistGit
repository name to signify that it is a container. A special character
is the option I propose.

Proposal:
    In the event that container images (Dockerfiles for now, others in
the future) and rpm spec files are to share the DistGit repositories,
the container images leading character would be a special character.
The decision upon which special character is open to the community at
large to decide but certain ones impose obvious limitations for shell
quoting and the like. For the sake of example I will use the special
character '-' as my marker for a container image DistGit repository.
Also in the example I'm going to use the cockpit-project[1] as it is
already something that is packaged as an rpm[2] in Fedora and
delivered as a container image[3] built from it's Fedora rpm set.

Example:
    Cockpit's rpm spec repo in DistGit would be stored as it is today:
        fedpkg clone cockpit

    Cockpit's Dockerfile repo in DistGit would be stored with a
leading special character:
        fedpkg clone -cockpit

    Here, the leading '-' signifies that it is the container image
DistGit repository which would make it an illegal name for an rpm by
the N-V-R convention.

The technical details of this are all open to change so feel free to
tell me I'm crazy and present a new idea and I'm happy to pursue it.

Looking forward to feedback.

Thank you,
-AdamM

[0] - https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service
[1] - http://cockpit-project.org/
[2] - https://apps.fedoraproject.org/packages/cockpit
[3] - https://hub.docker.com/r/cockpit/ws/

[Josh Boyer]

On Tue, Nov 10, 2015 at 1:08 PM, Adam Miller
<maxamillion(a)fedoraproject.org&gt; wrote:
>  Hello all,
>      In the Fedora 24 timeframe the Fedora Release Engineering group is
>  aiming to deliver the Layered Image Build Service[0] to allow Fedora
>  contributors to build containers images. In the first iteration of
>  this we're targeting Docker layered image support. Part of this will
>  be to allow Fedora contributors to maintain Dockerfiles much like we
>  maintain rpm spec files via DistGit.
>
>  However we need to make a couple of decisions about naming conventions
>  such that we can distinguish between rpms and container images as they
>  are stored in Fedora DistGit. Also, alternatively we could setup a
>  completely separate DistGit for container images that is disjoint from
>  the one used for rpms. I would love feedback on everyone's preferences
>  between those two options.
>
>  If we were to go with the former rather than the latter, we would need
>  to find a way to "namespace" container images so they can be
>  determined as different. I've thought about this a lot and I worry
>  about defining a namespace by some alphanumberic sequence because I
>  just know that at some point there will end up being a piece of
>  software in the ecosystem that we want to package as a rpm that will
>  share this pattern and result in problematic filtering. We could
>  accept that risk and simply say "this sequence is a reserved word" or
>  use a special character as the leading character in a DistGit
>  repository name to signify that it is a container. A special character
>  is the option I propose.
>
>  Proposal:
>      In the event that container images (Dockerfiles for now, others in
>  the future) and rpm spec files are to share the DistGit repositories,
>  the container images leading character would be a special character.
>  The decision upon which special character is open to the community at
>  large to decide but certain ones impose obvious limitations for shell
>  quoting and the like. For the sake of example I will use the special
>  character '-' as my marker for a container image DistGit repository.
>  Also in the example I'm going to use the cockpit-project[1] as it is
>  already something that is packaged as an rpm[2] in Fedora and
>  delivered as a container image[3] built from it's Fedora rpm set.
>
>  Example:
>      Cockpit's rpm spec repo in DistGit would be stored as it is today:
>          fedpkg clone cockpit
>
>      Cockpit's Dockerfile repo in DistGit would be stored with a
>  leading special character:
>          fedpkg clone -cockpit
>
>      Here, the leading '-' signifies that it is the container image
>  DistGit repository which would make it an illegal name for an rpm by
>  the N-V-R convention.
>
>  The technical details of this are all open to change so feel free to
>  tell me I'm crazy and present a new idea and I'm happy to pursue it. 
How much collaboration do we expect around the dockerfiles?  I would
think significantly more than an RPM spec file.  With an RPM, there
are only a handful of ways to build the software and adding patches
isn't a high-frequency/multi-maintainer event (usually).

If we expect dockerfiles to be more dynamic, then I would prefer a
separate git instance.  Then we don't have to worry about collisions
and hacks.  We can also set up a separate set of ACLs around who is
allow to commit to them, etc.

josh

[Adam Miller]

On Tue, Nov 10, 2015 at 12:35 PM, Josh Boyer <jwboyer(a)fedoraproject.org&gt; wrote:
>  On Tue, Nov 10, 2015 at 1:08 PM, Adam Miller
>  <maxamillion(a)fedoraproject.org&gt; wrote:
> > Hello all,
> >     In the Fedora 24 timeframe the Fedora Release Engineering group is
> > aiming to deliver the Layered Image Build Service[0] to allow Fedora
> > contributors to build containers images. In the first iteration of
> > this we're targeting Docker layered image support. Part of this will
> > be to allow Fedora contributors to maintain Dockerfiles much like we
> > maintain rpm spec files via DistGit.
> >
> > However we need to make a couple of decisions about naming conventions
> > such that we can distinguish between rpms and container images as they
> > are stored in Fedora DistGit. Also, alternatively we could setup a
> > completely separate DistGit for container images that is disjoint from
> > the one used for rpms. I would love feedback on everyone's preferences
> > between those two options.
> >
> > If we were to go with the former rather than the latter, we would need
> > to find a way to "namespace" container images so they can be
> > determined as different. I've thought about this a lot and I worry
> > about defining a namespace by some alphanumberic sequence because I
> > just know that at some point there will end up being a piece of
> > software in the ecosystem that we want to package as a rpm that will
> > share this pattern and result in problematic filtering. We could
> > accept that risk and simply say "this sequence is a reserved word" or
> > use a special character as the leading character in a DistGit
> > repository name to signify that it is a container. A special character
> > is the option I propose.
> >
> > Proposal:
> >     In the event that container images (Dockerfiles for now, others in
> > the future) and rpm spec files are to share the DistGit repositories,
> > the container images leading character would be a special character.
> > The decision upon which special character is open to the community at
> > large to decide but certain ones impose obvious limitations for shell
> > quoting and the like. For the sake of example I will use the special
> > character '-' as my marker for a container image DistGit repository.
> > Also in the example I'm going to use the cockpit-project[1] as it is
> > already something that is packaged as an rpm[2] in Fedora and
> > delivered as a container image[3] built from it's Fedora rpm set.
> >
> > Example:
> >     Cockpit's rpm spec repo in DistGit would be stored as it is today:
> >         fedpkg clone cockpit
> >
> >     Cockpit's Dockerfile repo in DistGit would be stored with a
> > leading special character:
> >         fedpkg clone -cockpit
> >
> >     Here, the leading '-' signifies that it is the container image
> > DistGit repository which would make it an illegal name for an rpm by
> > the N-V-R convention.
> >
> > The technical details of this are all open to change so feel free to
> > tell me I'm crazy and present a new idea and I'm happy to pursue it.
>
>  How much collaboration do we expect around the dockerfiles?  I would
>  think significantly more than an RPM spec file.  With an RPM, there
>  are only a handful of ways to build the software and adding patches
>  isn't a high-frequency/multi-maintainer event (usually).
>  
I'm honestly not entirely sure. I think that will just kind of
organically happen. I would like to try and get a sense of what people
would like to come out of it though so that we can cater to the
preferred workload.

>  If we expect dockerfiles to be more dynamic, then I would prefer a
>  separate git instance.  Then we don't have to worry about collisions
>  and hacks.  We can also set up a separate set of ACLs around who is
>  allow to commit to them, etc. 
That's definitely an option. I'm open to either but we can do that now
with what's surrounding DistGit.

-AdamM

>  josh
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

[Mathieu Bridon]

On Tue, 2015-11-10 at 12:08 -0600, Adam Miller wrote:
>  Hello all,
>      In the Fedora 24 timeframe the Fedora Release Engineering group
>  is
>  aiming to deliver the Layered Image Build Service[0] to allow Fedora
>  contributors to build containers images. In the first iteration of
>  this we're targeting Docker layered image support. Part of this will
>  be to allow Fedora contributors to maintain Dockerfiles much like we
>  maintain rpm spec files via DistGit.
>  
>  However we need to make a couple of decisions about naming
>  conventions
>  such that we can distinguish between rpms and container images as
>  they
>  are stored in Fedora DistGit. Also, alternatively we could setup a
>  completely separate DistGit for container images that is disjoint
>  from the one used for rpms. I would love feedback on everyone's
>  preferences between those two options. 
You could set it up in /srv/git/containers, in parallel to the current
/srv/git/rpms ?

That nicely avoids naming collisions.

It also should be quite asy now that dist-git is in ansible, you could
probably just factor out the path in the roles as a variable?

>      Cockpit's Dockerfile repo in DistGit would be stored with a
>  leading special character:
>          fedpkg clone -cockpit 
With the "-" as a leading character, you open yourself to nice errors
with the fact that it could be recognized as options. So you'd want to
type: fedpkg clone -- -cockpit

Alternatively, if you separate the two distgits, you could have a
fedcontainer tool, which would just be fedpkg but with its
configuration pointing to the container distgit, the container build
service, etc...

Or maybe just a: fedpkg --container clone cockpit ?

Most of the code (at least the one handling dist-git) could probably be
reused without any difference.

All in all, separating them seems much simpler to me.

You'd have a bit of work up front (to adapt the Ansible playbooks and
fedpkg slightly), but you pretty much avoid a whole class of issues
with naming collisions, cli options, and future surprises.


-- 
Mathieu

[Adam Miller]

On Tue, Nov 10, 2015 at 12:57 PM, Mathieu Bridon <bochecha(a)daitauha.fr&gt; wrote:
>  On Tue, 2015-11-10 at 12:08 -0600, Adam Miller wrote:
> > Hello all,
> >     In the Fedora 24 timeframe the Fedora Release Engineering group
> > is
> > aiming to deliver the Layered Image Build Service[0] to allow Fedora
> > contributors to build containers images. In the first iteration of
> > this we're targeting Docker layered image support. Part of this will
> > be to allow Fedora contributors to maintain Dockerfiles much like we
> > maintain rpm spec files via DistGit.
> >
> > However we need to make a couple of decisions about naming
> > conventions
> > such that we can distinguish between rpms and container images as
> > they
> > are stored in Fedora DistGit. Also, alternatively we could setup a
> > completely separate DistGit for container images that is disjoint
> > from the one used for rpms. I would love feedback on everyone's
> > preferences between those two options.
>
>  You could set it up in /srv/git/containers, in parallel to the current
>  /srv/git/rpms ?
>
>  That nicely avoids naming collisions.
>
>  It also should be quite asy now that dist-git is in ansible, you could
>  probably just factor out the path in the roles as a variable?
>
> >     Cockpit's Dockerfile repo in DistGit would be stored with a
> > leading special character:
> >         fedpkg clone -cockpit
>
>  With the "-" as a leading character, you open yourself to nice errors
>  with the fact that it could be recognized as options. So you'd want to
>  type: fedpkg clone -- -cockpit 
Yeah, all special characters have special oddities. Some of them we
have to consider from many targets and not just the shell so it's
probably not ideal.

>  Alternatively, if you separate the two distgits, you could have a
>  fedcontainer tool, which would just be fedpkg but with its
>  configuration pointing to the container distgit, the container build
>  service, etc...
>
>  Or maybe just a: fedpkg --container clone cockpit ? 
Definitely something along these lines is what we'd want to do. I like
the idea. +1

>  Most of the code (at least the one handling dist-git) could probably be
>  reused without any difference. 
Yes, it's likely something we'd want to maintain in the same code base
and have one just be a light wrapper of the other with the different
code paths as necessary.

>  All in all, separating them seems much simpler to me.
>
>  You'd have a bit of work up front (to adapt the Ansible playbooks and
>  fedpkg slightly), but you pretty much avoid a whole class of issues
>  with naming collisions, cli options, and future surprises.
>  
Agreed.

+1

-AdamM

>  --
>  Mathieu
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

[Tomas Tomecek]

Quoting Adam Miller (2015-11-10 19:08:25)
>      Cockpit's Dockerfile repo in DistGit would be stored with a
>  leading special character:
>          fedpkg clone -cockpit
>   
How about

$ fedpkg clone cockpit-docker


Pros

 * no need to change infrastructure

 * easy to use

 * repo name suggests content

Cons

 * not a different namespace (in fact same namespace as rpms)

 * someone can do `fedpkg clone cockpit-docker && cd cockpit-docker &&
fedpkg
   build` and get scared that it doesn't work

   * `build` could be aware of Dockerfile and run `container-build` actually

   * error could suggest using different build method


What's the way of building (qcow, docker base) images? Could that approach be
resued? (meaning, what's the way of storing kickstart files atm?)


~~
Tomáš Tomeček
Software Engineer
Developer Experience
UTC+2 (CEST)

[Adam Miller]

On Thu, Nov 12, 2015 at 4:57 AM, Tomas Tomecek <ttomecek(a)redhat.com&gt; wrote:
>  Quoting Adam Miller (2015-11-10 19:08:25)
> >     Cockpit's Dockerfile repo in DistGit would be stored with a
> > leading special character:
> >         fedpkg clone -cockpit
> >
>
>  How about
>
>  $ fedpkg clone cockpit-docker
>
>
>  Pros
>
>   * no need to change infrastructure
>
>   * easy to use
>
>   * repo name suggests content 
I do like this, but I almost think if we were going to go this route
we should have a policy around naming a container in DistGit with a
prefix. I'd then suggest something like "ctr" to mean
"container":

$ fedpkg clone ctr-cockpit

We would however need to request some kind of gating mechanism in
pkgdb to make whatever the prefix is be a "reserved word" that
wouldn't become the name of package in the future or else it would
cause havok.

Another thing that was mentioned to me was if we have separate DistGit
instances containing packages and containers named the same thing that
koji would be unhappy so that's definitely something we need to keep
in mind.

>  Cons
>
>   * not a different namespace (in fact same namespace as rpms)
>
>   * someone can do `fedpkg clone cockpit-docker && cd cockpit-docker &&
> fedpkg
>     build` and get scared that it doesn't work
>
>     * `build` could be aware of Dockerfile and run `container-build` actually
>
>     * error could suggest using different build method 
I'd vote for a detection of Dockerfile without a spec file error out
and suggest an alternative build method.

>  What's the way of building (qcow, docker base) images? Could that approach be
>  resued? (meaning, what's the way of storing kickstart files atm?) 
Currently the kickstarts are flattened and passed to the koji
image-build command as a parameter, they are all stored as part of the
spin-kickstarts[0] fedorahosted space and aren't really managed at all
like DistGit.

-AdamM

[0] - https://fedorahosted.org/spin-kickstarts/

>  ~~
>  Tomáš Tomeček
>  Software Engineer
>  Developer Experience
>  UTC+2 (CEST)

[Ray Strode]

Hi,

On Tue, Nov 10, 2015 at 1:08 PM, Adam Miller
<maxamillion(a)fedoraproject.org&gt; wrote:
>  If we were to go with the former rather than the latter, we would
> need
>  to find a way to "namespace" container images so they can be
>  determined as different. I've thought about this a lot and I worry
>  about defining a namespace by some alphanumberic sequence because I
>  just know that at some point there will end up being a piece of
>  software in the ecosystem that we want to package as a rpm that will
>  share this pattern and result in problematic filtering. We could
>  accept that risk and simply say "this sequence is a reserved word" or
>  use a special character as the leading character in a DistGit
>  repository name to signify that it is a container. 
git repositories normally use '/' to separate namespaces, so i'd propose

$ fedpkg clone containers/cockpit

and maybe add support for

$ fedpkg clone srpms/cockpit

at the same time.

This has the added benefit that cgit will automatically filter docker
reposistories when you visit, e.g,

http://pkgs.fedoraproject.org/cgit/containers/


--Ray

[Petr Spacek]

On 12.11.2015 21:01, Ray Strode wrote:
>  Hi,
>  
>  On Tue, Nov 10, 2015 at 1:08 PM, Adam Miller
>  <maxamillion(a)fedoraproject.org&gt; wrote:
> > If we were to go with the former rather than the latter, we would need
> > to find a way to "namespace" container images so they can be
> > determined as different. I've thought about this a lot and I worry
> > about defining a namespace by some alphanumberic sequence because I
> > just know that at some point there will end up being a piece of
> > software in the ecosystem that we want to package as a rpm that will
> > share this pattern and result in problematic filtering. We could
> > accept that risk and simply say "this sequence is a reserved word" or
> > use a special character as the leading character in a DistGit
> > repository name to signify that it is a container.
>  
>  git repositories normally use '/' to separate namespaces, so i'd propose
>  
>  $ fedpkg clone containers/cockpit
>  
>  and maybe add support for
>  
>  $ fedpkg clone srpms/cockpit
>  
>  at the same time.
>  
>  This has the added benefit that cgit will automatically filter docker
>  reposistories when you visit, e.g,
>  
>  http://pkgs.fedoraproject.org/cgit/containers/ 
+1, sounds reasonable.

-- 
Petr Spacek  @  Red Hat

[Ralph Bean]

On Thu, Nov 12, 2015 at 03:01:49PM -0500, Ray Strode wrote:
>  Hi,
>  
>  On Tue, Nov 10, 2015 at 1:08 PM, Adam Miller
>  <maxamillion(a)fedoraproject.org&gt; wrote:
>  > If we were to go with the former rather than the latter, we would need
>  > to find a way to "namespace" container images so they can be
>  > determined as different. I've thought about this a lot and I worry
>  > about defining a namespace by some alphanumberic sequence because I
>  > just know that at some point there will end up being a piece of
>  > software in the ecosystem that we want to package as a rpm that will
>  > share this pattern and result in problematic filtering. We could
>  > accept that risk and simply say "this sequence is a reserved word" or
>  > use a special character as the leading character in a DistGit
>  > repository name to signify that it is a container.
>  
>  git repositories normally use '/' to separate namespaces, so i'd propose
>  
>  $ fedpkg clone containers/cockpit
>  
>  and maybe add support for
>  
>  $ fedpkg clone srpms/cockpit
>  
>  at the same time.
>  
>  This has the added benefit that cgit will automatically filter docker
>  reposistories when you visit, e.g,
>  
>  http://pkgs.fedoraproject.org/cgit/containers/ 
I like this too.  Here are three thoughts:

Perhaps, we use 'dockerfiles' for the prefix instead of 'containers',
because presumably there will be some whole new way to build
containers in 2017, and we'll need to keep our dockerfiles/ repos
separate from our awesomefiles/ repos.

We could also use this opportunity to move the kickstarts (another
input to koji builds) away from https://fedorahosted.org/spin-kickstarts
and over to dist-git as well, with a namespace like 'kickstarts/kde'
and 'kickstarts/lxde'.

The existing rpm content could be moved to a 'specfiles/' namespace
(or maybe 'srpms/'?) but we could further add some apache httpd rules that
respond with a redirect to the 'srpms/' namespace if the requests to
the base namespaceless-namespace level are met with a 404. -- "when in
doubt, default to srpms/".  That might help keep some of our existing
tools working as-is without too much catastrophe.

[Adam Miller]

On Fri, Nov 13, 2015 at 3:47 PM, Ralph Bean <rbean(a)redhat.com&gt; wrote:
>  On Thu, Nov 12, 2015 at 03:01:49PM -0500, Ray Strode wrote:
> > Hi,
> >
> > On Tue, Nov 10, 2015 at 1:08 PM, Adam Miller
> > <maxamillion(a)fedoraproject.org&gt; wrote:
> > > If we were to go with the former rather than the latter, we would need
> > > to find a way to "namespace" container images so they can be
> > > determined as different. I've thought about this a lot and I worry
> > > about defining a namespace by some alphanumberic sequence because I
> > > just know that at some point there will end up being a piece of
> > > software in the ecosystem that we want to package as a rpm that will
> > > share this pattern and result in problematic filtering. We could
> > > accept that risk and simply say "this sequence is a reserved word" or
> > > use a special character as the leading character in a DistGit
> > > repository name to signify that it is a container.
> >
> > git repositories normally use '/' to separate namespaces, so i'd propose
> >
> > $ fedpkg clone containers/cockpit
> >
> > and maybe add support for
> >
> > $ fedpkg clone srpms/cockpit
> >
> > at the same time.
> >
> > This has the added benefit that cgit will automatically filter docker
> > reposistories when you visit, e.g,
> >
> > http://pkgs.fedoraproject.org/cgit/containers/
>
>  I like this too.  Here are three thoughts:
>
>  Perhaps, we use 'dockerfiles' for the prefix instead of 'containers',
>  because presumably there will be some whole new way to build
>  containers in 2017, and we'll need to keep our dockerfiles/ repos
>  separate from our awesomefiles/ repos.
>
>  We could also use this opportunity to move the kickstarts (another
>  input to koji builds) away from https://fedorahosted.org/spin-kickstarts
>  and over to dist-git as well, with a namespace like 'kickstarts/kde'
>  and 'kickstarts/lxde'.
>
>  The existing rpm content could be moved to a 'specfiles/' namespace
>  (or maybe 'srpms/'?) but we could further add some apache httpd rules that
>  respond with a redirect to the 'srpms/' namespace if the requests to
>  the base namespaceless-namespace level are met with a 404. -- "when in
>  doubt, default to srpms/".  That might help keep some of our existing
>  tools working as-is without too much catastrophe. 
+1

I'm a big fan of where this is going.

Maybe we should draft a proposal around this (maybe some notes in
gobby?) and then start a new thread under a new name to generate more
general discussion around this in the context of being able to deliver
many different kinds of artifact for Fedora.Next. My only fear of
continuing this topic thread is that folks who aren't interested in
containers or docker might not be tuned in because of the subject
line.

Thoughts?

-AdamM

>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

[Brian C. Lane]

On Tue, Nov 10, 2015 at 12:08:25PM -0600, Adam Miller wrote:
>  Hello all,
>      In the Fedora 24 timeframe the Fedora Release Engineering group is
>  aiming to deliver the Layered Image Build Service[0] to allow Fedora
>  contributors to build containers images. In the first iteration of
>  this we're targeting Docker layered image support. Part of this will
>  be to allow Fedora contributors to maintain Dockerfiles much like we
>  maintain rpm spec files via DistGit. 
So you're talking about one docker file per package, right? Or are these
going to pull from multiple packages and have the docker container be a
different level of object? Or both?

If you're talking about a 1:1 package:docker mapping then the only thing
that makes sense to me is to have them live in the current system next
to the .spec files. Anything else will easily introduce divergence from
the rpm packages.

I also see that 'container packager' and 'rpm packager' are used in the
build service page. I don't think these should be separate roles,
everyone should be a packager. If you have one role working on container
building for a package and another who only does rpm packaging we're
going to, again, end up with things diverging.

Some things I'd suggest (given I don't know the answers to the above
questions):

1. All bits have to come from koji built rpms. No rebuilding of bits
for docker, it just installs existing bits.

2. docker files for packages live in dist-git next to the package spec
file.

3. Any meta-containers collecting using packages are also an rpm
package, following whatever naming convention gets decided on.
fedora-docker-foo would make sense to me.

4. package level docker files are maintained by the same people who maintain
the package.

-- 
Brian C. Lane | Anaconda Team | IRC: bcl #anaconda | Port Orchard, WA (PST8PDT)

[Adam Miller]

On Fri, Nov 13, 2015 at 5:14 PM, Brian C. Lane <bcl(a)redhat.com&gt; wrote:
>  On Tue, Nov 10, 2015 at 12:08:25PM -0600, Adam Miller wrote:
> > Hello all,
> >     In the Fedora 24 timeframe the Fedora Release Engineering group is
> > aiming to deliver the Layered Image Build Service[0] to allow Fedora
> > contributors to build containers images. In the first iteration of
> > this we're targeting Docker layered image support. Part of this will
> > be to allow Fedora contributors to maintain Dockerfiles much like we
> > maintain rpm spec files via DistGit.
>
>  So you're talking about one docker file per package, right? Or are these
>  going to pull from multiple packages and have the docker container be a
>  different level of object? Or both? 
Both, it's mostly at the discretion of the person attempting to
deliver something in a container. Some may be simple "I added httpd on
top of the base Fedora Docker image" and others might be "I put all of
ownCloud into a container so you can just run the whole thing with
this one command"

That being said, I'm not married to the concept. It was just the idea
going into it. If the general consensus among the community is "one
Dockerfile to one package" we could go that route.

>  If you're talking about a 1:1 package:docker mapping then the only thing
>  that makes sense to me is to have them live in the current system next
>  to the .spec files. Anything else will easily introduce divergence from
>  the rpm packages. 
Absolutely, so far there has been the expectation that there will be
divergence because of the more "dynamic" nature of what containers
bring to the table.

>  I also see that 'container packager' and 'rpm packager' are used in the
>  build service page. I don't think these should be separate roles,
>  everyone should be a packager. If you have one role working on container
>  building for a package and another who only does rpm packaging we're
>  going to, again, end up with things diverging. 
I believe they will diverge naturally as time goes on. Also, I don' t
think everyone who has rpm packaging expertise is well versed in the
ways of Docker and vice versa.

>  Some things I'd suggest (given I don't know the answers to the above
>  questions):
>
>  1. All bits have to come from koji built rpms. No rebuilding of bits
>  for docker, it just installs existing bits. 
That is the plan, but configuration can take place inside the context
of the container image build process to allow for single-command
execution.

>  2. docker files for packages live in dist-git next to the package spec
>  file. 
We can, pending the agreement that's all we as a community/project use
Dockerfiles for. The original idea going in to this though is that we
could extend this to provide a more full featured solution for
services that would need more than one package. Even simple things
like httpd modules (mod_wsgi, mod_php, etc) would technically ship
more rpms than they produce in the docker image.

Also in the distant future (out of scope for the original offering),
we'd like to look at delivering multi-container apps via Nulecule[0]
specifications.

>  3. Any meta-containers collecting using packages are also an rpm
>  package, following whatever naming convention gets decided on.
>  fedora-docker-foo would make sense to me. 
I'm not sure I follow what you mean here, why would something that
only comes as a Dockerfile also be a rpm?

>  4. package level docker files are maintained by the same people who maintain
>  the package.
>  
This certainly makes sense if we end up going the package:container 1:1 route.

-AdamM

[0] - https://github.com/projectatomic/nulecule

>  --
>  Brian C. Lane | Anaconda Team | IRC: bcl #anaconda | Port Orchard, WA (PST8PDT)
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

[Stanislav Ochotnicky]

On Fri 13 Nov 2015 06:34:21 PM EST Adam Miller <maxamillion(a)fedoraproject.org&gt;
wrote:
> > 4. package level docker files are maintained by the same people
> who maintain
> > the package.
> >
>
>  This certainly makes sense if we end up going the package:container 1:1 route. 
This assumption is not correct IMO. One docker image generally
provides some functionality that can be used in different ways. It might
be OK for basic httpd/mysql/postgres setup but I think inevitably you'll
have people who will want to provide more usable/complete docker image
which will have all of them integrated without need of involving
Kubernetes or similar technologies.


--
Stanislav Ochotnicky <sochotnicky(a)redhat.com&gt;
Business System Analyst, PnT DevOps PMO Team - Brno

PGP: 7B087241
Red Hat Inc.                               http://cz.redhat.com

[Brian C. Lane]

On Fri, Nov 13, 2015 at 05:34:21PM -0600, Adam Miller wrote:
>  On Fri, Nov 13, 2015 at 5:14 PM, Brian C. Lane <bcl(a)redhat.com&gt;
> wrote:
>  > On Tue, Nov 10, 2015 at 12:08:25PM -0600, Adam Miller wrote:
>  >> Hello all,
>  >>     In the Fedora 24 timeframe the Fedora Release Engineering group is
>  >> aiming to deliver the Layered Image Build Service[0] to allow Fedora
>  >> contributors to build containers images. In the first iteration of
>  >> this we're targeting Docker layered image support. Part of this will
>  >> be to allow Fedora contributors to maintain Dockerfiles much like we
>  >> maintain rpm spec files via DistGit.
>  >
>  > So you're talking about one docker file per package, right? Or are these
>  > going to pull from multiple packages and have the docker container be a
>  > different level of object? Or both?
>  
>  Both, it's mostly at the discretion of the person attempting to
>  deliver something in a container. Some may be simple "I added httpd on
>  top of the base Fedora Docker image" and others might be "I put all of
>  ownCloud into a container so you can just run the whole thing with
>  this one command"
>  
>  That being said, I'm not married to the concept. It was just the idea
>  going into it. If the general consensus among the community is "one
>  Dockerfile to one package" we could go that route.
>  
>  >
>  > If you're talking about a 1:1 package:docker mapping then the only thing
>  > that makes sense to me is to have them live in the current system next
>  > to the .spec files. Anything else will easily introduce divergence from
>  > the rpm packages.
>  
>  Absolutely, so far there has been the expectation that there will be
>  divergence because of the more "dynamic" nature of what containers
>  bring to the table. 
Ok, that makes sense. So some packages may also have docker images but
most of this will be more along the lines of a spin for a specific
collections of packages built from our existing rpms.


>  > I also see that 'container packager' and 'rpm
> packager' are used in the
>  > build service page. I don't think these should be separate roles,
>  > everyone should be a packager. If you have one role working on container
>  > building for a package and another who only does rpm packaging we're
>  > going to, again, end up with things diverging.
>  
>  I believe they will diverge naturally as time goes on. Also, I don' t
>  think everyone who has rpm packaging expertise is well versed in the
>  ways of Docker and vice versa. 
I think packagers should be familiar with both if we're going to be
producing both. Current rpm maintainers have a body of knowledge
surrounding their packages and upstreams that should not be lost or
ignored when using them for Docker images.


>  > Some things I'd suggest (given I don't know the answers
> to the above
>  > questions):
>  >
>  > 1. All bits have to come from koji built rpms. No rebuilding of bits
>  > for docker, it just installs existing bits.
>  
>  That is the plan, but configuration can take place inside the context
>  of the container image build process to allow for single-command
>  execution. 
Sure, that's the big benefit of Docker, reasonable configuration of
existing packages makes sense. As long as that's where it stops and the
binaries are the same as what we produce with the rpms. I don't want to
see us rebuilding things for the Docker container and using different
binaries or pulling in external dependencies that aren't already
packaged as an rpm.


>  > 2. docker files for packages live in dist-git next to the
> package spec
>  > file.
>  
>  We can, pending the agreement that's all we as a community/project use
>  Dockerfiles for. The original idea going in to this though is that we
>  could extend this to provide a more full featured solution for
>  services that would need more than one package. Even simple things
>  like httpd modules (mod_wsgi, mod_php, etc) would technically ship
>  more rpms than they produce in the docker image.
>  
>  Also in the distant future (out of scope for the original offering),
>  we'd like to look at delivering multi-container apps via Nulecule[0]
>  specifications.
>  
>  >
>  > 3. Any meta-containers collecting using packages are also an rpm
>  > package, following whatever naming convention gets decided on.
>  > fedora-docker-foo would make sense to me.
>  
>  I'm not sure I follow what you mean here, why would something that
>  only comes as a Dockerfile also be a rpm? 
What I'm thinking is that dist-git doesn't have to always produce an
rpm. If the package has both, then a fedpkg build produces both, and we
can add fedpkg build --rpm and fedpkg build --docker. If a package only
has a docker file then it only produces a docker image. This reuses the
tooling we are all familiar with, and keeps everything in the same
place. Use a container specific naming convention and it can be expanded
to support other containers as they become popular.


>  > 4. package level docker files are maintained by the same people
> who maintain
>  > the package.
>  >
>  
>  This certainly makes sense if we end up going the package:container
>  1:1 route. 
I think it makes sense either way. Much of the packaging guidelines
should still apply to creating containers.

-- 
Brian C. Lane | Anaconda Team | IRC: bcl #anaconda | Port Orchard, WA (PST8PDT)

[Adam Miller]

Hello all,
    After consideration and discussion of feedback from this thread
with members of the Fedora Infrastructure and Fedora Apps team
(special thanks to pingou and threebean), the following is the
proposal we'd like to move forward with:

DistGit Restructure/Namespace
  - Move all current rpm packages that live under repositories/ to
live under repositories/rpms/
  - Create new namespaces for other things (i.e. - repositories/docker/)


Current DistGit layout:
    /srv/git/repositories/<pkg>.git
future:
    -> /srv/git/repositories/rpms/<pkg>.git
        (symlinked to the old location for a migration time period)
        => At one point, will break `git pull` if the remote url isn't fixed
    -> /srv/git/repositories/docker/<pkg>.git
    => /srv/git/repositories/<future_new_artifact>/<pkg>.git

This will require updates to different aspects of the Fedora
Apps/Infrastructure over time which will be addressed in phases.
Everything will continue to function with business as usual during the
transition periods with the symlinks in place for backwards
compatibility.

In the future, users of fedpkg can do the following:

    fedpkg clone <namespace>/<pkg_name>

    Examples:

        fedpkg clone rpms/foo
        fedpkg clone docker/bar

In the event the namespace is omitted, fedpkg will default to rpms/foo
in order to maintain function-alike backwards compatibility.

If this proposal is agreed upon by general consensus, the teams
involved will begin to scope out what systems need what modifications
and begin the migration, with updates along the way (in case folks are
interested).

I look forward to feedback, thank you.

-AdamM