Containers: we're asking for ponies, so here's a suggestion for the Minimum Viable (Modular) Pony Show

[Matthew Miller]

At DevConf.cz, I said that 2016 needed to be the year of the _show_
part of "show & tell" for what we started with the "rings"
proposal and
now what we're calling, vaguely, "modularity".

Some of that is scary, because anything really new always is. It's
important that we do this in a way that *both* brings useful new
functionality _and_ doesn't break what we have.

In the commercial world, there's a concept of "minimum viable product" --
not done, not perfect, but what it takes to actually start solving problems
and providing value. In Fedora, we're not in business in the same way, but
some of the same can apply. So, here's my suggestions for a "minimum viable
demo" for what a container-based module might look like in Fedora.

Particularly, when I've spoken to people about delivering parts of Fedora as
containers, the part about tracking security and proving updates becames
disturbingly hand-wavy, with "yeah, of course, we'll do something to cover
that". I think to be successful, we need to make sure that this is addressed
early. So, this proposal for a demo I'd like to see.

I've broken it down into three parts of increasing complexity and
usefulness: starting with "and I want a pony", then moving up to
"actually,
a unicorn", and then "also, could you put wings on that?"

So, first:


The Minimum Viable Pony
-----------------------


This is a basic web server module delivered as a container.

Part A: the basic module:

  * Answers on port 80 and serves out static web pages.

  * Content comes from /var/www/html... or another container, or whatever.

  * Doesn't even need to be configurable. It's a demo.

  * It's composed entirely from RPMs already existing in Fedora.


Part B: DNF Plugin to list CVEs

  * Works like
https://docs.fedoraproject.org/en-US/Fedora/17/html/Security_Guide/sect-S...

    - lists all RPMs on host system where a security update is available
      - can list by CVE
      - also by Bugzilla
      - also with link to update notice
      - with various breakdowns by severity level

    - **additionally** lists "modules" which need updates (with same links)
      - has option to list which specific RPMs within the module have the
        problem.


So that's a good start. As a sysadmin, I'm feeling a little less scared
about. But we also need...


To Take It to Unicorn Level
---------------------------

It's all well and good to fix the problem, but we also need to be able to
fix it. So, the next level is:

* Image is built in Fedora's new Layered Image Build Service
  https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service

* If there is a security vulnerability in any RPM that is in the image, it
  is automatically rebuilt and made available somewhere.

* And automatically tested!

  - Here, just a basic test that makes sure that input web content comes
    out. It's a proof-of-concept so no real in-depth testing is needed.

* And, then, a dnf plugin which actually downloads and applies the update.

Now, not only do we have comforting tooling (minimizing change pain), we
also have a real, practical improvement (there will be cake!). In the
current system, while we do hands-on testing of individual updates, they
aren't necessarily tested in the application to which they apply. For
example, if the web server here is Nginx and the security flaw in the SSL
library, current update testing does not guarantee that the update has been
tested *for that use*. But this will. Plus, in addition to whatever human
oversight, there will be an _automatic_ test.

So that's pretty good. But finally, to get to the glorious future, I'd also
like to see this unicorn become also a pegasus (my daughter tells me that
this makes it an "alicorn")...



Taking Flight
-------------

Same as above, but with two new things:

New Thing One:

  * Instead of RPMs, a module which uses some non-RPM upstream packaging
    format as part of its composition
    - Ruby gem, Python egg or wheel, Java jar... whatever

  * And the DNF list-security command still works
    - and the verbose flag now tells us about the CVEs or whatever for the
      non-RPM package

New Thing Two:

  * There is enough metadata and intelligence such that when the new
    container is updated, that update happens without downtime. This may
    involve multiple containers.

  * If the update fails locally for some reason (despite the best efforts at
    upstream testing), it's rolled back automatically



From here, I can imagine all sorts of other mythical creatures. But, you
know, for a start. What do you think?

As a user/sysadmin, is there something more you'd like to see? If
you're skeptical about containers, would this make you feel better? If
not, what might?

If you're excited about them, does this seem like a useful start?
What's missing?

And, of course, if you're a developer working on this stuff, can I have
ponies please?


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Šimon Lukašík]

On 02/08/2016 03:53 PM, Matthew Miller wrote:
>  At DevConf.cz, I said that 2016 needed to be the year of the _show_
>  part of "show & tell" for what we started with the "rings"
> proposal and
>  now what we're calling, vaguely, "modularity".
>
>  Some of that is scary, because anything really new always is. It's
>  important that we do this in a way that *both* brings useful new
>  functionality _and_ doesn't break what we have.
>
>  In the commercial world, there's a concept of "minimum viable product" --
>  not done, not perfect, but what it takes to actually start solving problems
>  and providing value. In Fedora, we're not in business in the same way, but
>  some of the same can apply. So, here's my suggestions for a "minimum viable
>  demo" for what a container-based module might look like in Fedora.
>
>  Particularly, when I've spoken to people about delivering parts of Fedora as
>  containers, the part about tracking security and proving updates becames
>  disturbingly hand-wavy, with "yeah, of course, we'll do something to cover
>  that". I think to be successful, we need to make sure that this is addressed
>  early. So, this proposal for a demo I'd like to see.
>
>  I've broken it down into three parts of increasing complexity and
>  usefulness: starting with "and I want a pony", then moving up to
> "actually,
>  a unicorn", and then "also, could you put wings on that?"
>
>  So, first:
>
>
>  The Minimum Viable Pony
>  -----------------------
>
>
>  This is a basic web server module delivered as a container.
>
>  Part A: the basic module:
>
>     * Answers on port 80 and serves out static web pages.
>
>     * Content comes from /var/www/html... or another container, or whatever.
>
>     * Doesn't even need to be configurable. It's a demo.
>
>     * It's composed entirely from RPMs already existing in Fedora.
>
>
>  Part B: DNF Plugin to list CVEs
>
>     * Works like
> https://docs.fedoraproject.org/en-US/Fedora/17/html/Security_Guide/sect-S...
>
>       - lists all RPMs on host system where a security update is available
>         - can list by CVE
>         - also by Bugzilla
>         - also with link to update notice
>         - with various breakdowns by severity level
>
>       - **additionally** lists "modules" which need updates (with same links)
>         - has option to list which specific RPMs within the module have the
>           problem.
>  
We are already able to do this kind of stuff with OpenSCAP. ( 
http://www.open-scap.org ).

Thing is just we have slightly different approach. Let me explain.

Running dnf/yum plugin is often regarded as a partial solution. It may 
be good enough for Fedora users. However, there are still use-cases. 
Where dnf/yum plugin may not be good enough:
  * disconnected system (a container without internet access)
  * cloned remote repositories (latest Fedora contains bugfixes, but my 
custom repo may not)
  * disabled local repositories (software was installed to the 
container, but the repofile is missing for some reason).

So, we try to examine what is actually installed on the system. Here is 
one example 
https://www.open-scap.org/resources/documentation/security-compliance-of-...

During the last months guys have integrated this solution more tightly 
with atomic, allowing the scanner to be actually inside super-privileged 
container. 
https://martin.preisler.me/2015/11/atomic-scan-and-openscap-daemon/

The problem we are currently facing in Fedora is that you need to have a 
data that you feed to the scanner. Hence, this CVE analysis can be done 
only for RHEL, SuSE and Debian systems. So, we are done, once we are 
able to develop plug-in to bodhi to generate the data feed for us.

Do you think this approach is sensible for the ponycorn? :)

>  So that's a good start. As a sysadmin, I'm feeling a little less scared
>  about. But we also need...
>
>
>  To Take It to Unicorn Level
>  ---------------------------
>
>  It's all well and good to fix the problem, but we also need to be able to
>  fix it. So, the next level is:
>
>  * Image is built in Fedora's new Layered Image Build Service
>     https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service
>
>  * If there is a security vulnerability in any RPM that is in the image, it
>     is automatically rebuilt and made available somewhere.
>
>  * And automatically tested!
>
>     - Here, just a basic test that makes sure that input web content comes
>       out. It's a proof-of-concept so no real in-depth testing is needed.
>  
Also some basic security checks should be in place before publishing the 
image. I think this represents good starting point: 
https://github.com/OpenSCAP/scap-security-guide/issues/589 (Don't be 
confused by RHEL7 label, we have the same thing prepared for Fedora in 
upstream).

~š.

>  * And, then, a dnf plugin which actually downloads and applies the
> update.
>
>  Now, not only do we have comforting tooling (minimizing change pain), we
>  also have a real, practical improvement (there will be cake!). In the
>  current system, while we do hands-on testing of individual updates, they
>  aren't necessarily tested in the application to which they apply. For
>  example, if the web server here is Nginx and the security flaw in the SSL
>  library, current update testing does not guarantee that the update has been
>  tested *for that use*. But this will. Plus, in addition to whatever human
>  oversight, there will be an _automatic_ test.
>
>  So that's pretty good. But finally, to get to the glorious future, I'd also
>  like to see this unicorn become also a pegasus (my daughter tells me that
>  this makes it an "alicorn")...
>
>
>
>  Taking Flight
>  -------------
>
>  Same as above, but with two new things:
>
>  New Thing One:
>
>     * Instead of RPMs, a module which uses some non-RPM upstream packaging
>       format as part of its composition
>       - Ruby gem, Python egg or wheel, Java jar... whatever
>
>     * And the DNF list-security command still works
>       - and the verbose flag now tells us about the CVEs or whatever for the
>         non-RPM package
>
>  New Thing Two:
>
>     * There is enough metadata and intelligence such that when the new
>       container is updated, that update happens without downtime. This may
>       involve multiple containers.
>
>     * If the update fails locally for some reason (despite the best efforts at
>       upstream testing), it's rolled back automatically
>
>
>
>   From here, I can imagine all sorts of other mythical creatures. But, you
>  know, for a start. What do you think?
>
>  As a user/sysadmin, is there something more you'd like to see? If
>  you're skeptical about containers, would this make you feel better? If
>  not, what might?
>
>  If you're excited about them, does this seem like a useful start?
>  What's missing?
>
>  And, of course, if you're a developer working on this stuff, can I have
>  ponies please?
>
>  

-- 
~š.

[Matthew Miller]

On Mon, Feb 08, 2016 at 04:37:40PM +0100, Šimon Lukašík wrote:
>  Thing is just we have slightly different approach. Let me explain.
Using OpenSCAP sounds great. Mostly, I wanted to focus on results and
what the user sees rather than implementation details.

The advantages you describe sound exciting, which falls under the
offering-carrots aspect: new functionality that we can give people that
they'll be actively want to switch to. However, I think we also need to
do the "we replaced your favorite coffee brand... and you don't even
know it" part wherever possible. *

So, I really _do_ want the existing command to work, but maybe it's
OpenSCAP underneath.

[...]
>  The problem we are currently facing in Fedora is that you need to
>  have a data that you feed to the scanner. Hence, this CVE analysis
>  can be done only for RHEL, SuSE and Debian systems. So, we are done,
>  once we are able to develop plug-in to bodhi to generate the data
>  feed for us. 
What is needed for that to happen?

>  Do you think this approach is sensible for the ponycorn? :) 
Yes, with the above notes.

Man, I missed a really good opportunity to call this message an "RFP".
:)



* I need to unify this metaphor... it's carrots and coffee rather than
carrots and sticks, since we have no sticks. But on the other hand, I
don't think horses like coffee.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Matthew Miller]

On Mon, Feb 08, 2016 at 11:14:44AM -0500, Matthew Miller wrote:
>  > Thing is just we have slightly different approach. Let me
> explain.
>  So, I really _do_ want the existing command to work, but maybe it's
>  OpenSCAP underneath. 
Which, by the way, is what Langdon suggested when I talked to him about
this at a DevConf dinner the other night. 

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Šimon Lukašík]

On 02/08/2016 05:14 PM, Matthew Miller wrote:
>  On Mon, Feb 08, 2016 at 04:37:40PM +0100, Šimon Lukašík wrote:
> > Thing is just we have slightly different approach. Let me explain.
>
>  Using OpenSCAP sounds great. Mostly, I wanted to focus on results and
>  what the user sees rather than implementation details.
>
>  The advantages you describe sound exciting, which falls under the
>  offering-carrots aspect: new functionality that we can give people that
>  they'll be actively want to switch to. However, I think we also need to
>  do the "we replaced your favorite coffee brand... and you don't even
>  know it" part wherever possible. * 
I always notice when my favorite coffee shops changes a brand!! However 
you are right, most people won't. :)

>  So, I really _do_ want the existing command to work, but maybe it's
>  OpenSCAP underneath.
>  
Do you think that this functionality should be integrated with existing 
dnf commands?

That would be beneficial for some, however we are trying to make the 
use-case even easier then before. You perhaps do not want to turn on all 
the containers and mount all the images and run the command, right?

>  [...]
> > The problem we are currently facing in Fedora is that you need to
> > have a data that you feed to the scanner. Hence, this CVE analysis
> > can be done only for RHEL, SuSE and Debian systems. So, we are done,
> > once we are able to develop plug-in to bodhi to generate the data
> > feed for us.
>
>  What is needed for that to happen?
>  
I think Bodhi has all the data it needs. We just need someone to write a 
script to query the data from Bodhi database and put it into that XML 
that OpenSCAP can parse.

We can review the examples us such data for other distributions:

   https://www.redhat.com/security/data/oval/
   https://support.novell.com/security/oval/
   http://people.canonical.com/~ubuntu-security/oval/
   https://www.debian.org/security/oval/

Best,

~š.

> > Do you think this approach is sensible for the ponycorn? :)
>
>  Yes, with the above notes.
>
>  Man, I missed a really good opportunity to call this message an "RFP".
>  :)
>
>
>
>  * I need to unify this metaphor... it's carrots and coffee rather than
>  carrots and sticks, since we have no sticks. But on the other hand, I
>  don't think horses like coffee.
>  

-- 
~š.

[Matthew Miller]

On Tue, Feb 09, 2016 at 10:05:30AM +0100, Šimon Lukašík wrote:
>  >The advantages you describe sound exciting, which falls under
> the
>  >offering-carrots aspect: new functionality that we can give people that
>  >they'll be actively want to switch to. However, I think we also need to
>  >do the "we replaced your favorite coffee brand... and you don't even
>  >know it" part wherever possible. *
>  I always notice when my favorite coffee shops changes a brand!!
>  However you are right, most people won't. :) 
Unless you tell them "This coffee shop no longer has coffee. Here, we
have chai! It's _way_ better."

It might even _be_ better.

>  >So, I really _do_ want the existing command to work, but maybe
> it's
>  >OpenSCAP underneath.
>  Do you think that this functionality should be integrated with
>  existing dnf commands? 
So, yeah, I do -- at least for the demo and basic functionality. It
might be that extended functionality better belongs in its own thing.
But, hey, part of the justification for all the investment in DNF is
supposed to be that it now has a defined plugin API.

>  That would be beneficial for some, however we are trying to make the
>  use-case even easier then before. You perhaps do not want to turn on
>  all the containers and mount all the images and run the command,
>  right? 
Yeah, probably not.


>  >>The problem we are currently facing in Fedora is that you
> need to
>  >>have a data that you feed to the scanner. Hence, this CVE analysis 
[...]
>  >What is needed for that to happen?
>  I think Bodhi has all the data it needs. We just need someone to
>  write a script to query the data from Bodhi database and put it into
>  that XML that OpenSCAP can parse. 
So what it sounds like is what we need is to find "someone". :)

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Šimon Lukašík]

On 02/08/2016 03:53 PM, Matthew Miller wrote:
>  At DevConf.cz, I said that 2016 needed to be the year of the _show_
>  part of "show & tell" for what we started with the "rings"
> proposal and
>  now what we're calling, vaguely, "modularity".
>
>  Some of that is scary, because anything really new always is. It's
>  important that we do this in a way that *both* brings useful new
>  functionality _and_ doesn't break what we have.
>
>  In the commercial world, there's a concept of "minimum viable product" --
>  not done, not perfect, but what it takes to actually start solving problems
>  and providing value. In Fedora, we're not in business in the same way, but
>  some of the same can apply. So, here's my suggestions for a "minimum viable
>  demo" for what a container-based module might look like in Fedora.
>
>  Particularly, when I've spoken to people about delivering parts of Fedora as
>  containers, the part about tracking security and proving updates becames
>  disturbingly hand-wavy, with "yeah, of course, we'll do something to cover
>  that". I think to be successful, we need to make sure that this is addressed
>  early. So, this proposal for a demo I'd like to see.
>
>  I've broken it down into three parts of increasing complexity and
>  usefulness: starting with "and I want a pony", then moving up to
> "actually,
>  a unicorn", and then "also, could you put wings on that?"
>
>  So, first:
>
>
>  The Minimum Viable Pony
>  -----------------------
>
>
>  This is a basic web server module delivered as a container.
>
>  Part A: the basic module:
>
>     * Answers on port 80 and serves out static web pages.
>
>     * Content comes from /var/www/html... or another container, or whatever.
>
>     * Doesn't even need to be configurable. It's a demo.
>
>     * It's composed entirely from RPMs already existing in Fedora.
>
>
>  Part B: DNF Plugin to list CVEs
>
>     * Works like
> https://docs.fedoraproject.org/en-US/Fedora/17/html/Security_Guide/sect-S...
>
>       - lists all RPMs on host system where a security update is available
>         - can list by CVE
>         - also by Bugzilla
>         - also with link to update notice
>         - with various breakdowns by severity level
>
>       - **additionally** lists "modules" which need updates (with same links)
>         - has option to list which specific RPMs within the module have the
>           problem.
>  
We are already able to do this kind of stuff with OpenSCAP. ( 
http://www.open-scap.org ).

Thing is just we have slightly different approach. Let me explain.

Running dnf/yum plugin is often regarded as a partial solution. It may 
be good enough for Fedora users. However, there are still use-cases. 
Where dnf/yum plugin may not be good enough:
  * disconnected system (a container without internet access)
  * cloned remote repositories (latest Fedora contains bugfixes, but my 
custom repo may not)
  * disabled local repositories (software was installed to the 
container, but the repofile is missing for some reason).

So, we try to examine what is actually installed on the system. Here is 
one example 
https://www.open-scap.org/resources/documentation/security-compliance-of-...

During the last months guys have integrated this solution more tightly 
with atomic, allowing the scanner to be actually inside super-privileged 
container. 
https://martin.preisler.me/2015/11/atomic-scan-and-openscap-daemon/

The problem we are currently facing in Fedora is that you need to have a 
data that you feed to the scanner. Hence, this CVE analysis can be done 
only for RHEL, SuSE and Debian systems. So, we are done, once we are 
able to develop plug-in to bodhi to generate the data feed for us.

Do you think this approach is sensible for the ponycorn? :)

>  So that's a good start. As a sysadmin, I'm feeling a little less scared
>  about. But we also need...
>
>
>  To Take It to Unicorn Level
>  ---------------------------
>
>  It's all well and good to fix the problem, but we also need to be able to
>  fix it. So, the next level is:
>
>  * Image is built in Fedora's new Layered Image Build Service
>     https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service
>
>  * If there is a security vulnerability in any RPM that is in the image, it
>     is automatically rebuilt and made available somewhere.
>
>  * And automatically tested!
>
>     - Here, just a basic test that makes sure that input web content comes
>       out. It's a proof-of-concept so no real in-depth testing is needed.
>  
Also some basic security checks should be in place before publishing the 
image. I think this represents good starting point: 
https://github.com/OpenSCAP/scap-security-guide/issues/589 (Don't be 
confused by RHEL7 label, we have the same thing prepared for Fedora in 
upstream).

~š.

>  * And, then, a dnf plugin which actually downloads and applies the
> update.
>
>  Now, not only do we have comforting tooling (minimizing change pain), we
>  also have a real, practical improvement (there will be cake!). In the
>  current system, while we do hands-on testing of individual updates, they
>  aren't necessarily tested in the application to which they apply. For
>  example, if the web server here is Nginx and the security flaw in the SSL
>  library, current update testing does not guarantee that the update has been
>  tested *for that use*. But this will. Plus, in addition to whatever human
>  oversight, there will be an _automatic_ test.
>
>  So that's pretty good. But finally, to get to the glorious future, I'd also
>  like to see this unicorn become also a pegasus (my daughter tells me that
>  this makes it an "alicorn")...
>
>
>
>  Taking Flight
>  -------------
>
>  Same as above, but with two new things:
>
>  New Thing One:
>
>     * Instead of RPMs, a module which uses some non-RPM upstream packaging
>       format as part of its composition
>       - Ruby gem, Python egg or wheel, Java jar... whatever
>
>     * And the DNF list-security command still works
>       - and the verbose flag now tells us about the CVEs or whatever for the
>         non-RPM package
>
>  New Thing Two:
>
>     * There is enough metadata and intelligence such that when the new
>       container is updated, that update happens without downtime. This may
>       involve multiple containers.
>
>     * If the update fails locally for some reason (despite the best efforts at
>       upstream testing), it's rolled back automatically
>
>
>
>   From here, I can imagine all sorts of other mythical creatures. But, you
>  know, for a start. What do you think?
>
>  As a user/sysadmin, is there something more you'd like to see? If
>  you're skeptical about containers, would this make you feel better? If
>  not, what might?
>
>  If you're excited about them, does this seem like a useful start?
>  What's missing?
>
>  And, of course, if you're a developer working on this stuff, can I have
>  ponies please?
>
>  

-- 
~š.

[Matthew Miller]

On Mon, Feb 08, 2016 at 09:53:39AM -0500, Matthew Miller wrote:
>  It's all well and good to fix the problem, but we also need to be
> able to
>  fix it. So, the next level is: 
Ohh, that was good self-editing. That was supposed to be "also need to
be able to do *more* than just fix it".

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader