Containers: we're asking for ponies, so here's a suggestion for the Minimum Viable (Modular) Pony Show
At DevConf.cz, I said that 2016 needed to be the year of the _show_ part of "show & tell" for what we started with the "rings" proposal and now what we're calling, vaguely, "modularity".
Some of that is scary, because anything really new always is. It's important that we do this in a way that *both* brings useful new functionality _and_ doesn't break what we have.
In the commercial world, there's a concept of "minimum viable product" -- not done, not perfect, but what it takes to actually start solving problems and providing value. In Fedora, we're not in business in the same way, but some of the same can apply. So, here's my suggestions for a "minimum viable demo" for what a container-based module might look like in Fedora.
Particularly, when I've spoken to people about delivering parts of Fedora as containers, the part about tracking security and proving updates becames disturbingly hand-wavy, with "yeah, of course, we'll do something to cover that". I think to be successful, we need to make sure that this is addressed early. So, this proposal for a demo I'd like to see.
I've broken it down into three parts of increasing complexity and usefulness: starting with "and I want a pony", then moving up to "actually, a unicorn", and then "also, could you put wings on that?"
So, first:
The Minimum Viable Pony -----------------------
This is a basic web server module delivered as a container.
Part A: the basic module:
* Answers on port 80 and serves out static web pages.
* Content comes from /var/www/html... or another container, or whatever.
* Doesn't even need to be configurable. It's a demo.
* It's composed entirely from RPMs already existing in Fedora.
Part B: DNF Plugin to list CVEs
* Works like https://docs.fedoraproject.org/en-US/Fedora/17/html/Security_Guide/sect-Secu...
- lists all RPMs on host system where a security update is available - can list by CVE - also by Bugzilla - also with link to update notice - with various breakdowns by severity level
- **additionally** lists "modules" which need updates (with same links) - has option to list which specific RPMs within the module have the problem.
So that's a good start. As a sysadmin, I'm feeling a little less scared about. But we also need...
To Take It to Unicorn Level ---------------------------
It's all well and good to fix the problem, but we also need to be able to fix it. So, the next level is:
* Image is built in Fedora's new Layered Image Build Service https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service
* If there is a security vulnerability in any RPM that is in the image, it is automatically rebuilt and made available somewhere.
* And automatically tested!
- Here, just a basic test that makes sure that input web content comes out. It's a proof-of-concept so no real in-depth testing is needed.
* And, then, a dnf plugin which actually downloads and applies the update.
Now, not only do we have comforting tooling (minimizing change pain), we also have a real, practical improvement (there will be cake!). In the current system, while we do hands-on testing of individual updates, they aren't necessarily tested in the application to which they apply. For example, if the web server here is Nginx and the security flaw in the SSL library, current update testing does not guarantee that the update has been tested *for that use*. But this will. Plus, in addition to whatever human oversight, there will be an _automatic_ test.
So that's pretty good. But finally, to get to the glorious future, I'd also like to see this unicorn become also a pegasus (my daughter tells me that this makes it an "alicorn")...
Taking Flight -------------
Same as above, but with two new things:
New Thing One:
* Instead of RPMs, a module which uses some non-RPM upstream packaging format as part of its composition - Ruby gem, Python egg or wheel, Java jar... whatever
* And the DNF list-security command still works - and the verbose flag now tells us about the CVEs or whatever for the non-RPM package
New Thing Two:
* There is enough metadata and intelligence such that when the new container is updated, that update happens without downtime. This may involve multiple containers.
* If the update fails locally for some reason (despite the best efforts at upstream testing), it's rolled back automatically
From here, I can imagine all sorts of other mythical creatures. But, you know, for a start. What do you think?
As a user/sysadmin, is there something more you'd like to see? If you're skeptical about containers, would this make you feel better? If not, what might?
If you're excited about them, does this seem like a useful start? What's missing?
And, of course, if you're a developer working on this stuff, can I have ponies please?
On 02/08/2016 03:53 PM, Matthew Miller wrote:
At DevConf.cz, I said that 2016 needed to be the year of the _show_ part of "show & tell" for what we started with the "rings" proposal and now what we're calling, vaguely, "modularity".
Some of that is scary, because anything really new always is. It's important that we do this in a way that *both* brings useful new functionality _and_ doesn't break what we have.
In the commercial world, there's a concept of "minimum viable product" -- not done, not perfect, but what it takes to actually start solving problems and providing value. In Fedora, we're not in business in the same way, but some of the same can apply. So, here's my suggestions for a "minimum viable demo" for what a container-based module might look like in Fedora.
Particularly, when I've spoken to people about delivering parts of Fedora as containers, the part about tracking security and proving updates becames disturbingly hand-wavy, with "yeah, of course, we'll do something to cover that". I think to be successful, we need to make sure that this is addressed early. So, this proposal for a demo I'd like to see.
I've broken it down into three parts of increasing complexity and usefulness: starting with "and I want a pony", then moving up to "actually, a unicorn", and then "also, could you put wings on that?"
So, first:
The Minimum Viable Pony
This is a basic web server module delivered as a container.
Part A: the basic module:
Answers on port 80 and serves out static web pages.
Content comes from /var/www/html... or another container, or whatever.
Doesn't even need to be configurable. It's a demo.
It's composed entirely from RPMs already existing in Fedora.
Part B: DNF Plugin to list CVEs
Works like https://docs.fedoraproject.org/en-US/Fedora/17/html/Security_Guide/sect-Secu...
lists all RPMs on host system where a security update is available
- can list by CVE
- also by Bugzilla
- also with link to update notice
- with various breakdowns by severity level
**additionally** lists "modules" which need updates (with same links)
- has option to list which specific RPMs within the module have the problem.
We are already able to do this kind of stuff with OpenSCAP. ( http://www.open-scap.org ).
Thing is just we have slightly different approach. Let me explain.
Running dnf/yum plugin is often regarded as a partial solution. It may be good enough for Fedora users. However, there are still use-cases. Where dnf/yum plugin may not be good enough: * disconnected system (a container without internet access) * cloned remote repositories (latest Fedora contains bugfixes, but my custom repo may not) * disabled local repositories (software was installed to the container, but the repofile is missing for some reason).
So, we try to examine what is actually installed on the system. Here is one example https://www.open-scap.org/resources/documentation/security-compliance-of-rhe...
During the last months guys have integrated this solution more tightly with atomic, allowing the scanner to be actually inside super-privileged container. https://martin.preisler.me/2015/11/atomic-scan-and-openscap-daemon/
The problem we are currently facing in Fedora is that you need to have a data that you feed to the scanner. Hence, this CVE analysis can be done only for RHEL, SuSE and Debian systems. So, we are done, once we are able to develop plug-in to bodhi to generate the data feed for us.
Do you think this approach is sensible for the ponycorn? :)
So that's a good start. As a sysadmin, I'm feeling a little less scared about. But we also need...
To Take It to Unicorn Level
It's all well and good to fix the problem, but we also need to be able to fix it. So, the next level is:
Image is built in Fedora's new Layered Image Build Service https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service
If there is a security vulnerability in any RPM that is in the image, it is automatically rebuilt and made available somewhere.
And automatically tested!
- Here, just a basic test that makes sure that input web content comes out. It's a proof-of-concept so no real in-depth testing is needed.
Also some basic security checks should be in place before publishing the image. I think this represents good starting point: https://github.com/OpenSCAP/scap-security-guide/issues/589 (Don't be confused by RHEL7 label, we have the same thing prepared for Fedora in upstream).
~š.
- And, then, a dnf plugin which actually downloads and applies the update.
Now, not only do we have comforting tooling (minimizing change pain), we also have a real, practical improvement (there will be cake!). In the current system, while we do hands-on testing of individual updates, they aren't necessarily tested in the application to which they apply. For example, if the web server here is Nginx and the security flaw in the SSL library, current update testing does not guarantee that the update has been tested *for that use*. But this will. Plus, in addition to whatever human oversight, there will be an _automatic_ test.
So that's pretty good. But finally, to get to the glorious future, I'd also like to see this unicorn become also a pegasus (my daughter tells me that this makes it an "alicorn")...
Taking Flight
Same as above, but with two new things:
New Thing One:
Instead of RPMs, a module which uses some non-RPM upstream packaging format as part of its composition
- Ruby gem, Python egg or wheel, Java jar... whatever
And the DNF list-security command still works
- and the verbose flag now tells us about the CVEs or whatever for the non-RPM package
New Thing Two:
There is enough metadata and intelligence such that when the new container is updated, that update happens without downtime. This may involve multiple containers.
If the update fails locally for some reason (despite the best efforts at upstream testing), it's rolled back automatically
From here, I can imagine all sorts of other mythical creatures. But, you know, for a start. What do you think?
As a user/sysadmin, is there something more you'd like to see? If you're skeptical about containers, would this make you feel better? If not, what might?
If you're excited about them, does this seem like a useful start? What's missing?
And, of course, if you're a developer working on this stuff, can I have ponies please?
On Mon, Feb 08, 2016 at 04:37:40PM +0100, Šimon Lukašík wrote:
Thing is just we have slightly different approach. Let me explain.
Using OpenSCAP sounds great. Mostly, I wanted to focus on results and what the user sees rather than implementation details.
The advantages you describe sound exciting, which falls under the offering-carrots aspect: new functionality that we can give people that they'll be actively want to switch to. However, I think we also need to do the "we replaced your favorite coffee brand... and you don't even know it" part wherever possible. *
So, I really _do_ want the existing command to work, but maybe it's OpenSCAP underneath.
[...]
The problem we are currently facing in Fedora is that you need to have a data that you feed to the scanner. Hence, this CVE analysis can be done only for RHEL, SuSE and Debian systems. So, we are done, once we are able to develop plug-in to bodhi to generate the data feed for us.
What is needed for that to happen?
Do you think this approach is sensible for the ponycorn? :)
Yes, with the above notes.
Man, I missed a really good opportunity to call this message an "RFP". :)
* I need to unify this metaphor... it's carrots and coffee rather than carrots and sticks, since we have no sticks. But on the other hand, I don't think horses like coffee.
On Mon, Feb 08, 2016 at 11:14:44AM -0500, Matthew Miller wrote:
Thing is just we have slightly different approach. Let me explain.
So, I really _do_ want the existing command to work, but maybe it's OpenSCAP underneath.
Which, by the way, is what Langdon suggested when I talked to him about this at a DevConf dinner the other night.
On 02/08/2016 05:14 PM, Matthew Miller wrote:
On Mon, Feb 08, 2016 at 04:37:40PM +0100, Šimon Lukašík wrote:
Thing is just we have slightly different approach. Let me explain.
Using OpenSCAP sounds great. Mostly, I wanted to focus on results and what the user sees rather than implementation details.
The advantages you describe sound exciting, which falls under the offering-carrots aspect: new functionality that we can give people that they'll be actively want to switch to. However, I think we also need to do the "we replaced your favorite coffee brand... and you don't even know it" part wherever possible. *
I always notice when my favorite coffee shops changes a brand!! However you are right, most people won't. :)
So, I really _do_ want the existing command to work, but maybe it's OpenSCAP underneath.
Do you think that this functionality should be integrated with existing dnf commands?
That would be beneficial for some, however we are trying to make the use-case even easier then before. You perhaps do not want to turn on all the containers and mount all the images and run the command, right?
[...]
The problem we are currently facing in Fedora is that you need to have a data that you feed to the scanner. Hence, this CVE analysis can be done only for RHEL, SuSE and Debian systems. So, we are done, once we are able to develop plug-in to bodhi to generate the data feed for us.
What is needed for that to happen?
I think Bodhi has all the data it needs. We just need someone to write a script to query the data from Bodhi database and put it into that XML that OpenSCAP can parse.
We can review the examples us such data for other distributions:
https://www.redhat.com/security/data/oval/ https://support.novell.com/security/oval/ http://people.canonical.com/~ubuntu-security/oval/ https://www.debian.org/security/oval/
Best,
~š.
Do you think this approach is sensible for the ponycorn? :)
Yes, with the above notes.
Man, I missed a really good opportunity to call this message an "RFP". :)
- I need to unify this metaphor... it's carrots and coffee rather than
carrots and sticks, since we have no sticks. But on the other hand, I don't think horses like coffee.
On Tue, Feb 09, 2016 at 10:05:30AM +0100, Šimon Lukašík wrote:
The advantages you describe sound exciting, which falls under the offering-carrots aspect: new functionality that we can give people that they'll be actively want to switch to. However, I think we also need to do the "we replaced your favorite coffee brand... and you don't even know it" part wherever possible. *
I always notice when my favorite coffee shops changes a brand!! However you are right, most people won't. :)
Unless you tell them "This coffee shop no longer has coffee. Here, we have chai! It's _way_ better."
It might even _be_ better.
So, I really _do_ want the existing command to work, but maybe it's OpenSCAP underneath.
Do you think that this functionality should be integrated with existing dnf commands?
So, yeah, I do -- at least for the demo and basic functionality. It might be that extended functionality better belongs in its own thing. But, hey, part of the justification for all the investment in DNF is supposed to be that it now has a defined plugin API.
That would be beneficial for some, however we are trying to make the use-case even easier then before. You perhaps do not want to turn on all the containers and mount all the images and run the command, right?
Yeah, probably not.
The problem we are currently facing in Fedora is that you need to have a data that you feed to the scanner. Hence, this CVE analysis
[...]
What is needed for that to happen?
I think Bodhi has all the data it needs. We just need someone to write a script to query the data from Bodhi database and put it into that XML that OpenSCAP can parse.
So what it sounds like is what we need is to find "someone". :)
On 02/08/2016 03:53 PM, Matthew Miller wrote:
At DevConf.cz, I said that 2016 needed to be the year of the _show_ part of "show & tell" for what we started with the "rings" proposal and now what we're calling, vaguely, "modularity".
Some of that is scary, because anything really new always is. It's important that we do this in a way that *both* brings useful new functionality _and_ doesn't break what we have.
In the commercial world, there's a concept of "minimum viable product" -- not done, not perfect, but what it takes to actually start solving problems and providing value. In Fedora, we're not in business in the same way, but some of the same can apply. So, here's my suggestions for a "minimum viable demo" for what a container-based module might look like in Fedora.
Particularly, when I've spoken to people about delivering parts of Fedora as containers, the part about tracking security and proving updates becames disturbingly hand-wavy, with "yeah, of course, we'll do something to cover that". I think to be successful, we need to make sure that this is addressed early. So, this proposal for a demo I'd like to see.
I've broken it down into three parts of increasing complexity and usefulness: starting with "and I want a pony", then moving up to "actually, a unicorn", and then "also, could you put wings on that?"
So, first:
The Minimum Viable Pony
This is a basic web server module delivered as a container.
Part A: the basic module:
Answers on port 80 and serves out static web pages.
Content comes from /var/www/html... or another container, or whatever.
Doesn't even need to be configurable. It's a demo.
It's composed entirely from RPMs already existing in Fedora.
Part B: DNF Plugin to list CVEs
Works like https://docs.fedoraproject.org/en-US/Fedora/17/html/Security_Guide/sect-Secu...
lists all RPMs on host system where a security update is available
- can list by CVE
- also by Bugzilla
- also with link to update notice
- with various breakdowns by severity level
**additionally** lists "modules" which need updates (with same links)
- has option to list which specific RPMs within the module have the problem.
We are already able to do this kind of stuff with OpenSCAP. ( http://www.open-scap.org ).
Thing is just we have slightly different approach. Let me explain.
Running dnf/yum plugin is often regarded as a partial solution. It may be good enough for Fedora users. However, there are still use-cases. Where dnf/yum plugin may not be good enough: * disconnected system (a container without internet access) * cloned remote repositories (latest Fedora contains bugfixes, but my custom repo may not) * disabled local repositories (software was installed to the container, but the repofile is missing for some reason).
So, we try to examine what is actually installed on the system. Here is one example https://www.open-scap.org/resources/documentation/security-compliance-of-rhe...
During the last months guys have integrated this solution more tightly with atomic, allowing the scanner to be actually inside super-privileged container. https://martin.preisler.me/2015/11/atomic-scan-and-openscap-daemon/
The problem we are currently facing in Fedora is that you need to have a data that you feed to the scanner. Hence, this CVE analysis can be done only for RHEL, SuSE and Debian systems. So, we are done, once we are able to develop plug-in to bodhi to generate the data feed for us.
Do you think this approach is sensible for the ponycorn? :)
So that's a good start. As a sysadmin, I'm feeling a little less scared about. But we also need...
To Take It to Unicorn Level
It's all well and good to fix the problem, but we also need to be able to fix it. So, the next level is:
Image is built in Fedora's new Layered Image Build Service https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service
If there is a security vulnerability in any RPM that is in the image, it is automatically rebuilt and made available somewhere.
And automatically tested!
- Here, just a basic test that makes sure that input web content comes out. It's a proof-of-concept so no real in-depth testing is needed.
Also some basic security checks should be in place before publishing the image. I think this represents good starting point: https://github.com/OpenSCAP/scap-security-guide/issues/589 (Don't be confused by RHEL7 label, we have the same thing prepared for Fedora in upstream).
~š.
- And, then, a dnf plugin which actually downloads and applies the update.
Now, not only do we have comforting tooling (minimizing change pain), we also have a real, practical improvement (there will be cake!). In the current system, while we do hands-on testing of individual updates, they aren't necessarily tested in the application to which they apply. For example, if the web server here is Nginx and the security flaw in the SSL library, current update testing does not guarantee that the update has been tested *for that use*. But this will. Plus, in addition to whatever human oversight, there will be an _automatic_ test.
So that's pretty good. But finally, to get to the glorious future, I'd also like to see this unicorn become also a pegasus (my daughter tells me that this makes it an "alicorn")...
Taking Flight
Same as above, but with two new things:
New Thing One:
Instead of RPMs, a module which uses some non-RPM upstream packaging format as part of its composition
- Ruby gem, Python egg or wheel, Java jar... whatever
And the DNF list-security command still works
- and the verbose flag now tells us about the CVEs or whatever for the non-RPM package
New Thing Two:
There is enough metadata and intelligence such that when the new container is updated, that update happens without downtime. This may involve multiple containers.
If the update fails locally for some reason (despite the best efforts at upstream testing), it's rolled back automatically
From here, I can imagine all sorts of other mythical creatures. But, you know, for a start. What do you think?
As a user/sysadmin, is there something more you'd like to see? If you're skeptical about containers, would this make you feel better? If not, what might?
If you're excited about them, does this seem like a useful start? What's missing?
And, of course, if you're a developer working on this stuff, can I have ponies please?
On Mon, Feb 08, 2016 at 09:53:39AM -0500, Matthew Miller wrote:
It's all well and good to fix the problem, but we also need to be able to fix it. So, the next level is:
Ohh, that was good self-editing. That was supposed to be "also need to be able to do *more* than just fix it".
-
Matthew Miller -
Šimon Lukašík