Hi security team. I'm working on
https://fedoraproject.org/wiki/Changes/VisibleCloud
which proposes promoting the Fedora Cloud image on basically equal footing with the desktop download. Daniel Berrange gave the useful feedback that while installation-based distribution allows one to install updates at build time, image-based distribution means that the image must be booted to apply updates, giving a window of insecurity. (Unless careful measures are taken.)
When there was a security issue with the previous Fedora image, we did do a fire-drill with an adhoc respin and pushed new images. Dan suggests that we develop (in coordination with the qa and release engineering teams) a security policy for updates to the cloud image.
Is this of interest?
----- Original Message -----
Hi security team. I'm working on
https://fedoraproject.org/wiki/Changes/VisibleCloud
which proposes promoting the Fedora Cloud image on basically equal footing with the desktop download. Daniel Berrange gave the useful feedback that while installation-based distribution allows one to install updates at build time, image-based distribution means that the image must be booted to apply updates, giving a window of insecurity. (Unless careful measures are taken.)
When there was a security issue with the previous Fedora image, we did do a fire-drill with an adhoc respin and pushed new images. Dan suggests that we develop (in coordination with the qa and release engineering teams) a security policy for updates to the cloud image.
Is this of interest?
I think this is of great interest to us. It's a whole new way of thinking about the distribution. New concepts like this always bring new challenges.
So needing to respin images is almost certainly going to happen. I suspect there isn't going to be an easy way to define what that is though. Some people might care about local root issues, remote root is obviously bad no matter what. What about system level denial of service? The attack surface potential here is going to be REALLY high. Our challenge will be to think of this not as a normal distribution, but as a cloud image (which I'm currently not doing in my head).
I'm unsure what I think about the concern with needing to boot an image to apply updates. This is true of a fresh install, no? This update problem will be dictated by what's running on an image at boot time.
Anyhow, I think this is a good conversation opener. If anyone has any ideas about what we should be worried about, thinking about, or if you have a clever idea, let us know.
Thanks Matthew.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/15/2013 09:16 PM, Josh Bressers wrote:
----- Original Message -----
Hi security team. I'm working on
https://fedoraproject.org/wiki/Changes/VisibleCloud
which proposes promoting the Fedora Cloud image on basically equal footing with the desktop download. Daniel Berrange gave the useful feedback that while installation-based distribution allows one to install updates at build time, image-based distribution means that the image must be booted to apply updates, giving a window of insecurity. (Unless careful measures are taken.)
When there was a security issue with the previous Fedora image, we did do a fire-drill with an adhoc respin and pushed new images. Dan suggests that we develop (in coordination with the qa and release engineering teams) a security policy for updates to the cloud image.
Is this of interest?
I think this is of great interest to us. It's a whole new way of thinking about the distribution. New concepts like this always bring new challenges.
Speaking as the SRT guy responsible for the RHT end of this fire drill this is something we want to avoid in future, and some things are already in progress, once I make some progress I was planning to talk to the Fedora (and possibly others).
So needing to respin images is almost certainly going to happen. I suspect there isn't going to be an easy way to define what that is though. Some people might care about local root issues, remote root is obviously bad no matter what. What about system level denial of service? The attack surface potential here is going to be REALLY high. Our challenge will be to think of this not as a normal distribution, but as a cloud image (which I'm currently not doing in my head).
One problem with AWS for example is even if you respin the image it has a new AMI (Amazon Machine Instance) ID, so any systems launching Fedora servers automatically will include that old AMI ID until it is manually updated. Only people going through the Marketplace or searching for "Fedora" will get the latest one.
I'm unsure what I think about the concern with needing to boot an image to apply updates. This is true of a fresh install, no? This update problem will be dictated by what's running on an image at boot time.
In general we are safe here, the default images just have OpenSSH exposed to the world, if that has a remote exploit we have much larger problems than our cloud images (like .. everything on the planet being hacked).
Anyhow, I think this is a good conversation opener. If anyone has any ideas about what we should be worried about, thinking about, or if you have a clever idea, let us know.
I had thought about a stub RPM like "cloud-update" that could contain cloud specific updates/etc, some problems crop up like do we do one per service provider (since we can't reliably detect which provider we're on without doing some potentially dangerous things), who maintains it, how do we handle situations where we push out an emergency update through the cloud-update thing and then push an actual updated cloud image, we need to ensure the image and the cloud-update don't conflict to name a few of the simpler problems.
- -- Kurt Seifried Red Hat Security Response Team (SRT) PGP: 0x5E267993 A90B F995 7350 148F 66BF 7554 160D 4553 5E26 7993
Speaking as the SRT guy responsible for the RHT end of this fire drill this is something we want to avoid in future, and some things are already in progress, once I make some progress I was planning to talk to the Fedora (and possibly others).
That fire drill will likely happen again at some point. It needs to be planned for.
One problem with AWS for example is even if you respin the image it has a new AMI (Amazon Machine Instance) ID, so any systems launching Fedora servers automatically will include that old AMI ID until it is manually updated. Only people going through the Marketplace or searching for "Fedora" will get the latest one.
I think there are two different problems here.
There is the issue of doing a respin of images (this includes what you download off a mirror). Then there is the issue of updating existing images. Neither problem is simple.
I had thought about a stub RPM like "cloud-update" that could contain cloud specific updates/etc, some problems crop up like do we do one per service provider (since we can't reliably detect which provider we're on without doing some potentially dangerous things), who maintains it, how do we handle situations where we push out an emergency update through the cloud-update thing and then push an actual updated cloud image, we need to ensure the image and the cloud-update don't conflict to name a few of the simpler problems.
I still don't really like this solution. The right thing to do is to break the problems apart into manageable bits.
I'd think the first problem is just understanding how should an image respin work? Some questions to answer:
* What sort of event could cause a respin? * Who is responsible for the respin event? * How do we communicate a respin event? * What do we do with the old images?
I'm sure there are plenty more questions, those are the ones that come to mind.
Thanks.
I'm sure there are plenty more questions, those are the ones that come to mind.
To keep this thread a bit alive, here's a reminder of some of our new challenges: http://missingm.co/2013/07/identical-droplets-in-the-digitalocean-regenerate...
On Tue, Jul 30, 2013 at 03:09:43PM -0400, Josh Bressers wrote:
To keep this thread a bit alive,
Yeah -- more from me after Flock, which I'm busy prepping for now.
here's a reminder of some of our new challenges: http://missingm.co/2013/07/identical-droplets-in-the-digitalocean-regenerate...
So, we do the right thing (generate the keys on start) with the official Fedora cloud images, but DigitalOcean produces their own images internally (for Ubuntu too), so it's possible that they've got issues there with Fedora too. They've graciously provided me with a gratis account so I'll look into it.
----- Original Message -----
From: "Matthew Miller" mattdm@fedoraproject.org To: "Josh Bressers" bressers@redhat.com Cc: security@lists.fedoraproject.org Sent: Tuesday, July 30, 2013 3:17:41 PM Subject: Re: cloud image updates (for f20 and beyond)
On Tue, Jul 30, 2013 at 03:09:43PM -0400, Josh Bressers wrote:
To keep this thread a bit alive,
Yeah -- more from me after Flock, which I'm busy prepping for now.
here's a reminder of some of our new challenges: http://missingm.co/2013/07/identical-droplets-in-the-digitalocean-regenerate...
So, we do the right thing (generate the keys on start) with the official Fedora cloud images, but DigitalOcean produces their own images internally (for Ubuntu too), so it's possible that they've got issues there with Fedora too. They've graciously provided me with a gratis account so I'll look into it.
I can help out if you want, Matt. I've spoken with them about their Fedora images in the past so I know one of their engineers reasonably well from those conversations.
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mattdm@fedoraproject.org -- security mailing list security@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/security
On Tue, Jul 30, 2013 at 03:33:19PM -0400, Sam Kottler wrote:
I can help out if you want, Matt. I've spoken with them about their Fedora images in the past so I know one of their engineers reasonably well from those conversations.
Cool -- I was most recently talking to Jeff Carr and someone named Bodie.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
On Mon, Jul 15, 2013 at 09:35:02PM -0400, Matthew Miller wrote:
Hi security team. I'm working on
https://fedoraproject.org/wiki/Changes/VisibleCloud
which proposes promoting the Fedora Cloud image on basically equal footing with the desktop download. Daniel Berrange gave the useful feedback that while installation-based distribution allows one to install updates at build time, image-based distribution means that the image must be booted to apply updates, giving a window of insecurity. (Unless careful measures are taken.)
Yeah, I can see this as being a concern. The risk will more than likely be a small due to the window of time involved but it's always a good to ship the fixes when they exist.
When there was a security issue with the previous Fedora image, we did do a fire-drill with an adhoc respin and pushed new images. Dan suggests that we develop (in coordination with the qa and release engineering teams) a security policy for updates to the cloud image.
Each CVE receives a CVSSv2 score in BZ. This *could* be used as a way to determine which vulnerability patches should go into your spin. Of course this may end up with more updates that needed being that you might be patching software that would necessarily run at boot time or be vulnerable immediately. It's a place to start, IMO, though.
- -- Eric
- -------------------------------------------------- Eric "Sparks" Christensen Fedora Project - Red Hat
sparks@redhat.com - sparks@fedoraproject.org 097C 82C3 52DF C64A 50C2 E3A3 8076 ABDE 024B B3D1 - --------------------------------------------------
security@lists.fedoraproject.org
-
Eric Christensen -
Josh Bressers -
Kurt Seifried -
Matthew Miller -
Sam Kottler