Greetings!
We sort of ran out of time in today's Cloud WG meeting, but I did want to ask:
What are your thoughts on AMI lifetimes? That is to say, how long should EC2 AMIs exist before they're deleted? A few points to consider:
- AMIs only cost us for storage, so it's not a *huge* cost to maintain a public AMI - At the same time, there are a lot of AMIs, since we build 2-4 per AWS region per build, and that number is growing - There are 9 regions now, and we have 2 virtualization types, and 2 volume types, as well (9 regions * 2 * 2 = 36 AMIs per Base image build, 18 for Atomic builds (since they are only available in HVM format)) - This total number will only grow larger as we add instance-store AMIs, and so on - This isn't even taking into account any costs we'll have once we secure a deal with other providers like HP, Rackspace, and GCE, to maintain public images on their services
I propose we have some sort of discussion regarding how long cloud image builds should be available on services like AWS. I suspect this will resolve to having different lifetimes for scratch, test, RC, final, and maybe other build types.
Any input is appreciated. We can certainly talk about this at next week's meeting, as well.
-- David
On Wed, Mar 18, 2015 at 3:38 PM, David Gay dgay@redhat.com wrote:
Greetings!
We sort of ran out of time in today's Cloud WG meeting, but I did want to ask:
What are your thoughts on AMI lifetimes? That is to say, how long should EC2 AMIs exist before they're deleted? A few points to consider:
- AMIs only cost us for storage, so it's not a *huge* cost to maintain a public AMI
- At the same time, there are a lot of AMIs, since we build 2-4 per AWS region per build, and that number is growing
- There are 9 regions now, and we have 2 virtualization types, and 2 volume types, as well (9 regions * 2 * 2 = 36 AMIs per Base image build, 18 for Atomic builds (since they are only available in HVM format))
- This total number will only grow larger as we add instance-store AMIs, and so on
- This isn't even taking into account any costs we'll have once we secure a deal with other providers like HP, Rackspace, and GCE, to maintain public images on their services
I propose we have some sort of discussion regarding how long cloud image builds should be available on services like AWS. I suspect this will resolve to having different lifetimes for scratch, test, RC, final, and maybe other build types.
Any input is appreciated. We can certainly talk about this at next week's meeting, as well.
-- David _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
"In God We Trust - All Others Bring Data" ;-)
Surely you or someone on the team must have some raw data on usage for the existing AMIs, including comparisons for how much usage Fedora AMIs get vs. CentOS AMIs and RHEL AMIs. Don't build / maintain what people aren't using!
If you have raw data, I'd be happy to explore it for you - email me off-list if you need an NDA or something like that.
I vote for different life-time for different purpose, precisely as David suggests: scratch, test, RC, Final (Beta). But let's also don't forget about policy for testing the amis: how long to keep amis that failed testing in particular release stage. Ideally, we should also keep track of released/available amis so people are able to easily find particular version and flavor of Fedora amis. My usual use case is: reproduce a bug on e.g. F20 + some software stack. I find F20 ami ideal for this but the issue is to locate it[1].
Cheers, milan
[1] http://thecloudmarket.com/
2015-03-19 5:11 GMT+01:00 M. Edward (Ed) Borasky znmeb@znmeb.net:
On Wed, Mar 18, 2015 at 3:38 PM, David Gay dgay@redhat.com wrote:
Greetings!
We sort of ran out of time in today's Cloud WG meeting, but I did want
to ask:
What are your thoughts on AMI lifetimes? That is to say, how long should
EC2 AMIs exist before they're deleted? A few points to consider:
- AMIs only cost us for storage, so it's not a *huge* cost to maintain a
public AMI
- At the same time, there are a lot of AMIs, since we build 2-4 per
AWS region per build, and that number is growing
- There are 9 regions now, and we have 2 virtualization types, and 2volume types, as well (9 regions * 2 * 2 = 36 AMIs per Base image build, 18 for Atomic builds (since they are only available in HVM format))
- This total number will only grow larger as we add instance-storeAMIs, and so on
- This isn't even taking into account any costs we'll have once we
secure a deal with other providers like HP, Rackspace, and GCE, to maintain public images on their services
I propose we have some sort of discussion regarding how long cloud image
builds should be available on services like AWS. I suspect this will resolve to having different lifetimes for scratch, test, RC, final, and maybe other build types.
Any input is appreciated. We can certainly talk about this at next
week's meeting, as well.
-- David _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
"In God We Trust - All Others Bring Data" ;-)
Surely you or someone on the team must have some raw data on usage for the existing AMIs, including comparisons for how much usage Fedora AMIs get vs. CentOS AMIs and RHEL AMIs. Don't build / maintain what people aren't using!
If you have raw data, I'd be happy to explore it for you - email me off-list if you need an NDA or something like that.
-- OSJourno: Robust Power Tools for Digital Journalists
http://www.znmeb.mobi/stories/osjourno-robust-power-tools-for-digital-journa...
Remember, if you're traveling to Bactria, Hump Day is Tuesday and Thursday. _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
On 03/19/2015 12:11 AM, M. Edward (Ed) Borasky wrote:
If you have raw data, I'd be happy to explore it for you - email me off-list if you need an NDA or something like that.
Any data we have should be shared with all or none. We don't have a Fedora NDA AFAIK.
Best,
jzb
On Thursday, March 19, 2015 08:45:36 AM Joe Brockmeier wrote:
On 03/19/2015 12:11 AM, M. Edward (Ed) Borasky wrote:
If you have raw data, I'd be happy to explore it for you - email me off-list if you need an NDA or something like that.
Any data we have should be shared with all or none. We don't have a Fedora NDA AFAIK.
Best,
jzb
AFAIK amazon does not make available any usage data at all.
Dennis
On Thu, Mar 19, 2015 at 7:09 AM, Dennis Gilmore dennis@ausil.us wrote:
On Thursday, March 19, 2015 08:45:36 AM Joe Brockmeier wrote:
On 03/19/2015 12:11 AM, M. Edward (Ed) Borasky wrote:
If you have raw data, I'd be happy to explore it for you - email me off-list if you need an NDA or something like that.
Any data we have should be shared with all or none. We don't have a Fedora NDA AFAIK.
Best,
jzb
AFAIK amazon does not make available any usage data at all.
Amazon is a vendor - either Red Hat or Fedora or CentOS is the customer. "We" are paying them for hosting and they should supply "us" with usage data.
On Thu, Mar 19, 2015 at 3:09 PM, Dennis Gilmore dennis@ausil.us wrote:
On Thursday, March 19, 2015 08:45:36 AM Joe Brockmeier wrote:
On 03/19/2015 12:11 AM, M. Edward (Ed) Borasky wrote:
If you have raw data, I'd be happy to explore it for you - email me off-list if you need an NDA or something like that.
Any data we have should be shared with all or none. We don't have a Fedora NDA AFAIK.
Best,
jzb
AFAIK amazon does not make available any usage data at all.
I think could get you data from HP Public Cloud if that is useful. Let me know.
...Juerg
Dennis _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
On 03/18/2015 06:38 PM, David Gay wrote:
Greetings!
We sort of ran out of time in today's Cloud WG meeting, but I did want to ask:
What are your thoughts on AMI lifetimes? That is to say, how long should EC2 AMIs exist before they're deleted? A few points to consider:
I feel like I should know this, but I don't.
If a user spins up an AMI and then it's deleted by the provider, do they still have their instance(s) or do they lose the ability to create new images?
That would color my response a bit.
Do we know how other projects handle theirs? If I go to spin up a Foo Linux release from 2 years ago, is the AMI still there?
At minimum, we should probably delete any AMIs that are no longer a supported version of Fedora, and I'd also be for deleting any TC, alpha, beta, etc. AMIs - especially once a release is published. So, for instance, any F21 alpha, beta, etc. AMIs can probably go to the great bit bucket in the sky at this point.
Also wonder if this is something we need to have ACK'ed by FESCo?
Best,
jzb
On 03/19/2015 08:54 AM, Joe Brockmeier wrote:
On 03/18/2015 06:38 PM, David Gay wrote:
Greetings!
We sort of ran out of time in today's Cloud WG meeting, but I did want to ask:
What are your thoughts on AMI lifetimes? That is to say, how long should EC2 AMIs exist before they're deleted? A few points to consider:
I feel like I should know this, but I don't.
If a user spins up an AMI and then it's deleted by the provider, do they still have their instance(s) or do they lose the ability to create new images?
The instances they already started would still run and be available, but they wouldn't be able to spin up anything new. If creating/killing instances is something they do a lot (autoscaling groups, worker farms, etc) then that could hose them just as surely as killing their existing instances.
That would color my response a bit.
Do we know how other projects handle theirs? If I go to spin up a Foo Linux release from 2 years ago, is the AMI still there?
At minimum, we should probably delete any AMIs that are no longer a supported version of Fedora, and I'd also be for deleting any TC, alpha, beta, etc. AMIs - especially once a release is published. So, for instance, any F21 alpha, beta, etc. AMIs can probably go to the great bit bucket in the sky at this point.
Also wonder if this is something we need to have ACK'ed by FESCo?
Best,
jzb
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
On Thu, Mar 19, 2015 at 1:58 PM, Ryan Brown rybrown@redhat.com wrote:
On 03/19/2015 08:54 AM, Joe Brockmeier wrote:
On 03/18/2015 06:38 PM, David Gay wrote:
Greetings!
We sort of ran out of time in today's Cloud WG meeting, but I did want to ask:
What are your thoughts on AMI lifetimes? That is to say, how long should EC2 AMIs exist before they're deleted? A few points to consider:
I feel like I should know this, but I don't.
If a user spins up an AMI and then it's deleted by the provider, do they still have their instance(s) or do they lose the ability to create new images?
The instances they already started would still run and be available, but they wouldn't be able to spin up anything new. If creating/killing instances is something they do a lot (autoscaling groups, worker farms, etc) then that could hose them just as surely as killing their existing instances.
We (HP Public Cloud) even had a customer yell at us after we changed the name of an image because that was what they were using in their tooling to spin up instances. I'd be very careful not to break customers and clearly communicate and have the image lifecycle officially documented. We always advise customers to take snapshots and use those instead of the base images so that the image is fully under their control. But customers don't always listen...
That would color my response a bit.
Do we know how other projects handle theirs? If I go to spin up a Foo Linux release from 2 years ago, is the AMI still there?
At minimum, we should probably delete any AMIs that are no longer a supported version of Fedora, and I'd also be for deleting any TC, alpha, beta, etc. AMIs - especially once a release is published. So, for instance, any F21 alpha, beta, etc. AMIs can probably go to the great bit bucket in the sky at this point.
Also wonder if this is something we need to have ACK'ed by FESCo?
Best,
jzb
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
-- Ryan Brown / Software Engineer, Openstack / Red Hat, Inc. _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
On 03/18/2015 06:38 PM, David Gay wrote:
Greetings!
We sort of ran out of time in today's Cloud WG meeting, but I did want to ask:
What are your thoughts on AMI lifetimes? That is to say, how long should EC2 AMIs exist before they're deleted? A few points to consider:
- AMIs only cost us for storage, so it's not a *huge* cost to maintain a public AMI
- At the same time, there are a lot of AMIs, since we build 2-4 per AWS region per build, and that number is growing
- There are 9 regions now, and we have 2 virtualization types, and 2 volume types, as well (9 regions * 2 * 2 = 36 AMIs per Base image build, 18 for Atomic builds (since they are only available in HVM format))
- This total number will only grow larger as we add instance-store AMIs, and so on
- This isn't even taking into account any costs we'll have once we secure a deal with other providers like HP, Rackspace, and GCE, to maintain public images on their services
I propose we have some sort of discussion regarding how long cloud image builds should be available on services like AWS. I suspect this will resolve to having different lifetimes for scratch, test, RC, final, and maybe other build types.
In my experience, folks expect AMIs to stick around for a long time. AMI IDs work their way into all sorts of places (scripts, ansible playbooks, CloudFormation templates, and a zillion others) so I think that deleting an AMI before the end of the supported lifetime of a release would make people sad*.
I think it's reasonable to only offer that support for release AMIs, and scratch/test/RC/etc AMIs would have to have a shorter lifetime.
My (very rough) calculations put the cost of storing 4 AMIs per region at around $5/month, so each "final" AMI built will cost $5 * 13-16 months, or $65-$80. Not expensive, but not exactly free. Costs will, of course, vary for other cloud providers.
Of course, there's no replacement for checking the metrics to see what folks actually use.
* or angry, because their scripts will be broken
On Wednesday, March 18, 2015 06:38:13 PM David Gay wrote:
Greetings!
We sort of ran out of time in today's Cloud WG meeting, but I did want to ask:
What are your thoughts on AMI lifetimes? That is to say, how long should EC2 AMIs exist before they're deleted? A few points to consider:
- AMIs only cost us for storage, so it's not a *huge* cost to maintain a
public AMI - At the same time, there are a lot of AMIs, since we build 2-4 per AWS region per build, and that number is growing - There are 9 regions now, and we have 2 virtualization types, and 2 volume types, as well (9 regions * 2 * 2 = 36 AMIs per Base image build, 18 for Atomic builds (since they are only available in HVM format)) - This total number will only grow larger as we add instance-store AMIs, and so on - This isn't even taking into account any costs we'll have once we secure a deal with other providers like HP, Rackspace, and GCE, to maintain public images on their services
your math is off, there should only be 9 Atomic as we only build it for x86_64 where we build the base for i386 and x86_64 so you have two arches by 2 image types by 9 regions
Dennis
I propose we have some sort of discussion regarding how long cloud image builds should be available on services like AWS. I suspect this will resolve to having different lifetimes for scratch, test, RC, final, and maybe other build types.
Any input is appreciated. We can certainly talk about this at next week's meeting, as well.
-- David _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
On 2015-03-18 15:38, David Gay wrote:
What are your thoughts on AMI lifetimes? That is to say, how long should EC2 AMIs exist before they're deleted? A few points to consider:
- AMIs only cost us for storage, so it's not a *huge* cost to
maintain a public AMI - At the same time, there are a lot of AMIs, since we build 2-4 per AWS region per build, and that number is growing - There are 9 regions now, and we have 2 virtualization types, and 2 volume types, as well (9 regions * 2 * 2 = 36 AMIs per Base image build, 18 for Atomic builds (since they are only available in HVM format)) - This total number will only grow larger as we add instance-store AMIs, and so on - This isn't even taking into account any costs we'll have once we secure a deal with other providers like HP, Rackspace, and GCE, to maintain public images on their services
I propose we have some sort of discussion regarding how long cloud image builds should be available on services like AWS. I suspect this will resolve to having different lifetimes for scratch, test, RC, final, and maybe other build types.
Different clouds have different norms. AWS is a cloud where anyone can share an image with the world without having to "secure a deal" with anyone. People there are used to sifting through the resulting giant lists of images. AFAIK, that isn't the case in any of the other clouds you listed -- those all use much shorter, curated lists, so for those it may certainly make sense to prune unsupported releases after a period of time.
In AWS people seem to essentially expect images to last forever and hardcode their IDs into templates and launch configurations all over the place that will break when the image goes away, so removing them is not a decision to take lightly. The best data I am aware of for that are the MirrorManager statistics for Fedora 8, which can show how long that release remained popular after we created the Cloud SIG and began releasing newer images ourselves. Perhaps someone with the appropriate level of infrastructure log access would be able to shed some more light on that. Also keep in mind that the storage costs for AWS images are somewhat nuanced, as some images are stored compressed or sparsely, some can share storage, and so on.
Now, for pre-release images, I think rotating them out after a period of time makes sense no matter which cloud it is. Exactly how long that should be could reasonably be cloud-dependent for stuff like betas, but probably not as much for nightlies or TCs.
Cool, thanks for the input everyone. A short summary of what's been discussed:
1. Everyone seems to agree that we should hope to get usage data from AWS, but as Dennis mentioned (and I expect), there isn't any usage data available for AMIs. If there is such a console, I've never seen one.
2. Q: "If a user spins up an AMI and then it's deleted by the provider, do they still have their instance(s) or do they lose the ability to create new images?" A: "The instances they already started would still run and be available, but they wouldn't be able to spin up anything new. If creating/killing instances is something they do a lot (autoscaling groups, worker farms, etc) then that could hose them just as surely as killing their existing instances."
3. We should probably look into how other projects handle their AMIs. I think the consensus here though is that whatever lifetime we have for releases, the alpha, beta, TC, RC, and other testing builds can -- and should -- be safely eliminated after the release. There's no good reason I can think of that someone would yell at us for deleting a test build AMI of a release that's already happened.
4. Anyone have an opinion on jzb wondering if we should run this by FESCo?
5. Regarding this exchange:
"at around $5/month, so each "final" AMI built will cost $5 * 13-16 months, or $65-$80. Not expensive, but not exactly free. Costs will, of course, vary for other cloud providers." "your math is off, there should only be 9 Atomic as we only build it for x86_64 where we build the base for i386 and x86_64 so you have two arches by 2 image types by 9 regions"
Whatever the costs will be per AMI, I can tell you that what I've heard from people in the cloud WG is that we want a number of different AMIs *per build*. A new build currently results in 6 AMIs: 2 for atomic (standard + gp2) and 4 for base (standard + gp2, both for HVM and paravirtual virtualization). I spoke with gholms some time back and I think we determined that we're also going to want instance-store AMIs, as well as *encrypted* EBS AMIs. So, maybe there should be some discussion on that with the full group, since that will result in a large number of AMIs. If we end up building that many different combinations of storage types, volume types, and virtualization types, we're talking a fair amount of AMIs being kept up during the release process, because of how many image builds go through Koji. Dennis mentioned to me that there is some sort of Koji bug that, if fixed, would builds be marked as either "real life" or "scratch", so we could at least cut down a bit on the number of AMIs being built.
I think this discussion should continue a bit more based on all that. However, I *do* move that I immediately delete at least all the alpha, beta, TC, and RC builds that were created back when we were working on F21.
-- David
On 03/24/2015 01:51 PM, David Gay wrote:
Cool, thanks for the input everyone. A short summary of what's been discussed:
- Everyone seems to agree that we should hope to get usage data from
AWS, but as Dennis mentioned (and I expect), there isn't any usage data available for AMIs. If there is such a console, I've never seen one.
- Q: "If a user spins up an AMI and then it's deleted by the
provider, do they still have their instance(s) or do they lose the ability to create new images?" A: "The instances they already started would still run and be available, but they wouldn't be able to spin up anything new. If creating/killing instances is something they do a lot (autoscaling groups, worker farms, etc) then that could hose them just as surely as killing their existing instances."
- We should probably look into how other projects handle their AMIs.
I think the consensus here though is that whatever lifetime we have for releases, the alpha, beta, TC, RC, and other testing builds can -- and should -- be safely eliminated after the release. There's no good reason I can think of that someone would yell at us for deleting a test build AMI of a release that's already happened.
- Anyone have an opinion on jzb wondering if we should run this by
FESCo?
- Regarding this exchange:
"at around $5/month, so each "final" AMI built will cost $5 * 13-16 months, or $65-$80. Not expensive, but not exactly free. Costs will, of course, vary for other cloud providers." "your math is off, there should only be 9 Atomic as we only build it for x86_64 where we build the base for i386 and x86_64 so you have two arches by 2 image types by 9 regions"
Whatever the costs will be per AMI, I can tell you that what I've heard from people in the cloud WG is that we want a number of different AMIs *per build*. A new build currently results in 6 AMIs: 2 for atomic (standard + gp2) and 4 for base (standard + gp2, both for HVM and paravirtual virtualization). I spoke with gholms some time back and I think we determined that we're also going to want instance-store AMIs, as well as *encrypted* EBS AMIs. So, maybe there
+1 for instance-store AMIs, they're incredibly useful. I think it's a good time to think about what AMIs we should be producing though, and talk to FESCO about just how much we're willing to spend on providing AMIs.
should be some discussion on that with the full group, since that will result in a large number of AMIs. If we end up building that many different combinations of storage types, volume types, and virtualization types, we're talking a fair amount of AMIs being kept up during the release process, because of how many image builds go through Koji. Dennis mentioned to me that there is some sort of Koji bug that, if fixed, would builds be marked as either "real life" or "scratch", so we could at least cut down a bit on the number of AMIs being built.
I think this discussion should continue a bit more based on all that. However, I *do* move that I immediately delete at least all the alpha, beta, TC, and RC builds that were created back when we were working on F21.
+1 sounds like a good plan for now.
-- David _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
-
David Gay -
Dennis Gilmore -
Garrett Holmstrom -
Joe Brockmeier -
Juerg Haefliger -
M. Edward (Ed) Borasky -
milanisko k -
Ryan Brown