On Wed, Nov 18, 2009 at 20:54:22 +0100, Christoph Wickert christoph.wickert@googlemail.com wrote:
I have no idea why we didn't spot this bug before. I have tested every nightly and they all were fine, but they were created on a different machine then the final images. Also installing LXDE from the package repositories works fine, so you can already install LXDE on Fedora 12.
This is something that would be good to understand for the future. We'd really like to have the final spins be tested. Maybe we need RCs for each spin? Releng will possibly want Spins SIG help if we go that route.
On Wed, 2009-11-18 at 14:15 -0600, Bruno Wolff III wrote:
This is something that would be good to understand for the future. We'd really like to have the final spins be tested. Maybe we need RCs for each spin? Releng will possibly want Spins SIG help if we go that route.
We did have some RCs that were published, but given that there were 18 live images to make, getting them all made, published, and wait for feedback on all of them between every RC would have been extremely limiting in our ability to have fast turnaround on RCs.
In order to accommodate so many spins, the spin process had to be very hands off from releng point of view. I have to trust that the spin owners are vigilant in ensuring their spin works, so that releng can compose and publish all of them in the very limited amount of time we have.
Basically we'll need spin owners to participate when we reach RC stage, and as each day passes, if any package gets tagged for the release that is in one of your spins, you'll have the responsibility of re-ensuring your spin works.
Am Mittwoch, den 18.11.2009, 15:08 -0800 schrieb Jesse Keating:
On Wed, 2009-11-18 at 14:15 -0600, Bruno Wolff III wrote:
This is something that would be good to understand for the future. We'd really like to have the final spins be tested. Maybe we need RCs for each spin? Releng will possibly want Spins SIG help if we go that route.
We did have some RCs that were published, but given that there were 18 live images to make, getting them all made, published, and wait for feedback on all of them between every RC would have been extremely limiting in our ability to have fast turnaround on RCs.
In order to accommodate so many spins, the spin process had to be very hands off from releng point of view. I have to trust that the spin owners are vigilant in ensuring their spin works, so that releng can compose and publish all of them in the very limited amount of time we have.
Basically we'll need spin owners to participate when we reach RC stage, and as each day passes, if any package gets tagged for the release that is in one of your spins, you'll have the responsibility of re-ensuring your spin works.
I can only test what I'm told to test. I tested all the nightlies and all worked. I didn't even know there were images at http://alt.fedoraproject.org/pub/alt/stage/12-RC.4/ and according to Bill these were not really the rc4 images but the final ones. So were were this rc images that you mentioned at the beginning of your mail?
For the next release I suggest to send a mail to every spin owner and tell him to test each image rel-eng makes. I'm sure owners and the SIG will be happy to help rel-eng wherever they can.
Regards, Christoph
On Thu, Nov 19, 2009 at 02:23:42 +0100, Christoph Wickert christoph.wickert@googlemail.com wrote:
I can only test what I'm told to test. I tested all the nightlies and all worked. I didn't even know there were images at http://alt.fedoraproject.org/pub/alt/stage/12-RC.4/ and according to Bill these were not really the rc4 images but the final ones. So were were this rc images that you mentioned at the beginning of your mail?
There were email messages about the RC candidates, though not on the Spins SIG list. I think the expectation was that testing nightlies would be good enough. Now we know differently. Though it would be nice to know what was different so that we can make the nightlies better.
They really were RC4 images. RC4 was the last release candidate and hence those images also became the final release. (Unlike the kernel release process, the Fedora release process has release candidates being release candidates.)
For the next release I suggest to send a mail to every spin owner and tell him to test each image rel-eng makes. I'm sure owners and the SIG will be happy to help rel-eng wherever they can.
I don't think sending individual messages to spin owners by releng can be expected. I think it should be expected that spin owners are subscribed to the spins-sig list and that maybe this list should be copied along with the other lists (I think the messages went out to the test list) when new RCs are announced.
On Thu, 2009-11-19 at 02:23 +0100, Christoph Wickert wrote:
I can only test what I'm told to test. I tested all the nightlies and all worked. I didn't even know there were images at http://alt.fedoraproject.org/pub/alt/stage/12-RC.4/ and according to Bill these were not really the rc4 images but the final ones. So were were this rc images that you mentioned at the beginning of your mail?
For the next release I suggest to send a mail to every spin owner and tell him to test each image rel-eng makes. I'm sure owners and the SIG will be happy to help rel-eng wherever they can.
RC4 was one that was published, but if you read the rest of my mail, you'll see that there just isn't enough time to A) produce every spin for every RC point (there were 3 RCs in one day I seem to recall), and B) stage them somewhere that the public can get to them, and C) wait for spin owners to read email, download giant files, and give feedback. There just isn't enough time for this. Spin owners need to be able to create their spin locally and test that.
Should I communicate more at each RC point, absolutely, that's on me. But there is no chance in hell I'll be able to create 10 live images, across 2 arches for each and ever RC.
On Thu, 2009-11-19 at 09:33 -0800, Jesse Keating wrote:
On Thu, 2009-11-19 at 02:23 +0100, Christoph Wickert wrote:
I can only test what I'm told to test. I tested all the nightlies and all worked. I didn't even know there were images at http://alt.fedoraproject.org/pub/alt/stage/12-RC.4/ and according to Bill these were not really the rc4 images but the final ones. So were were this rc images that you mentioned at the beginning of your mail?
For the next release I suggest to send a mail to every spin owner and tell him to test each image rel-eng makes. I'm sure owners and the SIG will be happy to help rel-eng wherever they can.
RC4 was one that was published, but if you read the rest of my mail, you'll see that there just isn't enough time to A) produce every spin for every RC point (there were 3 RCs in one day I seem to recall), and B) stage them somewhere that the public can get to them, and C) wait for spin owners to read email, download giant files, and give feedback. There just isn't enough time for this. Spin owners need to be able to create their spin locally and test that.
Should I communicate more at each RC point, absolutely, that's on me. But there is no chance in hell I'll be able to create 10 live images, across 2 arches for each and ever RC.
Note, I'm not trying to blame Christoph for this current situation. Its as much my fault as it is his. My statements above speak to the future of how things should work, not an expectation that this is what was being done for F12. We clearly had some communication gaps in F12 and that's what led to this situation.
On Thu, Nov 19, 2009 at 09:38:01 -0800, Jesse Keating jkeating@redhat.com wrote:
Note, I'm not trying to blame Christoph for this current situation. Its as much my fault as it is his. My statements above speak to the future of how things should work, not an expectation that this is what was being done for F12. We clearly had some communication gaps in F12 and that's what led to this situation.
I think the main things we want to do are make sure the nightly builds are being built the same way as the RCs. That will mininize the chances that we miss potential problems. (Though a last second change could still cause a problem, I think this is less likely to happen and to go unnoticed.) And to make sure the spin owners are aware of the nightly builds and are checking them for issues from time to time and frequently in the run up to the alpha, beta and release building periods. And if there are special build requirements such as for AOS, they need to be documented and we need to be extra careful about the nightly builds being built the same way as alpha, beta and RCs are.
-
Bruno Wolff III -
Christoph Wickert -
Jesse Keating