Hi folks! For those who aren't aware, Fedora openQA is set up to test the Atomic installer image nightly - that's the image that uses anaconda to deploy a fixed Atomic host payload. For each day's Rawhide, Branched and post-release stable composes, if an Atomic installer image is present, openQA will test it.
The 'compose check' emails sent to test@ and devel@ for Rawhide and Branched cover those tests, so if the image stops working for Rawhide or Branched, you should see a failure report in the 'compose check' emails. (At present the image doesn't even seem to be being *generated* for Rawhide, but whenever it does appear, it'll be tested). Currently we're not set up to email a report for post-release nightlies - I keep meaning to set this up, but haven't got around to it. Still, now the openQA deployment is public, you can easily check on the result each day if you want: usually, when you go to https://openqa.fedoraproject.org/%C2%A0, you should see a build like 'Build23_Postrelease_20151207' on the front page, with just one test. That's the test for today's F23 nightly compose. You can click on the build name and it'll show you the test, if it's failed, you can click on it for more details of what went wrong.
The test has been failing for the last little while due to an issue in the test itself (I won't bore you with the details, but let's say you learn all sorts of interesting things about console fonts when working on a screenshot-matching test automation system...), but I've just submitted a fix for that; if it passes review I'll land it tomorrow, and tests from 2015-12-08 onwards shouldn't have the problem. (The 2015-12-07 test probably will).
Thanks folks!
On Mon, Dec 7, 2015, at 08:39 PM, Adam Williamson wrote:
Hi folks! For those who aren't aware, Fedora openQA is set up to test the Atomic installer image nightly - that's the image that uses anaconda to deploy a fixed Atomic host payload. For each day's Rawhide, Branched and post-release stable composes, if an Atomic installer image is present, openQA will test it.
Thanks for working on this!
Where is the source code for the tests?
I'm also interested in figuring out how this intersects with the Anaconda upstream tests over time, as those cover things like kickstart automation.
On Mon, 2015-12-07 at 21:15 -0500, Colin Walters wrote:
On Mon, Dec 7, 2015, at 08:39 PM, Adam Williamson wrote:
Hi folks! For those who aren't aware, Fedora openQA is set up to test the Atomic installer image nightly - that's the image that uses anaconda to deploy a fixed Atomic host payload. For each day's Rawhide, Branched and post-release stable composes, if an Atomic installer image is present, openQA will test it.
Thanks for working on this!
Where is the source code for the tests?
https://bitbucket.org/rajcze/openqa_fedora
openQA is kind of its own animal - yell if you need any help working out how it works.
I'm also interested in figuring out how this intersects with the Anaconda upstream tests over time, as those cover things like kickstart automation.
There isn't really much of an intersection as things stand - they're just different processes. I'm fairly comfortable viewing the anaconda project's own tests as project-level CI-style test automation, while our use of openQA is much more at the distribution level and more about validating distribution composes as entire artefacts in themselves.
The most interesting property of openQA per se is that it's a rather different style of automation to things like kickstart testing: openQA basically runs a VM and then interacts with it exactly like an extremely bored human QA intern would, it expects certain things to show up on the screen then it types stuff and clicks - it uses some qemu features for interacting with the VM, but so far as the OS is concerned, it's almost entirely indistinguishable from a human being sitting there whacking a keyboard. So it can test things that don't really show up in more abstract forms of automated testing (but equally, it's not very good at testing some other things).
On Mon, 2015-12-07 at 19:00 -0800, Adam Williamson wrote:
On Mon, 2015-12-07 at 21:15 -0500, Colin Walters wrote:
On Mon, Dec 7, 2015, at 08:39 PM, Adam Williamson wrote:
Hi folks! For those who aren't aware, Fedora openQA is set up to test the Atomic installer image nightly - that's the image that uses anaconda to deploy a fixed Atomic host payload. For each day's Rawhide, Branched and post-release stable composes, if an Atomic installer image is present, openQA will test it.
Thanks for working on this!
Where is the source code for the tests?
https://bitbucket.org/rajcze/openqa_fedora
openQA is kind of its own animal - yell if you need any help working out how it works.
To expand on this a bit - there isn't exactly a 'cloud installer image test'. The test repo has a bunch of test steps, and there's a sort of front controller - main.pm - which causes each test run to actually load some combination of those test steps depending on the settings for that test. The settings for a test are kind of a combination of that test's own innate settings, the settings for the 'machine' it's running on, and possibly settings deriving from the image being tested; all of this stuff is defined in the 'templates' file.
You can see some of this stuff in the web UI. If you look at today's test - https://openqa.fedoraproject.org/tests/461%C2%A0- you can click 'Settings', and see the settings for that test - ARCH, BACKEND, BUILD, CANNED etc. Then you can go to main.pm and see that it will actually send the test process down particular paths depending on some of those settings. For instance, at line 122 there's a fork depending on whether 'KICKSTART' is set - for this test it isn't, so we go down one particular fork there.
You can see in the web UI the actual individual test steps that were scheduled - _boot_to_anaconda , _software_selection , disk_guided_empty , etc. Those are files in the tests/ subdirectory. The behaviour within test steps themselves can also depend on the test settings.
The actual 'test' being run on the image is called 'default_install' , and you can see in the 'templates' file the particular settings that are set for that test:
https://bitbucket.org/rajcze/openqa_fedora/src/35c42da79bbd68644933dd0ca1cc3...
since it's pretty much our 'default' test path, only one variable gets set (PACKAGE_SET). You can see further down in 'templates' that the other test definitions set more variables, and in main.pm (and sometimes in the individual test files) you can follow what changes those settings cause...
On Mon, Dec 7, 2015 at 6:39 PM, Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks! For those who aren't aware, Fedora openQA is set up to test the Atomic installer image nightly - that's the image that uses anaconda to deploy a fixed Atomic host payload.
It exists! I've been looking around for such a thing for a day and only found old blog posts. It's really non-obvious how to find it in koji. I can find lives. I can find other atomics. But somehow this one is like the G train in Brooklyn (OK the G *eventually* does show itself).
https://openqa.fedoraproject.org/ , you should see a build like 'Build23_Postrelease_20151207' on the front page,
OK I click on this, it shows the test, and that it failed. Any chance of it eventually linking to the image it tested so that it's possible to fall back to a manual test (?) I don't know if that's even useful. If it fails the auto test, all that probably matters is the fail summary. More likely is if it passes all auto tests to have a link to the image so a manual test can try to blow it up, right?
On Tue, 2015-12-08 at 11:26 -0700, Chris Murphy wrote:
On Mon, Dec 7, 2015 at 6:39 PM, Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks! For those who aren't aware, Fedora openQA is set up to test the Atomic installer image nightly - that's the image that uses anaconda to deploy a fixed Atomic host payload.
It exists! I've been looking around for such a thing for a day and only found old blog posts. It's really non-obvious how to find it in koji. I can find lives. I can find other atomics. But somehow this one is like the G train in Brooklyn (OK the G *eventually* does show itself).
It's not built by koji, which is why you can't find it there. It's an installer image, it's built by pungi, just like netinst and DVD images.
fedfind can find it, though. ;) That's what fedfind does! It finds fed(ora)!
[adamw@adam openqa_fedora (core-upload %)]$ fedfind images -m Postrelease INFO:fedfind.cli:Finding images for: Fedora 23 Postrelease 20151208 ... https://dl.fedoraproject.org/pub/alt/atomic/testing/23-20151208/Cloud_Atomic... https://dl.fedoraproject.org/pub/alt/atomic/testing/23-20151208/Cloud_Atomic... ...
the boot.iso in those results is the same image - just as for the other install trees boot.iso is the same image as netinst.iso, for the Cloud_Atomic install tree, boot.iso is the same image as the Cloud_Atomic .iso.
https://openqa.fedoraproject.org/ , you should see a build like 'Build23_Postrelease_20151207' on the front page,
OK I click on this, it shows the test, and that it failed. Any chance of it eventually linking to the image it tested so that it's possible to fall back to a manual test
So, answer one: it actually does. The ISO tested is on the Logs & Assets page, down at the bottom, under Assets. However, we'd really prefer if people didn't download ISOs that way - our openqa really isn't set up to be an image download server. I should actually send a patch upstream to disable it, and have our deployment do that...
Answer two: we *could* make openQA post a proper download URL for the ISO it tested, but it'd be a moderate amount of finicky work and it's probably not worth it. The thing to bear in mind is that our openQA setup uses fedfind to find the images, in fact that's 90% of the reason fedfind exists; anything openQA tests, you can find with fedfind.
(?) I don't know if that's even useful. If it fails the auto test, all that probably matters is the fail summary.
Not always - we try to have openQA upload as much info as possible so you can usually figure out the actual cause of a failure just from the stuff openQA uploads, but it's not always the case. Sometimes you do have to reproduce the test manually to figure out what's going on.
More likely is if it passes all auto tests to have a link to the image so a manual test can try to blow it up, right?
Eh, my take is that we don't/shouldn't exactly design our manual test processes around openQA. This testing (of the post-release nightly cloud images) is kind of a bonus thing I rigged up just because maxamillion asked and it wasn't too difficult; the main point of openQA is to aid in pre-release testing, and of course we have a more developed test process there, where we have the regular 'nomination' of nightly composes for manual testing, with the wiki pages with download links and all the rest of it. We could certainly stand to draw up a proper process for manual testing of post-release images, if we're going to be releasing them officially, which apparently we are, but I'm not the guy who's been keeping up on that stuff so I don't want to leap in, I'm sure some folks already have ideas for doing that.
On Tue, Dec 8, 2015 at 11:22 PM, Adam Williamson adamwill@fedoraproject.org wrote:
On Tue, 2015-12-08 at 11:26 -0700, Chris Murphy wrote:
On Mon, Dec 7, 2015 at 6:39 PM, Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks! For those who aren't aware, Fedora openQA is set up to test the Atomic installer image nightly - that's the image that uses anaconda to deploy a fixed Atomic host payload.
It exists! I've been looking around for such a thing for a day and only found old blog posts. It's really non-obvious how to find it in koji. I can find lives. I can find other atomics. But somehow this one is like the G train in Brooklyn (OK the G *eventually* does show itself).
It's not built by koji, which is why you can't find it there. It's an installer image, it's built by pungi, just like netinst and DVD images.
fedfind can find it, though. ;) That's what fedfind does! It finds fed(ora)!
I think I'd be helpful if the releng dashboard listed this build along with the other cloud images. I estimate my recall half-life for fedfind is about 15 days. Does anyone else think it'd be useful to have the nightly atomic installer ISO listed at https://apps.fedoraproject.org/releng-dash/ ? Or maybe even an Atomic specific section on the dashboard?
https://openqa.fedoraproject.org/ , you should see a build like 'Build23_Postrelease_20151207' on the front page,
OK I click on this, it shows the test, and that it failed. Any chance of it eventually linking to the image it tested so that it's possible to fall back to a manual test
So, answer one: it actually does. The ISO tested is on the Logs & Assets page, down at the bottom, under Assets.
When I click on Postrelease_20151209, I end up here https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=23...
There isn't anything down at the bottom, definitely no Logs & Assets page. Same with the other listings on the man openqa page... https://drive.google.com/open?id=0B_2Asp8DGjJ9cUpnY2ZuY0lIV2M
More likely is if it passes all auto tests to have a link to the image so a manual test can try to blow it up, right?
Eh, my take is that we don't/shouldn't exactly design our manual test processes around openQA. This testing (of the post-release nightly cloud images) is kind of a bonus thing I rigged up just because maxamillion asked and it wasn't too difficult; the main point of openQA is to aid in pre-release testing, and of course we have a more developed test process there, where we have the regular 'nomination' of nightly composes for manual testing, with the wiki pages with download links and all the rest of it. We could certainly stand to draw up a proper process for manual testing of post-release images, if we're going to be releasing them officially, which apparently we are, but I'm not the guy who's been keeping up on that stuff so I don't want to leap in, I'm sure some folks already have ideas for doing that.
Gotcha. Thanks!
On 12/09/2015 03:48 PM, Chris Murphy wrote:
Does anyone else think it'd be useful to have the nightly atomic installer ISO listed at https://apps.fedoraproject.org/releng-dash/ ? Or maybe even an Atomic specific section on the dashboard?
Yes! I totally think it would.
Best,
jzb
On Wed, Dec 9, 2015 at 2:02 PM, Joe Brockmeier jzb@redhat.com wrote:
On 12/09/2015 03:48 PM, Chris Murphy wrote:
Does anyone else think it'd be useful to have the nightly atomic installer ISO listed at https://apps.fedoraproject.org/releng-dash/ ? Or maybe even an Atomic specific section on the dashboard?
Yes! I totally think it would.
https://fedorahosted.org/fedora-infrastructure/ticket/5026
So that probably needs more clarity, now that I've already submitted it. I asked for a Cloud specific section rather than Atomic specific. So if Atomic specific is better organization, add that to the ticket.
On Wed, Dec 9, 2015 at 3:47 PM, Chris Murphy lists@colorremedies.com wrote:
On Wed, Dec 9, 2015 at 2:02 PM, Joe Brockmeier jzb@redhat.com wrote:
On 12/09/2015 03:48 PM, Chris Murphy wrote:
Does anyone else think it'd be useful to have the nightly atomic installer ISO listed at https://apps.fedoraproject.org/releng-dash/ ? Or maybe even an Atomic specific section on the dashboard?
Yes! I totally think it would.
https://fedorahosted.org/fedora-infrastructure/ticket/5026
So that probably needs more clarity, now that I've already submitted it. I asked for a Cloud specific section rather than Atomic specific. So if Atomic specific is better organization, add that to the ticket.
Fedfind finds these cloud specific products built nightly. The last one, Docker, is in its own category, I guess. So what items in this listing make sense to list in a hypothetical Cloud specific heading on the releng dashboard? Or should it only list Atomic specific builds?
https://dl.fedoraproject.org/pub/alt/atomic/testing/23-20151209/Cloud-Images... https://dl.fedoraproject.org/pub/alt/atomic/testing/23-20151209/Cloud-Images... https://dl.fedoraproject.org/pub/alt/atomic/testing/23-20151209/Cloud-Images... https://dl.fedoraproject.org/pub/alt/atomic/testing/23-20151209/Cloud-Images... https://dl.fedoraproject.org/pub/alt/atomic/testing/23-20151209/Cloud-Images... https://dl.fedoraproject.org/pub/alt/atomic/testing/23-20151209/Cloud-Images... https://dl.fedoraproject.org/pub/alt/atomic/testing/23-20151209/Cloud-Images... https://dl.fedoraproject.org/pub/alt/atomic/testing/23-20151209/Cloud-Images... https://dl.fedoraproject.org/pub/alt/atomic/testing/23-20151209/Cloud-Images... https://dl.fedoraproject.org/pub/alt/atomic/testing/23-20151209/Cloud-Images... https://dl.fedoraproject.org/pub/alt/atomic/testing/23-20151209/Cloud_Atomic...
https://dl.fedoraproject.org/pub/alt/atomic/testing/23-20151209/Docker/x86_6...
On Wed, 2015-12-09 at 13:48 -0700, Chris Murphy wrote:
On Tue, Dec 8, 2015 at 11:22 PM, Adam Williamson adamwill@fedoraproject.org wrote:
On Tue, 2015-12-08 at 11:26 -0700, Chris Murphy wrote:
On Mon, Dec 7, 2015 at 6:39 PM, Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks! For those who aren't aware, Fedora openQA is set up to test the Atomic installer image nightly - that's the image that uses anaconda to deploy a fixed Atomic host payload.
It exists! I've been looking around for such a thing for a day and only found old blog posts. It's really non-obvious how to find it in koji. I can find lives. I can find other atomics. But somehow this one is like the G train in Brooklyn (OK the G *eventually* does show itself).
It's not built by koji, which is why you can't find it there. It's an installer image, it's built by pungi, just like netinst and DVD images.
fedfind can find it, though. ;) That's what fedfind does! It finds fed(ora)!
I think I'd be helpful if the releng dashboard listed this build along with the other cloud images. I estimate my recall half-life for fedfind is about 15 days. Does anyone else think it'd be useful to have the nightly atomic installer ISO listed at https://apps.fedoraproject.org/releng-dash/ ? Or maybe even an Atomic specific section on the dashboard?
I don't have anything much to do with the dashboard; it does some of the same stuff as fedfind, but that's all I know.
So, answer one: it actually does. The ISO tested is on the Logs & Assets page, down at the bottom, under Assets.
When I click on Postrelease_20151209, I end up here https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=23...
There isn't anything down at the bottom, definitely no Logs & Assets page. Same with the other listings on the man openqa page... https://drive.google.com/open?id=0B_2Asp8DGjJ9cUpnY2ZuY0lIV2M
That's an overview page. The Logs & Assets tab is available for each individual *test* page. Here's the overview for today:
https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=23...
Click on the green dot and you see the individual test:
https://openqa.fedoraproject.org/tests/632
and there you see the Logs & Assets tab. The overview seems a bit odd in this particular case because we only run a single test on post- release nightlies ATM.
On Wed, Dec 9, 2015 at 3:00 PM, Adam Williamson adamwill@fedoraproject.org wrote:
That's an overview page. The Logs & Assets tab is available for each individual *test* page. Here's the overview for today:
https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=23...
Click on the green dot and you see the individual test:
OOH - OK it's not at all obvious the dot is a link. Is it possible for the test text itself having the link?
On Wed, Dec 9, 2015 at 3:44 PM, Chris Murphy lists@colorremedies.com wrote:
On Wed, Dec 9, 2015 at 3:00 PM, Adam Williamson adamwill@fedoraproject.org wrote:
That's an overview page. The Logs & Assets tab is available for each individual *test* page. Here's the overview for today:
https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=23...
Click on the green dot and you see the individual test:
OOH - OK it's not at all obvious the dot is a link. Is it possible for the test text itself having the link?
Or even the word "Details" to the right of the dot, which is the link.
On Wed, 2015-12-09 at 15:44 -0700, Chris Murphy wrote:
On Wed, Dec 9, 2015 at 3:00 PM, Adam Williamson adamwill@fedoraproject.org wrote:
That's an overview page. The Logs & Assets tab is available for each individual *test* page. Here's the overview for today:
https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=23...
Click on the green dot and you see the individual test:
OOH - OK it's not at all obvious the dot is a link. Is it possible for the test text itself having the link?
Well, all things are possible, but I don't think I care enough to do it, certainly not now (I'm in the middle of something else). https://github.com/os-autoinst/openQA/blob/master/templates/test/overview.ht... is the template for the page in question, if you feel like wrapping your head around Mojolicious HTML templating - http://mojolicio.us/perldoc/Mojolicious/Guides/Rendering#Embedded-Perl
-
Adam Williamson -
Chris Murphy -
Colin Walters -
Joe Brockmeier