May 2015 - qa-devel - Fedora mailing-lists

[Fedora QA] #436: SSH access to systems in Beaker lab
by fedora-badges 20 Apr '16

20 Apr '16

#436: SSH access to systems in Beaker lab --------------------------------------+--------------------- Reporter: atodorov | Owner: tflink Type: defect | Status: new Priority: major | Milestone: Component: Blocker bug tracker page | Version: Keywords: | Blocked By: Blocking: | --------------------------------------+--------------------- = bug description = Currently systems in Beaker lab can be accessed only through bastion.fp.o which is not as convenient as direct SSH into the system. There's also the question whether or not to open the systems directly to the Internet. This needs to be discussed with infra. Filing here so it doesn't get lost. -- Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/436> Fedora QA <http://fedorahosted.org/fedora-qa> Fedora Quality Assurance

1 4

Fedmsg Emitting
by Martin Krizek 03 Jun '15

03 Jun '15

Before we start sending fedmsgs we need to discuss a few things. We don't have to find solutions to all these problems, just keep them in mind when designing the solution we're going to start with: 1. How often do we send fedmsg a) per-task b) per-update c) per-build a) and b): we can list affected packages in a fedmsg. I am not sure if there are any limits when it comes to fedmsg size. Whether the infra folks would be more happy with less larger or more smaller fedmsgs (or it doesn't matter). I guess c) allows to easier filtering in FMN. 2. Who do we target: users, systems or both The issue here is with tasks that repeatedly test one update. Currently we check if there's a bodhi update comment with the same result already and if so, we don't post the comment again. To do something like that with fedmsgs we'd have to have a code running somewhere that would check against its database whether an incoming result is a duplicate or not. The question is where the code would run. Bodhi comes to mind since it already has information about updates and so is good for tasks that work with bodhi updates. However, there might be tasks that work with something else, like composes. In this case we'd probably have the code on taskotron systems. So if we target systems we'd just send all results in fedmsgs and let the systems consume them and do whatever they want to do with them (e.g. bodhi can squash all the tasks relevant to specific update and notify the maintainer of the package via fedmsg about the result). If we target users, we'd have to have some logic to limit rate of fedmsgs ourselves but that would mean hiding some of the results (although duplicates) from the world. So the question here is where to put the 'deduplication logic'. Emitting all results is the simplest solution as a starting point. Thoughts? Thanks, Martin

3 6

[Fedora QA] #468: Fedora 22 Translation (L10n) Test Day
by fedora-badges 02 Jun '15

02 Jun '15

#468: Fedora 22 Translation (L10n) Test Day --------------------------------------+------------------------ Reporter: anipeter | Owner: tflink Type: defect | Status: new Priority: major | Milestone: Fedora 22 Component: Blocker bug tracker page | Version: Keywords: | Blocked By: Blocking: | --------------------------------------+------------------------ Translation (L10n) Test Day for Fedora 22 proposed for March 17, being a date before the translation deadline. Thanks Ani Peter (ani) -- Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/468> Fedora QA <http://fedorahosted.org/fedora-qa> Fedora Quality Assurance

1 6

2015-06-01 @ 14:00 UTC - Fedora QA Devel Meeting
by Tim Flink 02 Jun '15

02 Jun '15

# Fedora QA Devel Meeting # Date: 2015-06-01 # Time: 14:00 UTC (https://fedoraproject.org/wiki/Infrastructure/UTCHowto) # Location: #fedora-meeting-1 on irc.freenode.net We'll be holding a QA devel meeting on Monday. It's been a while since we had a full planning meeting so this will be another one Tim Proposed agenda =============== Status Updates -------------- - Please have #info statements ready so we can get through this as quickly as possible Upcoming Items -------------- - Cloud Migrations - Phabricator Hosting Open Floor ---------- - TBD

2 2

New qadevel staging machine to poke at
by Tim Flink 01 Jun '15

01 Jun '15

I ended up spending some time on the qadevel-stg system yesterday and I think that I have the phabricator bits to the point where they're ready to be tested (self-signed cert used, usual warnings will pop up). https://phab.qadevel.stg.fedoraproject.org/ I've built new phabricator packages, loaded a snapshot of our production phabricator and this machine is capable of hosting git repositories (once you add your ssh key, write access is ssh-only), so I made snapshots of all the repos and they're currently available on qadevel.stg: https://phab.qadevel.stg.fedoraproject.org/diffusion/ I still need to get buildbot and a few other bits working before it's truely done but I do think it's almost to the point where we could migrate off our current cloud instance once final freeze is over. If you have some time, please poke at it and let me know what you think. Tim

2 1

Upcoming Changes to Cloud-Based Infrastructure (qadevel, testdays, taskotron-demo)
by Tim Flink 30 May '15

30 May '15

The new cloud infrastructure has been deployed in infra and at some point, we'll need to migrate the machines that we're using there to that new cloud. I'd like to have all this done sooner than later - definitely before F23 branches. I've listed my initial thoughts on the 3 services we have in the cloud below - thoughts, comments etc. would be appreciated. Tim taskotron-demo -------------- This will be pretty straight forward and should stay in the cloud. testdays -------- I think that josef has been putting some time into this as of late and if we're going to continue using it, I wonder if it should be moved into infra instead of to the new cloud. If not, I think that we should change the hostname to testdays.cloud.fedoraproject.org to work around the SSL issue that folks have been complaining about. qadevel ------- I'd prefer not to migrate this to the new cloud - I have a new stg host set up in infra and I think that this will be a good chance for us to migrate to that new setup. This will involve a new hostname, redirects and fixing old links but it was something that was going to happen eventually. I've not heard any feedback on whether to move our git repos from bitbucket to phabricator but I also think that's secondary to moving to a new machine.

2 1

Proposal to CANCEL - 2015-05-25 Fedora QA Devel Meeting
by Tim Flink 22 May '15

22 May '15

Many folks were focused on testing this week and with the US holiday on Monday, I won't be around to lead the meeting. Therefore, I propose that we cancel the qadevel meeting that we'd usually have on Monday. If there are objections to this and someone else wants to lead the meeting, reply here otherwise consider the meeting canceled. Tim

1 0

Proposal to CANCEL - 2015-05-18 Fedora QA Devel Meeting
by Tim Flink 15 May '15

15 May '15

With F22 final testing in full swing, I'm not sure there's much point in having a meeting so I'm proposing that we cancel the usual Monday meeting. If there are objections, reply here otherwise consider the meeting canceled. Tim

1 0

To RHEL or Not to RHEL?
by Tim Flink 14 May '15

14 May '15

This was brought up a little while ago and we decided to put off the discussion a little bit but I'd like to re-start the conversation before we get too much farther with disposable clients. My plan for how our hosts would be set up once we deploy support for disposable clients is this: - virthosts would have N buildslave processes running on them - each buildslave would launch VMs for disposable clients as needed - each virthost would have access to a shared filesystem used to store at least VM images, maybe logs and other data While virt-in-virt is possible, I'd prefer to avoid the extra complexity and performance penalty and figure that running on bare metal makes more sense. If we disable local task execution, there should be little risk of one task disrupting other stuff on that virthost that can't be easily reverted. All of our infrastructure outside of the database host are running Fedora (F21 at the moment) which is fine but I'm wondering about migrating to RHEL for everything that doesn't have to run Fedora. For observers of this discussion - I'm not in any way asserting that Fedora isn't capable of running our production services well. I'm simply acknowledging that it will likely be more expensive (in terms of human resources) to run Fedora on our production systems and I'm wondering whether it's wise to be spending that much of our rather limited resources to do it. In my mind, the big pros/cons for the two approaches are: -------------------------- Moving to RHEL -------------------------- Pros: * Less frequent updates, less downtime required for kernel updates etc. * Using the same bits that infra is already using and there should be less maintenance work on our part Cons: * Will make python and library compatibility more of an issue (if we're going to support tasks on el7 anyways, this point is kind of moot since we'll need to do it anyways) * No buildbot packages for el7 - I'm not willing to take on epel7 maintenance for buildbot 0.8.x and I'd rather not see packages for it in epel7, either. We'll be migrating to buildbot 0.9 before long and I'd prefer to avoid compat packages. * Migration and some code changes may be required * May need to take on maintenance of EPEL packages and/or deal with folks not testing updates before making version changes in EPEL (I've gotten burned by that before with blockerbugs) -------------------------- Staying With Fedora -------------------------- Pros: * Dogfooding * Production environment is closer to dev environments * Don't have to worry about as many compat packages * No EPEL * More testing of Fedora - we are QA after all Cons: * We'll probably hit problems before infra does since we'd be upgrading everything more often * More frequent migrations since Fedora releases are supported for 1 year * More frequent downtime for updates * Will need to be more diligent about keeping dev/stg on updates-testing so that we don't get any nasty surprises in production I think that infra would like to see us migrate to RHEL so that the Taskotron systems are more like everything else that supports Fedora but I don't think that they'd object to us running Fedora as long as we accept responsibility for keeping everything working and actually do it. If we do decide that we'd prefer to keep Fedora, we'll discuss it with them but I wanted to start the discussion here before bothering the infra folks with it. I suspect that our virthosts for Taskotron will be slightly different from infra's either way - we still need to figure out a shared filesystem for the images (gluster is the first thing that comes to mind but there are other options) and none of infra's virthosts have that. They do use gluster for a few things, so I suspect that it'd be less work to set that up on rhel than fedora. I'm a bit torn on this - as much as I'd like see us use Fedora, I think it's important to provide as much value to Fedora as we can and I'm not sure that our time is best spent admin-ing and upgrading systems. That being said, migrating to el7 wouldn't be trivial but once we get over the initial hurdle of packaging and deployment I think it will require less maintenance. What does everyone else think? Keep in mind that you'll be the folks helping with all of this :) Tim

4 9

watching projects in Phabricator, Herald rules simplified
by Kamil Paral 14 May '15

14 May '15

Hello everyone, in the last week I've spent a bit of time investigating the possibilities of how to watch for ticket/revision updates in Phabricator, and found out that some new features in Phab greatly simplify this. I have updated our Phab wiki page and included most common use cases: https://fedoraproject.org/wiki/QA/Phabricator In a nutshell: * you can now start "watching" projects which automatically sends you all tasks (tickets) updates. No more Herald rules needed. Unfortunately, this does not apply for revisions (patches). * you can now easily subscribe to tickets without a project defined (usually bug reports). This way you can help us classify and triage them and make sure they go unnoticed (that unfortunately happened in the past). * Taskotron lovers can now easily watch for all Taskotron-related new revisions (patches) by using the repos-taskotron meta-project [1]. We will make sure that this meta-project is always linked with all taskotron-related repositories, and you don't need to update your Herald rules for every new Taskotron sub-project. * You can now easily decide between a permanent item subscription or just one-time notification by choosing either "Add emails to CC <name>" or "Send an email to <name>". I have tested it for the last week and everything seems to work properly for me. If you have some troubles, ask. Kamil [1] https://phab.qadevel.cloud.fedoraproject.org/project/profile/23/

1 0

CI for Taskotron and Related Projects
by Tim Flink 13 May '15

13 May '15

I've just re-started the CI on qadevel (the future one, not cloud) that mkrizek set up a while back. It was destroyed when I started working on replacing qadevel.cloud but never finished it. The link is: https://qadevel.fedoraproject.org/buildmaster/waterfall Note that there will be ssl cert warnings since it's self signed. Once the thing gets rebuilt for real, those ssl warnings will go away. Tim

3 4

2015-05-11 @ 14:00 UTC - Fedora QA Devel Meeting
by Tim Flink 11 May '15

11 May '15

# Fedora QA Devel Meeting # Date: 2015-05-11 # Time: 14:00 UTC (https://fedoraproject.org/wiki/Infrastructure/UTCHowto) # Location: #fedora-meeting-1 on irc.freenode.net We'll be holding a QA devel meeting on Monday and I actually remembered to send out an announcement this time :) Tim Proposed agenda =============== Status Updates -------------- - Please have #info statements ready so we can get through this as quickly as possible Upcoming Items -------------- - To RHEL or not to RHEL? Open Floor ---------- - TBD

1 1

Planned Outage: Taskotron - 2015-05-11 19:00 UTC
by Tim Flink 08 May '15

08 May '15

Planned Outage: Taskotron - 2015-05-11 19:00:00 UTC There will be an outage starting at 2015-05-11 19:00:00 UTC, which will last no more than 3 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2015-05-11 19:00:00 UTC' ----------------- Reason for outage ----------------- With Fedora 22 final freeze starting on Tuesday, I want to get all the Taskotron hosts updated before that happens. I suspect that the outage will be closer to 1.5 hours but 3 hours leaves a safety net in case something unexpected Happens Any jobs which would have been scheduled during the outage will be recorded and run after the outage is complete. ----------------- Affected Services ----------------- Taskotron ------------------- Unaffected Services ------------------- Ask Fedora - http://ask.fedoraproject.org/ Badges - https://badges.fedoraproject.org/ BFO - http://boot.fedoraproject.org/ Blockerbugs - https://qa.fedoraproject.org/blockerbugs/ Bodhi - https://admin.fedoraproject.org/updates/ Buildsystem - http://koji.fedoraproject.org/ GIT / Source Control - pkgs.fedoraproject.org Darkserver - https://darkserver.fedoraproject.org/ DNS - ns-sb01.fedoraproject.org, ns02.fedoraproject.org, ns04.fedoraproject.org, ns05.fedoraproject.org Docs - http://docs.fedoraproject.org/ Elections - https://admin.fedoraproject.org/voting Email system Fedmsg busmon - http://apps.fedoraproject.org/busmon Fedora Account System - https://admin.fedoraproject.org/accounts/ Fedora Community - https://admin.fedoraproject.org/community/ Fedora Calendar - https://apps.fedoraproject.org/calendar/ Fedora Hosted - https://fedorahosted.org/ Fedora OpenID - https://id.fedoraproject.org/ Fedora People - http://fedorapeople.org/ Main Website - http://fedoraproject.org/ Mirror List - https://mirrors.fedoraproject.org/ Mirror Manager - https://admin.fedoraproject.org/mirrormanager/ Package Database - https://admin.fedoraproject.org/pkgdb/ Secondary Architectures Spins - http://spins.fedoraproject.org/ Start - http://start.fedoraproject.org/ Torrent - http://torrent.fedoraproject.org/ Wiki - http://fedoraproject.org/wiki/

1 1

Additional Taskotron Clients
by Tim Flink 08 May '15

08 May '15

For at least the second time in the last couple weeks, the queue in production was way too long (~120 at one point). To help with this in the future, I added 4 new x86_64 clients to production. Hopefully this'll help keep the queue length lower and wait times down to something reasonable. Tim

2 2

Fedora Projects Tooling Meeting
by Adam Miller 08 May '15

08 May '15

Hello all, As a result from the WhenIsGood I sent out[0], the meeting to discuss Fedora Projects Tooling has been set for 2015-05-13 at 14:00 UTC https://apps.fedoraproject.org/calendar/infrastructure/2015/5/13/?tzone=UTC… Hope to see everyone there! -AdamM [0] - https://lists.fedoraproject.org/pipermail/qa-devel/2015-May/001206.html

1 0

new project: taskotron-vmbuilder
by Kamil Paral 05 May '15

05 May '15

Related to our disposable clients initiative in Taskotron, I'm working on ways how to boot and manage those clients. In order to know how to boot them, I need to know how they are created (e.g. cloud images require special magic with cloud-init). That's why I spent some time in the past looking at different ways how to either create custom images or re-use existing ones. The simplest approach turned out to be (in my view) a tool called virt-builder. It's a part of libguestfs project and it's a super easy way to create Fedora disk images - they have them already pre-created and stored on the server, so it just downloads them and customize them a little bit per your requirements: $ virt-builder fedora-21 --root-password password:fedora --install vim-enhanced --update I decided to try to use it for creating the base image for our disposable clients. The idea is that virt-builder is run once a day on a server and it generates a fresh new VM disk image. We then use this disk image for all our disposable clients spawned in the future, until a new disk image is generated the next day. Rinse and repeat. The benefit of this approach (over e.g. official cloud images which might be updated once per month or twice per year) is: * the clients are always up-to-date * it's very light on bandwidth - instead of downloading updates for every client on every task run, it's done once per day, centrally * it's much faster - the clients just boot and execute the task, they don't spend time upgrading each boot * it's less error-prone - there is a never-ending story of failures when you communicate over network thousands times per day. We know it really well. By updating the client just once, the chance for a network error is much lower, and we can easily repeat it if it fails. Also, it's not a big deal if it completely fails some day, our tasks continue to be executed without a problem, just with a day-old packages installed. * can be integrated with fedmsg "updates pushed" notification to be almost instantly in sync with the latest updates * it helps with rescheduling tasks and reproducing issues - if we keep last X-number of VM images, we can easily replay past tasks with exactly the same package set as it had during that time Subsequently I realized that we will probably also want to have several different base disk images, not just one. For example, certain GNOME checks might require most of GNOME installed, which would mean downloading a lot of packages every time they are run. But with virt-install, we can pre-create different base images for different purposes, and then let the task define which image to use. So for GNOME checks, we would prepare a base image with full GNOME installed. To make all of this manageable, I created a wrapper around virt-builder and called it taskotron-vmbuilder (ideas for a better name are welcome). The code is available here: https://bitbucket.org/fedoraqa/taskotron-vmbuilder The concept is that you can define templates and then run vmbuilder which takes care of the rest - it uses virt-builder to create the image, it makes sure that libtaskotron is installed, it configures stuff like root password and ssh key. We can add any other functionality that is needed. The simplest way to try this is this: $ ./vmbuilder.py templates/f21-minimal.yaml There are sample templates for minimal installation and Server and Workstation products. For more details, look into READMEs, it's fairly well documented, I think. My idea is that we will run this daily on the server, store the resulting images somewhere, and update the pointer to the "current one". I still haven't created project and repo in Phab, but if you approve this as a way forward, I will. Thanks, Kamil

3 4

WhenIsGood - Scheduling a discussion around Fedora Projects Planning
by Adam Miller 04 May '15

04 May '15

Hello all, There was recently a discussion on the Fedora Rel-Eng list as the result of a Project Planning workflow[0]. As this was discussed further in irc it was decided that this is likely a conversation that should be had across more Fedora groups to see if there's a solution that we would all like to utilize. If we can all agree on a workflow and utility we'll hopefully remove the potential for duplicating efforts or requesting multiple solutions to the same problem be hosted by the Infrastructure team. In an attempt to not cross-post I am duplicating this post on the rel-eng, qa, infrastructure, docs, buildsys, and env-and-stacks mailing lists. I would like to request that if there are any discussions around this, please rely to the rel-eng list so we can consolidate the discussion. All interested parties please fill out the following WhenIsGood so that we can find a meeting time to discuss this. http://whenisgood.net/pip4kjd Thank you, -AdamM [0] - https://lists.fedoraproject.org/pipermail/rel-eng/2015-April/019806.html

1 0

22 Final TC1 live fail: timeout due to SMP performance bug?
by Adam Williamson 04 May '15

04 May '15

Hey folks - just thought I'd note this. The openQA run for Final TC1 Workstation live failed; after 'clicking' the 'Install' button on the welcome screen it waited 30 seconds for anaconda to install then timed out and failed. I tried locally and it took right around 30 secs for anaconda to run in a 2-CPU VM here too, but it *did* run. So I think nothing was broken, it just would have taken longer than openQA waited to launch. I think this is probably related to that bug about multi-CPU VMs running slowly - does anyone recall the bug number? When I tried a single-CPU VM instead, anaconda only took 14 seconds to launch. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net

2 3

2015-05-04 Fedora QA Devel Meeting Minutes
by Tim Flink 04 May '15

04 May '15

========================== #fedora-meeting-1: qadevel ========================== Minutes: http://meetbot.fedoraproject.org/fedora-meeting-1/2015-05-04/qadevel.2015-0… Minutes (text): http://meetbot.fedoraproject.org/fedora-meeting-1/2015-05-04/qadevel.2015-0… Log: http://meetbot.fedoraproject.org/fedora-meeting-1/2015-05-04/qadevel.2015-0… Meeting summary --------------- * Roll Call (tflink, 14:07:14) * status updates (tflink, 14:08:32) * sent remote execution patch for review (mkrizek, 14:09:31) * LINK: https://phab.qadevel.cloud.fedoraproject.org/D356 (mkrizek, 14:09:35) * fixed blacklisting releases in trigger (mkrizek, 14:09:41) * LINK: https://phab.qadevel.cloud.fedoraproject.org/D360 (mkrizek, 14:09:45) * now resuming work on fedmsg support (mkrizek, 14:09:50) * I published a new project taskotron-vmbuilder that should make it easy to create up-to-date VM images for different purposes. Awaiting feedback on the list. (kparal, 14:12:32) * no updates on taskotron-related tasks (jskladan, 14:14:59) * some updates to flask-skeleton app (alembic, foundation) (jskladan, 14:15:00) * spent a some time converting the "legacy" TG2 code of TestDaysAll to Flask (will need a day or two more) (jskladan, 14:15:00) * tflink status report (tflink, 14:17:48) * finally made progress on the mysterious depcheck 'inferior arch' issue - appears to be repodata related (tflink, 14:17:54) * submitted patches to abort 'inferior arch' failures with an explanation message and not report them to bodhi (tflink, 14:17:54) * talked to beaker devs about deployment strategies, databases (tflink, 14:18:35) * Next 2 Weeks (tflink, 14:19:54) * open floor (tflink, 14:21:48) Meeting ended at 14:23:14 UTC. Action Items ------------ Action Items, by person ----------------------- * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * tflink (29) * kparal (13) * mkrizek (12) * jskladan (7) * zodbot (3) Generated by `MeetBot`_ 0.1.4 .. _`MeetBot`: http://wiki.debian.org/MeetBot

1 0

[Fedora QA] #470: Can't connect to testdays app
by fedora-badges 01 May '15

01 May '15

#470: Can't connect to testdays app --------------------------------------+------------------------ Reporter: dustymabe | Owner: tflink Type: defect | Status: new Priority: major | Milestone: Fedora 22 Component: Blocker bug tracker page | Version: Keywords: | Blocked By: Blocking: | --------------------------------------+------------------------ = bug description = When I try to connect to http://testdays.qa.fedoraproject.org/testdays/all_events Firefox won't work because of the following: "This site uses HTTP Strict Transport Security (HSTS) to specify that Firefox only connect to it securely. As a result, it is not possible to add an exception for this certificate." Because: "The certificate is not trusted because it is self-signed." Does anyone else have this problem? -- Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/470> Fedora QA <http://fedorahosted.org/fedora-qa> Fedora Quality Assurance

1 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

qa-devel May 2015