DNF Migration testing overview

[Mike Ruckman]

Greetings Testers!

With the F22 switch from yum to dnf as a package manager, there was a need to
figure out what the scope would be for QA testing. Naturally, to define the
scope for a subset, I had to figure out (at least in a general sense) how
large the complete set is. There's a multitude of complication vectors with a
switch like this - especially since dnf is specifically designed to not be a
drop in replacement for yum.

First question: Who/what all uses yum?
--------------------------------------

Swapping out yum is the equivalent of someone coming up with a new form of
hemoglobin, then trying to figure out what all could go wrong when you make
the swap (tl;dr: Everything). The first to come to mind is anyone who installs
packages from the CLI. But then you have all the GUI front-ends (PackageKit
and Apper) and all the build tools - not to mention the installer.

Some places yum touches:
 - Oz/ImageFactory (Building Cloud images)
 - koji (RPM-based build system)
 - pungi (Build installation trees and isos)
 - ABRT (Automatic Bug Reporting Tool)
 - Anaconda (Fedora installer)
 - liveimage-creator (tool for creating live images)
 - cloud-init (tool for configuring freshly launched cloud images)
 - FedUp (tool for updating Fedora between releases)
 - Software Center and Apper (Default software installers)
 - rolekit (tool to easily deploy roles to a Server installation)
 - virt-builder and friends (tools for manipulating virtualized environments)

All of those make up a large portion of the Fedora ecosystem - everything from
composing images to users being able to update their software packages, and
this doesn't even touch the different ways each of the above actually *uses*
yum functionality - just that yum somehow impacts these tools.

Second question: What's QA responsible for?
-------------------------------------------

According to the QA wiki page: "Fedora QA is the project which covers all
testing of the software that makes up Fedora. It's our goal to continually
improve the quality of Fedora releases and updates." A more literal translation
of that passage would indicate that QA is responsible for all the testing, but
a more nuanced interpretation says that QA is responsible for releases and
updates to those releases. Luckily for us in QA, we seem to operate from the
more nuanced interpretation.

So, out of that list above, which things would QA need to test?
 - Anaconda (which we already test heavily)
 - ABRT
 - FedUp
 - Software Center/Apper
 - rolekit

While those are the things easily placed directly under QA for testing, other
teams would likely appreciate help in testing their tools. This doesn't
necessarily equate to more work for QA, since a lot of the feedback the other
teams (Release Engineering, Infra) get is from QA testing the output their
tools create - releases and updates.

Also, those 5 things aren't trivial to test either. Here's the partial
breakdown:

Anaconda:
 - GUI installation
   - Do all the offered mechanisms work?
 - Package selection (installing all the packages selected)
 - Remote repositories
 - text based installation
 - kickstart installation
   - This is non-trivial to test

ABRT:
 This tool uses a bunch of yum tricks and workarounds to get the needed
 information (AIUI)
 - We have 36 testcases for this (some will need updated)

FedUp:
 - F20-F22 upgrades
 - How to handle migrating existing systems from yum to dnf. Each will have
   different installed package sets due to utilizing different depsolvers.
 - This will require more testing during the F23 cycle than during the F22
   cycle

Software Center and Apper:
 These tools aren't just a front-end for yum/dnf, but tie into the libraries
 dnf uses. For the switch to dnf, this is hawkey and libsolv. It could be
 argued that this doesn't need to be tested as part of the migration, but they
 for sure fall under the Default application functionality Final criterion.

rolekit:
 We get a small pass for this one, since there's currently only two supported
 roles: Domain Controller and Database. rolekit relies on yum for installing
 package sets for the role, so it'll have to be ported and tested. Hopefully
 this requires minimal extra testing.

Third question: What do we currently have to work with?
---------------------------------------

Our wiki has a plethora of testcases and documentation we can utilize
throughout this transition. We should be able to use/re-purpose large portions
of our existing testcases to get good test coverage of the change. We might
also have to take a look at the Release criteria going forward to make sure
there aren't any yum-isms left over. All this means is that we don't have to
start from scratch when it comes to testing this.

Testcases:

  I count 54 testcases currently available to test ABRT, FedUp, Yum,
  Installation Repos, and kickstarts. Many of these will likely need revised
  and more will likely need added to get the coverage we need.

Matrix Changes:

  For the image sanity checks, we'll have to make sure a successful run with
  dnf is within the bounds required of yum. We'll also have to go through and
  make sure that existing instructions within testcases use dnf (and still
  work). There might be some F22 specific criteria we need to add in order to
  set a baseline for what has to work with dnf for the release.

Kickstarts:

  We already use kickstarts for most of the heavy lifting on the releng side
  of things, so we have access to a known set of things that have to work with
  kickstarts. Testing these should be pretty simple and easy to grow/use going
  forward.

Conclusion
==========

Migrating from yum to dnf isn't going to be an easy task. On the upside, it
will help us identify all the places we use yum without thinking about it and
may not actively test for. So while it's going to be a lot of work, it lets us
lay the groundwork for any other hemoglobin-esque changes Fedora sees in the
future. We've already had an Anaconda/DNF test day (more are likely to come)
and will probably need to schedule some test days for other large projects as
well.

There's probably stuff that I missed, or overestimated the difficulty on (I'm
not terribly familiar with all the areas this encompasses), so I don't think
this is a complete list. If you noticed something that should be here please
reply! It'll really help us to get as complete a list of things to test that
we can so we don't end up hitting a bunch of snags when we get to release.

I've put most of this email up on the wiki [0] - so feel free to add things
there as well.

Thanks!

[0] https://fedoraproject.org/wiki/QA:DNF_Migration_Test_Overview

-- 
// Mike 
--
Fedora QA
freenode: roshi
http://roshi.fedorapeople.org