On 14 February 2018 at 18:18, Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
On Wed, 2018-02-14 at 17:28 +0100, Petr Šplíchal wrote:
> Hi!
>
> During the last days there have been concerns raised regarding
> what is an appropriate content for the tests namespace. [1] My
> original idea was to enable sharing tests even across branches
> of the same component, not only for tests to be used by
> completely different packages. The initial examples might have
> been a bit misleading in this respect. One of the main points
> still holds:
>
> > * Tests might follow a different branching pattern than the
> > dist-git repo, leading to code duplication
>
> From the feedback from developers I feel they always keep on
> mind and care a lot about the maintenance costs. So it
> perfectly makes sense to me if they want to keep and maintain
> tests in a separate repo instead of merging/cherry-picking
> between dist-git branches.
>
> When, for a particular package, it is the most efficient way to
> maintain tests in a separate repo why should we discourage from
> this approach? There are packages where it makes more sense to
> store test code directly in dist-git. And it is still an
> option. But why should we enforce this for all?
>
> Please share your thoughts and real-life examples. For those
> who are not familiar with the topic there is a new wiki page
> with a summary of the Share Test Code approach [2].
I don't have any problem with the concept *in itself*; in fact I
know of another reason to have something like it. That is
'generic' tests, tests we want to run on all packages, or at
least a very large set of packages - like the tests currently
running in Taskotron, which are generally run on all packages
(rpmgrill, rpmdeplint...) or a large subset (the Python tests).
Yes, sharing tests useful for multiple packages makes sense in
general. Test code for generic tests can be stored basically
anywhere but not in specific components. However I see some
additional benefits of having a dedicated tests namespace:
* Source of inspiration and real-life examples (at one place)
* Potential for workflow automation (tests CI, fedpkg --tests)
* Easy cherry-picking/backporting upstream/downstream tests
The difference I see when comparing these shared tests to generic
tests mentioned above like rpmdeplint is that you don't want to
reference these from the individual package repos (e.g. using the
Standard Test Interface), but just run them on all packages.
If the expectation is to run tests only on a subset of rpms then a
question comes: How to store the list of relevant components or
criteria to decide which packages are to be tested or not. Here
the Flexible Metadata Format might come handy in the future:
https://github.com/psss/fmf
What I did think was maybe a concern is that we should figure
out in advance the squishy human consequences of different
technical approaches.
Basically this boiled down to "who is responsible for these
'shared' tests, and who gets to decide which 'shared' tests
can/should block packages?"
Looking at the proposal, I think it actually has at least
workable initial explicit/implicit answers to this, if I'm
reading it correctly. Anyone can create a shared test
repository (and is therefore "responsible" for maintaining it),
but the definition of which tests are run on which packages
remains in the package repositories. Thus the package
maintainer(s) retain the ultimate choice over which tests are
run against their packages (and thus can pick which shared tests
are run, and disable shared tests if they are no longer properly
testing their package).
Yes. This is exactly true. The maintainer of the package repo has
always the last word on what is to be executed for qualification.
It can be done by referencing only suitable tests from the shared
repo or selecting an appropriate branch or even a specific commit
to ensure that future test development does not break stable
gating (and only manually update reference after review of shared
test updates) or even completely disable tests by removing the
reference.
The question 'who decides which tests block which packages'
is
left a bit up in the air, but in fact no more so than it already
was for package-specific tests...
Right. Do we want to change this? Specify this more strictly?
Like exactly defining requirements which an Always Ready Operating
System has to meet? And then mapping these requirements to the
test coverage? (Here again FMF mentioned above might come handy.)
psss...
> --
> Adam Williamson
> Fedora QA Community Monkey
> IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
>
http://www.happyassassin.net