Re: Requiring package test instructions
On Sat, 2016-07-16 at 14:35 +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Wed, Jul 13, 2016 at 09:44:36AM -0700, Adam Williamson wrote:
>
> On Wed, 2016-07-13 at 11:41 +0200, Vít Ondruch wrote:
> >
> > If I should provide "how to test" information, it should be probably
> > script which should be run by AutoQA or something. But TBH, I have no
> > idea how to provide such script. There is precisely zero information
> > about this stuff on places I would expect (this one [1] as an example).
> > There is also zero support in tools such as fedpkg and dist-git.
>
> If you're willing to do this, that's great, and we would definitely
> like to enable it. I know the Taskotron (it hasn't been AutoQA for a
> while) team is working on enabling generic dist-git integration (which
> is basically what you're asking for), but I don't know the current
> status of that...CCing tflink and kparal who may have more information.
>
> >
> > I am pretty sure this would be more valuable then some "how to test"
> > document.
>
> I think it varies substantially by package, but I don't think we're
> realistically going to get sufficient automated testing for all
> packages, including ones that are important to users.
I don't think koschei has been mentioned in this thread yet.
For library type packages that's the testing we want to do, and if
I'm not mistaken, koschei compilations already are started for dependent
packages when an rpm hits updates-testing, if enabled for those packages.
This kind of testing complements the usability testing that is
suitable for leaf packages.
So adding testing instructions might help for some types of packages,
for other types we should be working on integrating koschei more
closely, in particular flagging the update that breaks builds.
Well, there can be functional aspects to library testing too; take GTK+
as an interesting case, you can have a GTK+ update which passes all its
internal tests and all or most packages rebuild fine against it, but it
changes some bit of CSS or something and makes text go white in
Firefox, or something like that. It's happened before. ;)
But yeah, there are definitely cases - as I said - where the style of
human testing that Bodhi is kinda designed around doesn't really fit,
and we could definitely improve things.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net