2:50 p.m.
One of the features for Taskotron that we've been planning since the
beginning was a way for contributors to maintain their own automated
tasks/tests which would be run during a package's lifecycle.
I'm happy to say that we're almost to this milestone and wanted to get
some feedback from devel@ on the specifics of what we're planning WRT
where these automated tasks will be stored and the execution modes that
we're planning to support. Our current plan is written up at:
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
The hope is that by making it easier for contributors to write
automated tasks and making the model completely self-service and
convention drive, there will be a lot more automated checks for
packages than we currently have for Fedora.
Please read through the wiki page I mentioned above and give us
feedback on whether what we're planning to implement is going to be
useful or if there are areas of the plan which could be improved.
Thanks,
Tim
3:11 p.m.
On Mon, Oct 03, 2016 at 01:50:33PM -0600, Tim Flink wrote:
One of the features for Taskotron that we've been planning since
the
beginning was a way for contributors to maintain their own automated
tasks/tests which would be run during a package's lifecycle.
I'm happy to say that we're almost to this milestone and wanted to get
some feedback from devel@ on the specifics of what we're planning WRT
where these automated tasks will be stored and the execution modes that
we're planning to support. Our current plan is written up at:
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
The hope is that by making it easier for contributors to write
automated tasks and making the model completely self-service and
convention drive, there will be a lot more automated checks for
packages than we currently have for Fedora.
Please read through the wiki page I mentioned above and give us
feedback on whether what we're planning to implement is going to be
useful or if there are areas of the plan which could be improved.
As someone who requested this feature in the past, it looks fine
to me.
However the page is a bit thin on actual details. For example, where
will the tests run? (eg. In a VM?) Does it have access to the built
RPMs? Are the built RPMs installed?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
3:35 p.m.
On Mon, 3 Oct 2016 21:11:58 +0100
"Richard W.M. Jones" <rjones(a)redhat.com> wrote:
On Mon, Oct 03, 2016 at 01:50:33PM -0600, Tim Flink wrote:
> One of the features for Taskotron that we've been planning since the
> beginning was a way for contributors to maintain their own automated
> tasks/tests which would be run during a package's lifecycle.
>
> I'm happy to say that we're almost to this milestone and wanted to
> get some feedback from devel@ on the specifics of what we're
> planning WRT where these automated tasks will be stored and the
> execution modes that we're planning to support. Our current plan is
> written up at:
>
>
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
>
> The hope is that by making it easier for contributors to write
> automated tasks and making the model completely self-service and
> convention drive, there will be a lot more automated checks for
> packages than we currently have for Fedora.
>
> Please read through the wiki page I mentioned above and give us
> feedback on whether what we're planning to implement is going to be
> useful or if there are areas of the plan which could be improved.
As someone who requested this feature in the past, it looks fine
to me.
However the page is a bit thin on actual details. For example, where
will the tests run? (eg. In a VM?) Does it have access to the built
RPMs? Are the built RPMs installed?
The tests will run in a VM which is spun up specifically for the tests
being run and will be destroyed after the tests have been completed.
The Fedora version running in that VM will reflect the applicable
version when possible (eg the release from koji builds).
The tests will have access to the built RPMs.
The built RPMs will not be installed prior to the tests starting.
That being said, most of that is up for some discussion if folks have
different/better ideas. We're anticipating anticipate change requests
to make the whole system once folks start using the system more.
There is documentation available for writing tasks [1] but there is
also an option of forgoing the detailed task specification if you have
an executable script which adheres to conventions listed in the wiki
page.
Tim
[1]https://qa.stg.fedoraproject.org/docs/libtaskotron/latest/writingtasks....
This link point does point to a stg system - we're in the middle of some
changes to how our docs are built and most changes haven't made it to
production yet - this is a more reliable link for the moment and is
updated as new releases are made.
3:21 p.m.
On Mon, Oct 03, 2016 at 01:50:33PM -0600, Tim Flink wrote:
One of the features for Taskotron that we've been planning since
the
beginning was a way for contributors to maintain their own automated
tasks/tests which would be run during a package's lifecycle.
I'm happy to say that we're almost to this milestone and wanted to get
some feedback from devel@ on the specifics of what we're planning WRT
where these automated tasks will be stored and the execution modes that
we're planning to support. Our current plan is written up at:
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
Sounds reasonable.
How does the environment in which those tasks are run look?
Would it be possible for example to run a test which launches
a qemu VM to test stuff that requires a functional installation
and a reboot?
Zbyszek
4:31 p.m.
On Mon, 3 Oct 2016 20:21:42 +0000
Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl> wrote:
On Mon, Oct 03, 2016 at 01:50:33PM -0600, Tim Flink wrote:
> One of the features for Taskotron that we've been planning since the
> beginning was a way for contributors to maintain their own automated
> tasks/tests which would be run during a package's lifecycle.
>
> I'm happy to say that we're almost to this milestone and wanted to
> get some feedback from devel@ on the specifics of what we're
> planning WRT where these automated tasks will be stored and the
> execution modes that we're planning to support. Our current plan is
> written up at:
>
>
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
Sounds reasonable.
How does the environment in which those tasks are run look?
Would it be possible for example to run a test which launches
a qemu VM to test stuff that requires a functional installation
and a reboot?
It will be a VM running updated Fedora.
Rebooting during execution isn't something we explicitly support but it
has been requested before. I've filed an RFE ticket to track the
feature:
https://phab.qadevel.cloud.fedoraproject.org/T849
Our intent is to get this into place before F26 knowing full well that
we're going to be missing features that some folks will need. We're not
quite stupid/arrogant enough to pretend that we know what's needed to
support all the different kinds of tasks that people will want to run.
Instead of guessing what people need, we're intending to start adding
features once there are use cases from people who know the components
they work on FAR better than we do.
Tim
2:09 p.m.
On Mon, Oct 03, 2016 at 08:21:42PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Oct 03, 2016 at 01:50:33PM -0600, Tim Flink wrote:
> One of the features for Taskotron that we've been planning since the
> beginning was a way for contributors to maintain their own automated
> tasks/tests which would be run during a package's lifecycle.
>
> I'm happy to say that we're almost to this milestone and wanted to get
> some feedback from devel@ on the specifics of what we're planning WRT
> where these automated tasks will be stored and the execution modes that
> we're planning to support. Our current plan is written up at:
>
>
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
Sounds reasonable.
How does the environment in which those tasks are run look?
Would it be possible for example to run a test which launches
a qemu VM to test stuff that requires a functional installation
and a reboot?
And related to this question, do we also need to define
"TestRequires" packages/dependencies?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
12:54 a.m.
On Tuesday, October 4, 2016 8:09:14 PM CEST Richard W.M. Jones wrote:
And related to this question, do we also need to define
"TestRequires" packages/dependencies?
Sounds like natural approach would be to install the built packages into
some minimal environment, and the packages itself should bring the
dependencies? OTOH, we should be able to limit the set of packages to install
in such case, somehow.
The TestRequires is interesting idea, too, FTR mentioned in rhbz#1134397.
Pavel
3:32 a.m.
On Wed, Oct 05, 2016 at 07:54:59AM +0200, Pavel Raiskup wrote:
On Tuesday, October 4, 2016 8:09:14 PM CEST Richard W.M. Jones
wrote:
> And related to this question, do we also need to define
> "TestRequires" packages/dependencies?
Sounds like natural approach would be to install the built packages into
some minimal environment, and the packages itself should bring the
dependencies? OTOH, we should be able to limit the set of packages to install
in such case, somehow.
The problem is suppose the tests need tool or framework 'X' in order
to run the tests.
If I was using RPM %check, then I would add:
BuildRequires: X
But (I assume?) BuildRequires won't be installed in this testing
environment, so I now have to add instead:
Requires: X
That has obvious problems.
The TestRequires is interesting idea, too, FTR mentioned in
rhbz#1134397.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
3:49 a.m.
On Wed, 5 Oct 2016 09:32:49 +0100
"Richard W.M. Jones" <rjones(a)redhat.com> wrote:
On Wed, Oct 05, 2016 at 07:54:59AM +0200, Pavel Raiskup wrote:
> On Tuesday, October 4, 2016 8:09:14 PM CEST Richard W.M. Jones
> wrote:
> > And related to this question, do we also need to define
> > "TestRequires" packages/dependencies?
>
> Sounds like natural approach would be to install the built packages
> into some minimal environment, and the packages itself should bring
> the dependencies? OTOH, we should be able to limit the set of
> packages to install in such case, somehow.
The problem is suppose the tests need tool or framework 'X' in order
to run the tests.
If I was using RPM %check, then I would add:
BuildRequires: X
But (I assume?) BuildRequires won't be installed in this testing
environment, so I now have to add instead:
Requires: X
That has obvious problems.
> The TestRequires is interesting idea, too, FTR mentioned in
> rhbz#1134397.
I haven't thought much about it yet how feasible it would be, but why
not "abuse" rpm infrastructure even for the tests? Introduce foo-test
package as a test-suite for package foo and do everything within its
spec file - install deps (as BRs), run the tests (in %check), run in
chroot, container or natively ... And package test results/logs into
the binary foo-test rpm :-) Or leave it without %files.
Dan
5:22 a.m.
On Wednesday, October 5, 2016 10:49:26 AM CEST Dan Horák wrote:
I haven't thought much about it yet how feasible it would be, but
why
not "abuse" rpm infrastructure even for the tests? Introduce foo-test
package as a test-suite for package foo and do everything within its
spec file - install deps (as BRs), run the tests (in %check), run in
chroot, container or natively ... And package test results/logs into
the binary foo-test rpm :-) Or leave it without %files.
Right, or more - it would be fine to have testsuite `make install`able
(automatically by Automake e.g.). Then anybody could create 'foo-test'
package and than end-users could do the testing themselves too,
transparently - the same way as taskotron would.
Pavel
8:35 a.m.
On 2016-10-05, Pavel Raiskup <praiskup(a)redhat.com> wrote:
Right, or more - it would be fine to have testsuite `make
install`able
(automatically by Automake e.g.). Then anybody could create 'foo-test'
package and than end-users could do the testing themselves too,
transparently - the same way as taskotron would.
The assumption about support by the build system is crucial.
We had/have few macros in Perl for building test subpackages, but it was
terrible. The reason was no support in the build system and thus in the
upstream. Therefore the tests had tendency to run fine at build-time
(sources and binaries in local directory available) but terribly after
installation (tests expected files out of source test subdirectory). And
upstream was usually hostile against our fixes because there was not use
case for the upstream.
-- Petr
12:09 p.m.
On Tue, 4 Oct 2016 20:09:14 +0100
"Richard W.M. Jones" <rjones(a)redhat.com> wrote:
On Mon, Oct 03, 2016 at 08:21:42PM +0000, Zbigniew
Jędrzejewski-Szmek
wrote:
> On Mon, Oct 03, 2016 at 01:50:33PM -0600, Tim Flink wrote:
> > One of the features for Taskotron that we've been planning since
> > the beginning was a way for contributors to maintain their own
> > automated tasks/tests which would be run during a package's
> > lifecycle.
> >
> > I'm happy to say that we're almost to this milestone and wanted
> > to get some feedback from devel@ on the specifics of what we're
> > planning WRT where these automated tasks will be stored and the
> > execution modes that we're planning to support. Our current plan
> > is written up at:
> >
> >
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
>
> Sounds reasonable.
>
> How does the environment in which those tasks are run look?
> Would it be possible for example to run a test which launches
> a qemu VM to test stuff that requires a functional installation
> and a reboot?
And related to this question, do we also need to define
"TestRequires" packages/dependencies?
There are 2 ways of installing dependencies for task execution:
1. List the dependencies in the formula so that they're installed
during setup
2. Install them with a script that runs as part of the task - you have
root access on the VM and can do whatever you need to for testing,
even rendering the machine non-bootable because the next task will
get a fresh VM created from a base image.
Does this answer your question or did I misunderstand what you were
asking?
Tim
3:35 p.m.
On Mon, 3 Oct 2016 13:50:33 -0600
Tim Flink <tflink(a)redhat.com> wrote:
One of the features for Taskotron that we've been planning since
the
beginning was a way for contributors to maintain their own automated
tasks/tests which would be run during a package's lifecycle.
I'm happy to say that we're almost to this milestone and wanted to get
some feedback from devel@ on the specifics of what we're planning WRT
where these automated tasks will be stored and the execution modes
that we're planning to support. Our current plan is written up at:
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
The hope is that by making it easier for contributors to write
automated tasks and making the model completely self-service and
convention drive, there will be a lot more automated checks for
packages than we currently have for Fedora.
Please read through the wiki page I mentioned above and give us
feedback on whether what we're planning to implement is going to be
useful or if there are areas of the plan which could be improved.
To clarify at the top you have "Seeing as dist-git will be moving to
pagure before long", the plan there is not to move dist-git, but to add
a partial pagure instance in front of it. This would have issues,
permissions, docs, etc all disabled and would basically just be a way
to add a PR workflow. The existing dist-git setup would be still there
exactly the same.
Another alternate here is that we could make taskotron a 'namespace'
like currently rpms/ and docker/ are. Then we would have
perhaps: /taskotron/rpms/foobar/ as the top level and all the rest is
the same. This would get us a seperate pkgdb entry for the taskotron
part of things (ie, it could have different maintainers, people allowed
to commit, etc). That would add to complexity however.
Is there any provision for tests that should be run on _all_
packages? or we would need to link the test into all repos, etc?
I have in mind a test that would get the checksum from the sources file
and see if it could compare it to upstream.
Finally, are the lifecycle points triggered via fedmsg? If so, it would
be nice to have a UPLOAD lifecycle where someone has uploaded a source
to lookaside cache. (The above test would probibly be better at upload
time than sources git commit time, but either could work).
Thanks for working on all this. It's awesome to see it start to come to
fruition!
kevin
4:57 p.m.
On Mon, 3 Oct 2016 14:35:00 -0600
Kevin Fenzi <kevin(a)scrye.com> wrote:
On Mon, 3 Oct 2016 13:50:33 -0600
Tim Flink <tflink(a)redhat.com> wrote:
> One of the features for Taskotron that we've been planning since the
> beginning was a way for contributors to maintain their own automated
> tasks/tests which would be run during a package's lifecycle.
>
> I'm happy to say that we're almost to this milestone and wanted to
> get some feedback from devel@ on the specifics of what we're
> planning WRT where these automated tasks will be stored and the
> execution modes that we're planning to support. Our current plan is
> written up at:
>
>
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
>
> The hope is that by making it easier for contributors to write
> automated tasks and making the model completely self-service and
> convention drive, there will be a lot more automated checks for
> packages than we currently have for Fedora.
>
> Please read through the wiki page I mentioned above and give us
> feedback on whether what we're planning to implement is going to be
> useful or if there are areas of the plan which could be improved.
To clarify at the top you have "Seeing as dist-git will be moving to
pagure before long", the plan there is not to move dist-git, but to
add a partial pagure instance in front of it. This would have issues,
permissions, docs, etc all disabled and would basically just be a way
to add a PR workflow. The existing dist-git setup would be still there
exactly the same.
Thanks for the clarification - the emphasis was on coming support for
PRs. I've reworded that part of the proposal to make it more clear that
dist-git isn't "moving to pagure".
Another alternate here is that we could make taskotron a
'namespace'
like currently rpms/ and docker/ are. Then we would have
perhaps: /taskotron/rpms/foobar/ as the top level and all the rest is
the same. This would get us a seperate pkgdb entry for the taskotron
part of things (ie, it could have different maintainers, people
allowed to commit, etc). That would add to complexity however.
That is an alternative that we had been looking at before we learned
that dist-git would be getting pull requests. The namespace we were
talking about was 'checks/rpms/foobar' which would also open the door
for things like 'checks/docker/foobar' or any other type of deliverable
which uses dist-git.
Our thought is that keeping all the checks in the same repo that spec
files live in is going to be easier to use than having to worry about
keeping 2 repos in sync with eachother.
That being said, we're fine with either storage paradigm and it doesn't
matter much if we look for tasks in a directory inside dist-git
branches or a separate repo which only holds tasks as long as there is
a single convention that is easy for most contributors and doesn't
involve something that's unmanageable for the Taskotron devs/admins.
Is there any provision for tests that should be run on _all_
packages? or we would need to link the test into all repos, etc?
I have in mind a test that would get the checksum from the sources
file and see if it could compare it to upstream.
At this point, that's pretty much "come talk to us and we'll get
something figured out". I suspect that things which run on all
builds/uploads are going to be the minority of tasks that people want
to run and for now, it's treated as a special case.
So, if you have ideas for checks that would run on all packages or on
groups of packages, please come find us. If I'm wrong and there are a
lot of more general checks that people want to run, we can speed up
plans to make them less of a special case.
Finally, are the lifecycle points triggered via fedmsg? If so, it
would be nice to have a UPLOAD lifecycle where someone has uploaded a
source to lookaside cache. (The above test would probibly be better
at upload time than sources git commit time, but either could work).
Everything that runs in Taskotron is triggered via fedmsg so we can add
pretty much anything which emits a fedmsg.
In this context, there are only so many things which make sense to have
in dist-git but that doesn't mean it can't be run.
I've started an explicit list of the lifecycle points that we'll be
supporting for the tasks stored in dist-git.
https://phab.qadevel.cloud.fedoraproject.org/T721
Thanks for working on all this. It's awesome to see it start to
come
to fruition!
This has been a wishlist item for as long as I can remember. It'll be
great to start new kinds of automation and help make Fedora better :)
Tim
12:08 p.m.
On Mon, 3 Oct 2016 15:57:14 -0600
Tim Flink <tflink(a)redhat.com> wrote:
Thanks for the clarification - the emphasis was on coming support
for
PRs. I've reworded that part of the proposal to make it more clear
that dist-git isn't "moving to pagure".
Thanks.
> Another alternate here is that we could make taskotron a 'namespace'
> like currently rpms/ and docker/ are. Then we would have
> perhaps: /taskotron/rpms/foobar/ as the top level and all the rest
> is the same. This would get us a seperate pkgdb entry for the
> taskotron part of things (ie, it could have different maintainers,
> people allowed to commit, etc). That would add to complexity
> however.
That is an alternative that we had been looking at before we learned
that dist-git would be getting pull requests. The namespace we were
talking about was 'checks/rpms/foobar' which would also open the door
for things like 'checks/docker/foobar' or any other type of
deliverable which uses dist-git.
Our thought is that keeping all the checks in the same repo that spec
files live in is going to be easier to use than having to worry about
keeping 2 repos in sync with eachother.
Fair enough. I just thought it would be worth mentioning.
That being said, we're fine with either storage paradigm and it
doesn't matter much if we look for tasks in a directory inside
dist-git branches or a separate repo which only holds tasks as long
as there is a single convention that is easy for most contributors
and doesn't involve something that's unmanageable for the Taskotron
devs/admins.
Yeah, I don't have much of a horse in this race either.
I guess it depends on how much maintainers will find tests being
added/edited/etc as noise or not.
> Is there any provision for tests that should be run on _all_
> packages? or we would need to link the test into all repos, etc?
> I have in mind a test that would get the checksum from the sources
> file and see if it could compare it to upstream.
At this point, that's pretty much "come talk to us and we'll get
something figured out". I suspect that things which run on all
builds/uploads are going to be the minority of tasks that people want
to run and for now, it's treated as a special case.
ok. Thats fair.
So, if you have ideas for checks that would run on all packages or
on
groups of packages, please come find us. If I'm wrong and there are a
lot of more general checks that people want to run, we can speed up
plans to make them less of a special case.
> Finally, are the lifecycle points triggered via fedmsg? If so, it
> would be nice to have a UPLOAD lifecycle where someone has uploaded
> a source to lookaside cache. (The above test would probibly be
> better at upload time than sources git commit time, but either
> could work).
Everything that runs in Taskotron is triggered via fedmsg so we can
add pretty much anything which emits a fedmsg.
Excellent.
...snip...
kevin
12:59 p.m.
On Tue, 4 Oct 2016 11:08:36 -0600
Kevin Fenzi <kevin(a)scrye.com> wrote:
<snip>
> That being said, we're fine with either storage paradigm and
it
> doesn't matter much if we look for tasks in a directory inside
> dist-git branches or a separate repo which only holds tasks as long
> as there is a single convention that is easy for most contributors
> and doesn't involve something that's unmanageable for the Taskotron
> devs/admins.
Yeah, I don't have much of a horse in this race either.
I guess it depends on how much maintainers will find tests being
added/edited/etc as noise or not.
That's one of the things that I'm hoping to get figured out in this
thread so we can wrap things up and get this into production for folks
to use.
Thanks for bringing it up explicitly.
Tim
<snip>
8:19 a.m.
On Mon, Oct 03, 2016 at 02:35:00PM -0600, Kevin Fenzi wrote:
Another alternate here is that we could make taskotron a
'namespace'
like currently rpms/ and docker/ are. Then we would have
perhaps: /taskotron/rpms/foobar/ as the top level and all the rest is
the same. This would get us a seperate pkgdb entry for the taskotron
part of things (ie, it could have different maintainers, people allowed
to commit, etc). That would add to complexity however.
How much complexity in terms of ongoing maintenance? I think having
different committers is a big plus. The big downside — other than
complexity — is that the in-package-dist-git approach is very obvious
to existing packagers, whereas the namespaces are still a sort of
easter egg. Maybe we could do something in fedpkg to make it more
discoverable?
Also, if we do go with a separate namespace, how about "tests" instead
of "taskotron", with tasktron a subdir of that, and "testcases"
another (for https://github.com/fedora-infra/bodhi/issues/942)?
Thanks for working on all this. It's awesome to see it start to
come to
fruition!
+1
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
1:16 p.m.
On Tue, 4 Oct 2016 09:19:15 -0400
Matthew Miller <mattdm(a)fedoraproject.org> wrote:
On Mon, Oct 03, 2016 at 02:35:00PM -0600, Kevin Fenzi wrote:
> Another alternate here is that we could make taskotron a 'namespace'
> like currently rpms/ and docker/ are. Then we would have
> perhaps: /taskotron/rpms/foobar/ as the top level and all the rest
> is the same. This would get us a seperate pkgdb entry for the
> taskotron part of things (ie, it could have different maintainers,
> people allowed to commit, etc). That would add to complexity
> however.
How much complexity in terms of ongoing maintenance? I think having
different committers is a big plus. The big downside — other than
complexity — is that the in-package-dist-git approach is very obvious
to existing packagers, whereas the namespaces are still a sort of
easter egg. Maybe we could do something in fedpkg to make it more
discoverable?
As far as I know, separate repos shouldn't be very expensive in terms
of maintenance once everything is in place. If we do decide to
have separate ACLs for the "rpm" and "check" repos, that will require
some work to make sure that changes affecting both repos are propagated
correctly.
That being said, I'm not sure that having separate ACLs is much of a
benefit for Fedora. Assuming we end up going forward with current plans
to gate packages based on results from automation, I don't see how
someone with commit privileges on the "check" repo could cause much
more damage than if that person had commit privileges on the "rpm" repo
assuming that the other maintainers are watching the commits coming in
(and I do assume that folks are paying attention).
Once pull requests on dist-git repos are possible, I don't see enough
advantages to having separate ACLs to justify the work to keep
everything synced.
Also, if we do go with a separate namespace, how about
"tests" instead
of "taskotron", with tasktron a subdir of that, and "testcases"
another (for https://github.com/fedora-infra/bodhi/issues/942)?
When it was discussed earlier, we had been leaning towards using
"rpms-checks" so that the concept could be expanded to things like the
docker layered images (docker-checks) or anything else that we end up
wanting to have similar repos for in the future.
That would certainly be one way to start improving the package-related
testcase documentation.
Tim
> Thanks for working on all this. It's awesome to see it start
to
> come to fruition!
+1
7:44 a.m.
On Wed, Oct 05, 2016 at 12:16:32PM -0600, Tim Flink wrote:
On Tue, 4 Oct 2016 09:19:15 -0400
Matthew Miller <mattdm(a)fedoraproject.org> wrote:
> On Mon, Oct 03, 2016 at 02:35:00PM -0600, Kevin Fenzi wrote:
> > Another alternate here is that we could make taskotron a 'namespace'
> > like currently rpms/ and docker/ are. Then we would have
> > perhaps: /taskotron/rpms/foobar/ as the top level and all the rest
> > is the same. This would get us a seperate pkgdb entry for the
> > taskotron part of things (ie, it could have different maintainers,
> > people allowed to commit, etc). That would add to complexity
> > however.
>
> How much complexity in terms of ongoing maintenance? I think having
> different committers is a big plus. The big downside — other than
> complexity — is that the in-package-dist-git approach is very obvious
> to existing packagers, whereas the namespaces are still a sort of
> easter egg. Maybe we could do something in fedpkg to make it more
> discoverable?
As far as I know, separate repos shouldn't be very expensive in terms
of maintenance once everything is in place. If we do decide to
have separate ACLs for the "rpm" and "check" repos, that will
require
some work to make sure that changes affecting both repos are propagated
correctly.
That being said, I'm not sure that having separate ACLs is much of a
benefit for Fedora.
Yes, I don't see the advantage of so much permission fragmentation.
I'd much prefer that maintainers of individual packages arrange this
between themselves (e.g. "Please only commit to tests"), using social means,
as makes sense for the particular group of people.
Maybe we should standardize the prefix for git commits: "taskotron: "
or "tests: " or whatever, so it's easy to stop in git history.
Zbyszek
9 a.m.
On Thu, Oct 06, 2016 at 12:44:02PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
Maybe we should standardize the prefix for git commits:
"taskotron: "
or "tests: " or whatever, so it's easy to stop in git history.
That'll have to be automated somehow or I don't think people will
remember.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
3:25 a.m.
On 10/03/2016 09:50 PM, Tim Flink wrote:
One of the features for Taskotron that we've been planning since
the
beginning was a way for contributors to maintain their own automated
tasks/tests which would be run during a package's lifecycle.
I'm happy to say that we're almost to this milestone and wanted to get
some feedback from devel@ on the specifics of what we're planning WRT
where these automated tasks will be stored and the execution modes that
we're planning to support. Our current plan is written up at:
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
The hope is that by making it easier for contributors to write
automated tasks and making the model completely self-service and
convention drive, there will be a lot more automated checks for
packages than we currently have for Fedora.
Please read through the wiki page I mentioned above and give us
feedback on whether what we're planning to implement is going to be
useful or if there are areas of the plan which could be improved.
Is there any way
how this can be run on RHEL too once there will be some
tests?
Is there possibility to run the tests in Beaker?
From the wiki it does not look in any way compatible with existing RHTS
tests implemented in BeakerLib. Are there any plans to draw closer these
initiatives in one way or the other?
Regards,
--
Jakub Jelen
Associate Software Engineer
Security Technologies
Red Hat
12:27 p.m.
On Tue, 4 Oct 2016 10:25:26 +0200
Jakub Jelen <jjelen(a)redhat.com> wrote:
On 10/03/2016 09:50 PM, Tim Flink wrote:
> One of the features for Taskotron that we've been planning since the
> beginning was a way for contributors to maintain their own automated
> tasks/tests which would be run during a package's lifecycle.
>
> I'm happy to say that we're almost to this milestone and wanted to
> get some feedback from devel@ on the specifics of what we're
> planning WRT where these automated tasks will be stored and the
> execution modes that we're planning to support. Our current plan is
> written up at:
>
>
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
>
> The hope is that by making it easier for contributors to write
> automated tasks and making the model completely self-service and
> convention drive, there will be a lot more automated checks for
> packages than we currently have for Fedora.
>
> Please read through the wiki page I mentioned above and give us
> feedback on whether what we're planning to implement is going to be
> useful or if there are areas of the plan which could be improved.
Is there any way how this can be run on RHEL too once there will be
some tests?
We plan to add support for EPEL in the future but it's not something
that we'll be working on immediately.
Is there possibility to run the tests in Beaker?
Not yet, no. Getting our Beaker instance working will need to happen
first but that's recently become more of a priority instead of
languishing in an almost-done state while waiting for me to have free
time to work on it.
We do plan to support running tests in Beaker from Taskotron but that
work has yet to be scoped out as we need a working Beaker installation
before it makes sense to work on that feature.
From the wiki it does not look in any way compatible with existing
RHTS tests implemented in BeakerLib. Are there any plans to draw
closer these initiatives in one way or the other?
Not specifically, no. One of the objectives that we have is to make
sure that test cases can be easily shared between Fedora and RHEL so
that Red Hat maintainers can share downstream testcases with upstream
without the need to maintain two separate tests.
This is a roundabout way of saying that while there are no specific
plans, it will happen eventually.
Tim
3:25 a.m.
New subject: RFC: Storing Automated Tasks/Tests In Dist-Git (git-submodules)
On Monday, October 3, 2016 1:50:33 PM CEST Tim Flink wrote:
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
...
Please read through the wiki page I mentioned above and give us
feedback on whether what we're planning to implement is going to be
useful
Useful, yes!
or if there are areas of the plan which could be improved.
Can we rather make the ./taskotron directory separate git submodule? I expect
that I'm going to play with that directory a lot, without being "that much
careful" as I'm with package directories; which might mean that for packaging
work there might be a bit unpleasant rush in git-log otherwise.
Pavel
12:40 p.m.
New subject: RFC: Storing Automated Tasks/Tests In Dist-Git (git-submodules)
I didn't notice that my reply went only to Pavel, resending to devel@
On Tue, 04 Oct 2016 10:25:46 +0200
Pavel Raiskup <praiskup(a)redhat.com> wrote:
On Monday, October 3, 2016 1:50:33 PM CEST Tim Flink wrote:
>
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
> ...
> Please read through the wiki page I mentioned above and give us
> feedback on whether what we're planning to implement is going to be
> useful
Useful, yes!
> or if there are areas of the plan which could be improved.
Can we rather make the ./taskotron directory separate git submodule?
I expect that I'm going to play with that directory a lot, without
being "that much careful" as I'm with package directories; which
might mean that for packaging work there might be a bit unpleasant
rush in git-log otherwise.
I'd rather avoid git submodules and subtrees because that's a lot of
extra complexity to make sure that the various pointers are updated at
the correct times.
I think that a better alternative to putting checks into the same
dist-git repo as packaging information would be to have differently
namespaced git repos (eg checks-rpms/libfoo for rpms/libfoo) and
enhance fedpkg to work with that second repository to make it look like
a subdirectory even though it's a separate git repo.
Tim
2:37 a.m.
New subject: RFC: Storing Automated Tasks/Tests In Dist-Git (git-submodules)
On Wednesday, October 5, 2016 11:40:44 AM CEST Tim Flink wrote:
I didn't notice that my reply went only to Pavel, resending to
devel@
On Tue, 04 Oct 2016 10:25:46 +0200
Pavel Raiskup <praiskup(a)redhat.com> wrote:
> On Monday, October 3, 2016 1:50:33 PM CEST Tim Flink wrote:
> >
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
> > ...
> > Please read through the wiki page I mentioned above and give us
> > feedback on whether what we're planning to implement is going to be
> > useful
>
> Useful, yes!
>
> > or if there are areas of the plan which could be improved.
>
> Can we rather make the ./taskotron directory separate git submodule?
> I expect that I'm going to play with that directory a lot, without
> being "that much careful" as I'm with package directories; which
> might mean that for packaging work there might be a bit unpleasant
> rush in git-log otherwise.
I'd rather avoid git submodules and subtrees because that's a lot of
extra complexity to make sure that the various pointers are updated at
the correct times.
I disagree with extra complexity. Sub-tree sounds like something which is
terribly important -- so we can have separate ACLs for package and tests,
I don't have to bother package maintainers when I develop tests ... etc.
And when we talk about sub-tree, git submodules bring *a lot* of control
over sub-trees *for free*. There's something like `git submodule update
--init --recursive` (could be done by 'fedpkg clone'?).
Simply you can develop subtree without touching "parent", and once you are
ready -- you just update hash. Also one sub-trees (test directories) can
be shared among packages (and not being able to share some testcases among
several packages would be 99% show-stopper for me).
I think that a better alternative to putting checks into the same
dist-git repo as packaging information would be to have differently
namespaced git repos (eg checks-rpms/libfoo for rpms/libfoo) and
enhance fedpkg to work with that second repository to make it look like
a subdirectory even though it's a separate git repo.
I'm not against it, as I'm not going to hack that :) but this is a lot of
expensive complexity, when submodules are here clearly for this purpose.
If you wanted to make git-bisect working, this is going to be terrible (by
bisecting I mean that I want to see "tests" in the version compatible with
actual git-hash). So if you choose whatever you want, and we talk about
subtrees -- keep bisecting working, please.
Pavel
Tim
3:01 a.m.
New subject: RFC: Storing Automated Tasks/Tests In Dist-Git (git-submodules)
On Tue, Oct 18, 2016 at 09:37:45AM +0200, Pavel Raiskup wrote:
On Wednesday, October 5, 2016 11:40:44 AM CEST Tim Flink wrote:
> I didn't notice that my reply went only to Pavel, resending to devel@
>
> On Tue, 04 Oct 2016 10:25:46 +0200
> Pavel Raiskup <praiskup(a)redhat.com> wrote:
>
> > On Monday, October 3, 2016 1:50:33 PM CEST Tim Flink wrote:
> > >
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
> > > ...
> > > Please read through the wiki page I mentioned above and give us
> > > feedback on whether what we're planning to implement is going to be
> > > useful
> >
> > Useful, yes!
> >
> > > or if there are areas of the plan which could be improved.
> >
> > Can we rather make the ./taskotron directory separate git submodule?
> > I expect that I'm going to play with that directory a lot, without
> > being "that much careful" as I'm with package directories; which
> > might mean that for packaging work there might be a bit unpleasant
> > rush in git-log otherwise.
>
> I'd rather avoid git submodules and subtrees because that's a lot of
> extra complexity to make sure that the various pointers are updated at
> the correct times.
I disagree with extra complexity. Sub-tree sounds like something which is
terribly important -- so we can have separate ACLs for package and tests,
I don't have to bother package maintainers when I develop tests ... etc.
And when we talk about sub-tree, git submodules bring *a lot* of control
over sub-trees *for free*. There's something like `git submodule update
--init --recursive` (could be done by 'fedpkg clone'?).
Simply you can develop subtree without touching "parent", and once you are
ready -- you just update hash. Also one sub-trees (test directories) can
be shared among packages (and not being able to share some testcases among
several packages would be 99% show-stopper for me).
> I think that a better alternative to putting checks into the same
> dist-git repo as packaging information would be to have differently
> namespaced git repos (eg checks-rpms/libfoo for rpms/libfoo) and
> enhance fedpkg to work with that second repository to make it look like
> a subdirectory even though it's a separate git repo.
I'm not against it, as I'm not going to hack that :) but this is a lot of
expensive complexity, when submodules are here clearly for this purpose.
And are you going to hack on submodules? (both as user and to help infra getting
it set-up?)
Pierre
3:10 a.m.
New subject: RFC: Storing Automated Tasks/Tests In Dist-Git (git-submodules)
On Tuesday, October 18, 2016 10:01:30 AM CEST Pierre-Yves Chibon wrote:
> I'm not against it, as I'm not going to hack that :) but
this is a lot of
> expensive complexity, when submodules are here clearly for this purpose.
And are you going to hack on submodules? (both as user and to help infra getting
it set-up?)
Sorry pingou, wrongly written. I'm not going to hack on _this proposal_,
so I'm not against it. No doubts, we need that.
I'm going to participate at least as a package maintainer and test maintainer!
Pavel
8:07 a.m.
Hi Tim,
How about this use case:
Let say I have Ruby on Rails. This framework is much broader then one
rubygem-rails package. Where test for such framework will be stored?
How about tests, which might cover multiple versions of components? Lets
say I will have some generic test cases which should run on Python 2 as
well as Python 3 (or MySql as well as MariadDB). Where such tests will
be stored?
Vít
Dne 3.10.2016 v 21:50 Tim Flink napsal(a):
One of the features for Taskotron that we've been planning since
the
beginning was a way for contributors to maintain their own automated
tasks/tests which would be run during a package's lifecycle.
I'm happy to say that we're almost to this milestone and wanted to get
some feedback from devel@ on the specifics of what we're planning WRT
where these automated tasks will be stored and the execution modes that
we're planning to support. Our current plan is written up at:
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
The hope is that by making it easier for contributors to write
automated tasks and making the model completely self-service and
convention drive, there will be a lot more automated checks for
packages than we currently have for Fedora.
Please read through the wiki page I mentioned above and give us
feedback on whether what we're planning to implement is going to be
useful or if there are areas of the plan which could be improved.
Thanks,
Tim
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
12:42 p.m.
On Tue, 4 Oct 2016 15:07:28 +0200
Vít Ondruch <vondruch(a)redhat.com> wrote:
Hi Tim,
How about this use case:
Let say I have Ruby on Rails. This framework is much broader then one
rubygem-rails package. Where test for such framework will be stored?
Honestly, it depends on where you want those test cases to be stored.
This particular feature is primarily designed to test individual
packages and there are definitely important use cases which don't fit
incredibly well. The only requirement from Taskotron is that the test
cases are stored in a clone-able git repository.
For now, this would fall into the category of "come find us and we'll
help you get something set up". There will eventually be a better path
towards contributing not-just-package-specific tasks but we have to
start somewhere :)
How about tests, which might cover multiple versions of components?
Lets say I will have some generic test cases which should run on
Python 2 as well as Python 3 (or MySql as well as MariadDB). Where
such tests will be stored?
If I'm understanding you correctly, it would fall into the same case as
RoR - come find us and we'll help get something set up.
Tim
Vít
Dne 3.10.2016 v 21:50 Tim Flink napsal(a):
> One of the features for Taskotron that we've been planning since the
> beginning was a way for contributors to maintain their own automated
> tasks/tests which would be run during a package's lifecycle.
>
> I'm happy to say that we're almost to this milestone and wanted to
> get some feedback from devel@ on the specifics of what we're
> planning WRT where these automated tasks will be stored and the
> execution modes that we're planning to support. Our current plan is
> written up at:
>
>
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
>
> The hope is that by making it easier for contributors to write
> automated tasks and making the model completely self-service and
> convention drive, there will be a lot more automated checks for
> packages than we currently have for Fedora.
>
> Please read through the wiki page I mentioned above and give us
> feedback on whether what we're planning to implement is going to be
> useful or if there are areas of the plan which could be improved.
>
> Thanks,
>
> Tim
>
>
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to
> devel-leave(a)lists.fedoraproject.org
11:46 p.m.
On Mon, 3 Oct 2016 13:50:33 -0600
Tim Flink <tflink(a)redhat.com> wrote:
One of the features for Taskotron that we've been planning since
the
beginning was a way for contributors to maintain their own automated
tasks/tests which would be run during a package's lifecycle.
I'm happy to say that we're almost to this milestone and wanted to get
some feedback from devel@ on the specifics of what we're planning WRT
where these automated tasks will be stored and the execution modes
that we're planning to support. Our current plan is written up at:
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
The hope is that by making it easier for contributors to write
automated tasks and making the model completely self-service and
convention drive, there will be a lot more automated checks for
packages than we currently have for Fedora.
Please read through the wiki page I mentioned above and give us
feedback on whether what we're planning to implement is going to be
useful or if there are areas of the plan which could be improved.
Several folks have brought up concerns (off list) about our plan to use
sub-directories inside the rpms/ dist-git repos instead of separate
namespaces. The possibility of using namespaces (which are effectively
separate git repositories) did come up but I want to make sure this
discussion topic comes up in a more obvious fashion.
As I understand it, the primary concern is around having non-rpm stuff
in the repo and the commit history. I'm aware of two reasons for that
concern:
- Red Hat uses separate repos internally to hold tests for RHEL
packages and putting checks/tests into the rpm repo will make it
more painful farther down the road for RHEL package maintainers.
- Adding the checks/tests into the same repo increases the size of
the repo and could end up requiring more effort to search through
those commits for a specific change
- If there are other concerns, feel free to bring them up.
We chose the directory-in-dist-git option because it seems like the
better, more simple option. It requires fewer steps to get checks/tests
pushed, no extra tooling modification, makes the checks/tests easier to
find and would make for a less complex system overall.
That being said, I don't maintain many packages and I'm not going to
pretend that I know what's best for maintainers. So long as the end
system is consistent, easy to use, makes sense and is feasible to
complete with resources we have, I don't have a strong opinion on
whether we use directories or namespaces to hold checks/tests.
Which brings me to the question that I'd like to get some feedback
on: would it be preferable to store checks/tests within directories of
existing dist-git repos or create a new namespace to store checks/tests
and fiddle around with tooling etc. to hide some of the complexity that
may bring?
To be 100% clear from the start, this is not something that is being
dictated from within Red Hat. I do think that it would be unwise to
make more work for our downstream projects without a good reason to do
so but at the same time, I'm not really interested in creating more
work/complexity for Fedora maintainers unless there's a good reason to
do.
Tim
4:16 a.m.
On 10/17/2016 06:46 AM, Tim Flink wrote:
On Mon, 3 Oct 2016 13:50:33 -0600
Tim Flink <tflink(a)redhat.com> wrote:
> One of the features for Taskotron that we've been planning since the
> beginning was a way for contributors to maintain their own automated
> tasks/tests which would be run during a package's lifecycle.
>
> I'm happy to say that we're almost to this milestone and wanted to get
> some feedback from devel@ on the specifics of what we're planning WRT
> where these automated tasks will be stored and the execution modes
> that we're planning to support. Our current plan is written up at:
>
>
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
>
> The hope is that by making it easier for contributors to write
> automated tasks and making the model completely self-service and
> convention drive, there will be a lot more automated checks for
> packages than we currently have for Fedora.
>
> Please read through the wiki page I mentioned above and give us
> feedback on whether what we're planning to implement is going to be
> useful or if there are areas of the plan which could be improved.
Several folks have brought up concerns (off list) about our plan to use
sub-directories inside the rpms/ dist-git repos instead of separate
namespaces. The possibility of using namespaces (which are effectively
separate git repositories) did come up but I want to make sure this
discussion topic comes up in a more obvious fashion.
As I understand it, the primary concern is around having non-rpm stuff
in the repo and the commit history. I'm aware of two reasons for that
concern:
- Red Hat uses separate repos internally to hold tests for RHEL
packages and putting checks/tests into the rpm repo will make it
more painful farther down the road for RHEL package maintainers.
I'd like to describe Red Hat's dist-git structure/design. There are two
relevant namespaces:
- rpms/* for regular repositories used for building rpms
- tests/* for storing tests
rpms repositories have product branches similarly to Fedora's dist-git.
Whereas tests repositories have only master branch. There is no global
policy how tests repositories are structured and what goes in.
Everything is on QA group who manages set of repositories.
From workflow point of view:
- for every rpms/ repository there is tests/ repository created
automatically.
- if there are any special operations done on rpms/ repository tests/
repository is also considered here. E.g. removal, deprecation, etc.
- rhpkg (preferred tool for working with dist-git repos - corresponds
with fedpkg) has separate command to checkout tests - "rhpkg tests".
This would roughly correspond to "fedpkg co tests/foo".
Side note about the tooling:
- dist-git repository management on server side started with the same
code base. So as long as Fedora doesn't diverge too much (e.g. pagure
isn't considered internally yet) porting these changes can be fairly
trivial.
- rhpkg, similarly to fedpkg gets majority of the code from pyrpkg
library. If it turns out tests/ support makes a sense in other client
tools pyrpkg may get this code thus get this feature easily. (I'm CCing
cqi who is working on rhpkg/rpkg/fedpkg code and may provide more
insight here).
- Adding the checks/tests into the same repo increases the size of
the repo and could end up requiring more effort to search through
those commits for a specific change
- If there are other concerns, feel free to bring them up.
We chose the directory-in-dist-git option because it seems like the
better, more simple option. It requires fewer steps to get checks/tests
pushed, no extra tooling modification, makes the checks/tests easier to
find and would make for a less complex system overall.
That being said, I don't maintain many packages and I'm not going to
pretend that I know what's best for maintainers. So long as the end
system is consistent, easy to use, makes sense and is feasible to
complete with resources we have, I don't have a strong opinion on
whether we use directories or namespaces to hold checks/tests.
Which brings me to the question that I'd like to get some feedback
on: would it be preferable to store checks/tests within directories of
existing dist-git repos or create a new namespace to store checks/tests
and fiddle around with tooling etc. to hide some of the complexity that
may bring?
I'm wondering if you can estimate what amount of the work is needed here
which excuses mixing up content in the design.
--
Pavol Babincak
Release Engineering, Red Hat
10:42 a.m.
On Mon, 17 Oct 2016 11:16:22 +0200
Pavol Babincak <pbabinca(a)redhat.com> wrote:
On 10/17/2016 06:46 AM, Tim Flink wrote:
> On Mon, 3 Oct 2016 13:50:33 -0600
> Tim Flink <tflink(a)redhat.com> wrote:
>
>> One of the features for Taskotron that we've been planning since
>> the beginning was a way for contributors to maintain their own
>> automated tasks/tests which would be run during a package's
>> lifecycle.
>>
>> I'm happy to say that we're almost to this milestone and wanted to
>> get some feedback from devel@ on the specifics of what we're
>> planning WRT where these automated tasks will be stored and the
>> execution modes that we're planning to support. Our current plan
>> is written up at:
>>
>>
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
>>
>> The hope is that by making it easier for contributors to write
>> automated tasks and making the model completely self-service and
>> convention drive, there will be a lot more automated checks for
>> packages than we currently have for Fedora.
>>
>> Please read through the wiki page I mentioned above and give us
>> feedback on whether what we're planning to implement is going to be
>> useful or if there are areas of the plan which could be improved.
>
> Several folks have brought up concerns (off list) about our plan to
> use sub-directories inside the rpms/ dist-git repos instead of
> separate namespaces. The possibility of using namespaces (which are
> effectively separate git repositories) did come up but I want to
> make sure this discussion topic comes up in a more obvious fashion.
>
> As I understand it, the primary concern is around having non-rpm
> stuff in the repo and the commit history. I'm aware of two reasons
> for that concern:
>
> - Red Hat uses separate repos internally to hold tests for RHEL
> packages and putting checks/tests into the rpm repo will make it
> more painful farther down the road for RHEL package
> maintainers.
I'd like to describe Red Hat's dist-git structure/design. There are
two relevant namespaces:
- rpms/* for regular repositories used for building rpms
- tests/* for storing tests
rpms repositories have product branches similarly to Fedora's
dist-git.
Whereas tests repositories have only master branch. There is no
global policy how tests repositories are structured and what goes in.
Everything is on QA group who manages set of repositories.
One of the differences in Fedora is that I expect most check/test
contributions will come from package maintainers instead of dedicated
QA folks. At this time, there just aren't enough available person hours
among the Fedora QA folks to match the number of packages and
components which are in Fedora.
From workflow point of view:
- for every rpms/ repository there is tests/ repository created
automatically.
- if there are any special operations done on rpms/ repository tests/
repository is also considered here. E.g. removal, deprecation, etc.
- rhpkg (preferred tool for working with dist-git repos - corresponds
with fedpkg) has separate command to checkout tests - "rhpkg tests".
This would roughly correspond to "fedpkg co tests/foo".
Side note about the tooling:
- dist-git repository management on server side started with the same
code base. So as long as Fedora doesn't diverge too much (e.g. pagure
isn't considered internally yet) porting these changes can be fairly
trivial.
- rhpkg, similarly to fedpkg gets majority of the code from pyrpkg
library. If it turns out tests/ support makes a sense in other client
tools pyrpkg may get this code thus get this feature easily. (I'm
CCing cqi who is working on rhpkg/rpkg/fedpkg code and may provide
more insight here).
Is that code in rhpkg or rpkg (the parent of both rhpkg and fedpkg,
assuming that I got the name right)?
> - Adding the checks/tests into the same repo increases the
size of
> the repo and could end up requiring more effort to search
> through those commits for a specific change
>
> - If there are other concerns, feel free to bring them up.
>
> We chose the directory-in-dist-git option because it seems like the
> better, more simple option. It requires fewer steps to get
> checks/tests pushed, no extra tooling modification, makes the
> checks/tests easier to find and would make for a less complex
> system overall.
>
> That being said, I don't maintain many packages and I'm not going to
> pretend that I know what's best for maintainers. So long as the end
> system is consistent, easy to use, makes sense and is feasible to
> complete with resources we have, I don't have a strong opinion on
> whether we use directories or namespaces to hold checks/tests.
>
> Which brings me to the question that I'd like to get some feedback
> on: would it be preferable to store checks/tests within directories
> of existing dist-git repos or create a new namespace to store
> checks/tests and fiddle around with tooling etc. to hide some of
> the complexity that may bring?
>
I'm wondering if you can estimate what amount of the work is needed
here which excuses mixing up content in the design.
I'm not sure that I understand what you're asking here so I'll rephrase
what I think you're asking - please correct anything I've mixed up.
"How much extra work/complexity for Fedora maintainers is enough to
justify doing something different from what RHEL maintainers are
already doing?"
Unfortunately, I don't have a wonderfully objective, short answer to
this but I'll attempt a longer-winded answer.
To me, one of the most applicable differences between RHEL and Fedora
is that as a community project, there is not really a way to make
contributors do something.
If we had an army of dedicated QA folks who were doing most of the
package-level testing, it would make more sense to separate
checks/tests and rpm data (spec files, patches etc.). Since we don't
have that kind of an army in Fedora, it will be package maintainers who
are writing most of the automated jobs for Fedora packages.
The more steps and levels of indirection we add to the process of
contributing automated checks/tests for Fedora packages, the higher the
barrier to entry will be. I'm under the impression that most packagers
are plenty busy as it is and raising the barrier to entry for
contributing checks/tests at all will reduce the number of checks/tests
contributed.
Rephrasing my answer as a question: how many package level checks/tests
in Fedora are acceptable to lose in exchange for closer alignment with
downstream projects?
I don't know of a way to empirically answer any of those questions
which is why I'm asking for feedback here from people who probably have
a better idea of what level of annoyance they're willing to tolerate
than I do. I'm hoping to hear from both folks who deal with both Fedora
and downstream in addition to folks who don't work with downstream.
Tim
10:56 a.m.
On Mon, Oct 17, 2016 at 09:42:51AM -0600, Tim Flink wrote:
One of the differences in Fedora is that I expect most check/test
contributions will come from package maintainers instead of dedicated
QA folks. At this time, there just aren't enough available person hours
among the Fedora QA folks to match the number of packages and
components which are in Fedora.
I *would* love to build up a community of people interested in creating
these tests. It seems like a relatively easy way to get involved. This,
to me, argues for the separate repo with distinct (more relaxed)
permissions. We don't have that "army" (as you say) now, but maybe if
we had an easy way to funnel new contributors to it, we could.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
11:38 a.m.
On Mon, Oct 17, 2016 at 11:56:35AM -0400, Matthew Miller wrote:
On Mon, Oct 17, 2016 at 09:42:51AM -0600, Tim Flink wrote:
> One of the differences in Fedora is that I expect most check/test
> contributions will come from package maintainers instead of dedicated
> QA folks. At this time, there just aren't enough available person hours
> among the Fedora QA folks to match the number of packages and
> components which are in Fedora.
I *would* love to build up a community of people interested in creating
these tests. It seems like a relatively easy way to get involved. This,
to me, argues for the separate repo with distinct (more relaxed)
permissions. We don't have that "army" (as you say) now, but maybe if
we had an easy way to funnel new contributors to it, we could.
Writing any tests closely precedes writing fixes for the bugs
uncovered by those tests ;) So asking people to write tests, which
will then (sometimes) fail, and having a separate process for the
fixes that would need to touch the main repo, would be an exercise in
frustration.
It's a good principle to require both tests and fixes required for
those tests to pass to be submitted and merged as a single pull request.
I'd love to see a PR that adds a test for one of my packages, exposes
some bugs, but immediately fixes any fallout. I would be less thrilled
to have tests committed which will fail on the next rebuild, leaving me to
fix the package (or manually override the tests).
In the end, I'd prefer to get the first few test cases as pagure PRs,
or patches in the bugzilla while pagure is not yet integrated, and
then just allow that person to have commit rights to the main repo.
Zbyszek
11:45 a.m.
On Mon, Oct 17, 2016 at 04:38:28PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
It's a good principle to require both tests and fixes required
for
those tests to pass to be submitted and merged as a single pull request.
I'd love to see a PR that adds a test for one of my packages, exposes
some bugs, but immediately fixes any fallout. I would be less thrilled
to have tests committed which will fail on the next rebuild, leaving me to
fix the package (or manually override the tests).
As I'm imaginging it, tests added in this way would be non-blocking,
and would be expected to pass initially.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
12:18 p.m.
On Mon, Oct 17, 2016 at 12:45:30PM -0400, Matthew Miller wrote:
On Mon, Oct 17, 2016 at 04:38:28PM +0000, Zbigniew Jędrzejewski-Szmek
wrote:
> It's a good principle to require both tests and fixes required for
> those tests to pass to be submitted and merged as a single pull request.
> I'd love to see a PR that adds a test for one of my packages, exposes
> some bugs, but immediately fixes any fallout. I would be less thrilled
> to have tests committed which will fail on the next rebuild, leaving me to
> fix the package (or manually override the tests).
As I'm imaginging it, tests added in this way would be non-blocking,
and would be expected to pass initially.
"feel good" tests ;) But seriously, why should people write only tests
which pass?
Zbyszek
12:51 p.m.
On Mon, 17 Oct 2016 17:18:25 +0000
Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl> wrote:
On Mon, Oct 17, 2016 at 12:45:30PM -0400, Matthew Miller wrote:
> On Mon, Oct 17, 2016 at 04:38:28PM +0000, Zbigniew
> Jędrzejewski-Szmek wrote:
> > It's a good principle to require both tests and fixes required for
> > those tests to pass to be submitted and merged as a single pull
> > request. I'd love to see a PR that adds a test for one of my
> > packages, exposes some bugs, but immediately fixes any fallout. I
> > would be less thrilled to have tests committed which will fail on
> > the next rebuild, leaving me to fix the package (or manually
> > override the tests).
>
> As I'm imaginging it, tests added in this way would be non-blocking,
> and would be expected to pass initially.
"feel good" tests ;) But seriously, why should people write only tests
which pass?
I think this starts getting into the question about what we want to do
with the results.
If we are going to start gating builds based on results from
automation, we will need some way to indicate which results contribute
to a failure or a pass as part of that gating process.
The simple way to do this is to say "every test needs to pass for
gating to happen without manual override" but I'm not sure that's the
best solution here and I'm not aware of any existing proposals on how
to manage that in a more granular way for build gating.
I'm of the opinion that it would be best to get the checks/tests
running and once we have enough of those to justify gating builds, we
can start figuring out what kinds of things should fail a build.
Even if we're not gating builds right away, more testing is being done
and forward progress is being made.
Tim
1:03 p.m.
On Mon, Oct 17, 2016 at 05:18:25PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
> > those tests to pass to be submitted and merged as a single
pull request.
> > I'd love to see a PR that adds a test for one of my packages, exposes
> > some bugs, but immediately fixes any fallout. I would be less thrilled
> > to have tests committed which will fail on the next rebuild, leaving me to
> > fix the package (or manually override the tests).
> As I'm imaginging it, tests added in this way would be non-blocking,
> and would be expected to pass initially.
"feel good" tests ;) But seriously, why should people write only tests
which pass?
Because then if they stop passing, it's worth looking at.
I expect other tests to be written with more deep knowledge targetting
regressions and etc.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
12:57 p.m.
On Mon, 17 Oct 2016 11:56:35 -0400
Matthew Miller <mattdm(a)fedoraproject.org> wrote:
On Mon, Oct 17, 2016 at 09:42:51AM -0600, Tim Flink wrote:
> One of the differences in Fedora is that I expect most check/test
> contributions will come from package maintainers instead of
> dedicated QA folks. At this time, there just aren't enough
> available person hours among the Fedora QA folks to match the
> number of packages and components which are in Fedora.
I *would* love to build up a community of people interested in
creating these tests. It seems like a relatively easy way to get
involved. This, to me, argues for the separate repo with distinct
(more relaxed) permissions. We don't have that "army" (as you say)
now, but maybe if we had an easy way to funnel new contributors to
it, we could.
If we do keep going toward the goal of having more automation support
for testing and gating builds from koji based on results from that
automation, I don't understand how it makes sense to let more people
have write access to the checks/tests used to determine whether a build
is acceptable or not.
I agree with Zbyszek that PRs which can contain packaging fixes with
tests is the best way to handle new contributions. Keeping the ACLs and
PRs separate for the two things seems like a lot of unnecessary
extra work to me.
Tim
1:04 p.m.
On Mon, Oct 17, 2016 at 11:57:03AM -0600, Tim Flink wrote:
If we do keep going toward the goal of having more automation
support
for testing and gating builds from koji based on results from that
automation, I don't understand how it makes sense to let more people
have write access to the checks/tests used to determine whether a build
is acceptable or not.
I agree with Zbyszek that PRs which can contain packaging fixes with
tests is the best way to handle new contributions. Keeping the ACLs and
PRs separate for the two things seems like a lot of unnecessary
extra work to me.
Okay; I'll defer to your expertise on this.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
3:21 a.m.
Dne 17.10.2016 v 17:42 Tim Flink napsal(a):
On Mon, 17 Oct 2016 11:16:22 +0200
Pavol Babincak <pbabinca(a)redhat.com> wrote:
> On 10/17/2016 06:46 AM, Tim Flink wrote:
>> On Mon, 3 Oct 2016 13:50:33 -0600
>> Tim Flink <tflink(a)redhat.com> wrote:
>>
>>> One of the features for Taskotron that we've been planning since
>>> the beginning was a way for contributors to maintain their own
>>> automated tasks/tests which would be run during a package's
>>> lifecycle.
>>>
>>> I'm happy to say that we're almost to this milestone and wanted to
>>> get some feedback from devel@ on the specifics of what we're
>>> planning WRT where these automated tasks will be stored and the
>>> execution modes that we're planning to support. Our current plan
>>> is written up at:
>>>
>>>
https://phab.qadevel.cloud.fedoraproject.org/w/taskotron/new_distgit_task...
>>>
>>> The hope is that by making it easier for contributors to write
>>> automated tasks and making the model completely self-service and
>>> convention drive, there will be a lot more automated checks for
>>> packages than we currently have for Fedora.
>>>
>>> Please read through the wiki page I mentioned above and give us
>>> feedback on whether what we're planning to implement is going to be
>>> useful or if there are areas of the plan which could be improved.
>> Several folks have brought up concerns (off list) about our plan to
>> use sub-directories inside the rpms/ dist-git repos instead of
>> separate namespaces. The possibility of using namespaces (which are
>> effectively separate git repositories) did come up but I want to
>> make sure this discussion topic comes up in a more obvious fashion.
>>
>> As I understand it, the primary concern is around having non-rpm
>> stuff in the repo and the commit history. I'm aware of two reasons
>> for that concern:
>>
>> - Red Hat uses separate repos internally to hold tests for RHEL
>> packages and putting checks/tests into the rpm repo will make it
>> more painful farther down the road for RHEL package
>> maintainers.
> I'd like to describe Red Hat's dist-git structure/design. There are
> two relevant namespaces:
>
> - rpms/* for regular repositories used for building rpms
> - tests/* for storing tests
>
> rpms repositories have product branches similarly to Fedora's
> dist-git.
>
> Whereas tests repositories have only master branch. There is no
> global policy how tests repositories are structured and what goes in.
> Everything is on QA group who manages set of repositories.
One of the differences in Fedora is that I expect most check/test
contributions will come from package maintainers instead of dedicated
QA folks. At this time, there just aren't enough available person hours
among the Fedora QA folks to match the number of packages and
components which are in Fedora.
Well, my hopes are that Red Hat QA folks will get more involved in
Fedora testing. Actually you should really talk more to them.
> From workflow point of view:
> - for every rpms/ repository there is tests/ repository created
> automatically.
> - if there are any special operations done on rpms/ repository tests/
> repository is also considered here. E.g. removal, deprecation, etc.
> - rhpkg (preferred tool for working with dist-git repos - corresponds
> with fedpkg) has separate command to checkout tests - "rhpkg tests".
> This would roughly correspond to "fedpkg co tests/foo".
>
>
> Side note about the tooling:
> - dist-git repository management on server side started with the same
> code base. So as long as Fedora doesn't diverge too much (e.g. pagure
> isn't considered internally yet) porting these changes can be fairly
> trivial.
>
> - rhpkg, similarly to fedpkg gets majority of the code from pyrpkg
> library. If it turns out tests/ support makes a sense in other client
> tools pyrpkg may get this code thus get this feature easily. (I'm
> CCing cqi who is working on rhpkg/rpkg/fedpkg code and may provide
> more insight here).
Is that code in rhpkg or rpkg (the parent of both rhpkg and fedpkg,
assuming that I got the name right)?
>> - Adding the checks/tests into the same repo increases the size of
>> the repo and could end up requiring more effort to search
>> through those commits for a specific change
>>
>> - If there are other concerns, feel free to bring them up.
>>
>> We chose the directory-in-dist-git option because it seems like the
>> better, more simple option. It requires fewer steps to get
>> checks/tests pushed, no extra tooling modification, makes the
>> checks/tests easier to find and would make for a less complex
>> system overall.
>>
>> That being said, I don't maintain many packages and I'm not going to
>> pretend that I know what's best for maintainers. So long as the end
>> system is consistent, easy to use, makes sense and is feasible to
>> complete with resources we have, I don't have a strong opinion on
>> whether we use directories or namespaces to hold checks/tests.
>>
>> Which brings me to the question that I'd like to get some feedback
>> on: would it be preferable to store checks/tests within directories
>> of existing dist-git repos or create a new namespace to store
>> checks/tests and fiddle around with tooling etc. to hide some of
>> the complexity that may bring?
>>
> I'm wondering if you can estimate what amount of the work is needed
> here which excuses mixing up content in the design.
I'm not sure that I understand what you're asking here so I'll rephrase
what I think you're asking - please correct anything I've mixed up.
"How much extra work/complexity for Fedora maintainers is enough to
justify doing something different from what RHEL maintainers are
already doing?"
Unfortunately, I don't have a wonderfully objective, short answer to
this but I'll attempt a longer-winded answer.
To me, one of the most applicable differences between RHEL and Fedora
is that as a community project, there is not really a way to make
contributors do something.
If we had an army of dedicated QA folks who were doing most of the
package-level testing, it would make more sense to separate
checks/tests and rpm data (spec files, patches etc.). Since we don't
have that kind of an army in Fedora, it will be package maintainers who
are writing most of the automated jobs for Fedora packages.
The more steps and levels of indirection we add to the process of
contributing automated checks/tests for Fedora packages, the higher the
barrier to entry will be. I'm under the impression that most packagers
are plenty busy as it is and raising the barrier to entry for
contributing checks/tests at all will reduce the number of checks/tests
contributed.
Rephrasing my answer as a question: how many package level checks/tests
in Fedora are acceptable to lose in exchange for closer alignment with
downstream projects?
I don't know of a way to empirically answer any of those questions
which is why I'm asking for feedback here from people who probably have
a better idea of what level of annoyance they're willing to tolerate
than I do. I'm hoping to hear from both folks who deal with both Fedora
and downstream in addition to folks who don't work with downstream.
Tim
So let me make additional point. If we had two separate namespaces, we
can always make it appear it is one namespace. You can always checkout
one repository into another, we do it internally via rhpkg, we can use
submodules, in worst case, we can even merge the two repositories. While
if we start to use single repository which will mix everything, there is
no way to separate the history.
Vít
5:53 a.m.
Dne 17.10.2016 v 11:16 Pavol Babincak napsal(a):
Whereas tests repositories have only master branch.
I spoke to Pavol today a bit and I realized that I missed this peace of
the information, which comes quite essential to me and worth of
highlighting.
This is very important, since if there is only single branch used to
test all possible versions of the package, it allows us to find
regressions between different versions of packages. E.g. if there is
regression in Rawhide package, which is fixed and covered by some test
case, it allows us to automatically use this test case for
older/different versions of packages, without
backporting/merging/cherry-picking the test case.
IOW the same test suite which is used for testing RHEL7 is used for
testing RHEL6 and possible RHSCL.
Vít
1:18 a.m.
Dne 17.10.2016 v 06:46 Tim Flink napsal(a):
Which brings me to the question that I'd like to get some
feedback
on: would it be preferable to store checks/tests within directories of
existing dist-git repos or create a new namespace to store checks/tests
and fiddle around with tooling etc.
I prefer different namespace for tests.
--
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
2757
days inactive
2773
days old
42 comments
13 participants
participants (13)
-
Dan Horák -
Jakub Jelen -
Kevin Fenzi -
Matthew Miller -
Miroslav Suchý -
Pavel Raiskup -
Pavol Babincak -
Petr Pisar -
Pierre-Yves Chibon -
Richard W.M. Jones -
Tim Flink -
Vít Ondruch -
Zbigniew Jędrzejewski-Szmek