This is an initial take on stuff that was discussed in person during Tim's stay in Brno. Sending to the list for additional discussion/fine-tuning.
= What =
Talking rpmgrill-like checks, there will be a need to be able to facilitate some kind of structure for representing that a check is composed of multiple subchecks, for example:
check - FAILED subcheck1 - PASSED subcheck2 - PASSED subcheck3 - FAILED subcheck4 - PASSED
!IMPORTANT: ResultsDB will not be responsible for computing the result value for an "upper level" Result from the subchecks - this is the check's (check developer's) responsibility.
This could (should?) be done on two levels: * physicall nesting the Results as such in the database structure * namespacing Testcases
For the start, we decided to go with the simplistic approach of nesting the Testcases via a simple namespacing - thus allowing a frontend/query tool to reconstruct the structure at least to some extent e.g. by relying on a premise, that Results that are a part of one Job can be converted to a tree-like structure, based on the Testcase namespacing, at least to some extent, if needed.
== Namespace structure ==
We'll be providing some top-level namespaces (list not yet final): * app * fedoraqa * package * scratch (?)
These will the further split to facilitate for a finer level of granularity, e.g.:
app testdays powermanagement pm-suspendr fedoraqa depcheck rpmgrill package <pkgname> unit func
Everything below the top-level will be 100% user defined. We might have recommendations for specific namespaces (like package.<pkgname>), but we won't be enforcing them.
The structure will be implemented (at least in the initial implementation) just via the Testcase.name attribute in the DB, using dots as a separator. Later on, we can easily add an easy way of using wildcards for searching (e.g. app.testdays.*.pm-suspendr)
!IMPORTANT: the namespaces are not to be used to represent "additional data" about the underlying result such as architecture, item under test, etc. This is what the Result's extra-data (ResultData) is there for.
NOTE: Although we do not encourage to store the results to the finest granularity "just because" (e.g. individual results of a unittest testsuite), we leave it to the check-developer's judgement. If there is a usecase for it, let them do it, we don't care, as long as the DB is not extremely overloaded.
== Authentication/Authorization ==
We'll be continuing with the "expect no malice" approach we have right now. There will be just a simple limitation in libtaskotron:
check git clone if cloned: only allow non-pkg namespace if __our__ repo else: do whatever, don't care
in libtaskotron: check the git checkout like listed above have whitelisted napespace repos in config
!FIXME: the mechanism above is just copied from tflink's notes, I can't remember the details :/
== TODOs ==
* Change our checks to use the fedoraqa namespace * Implement repo checking in libtaskotron * Write docs for how to report stuff to ResultsDB * Come up with root nodes for namespaces
This is an initial take on stuff that was discussed in person during Tim's stay in Brno. Sending to the list for additional discussion/fine-tuning.
= What =
Talking rpmgrill-like checks, there will be a need to be able to facilitate some kind of structure for representing that a check is composed of multiple subchecks, for example:
check - FAILED subcheck1 - PASSED subcheck2 - PASSED subcheck3 - FAILED subcheck4 - PASSED!IMPORTANT: ResultsDB will not be responsible for computing the result value for an "upper level" Result from the subchecks - this is the check's (check developer's) responsibility.
I just realized this can go hand-in-hand with the new create_report directive that we introduced lately. If the generated report contains all the reported results and links to respective log files (if there are separate log files for individual subchecks, might or might not be true), this way even if the user is notified about the subcheck3 result, he can still click through and see the other check/subcheck results and their logs easily as well.
This could (should?) be done on two levels: * physicall nesting the Results as such in the database structure * namespacing Testcases
For the start, we decided to go with the simplistic approach of nesting the Testcases via a simple namespacing - thus allowing a frontend/query tool to reconstruct the structure at least to some extent e.g. by relying on a premise, that Results that are a part of one Job can be converted to a tree-like structure, based on the Testcase namespacing, at least to some extent, if needed.
== Namespace structure ==
We'll be providing some top-level namespaces (list not yet final): * app
This seems obvious at first, but the more I think about it, the more I'm not sure whether it wouldn't be clearer to use something like qa.testdays instead of app.testdays and team.cloud.tunir instead of app.tunir. Because at least it's obvious who stands behind it and we don't need to decide which "app" is high-profile enough to be put into "app" top namespace instead of the generic user/team namespaces. But if we have a clear idea what goes here and what doesn't, I'm OK with it.
* fedoraqa
I'd probably use just "qa". "fedora" seems superfluous. Or even "team.qa", see below.
* package
Maybe just "pkg"?
* scratch (?)
I'd also add "fas"/"user" and "team". This way everyone can use his/her own namespace (like user home on wiki) for experimenting or custom tasks, and we also make team ownership clearer ("team.infra.task" instead of "user.pingou.task"). With this, we might not even need "scratch", because everyone can play is his/her own user namespace (but we might find some other uses).
These will the further split to facilitate for a finer level of granularity, e.g.:
app testdays powermanagement pm-suspendr fedoraqa depcheck rpmgrill package <pkgname> unit funcEverything below the top-level will be 100% user defined. We might have recommendations for specific namespaces (like package.<pkgname>), but we won't be enforcing them.
By top-level you mean whole "pkg.<pkgname>" in this case, right? Because it would be weird to allow people to arbitrarily create "pkg.foo" namespace if there's no "foo" package in Fedora (and would introduce issues in the future, if such package appeared). So "pkg.<pkgname>" would be the fixed part, and anything below would be the flexible part, right? The same with "user.<username>" and "team.<teamname>".
The structure will be implemented (at least in the initial implementation) just via the Testcase.name attribute in the DB, using dots as a separator. Later on, we can easily add an easy way of using wildcards for searching (e.g. app.testdays.*.pm-suspendr)
!IMPORTANT: the namespaces are not to be used to represent "additional data" about the underlying result such as architecture, item under test, etc. This is what the Result's extra-data (ResultData) is there for.
NOTE: Although we do not encourage to store the results to the finest granularity "just because" (e.g. individual results of a unittest testsuite), we leave it to the check-developer's judgement. If there is a usecase for it, let them do it, we don't care, as long as the DB is not extremely overloaded.
This is a very good point and in the documentation I'd like to stress out "please don't create dozens of namespaces needlessly". We should explain when it makes sense to use subcheck namespaces and when it does not. For example, if "rpmgrill" executes many different checks and we consider some of them release-critical and some of them not, which means we want to listen for them individually over fedpkg (and package maintainers as well), it makes sense to have qa.rpmgrill.rpm_sanity, qa.rpmgrill.rpmlint, qa.rpmgrill.security, etc. However, if openssh test suite has 27 steps/test functions, it does not make sense to create pkg.openssh.step1 through pkg.openssh.step27 namespaces, if we only care about the final result, or if we process it in a single place only. If developers want the step separation but they process it in a single place, it's much better to create a parseable ResultYAML/JUnit file with a well-defined name as one of the artifacts (we can do that automatically for every task), and they can download it from the output directory and parse it. It's the same amount of work and it does not put 27 times heavier demand on our database and fedmsg.
== Authentication/Authorization ==
We'll be continuing with the "expect no malice" approach we have right now. There will be just a simple limitation in libtaskotron:
check git clone if cloned: only allow non-pkg namespace if __our__ repo else: do whatever, don't care
in libtaskotron: check the git checkout like listed above have whitelisted napespace repos in config
!FIXME: the mechanism above is just copied from tflink's notes, I can't remember the details :/
I'm not sure I understand this correctly, but I believe the idea was to give out namespace write-access based on git source URL. So tasks from dist-git system would only have write access to pkg.<pkgname>, where <pkgname> matches the package name in their dist-git URL. Tasks from QA repos would have write-access to qa namespace. Etc.
On Mon, 8 Feb 2016 06:54:17 -0500 (EST) Josef Skladanka jskladan@redhat.com wrote:
This is an initial take on stuff that was discussed in person during Tim's stay in Brno. Sending to the list for additional discussion/fine-tuning. = What =
Talking rpmgrill-like checks, there will be a need to be able to facilitate some kind of structure for representing that a check is composed of multiple subchecks, for example:
check - FAILED subcheck1 - PASSED subcheck2 - PASSED subcheck3 - FAILED subcheck4 - PASSED!IMPORTANT: ResultsDB will not be responsible for computing the result value for an "upper level" Result from the subchecks - this is the check's (check developer's) responsibility.
Are we going to require an overall status for the grouped subchecks? Would ResultsDB do any of the grouping outside of facilitating the namespaces?
This could (should?) be done on two levels: * physicall nesting the Results as such in the database structure * namespacing Testcases
For the start, we decided to go with the simplistic approach of nesting the Testcases via a simple namespacing - thus allowing a frontend/query tool to reconstruct the structure at least to some extent e.g. by relying on a premise, that Results that are a part of one Job can be converted to a tree-like structure, based on the Testcase namespacing, at least to some extent, if needed.
== Namespace structure ==
We'll be providing some top-level namespaces (list not yet final): * app * fedoraqa * package * scratch (?)
These will the further split to facilitate for a finer level of granularity, e.g.:
app testdays powermanagement pm-suspendr fedoraqa depcheck rpmgrill package <pkgname> unit funcEverything below the top-level will be 100% user defined. We might have recommendations for specific namespaces (like package.<pkgname>), but we won't be enforcing them.
The structure will be implemented (at least in the initial implementation) just via the Testcase.name attribute in the DB, using dots as a separator. Later on, we can easily add an easy way of using wildcards for searching (e.g. app.testdays.*.pm-suspendr)
!IMPORTANT: the namespaces are not to be used to represent "additional data" about the underlying result such as architecture, item under test, etc. This is what the Result's extra-data (ResultData) is there for.
NOTE: Although we do not encourage to store the results to the finest granularity "just because" (e.g. individual results of a unittest testsuite), we leave it to the check-developer's judgement. If there is a usecase for it, let them do it, we don't care, as long as the DB is not extremely overloaded.
To nitpick a bit, I had to read the first part of that a couple times before understanding.
Maybe something like:
Although the check developer has the final say over the granularity of results stored, we do not suggest storing results simply for the sake of having them
That being said, how many results are we talking about here if the DB would get overloaded?
== Authentication/Authorization ==
We'll be continuing with the "expect no malice" approach we have right now. There will be just a simple limitation in libtaskotron:
check git clone if cloned: only allow non-pkg namespace if __our__ repo else: do whatever, don't care
in libtaskotron: check the git checkout like listed above have whitelisted napespace repos in config
!FIXME: the mechanism above is just copied from tflink's notes, I can't remember the details :/
My memory of the idea is:
The check that we'd do for non-pkg namespaces would be to look at the git checkout during task execution - if that repo was cloned from a set of whitelisted repos (kept in config), it would be allowed to use non-pkg namespaces when reporting to resultsdb.
It's certainly not perfect and could be worked around without too much trouble but I think it'll be enough so long as we have sane behavior in the case where we don't even find a repo to look at. Putting the repo-checking code into libtaskotron should decrease the amount of work and error cases that we have to deal with (assuming non-malicious users).
Tim
== TODOs ==
- Change our checks to use the fedoraqa namespace
- Implement repo checking in libtaskotron
- Write docs for how to report stuff to ResultsDB
- Come up with root nodes for namespaces
qa-devel mailing list qa-devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org
On Mon, 8 Feb 2016 06:54:17 -0500 (EST) Josef Skladanka jskladan@redhat.com wrote:
== Namespace structure ==
We'll be providing some top-level namespaces (list not yet final): * app * fedoraqa * package * scratch (?)
These will the further split to facilitate for a finer level of granularity, e.g.:
app testdays powermanagement pm-suspendr fedoraqa depcheck rpmgrill package <pkgname> unit func
My thought on namespaces would be something like:
high level:
team user package release test (or dev)
team and user are self explanatory; each package would be in the package namespace, release would cover release-validation testing, and test would be reserved for taskotron unit/self testing.
I'm thinking that we could do, for example:
team.qa.testdays.<test-name>
or
team.desktop.func.<some-test>
or
release.validation.openqa
Am I on the right track here, or just wandering in the weeds?
John.
== Namespace structure ==
We'll be providing some top-level namespaces (list not yet final): * app * fedoraqa * package * scratch (?)
These will the further split to facilitate for a finer level of granularity, e.g.:
app testdays powermanagement pm-suspendr fedoraqa depcheck rpmgrill package <pkgname> unit func
I'm not sure if "func" and "unit" were meant as mandatory in the original proposal, but I'd drop them. Let each package maintainer organize their tests however they see it logical/useful. I don't see any benefit this separation would bring for us. Is there?
My thought on namespaces would be something like:
high level:
team user package release test (or dev)
I like test/dev/scratch. I realized it could be useful for experimentation, we could for example omit sending fedmsgs for these namespaces, and prune these results in resultsdb more often, and yet people would be able to run something in an experimental mode including seeing results in resultsdb. (Of course for basic task development, we should recommend disabling resultsdb submission).
team and user are self explanatory; each package would be in the package namespace, release would cover release-validation testing,
I'm not sure about "release", I see the same problem as with "app", it's hard to decide what goes in there and what goes into "team.qa" or elsewhere.
and test would be reserved for taskotron unit/self testing.
Ah, I understood test/dev differently. For taskotron testing, we run staging and development instance with separate resultsdb instances. So I guess a namespace for this purpose is not needed.
I'm thinking that we could do, for example:
team.qa.testdays.<test-name>
or
team.desktop.func.<some-test>
or
release.validation.openqa
Am I on the right track here, or just wandering in the weeds?
John.
That's how I imagine it. In the beginning, it might be a wild west, but once we have some permission model in place, I'd like to pre-define just a very small skeleton of top-level namespace prefixes (pkg.<pkgname>, team.<teamname>, user.<fasname>, etc) and let people do whatever they like with it.
Thanks for feedback, pirate.
check - FAILED subcheck1 - PASSED subcheck2 - PASSED subcheck3 - FAILED subcheck4 - PASSED!IMPORTANT: ResultsDB will not be responsible for computing the result value for an "upper level" Result from the subchecks - this is the check's (check developer's) responsibility.
Are we going to require an overall status for the grouped subchecks?
Unless some part of our code (resultsdb) requires it, I'd not mandate it. Some checks (like rpmgrill) might be composed of smaller checks, and it might not make much sense to try to create an overall result out of that (unless it is used for gating, or similar).
NOTE: Although we do not encourage to store the results to the finest granularity "just because" (e.g. individual results of a unittest testsuite), we leave it to the check-developer's judgement. If there is a usecase for it, let them do it, we don't care, as long as the DB is not extremely overloaded.
To nitpick a bit, I had to read the first part of that a couple times before understanding.
Maybe something like:
Although the check developer has the final say over the granularity of results stored, we do not suggest storing results simply for the sake of having them
Was my version too long? :-) Because I think we should include some explanation why they would want or not want store and emit detailed results. And some examples are the best way for guiding people. That's what I tried to provide in my previous email.
qa-devel@lists.fedoraproject.org
-
John Dulaney -
Josef Skladanka -
Kamil Paral -
Tim Flink