11:12 a.m.
Hi Everyone,
I have been working on a small tool named protron [1] that listens to pagure
pull-request messages, triggers a build and report build status as well as
taskotron/resultsdb test results to the pull-request.
Since protron kicks off the koji build, it knows its task identifier and keeps
it in memory so that it can later recognize a koji build it kicked versus a koji
build from something/someone else.
However, taskotron/resultsdb message do not include the koji task it was running
the tests against. So protron queries koji taskinfo <id> to get the package
nevr this task was about and keep then that info in memory to recognize
taskotron messages of interest.
In our regular workflow this isn't a problem because of the nevra restriction on
koji is such that we cannot build the same package twice, but with the PR model
coming to dist-git, we will need to somehow lift that constraint. There could
very well be two persons or two systems proposing a pull-requests doing the same
update (say the-new-hotness bumps foo to 1.2.3 while missing something, while a
contributor comes by and do the same update to 1.2.3 with the rebase of the
patch or so), both PRs will be building foo-1.2.3 and this will confuse protron:
to which PR corresponds foo-1.2.3?
If the PR are created with sufficient time in between, this will not be a
problem, the newer PR will override the info in memory of the old one.
However, if the two PRs are created sufficiently close to one another (so that
the tests for the first PR aren't finished running), this will be an issue.
To avoid this, one way would be to add to the taskotron message sent the task id
of the build taskotron was testing.
Would this be something that is doable?
Thanks in advance,
Pierre
4:18 p.m.
On Fri, 2017-03-03 at 18:12 +0100, Pierre-Yves Chibon wrote:
To avoid this, one way would be to add to the taskotron message sent
the task id
of the build taskotron was testing.
Would this be something that is doable?
So I think it should be *possible*, but taking a look at the current
Taskotron setup, I think it'd need a change in the Koji triggers. The
Koji triggers don't currently include the Koji task ID in the trigger
data, so when the task runs, it doesn't have that information available
to include in the report that gets submitted to ResultsDB.
This is the kind of thing we could put into resultsdb_conventions quite
nicely, in fact - a convention for 'result of a test associated with a
Koji task', which would store info like the Koji task ID. Just like the
convention for results associated with a compose that has productmd
metadata stores relevant bits of the productmd metadata into the
extradata like this:
productmd.image.type dvd
productmd.image.subvariant Server
productmd.compose.name Fedora
etc., we could do:
koji.id 123
koji.method build
koji.build foo-1.2-3
kinda thing. (or maybe koji.task.id if we want to have differing
conventions for koji tasks, builds etc.)
I don't think this would be much work to change in Taskotron, but I may
be missing something :)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
1:21 p.m.
On Fri, Mar 03, 2017 at 02:18:03PM -0800, Adam Williamson wrote:
On Fri, 2017-03-03 at 18:12 +0100, Pierre-Yves Chibon wrote:
> To avoid this, one way would be to add to the taskotron message sent the task id
> of the build taskotron was testing.
>
> Would this be something that is doable?
So I think it should be *possible*, but taking a look at the current
Taskotron setup, I think it'd need a change in the Koji triggers. The
Koji triggers don't currently include the Koji task ID in the trigger
data, so when the task runs, it doesn't have that information available
to include in the report that gets submitted to ResultsDB.
This is the kind of thing we could put into resultsdb_conventions quite
nicely, in fact - a convention for 'result of a test associated with a
Koji task', which would store info like the Koji task ID. Just like the
convention for results associated with a compose that has productmd
metadata stores relevant bits of the productmd metadata into the
extradata like this:
productmd.image.type dvd
productmd.image.subvariant Server
productmd.compose.name Fedora
etc., we could do:
koji.id 123
koji.method build
koji.build foo-1.2-3
kinda thing. (or maybe koji.task.id if we want to have differing
conventions for koji tasks, builds etc.)
I don't think this would be much work to change in Taskotron, but I may
be missing something :)
If I read things correctly taskotron already reports the bodhi update it tested,
so adjusting for koji should be feasible, but not knowing the source code I
can't quite estimate how easy/hard it would be :)
Pierre
2:38 a.m.
Hi Everyone,
I have been working on a small tool named protron [1] that listens to pagure
pull-request messages, triggers a build and report build status as well as
taskotron/resultsdb test results to the pull-request.
Hi pingou, you forgot to include the [1] link, but it was easy enough to find :-)
https://pagure.io/protron
Great choice of a name! :)
Since protron kicks off the koji build, it knows its task identifier and
keeps
it in memory so that it can later recognize a koji build it kicked versus a
koji
build from something/someone else.
However, taskotron/resultsdb message do not include the koji task it was
running
the tests against.
The code to process Koji scratch builds is very experimental and I don't know if
it's already deployed somewhere, but the published result should definitely include
Koji task ID. It should be even reported as the "item" (and the type of the
result is "koji_task"). Can you link to results which fail to do that? I can
look for bugs. (Provided we even reported such results already, again, I'm not sure.)
So protron queries koji taskinfo <id> to get the package
nevr this task was about and keep then that info in memory to recognize
taskotron messages of interest.
In our regular workflow this isn't a problem because of the nevra restriction
on
koji is such that we cannot build the same package twice, but with the PR
model
coming to dist-git, we will need to somehow lift that constraint. There could
very well be two persons or two systems proposing a pull-requests doing the
same
update (say the-new-hotness bumps foo to 1.2.3 while missing something, while
a
contributor comes by and do the same update to 1.2.3 with the rebase of the
patch or so), both PRs will be building foo-1.2.3 and this will confuse
protron:
to which PR corresponds foo-1.2.3?
Yes, we're aware that Koji scrach builds don't need to have unique NVRs, and
that's why we need to handle it differently from standard Koji builds. Our goal is to
reference them by task ID (because there seems to be no better identifier specific to koji
scratch builds, it's just a generic task).
If the PR are created with sufficient time in between, this will not be a
problem, the newer PR will override the info in memory of the old one.
However, if the two PRs are created sufficiently close to one another (so
that
the tests for the first PR aren't finished running), this will be an issue.
To avoid this, one way would be to add to the taskotron message sent the task
id
of the build taskotron was testing.
Would this be something that is doable?
It should work out of the box once this is deployed :) If you see issues with some
experimental deployment, let us know.
It might be better to use qa-devel for these kind of discussions, I'm not sure if
everyone is subscribed here. IIUC this list is more about resultsdb conventions and stuff.
(Adam, your list hijacks our qa-devel conversations!:))
9:13 a.m.
> However, taskotron/resultsdb message do not include the koji
task it was
> running
> the tests against.
The code to process Koji scratch builds is very experimental and I don't know
if it's already deployed somewhere, but the published result should
definitely include Koji task ID. It should be even reported as the "item"
(and the type of the result is "koji_task"). Can you link to results which
fail to do that? I can look for bugs. (Provided we even reported such
results already, again, I'm not sure.)
I realized you didn't actually mention *scratch* builds in your message. In case
you're using standard Koji builds, then yes, that would be a problem. But I can't
imagine we would want to use standard Koji builds for this, right? That doesn't seem
right. And for scratch builds, the implementation should already exist as described above.
So if there's any reason why you wouldn't want to use scratch builds, please shout
and we can discuss. Otherwise we should be good to go here.
1:33 p.m.
On Wed, Mar 08, 2017 at 10:13:14AM -0500, Kamil Paral wrote:
> > However, taskotron/resultsdb message do not include the
koji task it was
> > running
> > the tests against.
>
> The code to process Koji scratch builds is very experimental and I don't know
> if it's already deployed somewhere, but the published result should
> definitely include Koji task ID. It should be even reported as the "item"
> (and the type of the result is "koji_task"). Can you link to results
which
> fail to do that? I can look for bugs. (Provided we even reported such
> results already, again, I'm not sure.)
I realized you didn't actually mention *scratch* builds in your message. In case
you're using standard Koji builds, then yes, that would be a problem. But I can't
imagine we would want to use standard Koji builds for this, right? That doesn't seem
right. And for scratch builds, the implementation should already exist as described above.
So if there's any reason why you wouldn't want to use scratch builds, please shout
and we can discuss. Otherwise we should be good to go here.
I did indeed no mention scratch builds as I was speaking about real builds :)
I would want to use standard koji builds for this as we will not be able to do
chain-builds in scrach builds.
Our current idea is to make creating a tag in koji much cheaper so that we can
pretty much create a new tag for each pull-request, redefine the dist-tag in
those tags and build the rpms in there to run our tests on. This way is a PR
depends on another one, we can just either use the other's package tag or use
tag inheritance or so.
Redefining the dist-tag would allow going around the nevr uniqness restriction
in koji.
Not that I'm not suggesting we change one of the existing keys in the message
but rather that we had a new one (we expect the message format rather than
redefine it).
Does this help?
Pierre
7:15 a.m.
> I realized you didn't actually mention *scratch* builds in
your message. In
> case you're using standard Koji builds, then yes, that would be a problem.
> But I can't imagine we would want to use standard Koji builds for this,
> right? That doesn't seem right. And for scratch builds, the implementation
> should already exist as described above. So if there's any reason why you
> wouldn't want to use scratch builds, please shout and we can discuss.
> Otherwise we should be good to go here.
I did indeed no mention scratch builds as I was speaking about real builds :)
Ok, so none of what I said applies.
I would want to use standard koji builds for this as we will not be able to
do
chain-builds in scrach builds.
Aaand, would that be hard to implement? Because I believe it would be waaaay more easier
for us to do this with scratch builds :)
Our current idea is to make creating a tag in koji much cheaper so
that we
can
pretty much create a new tag for each pull-request, redefine the dist-tag in
those tags and build the rpms in there to run our tests on. This way is a PR
depends on another one, we can just either use the other's package tag or use
tag inheritance or so.
Redefining the dist-tag would allow going around the nevr uniqness
restriction
in koji.
Not that I'm not suggesting we change one of the existing keys in the message
but rather that we had a new one (we expect the message format rather than
redefine it).
The problem is that the tasks themselves are in control of creating the result message.
The only thing we can count on to be included is testcase and item and outcome. We can of
course modify all tasks we own to make a query at the end, figure out Koji task ID (or
perhaps receive it from the trigger right away), and include it in the result as an
additional field. But you can't rely on that. Any third-party koji_build task (however
useful it is) might not include the same information.
So I see two options here:
a) Make the NVR very unlikely to collide. For example use a timestamp (down to a
millisecond) in the Release field.
b) Rely on tasks providing "task-id" field in results. If there's no such
field, ignore the task.
The first one seems better. The second one wouldn't be that bad for generic tasks (in
the long run, we will probably not want to run "all available koji_build tasks"
anyway, just set a selected set of those we consider useful), but it would be hostile
again package-specific tasks in dist-git (every single task of every single package would
need to do the right thing). We definitely want to create a better experience for dist-git
tasks in the future (shield you from all the implementation stuff and automatically do
everything needed for you), but we're not there yet.
So those are the short-term solution I can see. Long-term, maybe we could re-think how we
create results and perhaps separate taskotron-provided results field from task-provided
results field, and therefore ensure information like task-id is always present. But
that's definitely not something we can do quickly.
9:16 a.m.
On Thu, Mar 09, 2017 at 08:15:17AM -0500, Kamil Paral wrote:
> I would want to use standard koji builds for this as we will not
be able to
> do chain-builds in scrach builds.
Aaand, would that be hard to implement? Because I believe it would be waaaay more easier
for us to do this with scratch builds :)
This is difficult to implement from a security PoV. :(
2599
days inactive
2609
days old
resultsdb-users@lists.fedoraproject.org
7 comments
4 participants
participants (4)
-
Adam Williamson -
Kamil Paral -
Pierre-Yves Chibon -
Ralph Bean