8:29 p.m.
Gang,
I finally got some work done on the CI task for Taskotron in Taskotron. The
idea here is that after each commit (of a relevant project - trigger,
execdb, resultsdb, libtaskotron) to pagure, we will run the whole stack in
docker containers, and execute a known "phony" task, to see whether it all
goes fine.
The way I devised is that I'll build a 'testsuite' container based on the
Trigger, and instead of running the fedmsg hub, I'll just use the CLI to
"replay" what would happen on a known, predefined fedmsg.
The testsuite will then watch execdb and resultsdb, whether everything went
fine.
It is not at all finished, but I started hacking on it here:
https://pagure.io/taskotron/task-taskotron-ci
I hope to finish it (to a point where it runs the phony task) till the end
of the week. At that point, I'd be glad for any actual, sensible task ideas
to ideally test as much of the capabilities of the
libtaskotron/execdb/resultsdb as possible.
The only problem with this kind of testing is, that we still don't really
have a good way to test trigger, as it is tied to external events. My idea
here was that I could add something like wiki edit consumer, and trigger
tasks off of that, making that one "triggering" edit from inside the
testsuite. But As it's almost 4am here, I'm not sure it is the best idea.
Once again, I'll be glad for any input/ideas/evil laughter.
Joza
10:58 a.m.
On Thu, Feb 09, 2017 at 03:29:13AM +0100, Josef Skladanka wrote:
I finally got some work done on the CI task for Taskotron in
Taskotron. The
idea here is that after each commit (of a relevant project - trigger,
execdb, resultsdb, libtaskotron) to pagure, we will run the whole stack in
docker containers, and execute a known "phony" task, to see whether it all
goes fine.
This is excellent. I'd love, eventually, to get to a point where we can
run the checks _pre_ commit and gate on them. Is there a path from this
to that?
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
12:31 a.m.
On Thu, Feb 9, 2017 at 5:58 PM, Matthew Miller <mattdm(a)fedoraproject.org>
wrote:
On Thu, Feb 09, 2017 at 03:29:13AM +0100, Josef Skladanka wrote:
> I finally got some work done on the CI task for Taskotron in Taskotron.
The
> idea here is that after each commit (of a relevant project - trigger,
> execdb, resultsdb, libtaskotron) to pagure, we will run the whole stack
in
> docker containers, and execute a known "phony" task, to see whether it
all
> goes fine.
This is excellent. I'd love, eventually, to get to a point where we can
run the checks _pre_ commit and gate on them. Is there a path from this
to that?
Absolutely, that is the goal.
Generally speaking, we'd like to run tests on Pagure's PRs.
For taskotron specifically, we'll need to figure out some Phabricator
plugin that fires off a fedmsg (or calls some API, whatever) on new
Differential request, but generally it is the same idea.
Joza
5:38 a.m.
So, the repo now has working PoC
https://pagure.io/taskotron/task-taskotron-ci
Readme contains example on how to run the task.
Works on my setup, and I'd be glad if somebody else tried.
J.
On Fri, Feb 10, 2017 at 7:31 AM, Josef Skladanka <jskladan(a)redhat.com>
wrote:
On Thu, Feb 9, 2017 at 5:58 PM, Matthew Miller <mattdm(a)fedoraproject.org>
wrote:
> On Thu, Feb 09, 2017 at 03:29:13AM +0100, Josef Skladanka wrote:
> > I finally got some work done on the CI task for Taskotron in Taskotron.
> The
> > idea here is that after each commit (of a relevant project - trigger,
> > execdb, resultsdb, libtaskotron) to pagure, we will run the whole stack
> in
> > docker containers, and execute a known "phony" task, to see whether
it
> all
> > goes fine.
>
> This is excellent. I'd love, eventually, to get to a point where we can
> run the checks _pre_ commit and gate on them. Is there a path from this
> to that?
Absolutely, that is the goal.
Generally speaking, we'd like to run tests on Pagure's PRs.
For taskotron specifically, we'll need to figure out some Phabricator
plugin that fires off a fedmsg (or calls some API, whatever) on new
Differential request, but generally it is the same idea.
Joza
6:20 a.m.
So, the repo now has working PoC
https://pagure.io/taskotron/task-taskotron-ci
Readme contains example on how to run the task.
Works on my setup, and I'd be glad if somebody else tried.
I tried, and it didn't eat my laptop. Also it worked. Good job.
/me throws cookies at jskladan
1:46 p.m.
The only problem with this kind of testing is, that we still
don't really
have a good way to test trigger, as it is tied to external events. My idea
here was that I could add something like wiki edit consumer, and trigger
tasks off of that, making that one "triggering" edit from inside the
testsuite.
In my experience, some fedmsg notifications are quite delayed from time to time, easily by
several hours. But I don't know if the delay is in actual fedmsg bus, or just in FMN
(which I use to get notified on IRC). I suspect rather FMN. But if that turned out to be a
fedmsg delay, that might make this approach impractical (or at least less reliable). So
something to consider and possibly investigate.
I think it would be better to fake the input data. Either by having our own fedmsg
producer and/or bus (I don't know precisely how fedmsg works and how hard is to set up
something like that), or by making some code adjustments in trigger that will allow us to
bypass the fedmsg reception and replace it with our own fedmsg, but cover as much
surrounding code as possible.
1:51 p.m.
On Tue, 2017-02-14 at 14:46 -0500, Kamil Paral wrote:
> The only problem with this kind of testing is, that we still
don't really
> have a good way to test trigger, as it is tied to external events. My idea
> here was that I could add something like wiki edit consumer, and trigger
> tasks off of that, making that one "triggering" edit from inside the
> testsuite.
In my experience, some fedmsg notifications are quite delayed from
time to time, easily by several hours. But I don't know if the delay
is in actual fedmsg bus, or just in FMN (which I use to get notified
on IRC). I suspect rather FMN. But if that turned out to be a fedmsg
delay, that might make this approach impractical (or at least less
reliable). So something to consider and possibly investigate.
I believe it's FMN. I've noticed the same delay with FMN notifications,
but I've never noticed any delay with the actual fedmsg notifications
used by all the stuff we've built around release validation (compose
complete fedmsgs, openQA test complete fedmsgs etc.)
I think it would be better to fake the input data. Either by having
our own fedmsg producer and/or bus (I don't know precisely how fedmsg
works and how hard is to set up something like that),
Are you aware of fedmsg-dg-replay? It's a fairly easy way to 'replay'
fedmsgs for testing. All you need (IIRC) is the fedmsg-relay service
running on the same system, and you can run
'fedmsg-dg-replay --msg-id <somemsgid>' and the message will be
replayed on the local bus (if the message is .prod. or .stg. , this
will be changed to .dev. , and the message will not be signed).
I don't remember exactly how the taskotron trigger code works, but it
shouldn't be too hard to configure a trigger to accept unsigned
messages with .dev. topics, for the purpose of testing with -replay. In
the fedmsg consumers I've written I have a pattern of providing three
consumers, one which listens for .prod. messages, one for .stg.
messages, and one which listens for for .dev. messages and is intended
for testing with fedmsg-dg-replay.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
2:41 a.m.
On Tue, 2017-02-14 at 14:46 -0500, Kamil Paral wrote:
> > The only problem with this kind of testing is, that we still don't really
> > have a good way to test trigger, as it is tied to external events. My
> > idea
> > here was that I could add something like wiki edit consumer, and trigger
> > tasks off of that, making that one "triggering" edit from inside the
> > testsuite.
>
> In my experience, some fedmsg notifications are quite delayed from
> time to time, easily by several hours. But I don't know if the delay
> is in actual fedmsg bus, or just in FMN (which I use to get notified
> on IRC). I suspect rather FMN. But if that turned out to be a fedmsg
> delay, that might make this approach impractical (or at least less
> reliable). So something to consider and possibly investigate.
I believe it's FMN. I've noticed the same delay with FMN notifications,
but I've never noticed any delay with the actual fedmsg notifications
used by all the stuff we've built around release validation (compose
complete fedmsgs, openQA test complete fedmsgs etc.)
> I think it would be better to fake the input data. Either by having
> our own fedmsg producer and/or bus (I don't know precisely how fedmsg
> works and how hard is to set up something like that),
Are you aware of fedmsg-dg-replay? It's a fairly easy way to 'replay'
fedmsgs for testing. All you need (IIRC) is the fedmsg-relay service
running on the same system, and you can run
'fedmsg-dg-replay --msg-id <somemsgid>' and the message will be
replayed on the local bus (if the message is .prod. or .stg. , this
will be changed to .dev. , and the message will not be signed).
I don't remember exactly how the taskotron trigger code works, but it
shouldn't be too hard to configure a trigger to accept unsigned
messages with .dev. topics, for the purpose of testing with -replay. In
the fedmsg consumers I've written I have a pattern of providing three
consumers, one which listens for .prod. messages, one for .stg.
messages, and one which listens for for .dev. messages and is intended
for testing with fedmsg-dg-replay.
Great info, thanks.
5:59 a.m.
On Tue, Feb 14, 2017 at 8:51 PM, Adam Williamson <adamwill(a)fedoraproject.org
wrote:
Are you aware of fedmsg-dg-replay? It's a fairly easy way to
'replay'
fedmsgs for testing. All you need (IIRC) is the fedmsg-relay service
running on the same system, and you can run
I am, but it has this bad quality of changing the topic, so we would have
to change the consumers' topics, so we would need to change those too. Or
make it configurable in some way...
I'd rather do it the way I have it now - using the trigger's internal
replay functionality instead of doing unnecessary complicated changes just
for the sake of using it to test stuff once in a while.
J.
10:55 a.m.
On Wed, 2017-02-15 at 12:59 +0100, Josef Skladanka wrote:
On Tue, Feb 14, 2017 at 8:51 PM, Adam Williamson
<adamwill(a)fedoraproject.org
> wrote:
> Are you aware of fedmsg-dg-replay? It's a fairly easy way to 'replay'
> fedmsgs for testing. All you need (IIRC) is the fedmsg-relay service
> running on the same system, and you can run
>
I am, but it has this bad quality of changing the topic, so we would have
to change the consumers' topics, so we would need to change those too. Or
make it configurable in some way...
I'd rather do it the way I have it now - using the trigger's internal
replay functionality instead of doing unnecessary complicated changes just
for the sake of using it to test stuff once in a while.
I was thinking that it's probably not that difficult to set up a
testing fedmsg bus as a test fixture with some canned messages that can
be replayed on request, but I haven't looked at doing it so I really
don't know how much work it is. I wonder if fedmsg's test suite does
it.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
4:52 p.m.
On Wed, Feb 15, 2017 at 5:55 PM, Adam Williamson <adamwill(a)fedoraproject.org
wrote:
> On Wed, 2017-02-15 at 12:59 +0100, Josef Skladanka wrote:
> > On Tue, Feb 14, 2017 at 8:51 PM, Adam Williamson <
> adamwill(a)fedoraproject.org
> > wrote:
> > > Are you aware
of fedmsg-dg-replay? It's a fairly easy way to 'replay'
> > > fedmsgs for testing. All you need (IIRC) is the fedmsg-relay service
> > > running on the same system, and you can run
> > >
> >
> > I am, but it has this bad quality of changing the topic, so we would have
> > to change the consumers' topics, so we would need to change those too. Or
> > make it configurable in some way...
> > I'd rather do it the way I have it now - using the trigger's internal
> > replay functionality instead of doing unnecessary complicated changes
> just
> > for the sake of using it to test stuff once in a while.
>
> I was thinking that it's probably not that difficult to set up a
> testing fedmsg bus as a test fixture with some canned messages that can
> be replayed on request, but I haven't looked at doing it so I really
> don't know how much work it is. I wonder if fedmsg's test suite does
> it.
>
>
Ah so, I did not get that at the first read, and now it is obvious even
from the previous email *facepalm*. Yeah, that would make sense, I guess.
We'll see about that, at the moment, we have bigger fish to fry, but in the
end I'd like to have this stuff covered too.
Thanks for the good idea, though!
joza
2626
days inactive
2632
days old
qa-devel@lists.fedoraproject.org
10 comments
4 participants
participants (4)
-
Adam Williamson -
Josef Skladanka -
Kamil Paral -
Matthew Miller