How to increase the test timeout

[Tom Stellard]

Hi,

I've tried to increase the test timeout for the llvm package by adding
a duration.fmf file to my tests/ directory.

The file contents look like this:

[tstellar@pkg tests]$ cat duration.fmf 
---
duration: 6h

This did not seem to work for the latest test run:
https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pi...

Is this the correct way to specify a timeout?

-Tom

[Petr Šplíchal]

Hi Tom,

as far as I know the timeout in the Fedora CI pipeline is hardcoded to 4
hours
https://github.com/CentOS-PaaS-SIG/ci-pipeline/blob/master/config/Dockerfiles/singlehost-test/package-test.sh#L194
and I don't think there is currently a way how this default can be
configured. Bruno, can you confirm this?

The duration attribute
https://pagure.io/fedora-ci/metadata/blob/master/f/l1/duration.fmf you
mention is not implemented yet (see the status field). It is part of the L1
metadata proposal which is still work in progress. On the Flock last week
we've just presented
https://flock2019.sched.com/event/3bf12a7b3ddda508c2dab6562d1ef533 the
first PoC implementing L1 & L2 metadata.

psss...

On Thu, 8 Aug 2019 at 18:35, Tom Stellard  wrote:
> Hi,
>
> I've tried to increase the test timeout for the llvm package by adding
> a duration.fmf file to my tests/ directory.
>
> The file contents look like this:
> [tstellar@pkg tests]$ cat duration.fmf
>
> duration: 6h
>
> This did not seem to work for the latest test run:
>
> https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pi...
>
> Is this the correct way to specify a timeout?
>
> -Tom
> _______________________________________________
> CI mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Bruno Goncalves]

On Mon, Aug 12, 2019 at 2:39 PM Petr Šplíchal  wrote:
> Hi Tom,
>
> as far as I know the timeout in the Fedora CI pipeline is hardcoded to 4 hours and I don't think there is currently a way how this default can be configured. Bruno, can you confirm this?

Correct, this is hardcoded in the pipeline, we currently don't have a
way to dynamically change it. We didn't expect any test playbook to
take longer than 4hrs to execute. Would be possible to split the test
in separate tests*.yml playbook? As each tests*.yml would run in a
clean environment and each one would have 4hrs as timeout.

Bruno
> The duration attribute you mention is not implemented yet (see the status field). It is part of the L1 metadata proposal which is still work in progress. On the Flock last week we've just presented the first PoC implementing L1 & L2 metadata.
>
> psss...
>
> On Thu, 8 Aug 2019 at 18:35, Tom Stellard  wrote:
> > Hi,
> >
> > I've tried to increase the test timeout for the llvm package by adding
> > a duration.fmf file to my tests/ directory.
> >
> > The file contents look like this:
> > [tstellar@pkg tests]$ cat duration.fmf
> >
> > duration: 6h
> >
> > This did not seem to work for the latest test run:
> > https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pi...
> >
> > Is this the correct way to specify a timeout?
> >
> > -Tom
> > _______________________________________________
> > CI mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Dominik Perpeet]

What would be the impact of just increasing the hardcoded max duration to 6
hours?

-Dominik

On Mon, Aug 12, 2019 at 2:54 PM Bruno Goncalves  wrote:
> On Mon, Aug 12, 2019 at 2:39 PM Petr Šplíchal  wrote:
> > Hi Tom,
> >
> > as far as I know the timeout in the Fedora CI pipeline is hardcoded to 4
>
> hours and I don't think there is currently a way how this default can be
> configured. Bruno, can you confirm this?
> >
>
> Correct, this is hardcoded in the pipeline, we currently don't have a
> way to dynamically change it. We didn't expect any test playbook to
> take longer than 4hrs to execute. Would be possible to split the test
> in separate tests*.yml playbook? As each tests*.yml would run in a
> clean environment and each one would have 4hrs as timeout.
>
> Bruno
> > The duration attribute you mention is not implemented yet (see the
>
> status field). It is part of the L1 metadata proposal which is still work
> in progress. On the Flock last week we've just presented the first PoC
> implementing L1 & L2 metadata.
> > psss...
> >
> > On Thu, 8 Aug 2019 at 18:35, Tom Stellard  wrote:
> > > Hi,
> > >
> > > I've tried to increase the test timeout for the llvm package by adding
> > > a duration.fmf file to my tests/ directory.
> > >
> > > The file contents look like this:
> > > [tstellar@pkg tests]$ cat duration.fmf
> > >
> > > duration: 6h
> > >
> > > This did not seem to work for the latest test run:
>
> https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pi...
> > > Is this the correct way to specify a timeout?
> > >
> > > -Tom
> > > _______________________________________________
> > > CI mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct:
>
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail]
> _______________________________________________
> CI mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Bruno Goncalves]

On Mon, Aug 12, 2019 at 3:41 PM Dominik Perpeet  wrote:
> What would be the impact of just increasing the hardcoded max duration to 6 hours?

Just some tests when they hung or the system hungs it will take 2 more
hours to have the results. It will also take 2 more hours to have the
resources free. If that's fine I see no problem to raise the timeout.

We just need a new issue requesting it on https://pagure.io/fedora-ci/general/
> -Dominik
>
> On Mon, Aug 12, 2019 at 2:54 PM Bruno Goncalves  wrote:
> > On Mon, Aug 12, 2019 at 2:39 PM Petr Šplíchal  wrote:
> > > Hi Tom,
> > >
> > > as far as I know the timeout in the Fedora CI pipeline is hardcoded to 4 hours and I don't think there is currently a way how this default can be configured. Bruno, can you confirm this?
> >
> > Correct, this is hardcoded in the pipeline, we currently don't have a
> > way to dynamically change it. We didn't expect any test playbook to
> > take longer than 4hrs to execute. Would be possible to split the test
> > in separate tests*.yml playbook? As each tests*.yml would run in a
> > clean environment and each one would have 4hrs as timeout.
> >
> > Bruno
> > > The duration attribute you mention is not implemented yet (see the status field). It is part of the L1 metadata proposal which is still work in progress. On the Flock last week we've just presented the first PoC implementing L1 & L2 metadata.
> > >
> > > psss...
> > >
> > > On Thu, 8 Aug 2019 at 18:35, Tom Stellard  wrote:
> > > > Hi,
> > > >
> > > > I've tried to increase the test timeout for the llvm package by adding
> > > > a duration.fmf file to my tests/ directory.
> > > >
> > > > The file contents look like this:
> > > > [tstellar@pkg tests]$ cat duration.fmf
> > > >
> > > > duration: 6h
> > > >
> > > > This did not seem to work for the latest test run:
> > > > https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pi...
> > > >
> > > > Is this the correct way to specify a timeout?
> > > >
> > > > -Tom
> > > > _______________________________________________
> > > > CI mailing list -- [e-mail]
> > > > To unsubscribe send an email to [e-mail]
> > > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> >
> > CI mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
>
> CI mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Dominik Perpeet]

I think this would be a good discussion to have in the Fedora CI SIG - or
even see if others have an opinion.

Waiting additional time doesn't seem like that much fun... we should
probably scope the work necessary to make the timeout more dynamic. Maybe
it'd even be easier for Tom and others to contribute to that rather than
trying to find a workaround.

-Dominik

On Mon, Aug 12, 2019 at 3:45 PM Bruno Goncalves  wrote:
> On Mon, Aug 12, 2019 at 3:41 PM Dominik Perpeet 
> wrote:
> > What would be the impact of just increasing the hardcoded max duration
>
> to 6 hours?
>
> Just some tests when they hung or the system hungs it will take 2 more
> hours to have the results. It will also take 2 more hours to have the
> resources free. If that's fine I see no problem to raise the timeout.
>
> We just need a new issue requesting it on
> https://pagure.io/fedora-ci/general/
> > -Dominik
> >
> > On Mon, Aug 12, 2019 at 2:54 PM Bruno Goncalves
>
> wrote:
> > > On Mon, Aug 12, 2019 at 2:39 PM Petr Šplíchal
>
> wrote:
> > > > Hi Tom,
> > > >
> > > > as far as I know the timeout in the Fedora CI pipeline is hardcoded
>
> to 4 hours and I don't think there is currently a way how this default can
> be configured. Bruno, can you confirm this?
> > > >
> > >
> > > Correct, this is hardcoded in the pipeline, we currently don't have a
> > > way to dynamically change it. We didn't expect any test playbook to
> > > take longer than 4hrs to execute. Would be possible to split the test
> > > in separate tests*.yml playbook? As each tests*.yml would run in a
> > > clean environment and each one would have 4hrs as timeout.
> > >
> > > Bruno
> > > > The duration attribute you mention is not implemented yet (see the
>
> status field). It is part of the L1 metadata proposal which is still work
> in progress. On the Flock last week we've just presented the first PoC
> implementing L1 & L2 metadata.
> > > > psss...
> > > >
> > > > On Thu, 8 Aug 2019 at 18:35, Tom Stellard
>
> wrote:
> > > > > Hi,
> > > > >
> > > > > I've tried to increase the test timeout for the llvm package by
>
> adding
> > > > > a duration.fmf file to my tests/ directory.
> > > > >
> > > > > The file contents look like this:
> > > > > [tstellar@pkg tests]$ cat duration.fmf
> > > > >
> > > > > duration: 6h
> > > > >
> > > > > This did not seem to work for the latest test run:
>
> https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pi...
> > > > > Is this the correct way to specify a timeout?
> > > > >
> > > > > -Tom
> > > > > _______________________________________________
> > > > > CI mailing list -- [e-mail]
> > > > > To unsubscribe send an email to [e-mail]
> > > > > Fedora Code of Conduct:
>
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > > List Guidelines:
>
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > > List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail]
> > > CI mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct:
>
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail]
> > CI mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct:
>
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail]
> _______________________________________________
> CI mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Tom Stellard]

On 08/12/2019 05:54 AM, Bruno Goncalves wrote:
> On Mon, Aug 12, 2019 at 2:39 PM Petr Šplíchal  wrote:
> > Hi Tom,
> >
> > as far as I know the timeout in the Fedora CI pipeline is hardcoded to 4 hours and I don't think there is currently a way how this default can be configured. Bruno, can you confirm this?
>
> Correct, this is hardcoded in the pipeline, we currently don't have a
> way to dynamically change it. We didn't expect any test playbook to
> take longer than 4hrs to execute. Would be possible to split the test
> in separate tests*.yml playbook? As each tests*.yml would run in a
> clean environment and each one would have 4hrs as timeout.

I will try splitting the test out into a separate playbook.

Thanks,
Tom
> Bruno
> > The duration attribute you mention is not implemented yet (see the status field). It is part of the L1 metadata proposal which is still work in progress. On the Flock last week we've just presented the first PoC implementing L1 & L2 metadata.
> >
> > psss...
> >
> > On Thu, 8 Aug 2019 at 18:35, Tom Stellard  wrote:
> > > Hi,
> > >
> > > I've tried to increase the test timeout for the llvm package by adding
> > > a duration.fmf file to my tests/ directory.
> > >
> > > The file contents look like this:
> > > [tstellar@pkg tests]$ cat duration.fmf
> > >
> > > duration: 6h
> > >
> > > This did not seem to work for the latest test run:
> > > https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pi...
> > >
> > > Is this the correct way to specify a timeout?
> > >
> > > -Tom
> > > _______________________________________________
> > > CI mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
>
> CI mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Tom Stellard]

On 08/12/2019 08:24 AM, Tom Stellard wrote:
> On 08/12/2019 05:54 AM, Bruno Goncalves wrote:
> > On Mon, Aug 12, 2019 at 2:39 PM Petr Šplíchal  wrote:
> > > Hi Tom,
> > >
> > > as far as I know the timeout in the Fedora CI pipeline is hardcoded to 4 hours and I don't think there is currently a way how this default can be configured. Bruno, can you confirm this?
> >
> > Correct, this is hardcoded in the pipeline, we currently don't have a
> > way to dynamically change it. We didn't expect any test playbook to
> > take longer than 4hrs to execute. Would be possible to split the test
> > in separate tests*.yml playbook? As each tests*.yml would run in a
> > clean environment and each one would have 4hrs as timeout.
>
> I will try splitting the test out into a separate playbook.

This wasn't enough to get me under the limit.  I'm trying to
do a mockbuild of the llvm package for one of the tests
which is what is taking so long.  I'm not even sure
if the suggested 6 hour timeout would be enough.

Is there some other change that can be made to accommodate
doing mockbuids as CI tests?  Like maybe a specific mockbuild
test type?  I am interested in adding more of these types of
tests for testing clang and they could be useful for other compilers,
like gcc, golang, rust, etc.

In addition to extra timeout these tests often need extra memory too
(which I am currently requesting using the provision.fmf) and maybe
extra cores (which might help with the timeouts).

I do plan on attending the first 30 minutes of the CI SIG tomorrow
and can discuss this more at the meeting if needed.

-Tom

On brew it takes 1.5
hours to do a build on a 6 core machine.  Is there
> Thanks,
> Tom
> > Bruno
> > > The duration attribute you mention is not implemented yet (see the status field). It is part of the L1 metadata proposal which is still work in progress. On the Flock last week we've just presented the first PoC implementing L1 & L2 metadata.
> > >
> > > psss...
> > >
> > > On Thu, 8 Aug 2019 at 18:35, Tom Stellard  wrote:
> > > > Hi,
> > > >
> > > > I've tried to increase the test timeout for the llvm package by adding
> > > > a duration.fmf file to my tests/ directory.
> > > >
> > > > The file contents look like this:
> > > > [tstellar@pkg tests]$ cat duration.fmf
> > > >
> > > > duration: 6h
> > > >
> > > > This did not seem to work for the latest test run:
> > > > https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pi...
> > > >
> > > > Is this the correct way to specify a timeout?
> > > >
> > > > -Tom
> > > > _______________________________________________
> > > > CI mailing list -- [e-mail]
> > > > To unsubscribe send an email to [e-mail]
> > > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> >
> > CI mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]