[PATCH] CI: Move intgcheck to essential set

[Nikolai Kondrashov]

Hi everyone,

We had an integration test issue sneak in to our public repo, because
gatekeepers only run essential CI test set by default.

The attached patch moves the integration tests to that set to avoid the issue
in the future and to also have more bugs caught, potentially.

Nick

[Nikolai Kondrashov]

On 06/15/2015 02:22 PM, Nikolai Kondrashov wrote:
> Hi everyone,
>
> We had an integration test issue sneak in to our public repo, because
> gatekeepers only run essential CI test set by default.
>
> The attached patch moves the integration tests to that set to avoid the issue
> in the future and to also have more bugs caught, potentially.

Here is a CI result: http://sssd-ci.duckdns.org/logs/job/17/41/summary.html

Nick

[Lukas Slebodnik]

On (15/06/15 14:22), Nikolai Kondrashov wrote:
> Hi everyone,
>
> We had an integration test issue sneak in to our public repo, because
> gatekeepers only run essential CI test set by default.
>
> The attached patch moves the integration tests to that set to avoid the issue
> in the future and to also have more bugs caught, potentially.
>
> Nick
> > From 47ca208fe7f660eeaad5ab65ff8f6da78baee81b Mon Sep 17 00:00:00 2001
>
> From: Nikolai Kondrashov 
> Date: Mon, 15 Jun 2015 13:26:08 +0300
> Subject: [PATCH] CI: Move intgcheck to essential set
>
> Move integration test run ("intgcheck") to "essential" test set.  It
> doesn't take long - 2.5 minutes at most, and it can (potentially) catch
> a wide range of problems. This will also make integration tests run on
> default push to CI.
>
> contrib/ci/run | 16 ++++++++--------
> 1 file changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/contrib/ci/run b/contrib/ci/run
> index 5f668ff..a07ac5c 100755
> --- a/contrib/ci/run
> +++ b/contrib/ci/run
> @@ -217,15 +217,15 @@ function build_debug()
>     mv "$test_dir" ci-test-dir
>     ((status == 0))
> if "$MODERATE"; then
>    if "$DEPS_INTGCHECK_SATISFIED"; then
>        printf -v intgcheck_configure_args " %q" \
>                    "${CONFIGURE_ARG_LIST[@]}"
>        stage make-intgcheck make -j $CPU_NUM intgcheck \
>                                  INTGCHECK_CONFIGURE_FLAGS=" \
>                                    $intgcheck_configure_args"
>    fi
>
> if "$DEPS_INTGCHECK_SATISFIED"; then
>    printf -v intgcheck_configure_args " %q" \
>                "${CONFIGURE_ARG_LIST[@]}"
>    stage make-intgcheck make -j $CPU_NUM intgcheck \
>                              INTGCHECK_CONFIGURE_FLAGS=" \
>                                $intgcheck_configure_args"
>
> fi
>
> if "$MODERATE"; then
>    test_dir_distcheck=`mktemp --directory /dev/shm/ci-test-dir.XXXXXXXX`
>    # Single thread due to https://fedorahosted.org/sssd/ticket/2354
>    status=0

It's just a partial solution.
If we want to have have nice green tests in post commit execution,
Then developers have to run the same test set.

I would prefer if developers ran CI script locally with all tests (rigorrous).
But unfortunatelly, there are developers who don't want to do it.

We had a simillar discussion few months ago (Mar 23)
because make distcheck failed. (commits 50b8a36 0ec41ab 08f8328)

So the main question is:
Do we want to have green tests in post commit execution?
or we can have failed tests and we can fix problems caused by
make distcheck or wrong spec file later.

LS

[Nikolai Kondrashov]

On 06/15/2015 02:51 PM, Lukas Slebodnik wrote:
> On (15/06/15 14:22), Nikolai Kondrashov wrote:
> > Hi everyone,
> >
> > We had an integration test issue sneak in to our public repo, because
> > gatekeepers only run essential CI test set by default.
> >
> > The attached patch moves the integration tests to that set to avoid the issue
> > in the future and to also have more bugs caught, potentially.
> >
> > Nick
> > > From 47ca208fe7f660eeaad5ab65ff8f6da78baee81b Mon Sep 17 00:00:00 2001
> >
> > From: Nikolai Kondrashov 
> > Date: Mon, 15 Jun 2015 13:26:08 +0300
> > Subject: [PATCH] CI: Move intgcheck to essential set
> >
> > Move integration test run ("intgcheck") to "essential" test set.  It
> > doesn't take long - 2.5 minutes at most, and it can (potentially) catch
> > a wide range of problems. This will also make integration tests run on
> > default push to CI.
> >
> > contrib/ci/run | 16 ++++++++--------
> > 1 file changed, 8 insertions(+), 8 deletions(-)
> >
> > diff --git a/contrib/ci/run b/contrib/ci/run
> > index 5f668ff..a07ac5c 100755
> > --- a/contrib/ci/run
> > +++ b/contrib/ci/run
> > @@ -217,15 +217,15 @@ function build_debug()
> >      mv "$test_dir" ci-test-dir
> >      ((status == 0))
> > if "$MODERATE"; then
> >    if "$DEPS_INTGCHECK_SATISFIED"; then
> >        printf -v intgcheck_configure_args " %q" \
> >                    "${CONFIGURE_ARG_LIST[@]}"
> >        stage make-intgcheck make -j $CPU_NUM intgcheck \
> >                                  INTGCHECK_CONFIGURE_FLAGS=" \
> >                                    $intgcheck_configure_args"
> >    fi
> >
> > if "$DEPS_INTGCHECK_SATISFIED"; then
> >    printf -v intgcheck_configure_args " %q" \
> >                "${CONFIGURE_ARG_LIST[@]}"
> >    stage make-intgcheck make -j $CPU_NUM intgcheck \
> >                              INTGCHECK_CONFIGURE_FLAGS=" \
> >                                $intgcheck_configure_args"
> >
> > fi
> >
> > if "$MODERATE"; then
> >     test_dir_distcheck=`mktemp --directory /dev/shm/ci-test-dir.XXXXXXXX`
> >     # Single thread due to https://fedorahosted.org/sssd/ticket/2354
> >     status=0
>
> It's just a partial solution.
> If we want to have have nice green tests in post commit execution,
> Then developers have to run the same test set.
>
> I would prefer if developers ran CI script locally with all tests (rigorrous).
> But unfortunatelly, there are developers who don't want to do it.
>
> We had a simillar discussion few months ago (Mar 23)
> because make distcheck failed. (commits 50b8a36 0ec41ab 08f8328)
>
> So the main question is:
> Do we want to have green tests in post commit execution?
> or we can have failed tests and we can fix problems caused by
> make distcheck or wrong spec file later.

I would also like all tests being run *before* pushing to the public repo.

But if that's too burdensome for whatever reason, then it is better to have
some extra tests run *after* pushing, than not have them running at all.

We should consider the effort required to run all these tests before pushing
vs the impact and frequency of the test failures caught.

Nick

[Lukas Slebodnik]

On (15/06/15 16:03), Nikolai Kondrashov wrote:
> On 06/15/2015 02:51 PM, Lukas Slebodnik wrote:
> > On (15/06/15 14:22), Nikolai Kondrashov wrote:
> > > Hi everyone,
> > >
> > > We had an integration test issue sneak in to our public repo, because
> > > gatekeepers only run essential CI test set by default.
> > >
> > > The attached patch moves the integration tests to that set to avoid the issue
> > > in the future and to also have more bugs caught, potentially.
> > >
> > > Nick
> > > > From 47ca208fe7f660eeaad5ab65ff8f6da78baee81b Mon Sep 17 00:00:00 2001
> > >
> > > From: Nikolai Kondrashov 
> > > Date: Mon, 15 Jun 2015 13:26:08 +0300
> > > Subject: [PATCH] CI: Move intgcheck to essential set
> > >
> > > Move integration test run ("intgcheck") to "essential" test set.  It
> > > doesn't take long - 2.5 minutes at most, and it can (potentially) catch
> > > a wide range of problems. This will also make integration tests run on
> > > default push to CI.
> > >
> > > contrib/ci/run | 16 ++++++++--------
> > > 1 file changed, 8 insertions(+), 8 deletions(-)
> > >
> > > diff --git a/contrib/ci/run b/contrib/ci/run
> > > index 5f668ff..a07ac5c 100755
> > > --- a/contrib/ci/run
> > > +++ b/contrib/ci/run
> > > @@ -217,15 +217,15 @@ function build_debug()
> > >     mv "$test_dir" ci-test-dir
> > >     ((status == 0))
> > > if "$MODERATE"; then
> > >    if "$DEPS_INTGCHECK_SATISFIED"; then
> > >        printf -v intgcheck_configure_args " %q" \
> > >                    "${CONFIGURE_ARG_LIST[@]}"
> > >        stage make-intgcheck make -j $CPU_NUM intgcheck \
> > >                                  INTGCHECK_CONFIGURE_FLAGS=" \
> > >                                    $intgcheck_configure_args"
> > >    fi
> > >
> > > if "$DEPS_INTGCHECK_SATISFIED"; then
> > >    printf -v intgcheck_configure_args " %q" \
> > >                "${CONFIGURE_ARG_LIST[@]}"
> > >    stage make-intgcheck make -j $CPU_NUM intgcheck \
> > >                              INTGCHECK_CONFIGURE_FLAGS=" \
> > >                                $intgcheck_configure_args"
> > >
> > > fi
> > >
> > > if "$MODERATE"; then
> > >    test_dir_distcheck=`mktemp --directory /dev/shm/ci-test-dir.XXXXXXXX`
> > >    # Single thread due to https://fedorahosted.org/sssd/ticket/2354
> > >    status=0
> >
> > It's just a partial solution.
> > If we want to have have nice green tests in post commit execution,
> > Then developers have to run the same test set.
> >
> > I would prefer if developers ran CI script locally with all tests (rigorrous).
> > But unfortunatelly, there are developers who don't want to do it.
> >
> > We had a simillar discussion few months ago (Mar 23)
> > because make distcheck failed. (commits 50b8a36 0ec41ab 08f8328)
> >
> > So the main question is:
> > Do we want to have green tests in post commit execution?
> > or we can have failed tests and we can fix problems caused by
> > make distcheck or wrong spec file later.
>
> I would also like all tests being run *before* pushing to the public repo.

Developrs used to sending links to CI results.
We can switch default from essentials to rigorous and we will have
almost the same effect as "all tests being run *before* pushing to the public
repo"

If someone wants to have faster tests he can still override default
to essential and have results faster.

LS

[Nikolai Kondrashov]

On 06/15/2015 04:24 PM, Lukas Slebodnik wrote:
> On (15/06/15 16:03), Nikolai Kondrashov wrote:
> > On 06/15/2015 02:51 PM, Lukas Slebodnik wrote:
> > > On (15/06/15 14:22), Nikolai Kondrashov wrote:
> > > > Hi everyone,
> > > >
> > > > We had an integration test issue sneak in to our public repo, because
> > > > gatekeepers only run essential CI test set by default.
> > > >
> > > > The attached patch moves the integration tests to that set to avoid the issue
> > > > in the future and to also have more bugs caught, potentially.
> > > >
> > > > Nick
> > > > > From 47ca208fe7f660eeaad5ab65ff8f6da78baee81b Mon Sep 17 00:00:00 2001
> > > >
> > > > From: Nikolai Kondrashov 
> > > > Date: Mon, 15 Jun 2015 13:26:08 +0300
> > > > Subject: [PATCH] CI: Move intgcheck to essential set
> > > >
> > > > Move integration test run ("intgcheck") to "essential" test set.  It
> > > > doesn't take long - 2.5 minutes at most, and it can (potentially) catch
> > > > a wide range of problems. This will also make integration tests run on
> > > > default push to CI.
> > > >
> > > > contrib/ci/run | 16 ++++++++--------
> > > > 1 file changed, 8 insertions(+), 8 deletions(-)
> > > >
> > > > diff --git a/contrib/ci/run b/contrib/ci/run
> > > > index 5f668ff..a07ac5c 100755
> > > > --- a/contrib/ci/run
> > > > +++ b/contrib/ci/run
> > > > @@ -217,15 +217,15 @@ function build_debug()
> > > >      mv "$test_dir" ci-test-dir
> > > >      ((status == 0))
> > > > if "$MODERATE"; then
> > > >    if "$DEPS_INTGCHECK_SATISFIED"; then
> > > >        printf -v intgcheck_configure_args " %q" \
> > > >                    "${CONFIGURE_ARG_LIST[@]}"
> > > >        stage make-intgcheck make -j $CPU_NUM intgcheck \
> > > >                                  INTGCHECK_CONFIGURE_FLAGS=" \
> > > >                                    $intgcheck_configure_args"
> > > >    fi
> > > >
> > > > if "$DEPS_INTGCHECK_SATISFIED"; then
> > > >    printf -v intgcheck_configure_args " %q" \
> > > >                "${CONFIGURE_ARG_LIST[@]}"
> > > >    stage make-intgcheck make -j $CPU_NUM intgcheck \
> > > >                              INTGCHECK_CONFIGURE_FLAGS=" \
> > > >                                $intgcheck_configure_args"
> > > >
> > > > fi
> > > >
> > > > if "$MODERATE"; then
> > > >     test_dir_distcheck=`mktemp --directory /dev/shm/ci-test-dir.XXXXXXXX`
> > > >     # Single thread due to https://fedorahosted.org/sssd/ticket/2354
> > > >     status=0
> > >
> > > It's just a partial solution.
> > > If we want to have have nice green tests in post commit execution,
> > > Then developers have to run the same test set.
> > >
> > > I would prefer if developers ran CI script locally with all tests (rigorrous).
> > > But unfortunatelly, there are developers who don't want to do it.
> > >
> > > We had a simillar discussion few months ago (Mar 23)
> > > because make distcheck failed. (commits 50b8a36 0ec41ab 08f8328)
> > >
> > > So the main question is:
> > > Do we want to have green tests in post commit execution?
> > > or we can have failed tests and we can fix problems caused by
> > > make distcheck or wrong spec file later.
> >
> > I would also like all tests being run *before* pushing to the public repo.
>
> Developrs used to sending links to CI results.
> We can switch default from essentials to rigorous and we will have
> almost the same effect as "all tests being run *before* pushing to the public
> repo"
>
> If someone wants to have faster tests he can still override default
> to essential and have results faster.

I'm fine with this. But then, I don't often push to CI. What about others?

Nick

[Jakub Hrozek]

On Mon, Jun 15, 2015 at 04:25:16PM +0300, Nikolai Kondrashov wrote:
> I'm fine with this. But then, I don't often push to CI. What about others?

I think it's worth trying out. I prefer precision over speed, because
the history already shown we're not great at running the strictest set.

[Nikolai Kondrashov]

On 06/15/2015 04:57 PM, Jakub Hrozek wrote:
> On Mon, Jun 15, 2015 at 04:25:16PM +0300, Nikolai Kondrashov wrote:
> > I'm fine with this. But then, I don't often push to CI. What about others?
>
> I think it's worth trying out. I prefer precision over speed, because
> the history already shown we're not great at running the strictest set.

Alright, unless there's any opposition, I'll change the default to "rigorous"
tomorrow.

Nick

[Nikolai Kondrashov]

On 06/15/2015 04:59 PM, Nikolai Kondrashov wrote:
> On 06/15/2015 04:57 PM, Jakub Hrozek wrote:
> > On Mon, Jun 15, 2015 at 04:25:16PM +0300, Nikolai Kondrashov wrote:
> > > I'm fine with this. But then, I don't often push to CI. What about others?
> >
> > I think it's worth trying out. I prefer precision over speed, because
> > the history already shown we're not great at running the strictest set.
>
> Alright, unless there's any opposition, I'll change the default to "rigorous"
> tomorrow.

Done.

Nick