CI projects in Copr

[Miroslav Suchý]

Hi,
I am gathering informations about various use of CI with Copr. Do you use Copr for building packages for nightlies? For 
building packages before pull request is merged? Do you have your set up described somewhere? What is the name of your 
project?

Please let me know. Either here or via private reply.
It will help me to understand your use of Copr and to make Copr better.

Thanks in advance.

Miroslav Suchy

[Igor Gnatenko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, 2017-08-23 at 13:47 +0200, Miroslav Suchý wrote:
> Hi,

Hi,
> I am gathering informations about various use of CI with Copr. Do you
> use Copr for building packages for nightlies? For 
> building packages before pull request is merged? Do you have your set
> up described somewhere? What is the name of your 
> project?

We do! Both. Our project name is: DNF 😛

So our setup is pretty simple, our tool clones multiple repositories
locally, builds SRPMs for them (it might be spec in upstream or
upstream repo + downstream spec) and then creates temporary COPR
repository and builds RPMs in there. Then we download those RPMs
locally and do something with them (build container with them and run
many tests).
> Please let me know. Either here or via private reply.
> It will help me to understand your use of Copr and to make Copr
> better.

I would like to see:
* only one and supported / good API
* probably ability for some good integration with custom tools as we
use (rpm-gitoverlay)
* or ability to push custom specs / sources into git repo so we don't
need to build SRPMs locally, we would just do git commit + git push and
COPR would build everything
* "Temporary" repositories -- repositories which automatically getting
removed after some time, we need such repos for pull request testing
and after few weeks no one will look into / use them, so it can be
safely removed
* Faster builds (for 7-8 packages we are building, it takes ~20 minutes
to build because we do it sequentially, but we can't do anything about
that), probably some re-use of builders or faster spawning builds would
help here, not sure
* auto-rebuilds like koschei does (but I don't really want to see
koschei as separate service, it should be built-in for better
integration). Ideally this should be in Fedora/Koji as well, but it
really depends what's COPR is aiming to do
> Thanks in advance.
>
> Miroslav Suchy
> _______________________________________________
> copr-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail].o
> rg

- -- 
- -Igor Gnatenko
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCAAdFiEEhLFO09aHZVqO+CM6aVcUvRu8X0wFAlmdgHgACgkQaVcUvRu8
> X0w63w/9GS6eMuw2WIrbNSZeZbmxKg2kWIG5NwLHLQrHUEmEcAPRUa2X8+rao62w
> XQduQeA3tWR7Hktxr5GFGYxVe/5zQ5uFQQrfTyJKan8rryzOvs8xPbcNbH6VcWyJ
> qVAzbbQMIAmHFZItHmvxrCbJgRYmlTCQnDGSpYVrwfnXiGSq/vHu3AvnxBUdZuSz
> PFaDP1kfgfS/BpB9hD91AiuAKySCcfc74DpMVA3NXIbldum5UX8HtVID0EWglKf7
> XmGttLQ5ng1vxZqRVf/6fFt2frj5ZYlEUlRYiLHk66uNGJ0aP1DkAAPCVaNoVtqw
> A7ZXWjVxRJshYZnEZuKLtQvq1QYAvNjT3zcoiJ8Pf7E8TXHZsEn//Cu2Lyjtzl3b
> NiqrHQaZa5nwKB4NYfZY1EZFcvNSHZjDyN67iaisoVU4swmoCIvak5hOjhuxNzdg
> Clcel8BXzI0S5XwuvrZBOfDBXa6bs6Ffkwj5EQmiXQH013zGAGg+RMAxRdAnRiqF
> eEthDiXlMcEWmt/FRjy/z5t4n4VXsAtgytT+zTksaoXQAJdJJAL0J43fYUfqGTyS
> Nq1kZ/Li7KExCa8B9jZ1VViIuZruYlqDtUQWdNBRm2vKPvngcL56PhmRkMUq5Btu
> ExCXvfMXaRmFSLQhej5q/K182KpycawR4Z1C/NVzm92iWOUMCIw=
> =VqpT
> -----END PGP SIGNATURE-----

[Michal Novotny]

Hello Igor,

On Wed, Aug 23, 2017 at 3:17 PM, Igor Gnatenko <
[e-mail]> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On Wed, 2017-08-23 at 13:47 +0200, Miroslav Suchý wrote:
> > Hi,
>
> Hi,
> > I am gathering informations about various use of CI with Copr. Do you
> > use Copr for building packages for nightlies? For
> > building packages before pull request is merged? Do you have your set
> > up described somewhere? What is the name of your
> > project?
>
> We do! Both. Our project name is: DNF 😛
>
> So our setup is pretty simple, our tool clones multiple repositories
> locally, builds SRPMs for them (it might be spec in upstream or
> upstream repo + downstream spec) and then creates temporary COPR
> repository and builds RPMs in there. Then we download those RPMs
> locally and do something with them (build container with them and run
> many tests).
> > Please let me know. Either here or via private reply.
> > It will help me to understand your use of Copr and to make Copr
> > better.
>
> I would like to see:
> only one and supported / good API

Yes, we will need to work on this...very soon.
> probably ability for some good integration with custom tools as we
>
> use (rpm-gitoverlay)

I will need to study this more.
> or ability to push custom specs / sources into git repo so we don't
>
> need to build SRPMs locally, we would just do git commit + git push and
> COPR would build everything

This should be possible with our SCM source types. It would be nice to
have just one SCM source type which is a goal but even now that should
be possible. Please, let me know if you see any blocker in this. That could
be very interesting input.
> "Temporary" repositories -- repositories which automatically getting
>
> removed after some time, we need such repos for pull request testing
> and after few weeks no one will look into / use them, so it can be
> safely removed

Well, we would need to have a way to determine how long to keep certain
repository.
> Faster builds (for 7-8 packages we are building, it takes ~20 minutes
>
> to build because we do it sequentially, but we can't do anything about
> that), probably some re-use of builders or faster spawning builds would
> help here, not sure

We can certainly work on faster build spawning (builder reusing is already
in place). Also automatic build ordering in batch builds could also be of
use
here? In any case, I agree this would help a lot.

* auto-rebuilds like koschei does (but I don't really want to see
> koschei as separate service, it should be built-in for better
> integration). Ideally this should be in Fedora/Koji as well, but it
> really depends what's COPR is aiming to do

I don't think it needs to be built-in, especially if it should serve
multiple clients.
Well, anyway this is also something we have in mind :).

Thanks for this reply.
clime
> > Thanks in advance.
> >
> > Miroslav Suchy
> > _______________________________________________
> > copr-devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail].o
> > rg
> -Igor Gnatenko
>
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCAAdFiEEhLFO09aHZVqO+CM6aVcUvRu8X0wFAlmdgHgACgkQaVcUvRu8
> X0w63w/9GS6eMuw2WIrbNSZeZbmxKg2kWIG5NwLHLQrHUEmEcAPRUa2X8+rao62w
> XQduQeA3tWR7Hktxr5GFGYxVe/5zQ5uFQQrfTyJKan8rryzOvs8xPbcNbH6VcWyJ
> qVAzbbQMIAmHFZItHmvxrCbJgRYmlTCQnDGSpYVrwfnXiGSq/vHu3AvnxBUdZuSz
> PFaDP1kfgfS/BpB9hD91AiuAKySCcfc74DpMVA3NXIbldum5UX8HtVID0EWglKf7
> XmGttLQ5ng1vxZqRVf/6fFt2frj5ZYlEUlRYiLHk66uNGJ0aP1DkAAPCVaNoVtqw
> A7ZXWjVxRJshYZnEZuKLtQvq1QYAvNjT3zcoiJ8Pf7E8TXHZsEn//Cu2Lyjtzl3b
> NiqrHQaZa5nwKB4NYfZY1EZFcvNSHZjDyN67iaisoVU4swmoCIvak5hOjhuxNzdg
> Clcel8BXzI0S5XwuvrZBOfDBXa6bs6Ffkwj5EQmiXQH013zGAGg+RMAxRdAnRiqF
> eEthDiXlMcEWmt/FRjy/z5t4n4VXsAtgytT+zTksaoXQAJdJJAL0J43fYUfqGTyS
> Nq1kZ/Li7KExCa8B9jZ1VViIuZruYlqDtUQWdNBRm2vKPvngcL56PhmRkMUq5Btu
> ExCXvfMXaRmFSLQhej5q/K182KpycawR4Z1C/NVzm92iWOUMCIw=
> =VqpT
> -----END PGP SIGNATURE-----
> _______________________________________________
> copr-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Pavel Raiskup]

Good ideas!  Thanks.

On Thursday, August 24, 2017 10:30:35 AM CEST Michal Novotny wrote:
> On Wed, Aug 23, 2017 at 3:17 PM, Igor Gnatenko
> [e-mail]> wrote:
> > only one and supported / good API
>
> Yes, we will need to work on this...very soon.

+1, let's create an API which works for everybody.  I mean, so far there were
rather "declarative" ways how to specify what should happen.  The problem is
that if you have declarative tool/language/whatever, one needs to pay attention
to standardizing something .... (and it is impossible to standardize upstream
habits across the universe).
> > probably ability for some good integration with custom tools as we
> >
> > use (rpm-gitoverlay)
>
> I will need to study this more.

-1, let's allow users to use whatever tool they want, don't concentrate on
particular tool.
> > Faster builds (for 7-8 packages we are building, it takes ~20 minutes
> >
> > to build because we do it sequentially, but we can't do anything about
> > that), probably some re-use of builders or faster spawning builds would
> > help here, not sure
>
> We can certainly work on faster build spawning (builder reusing is already in
> place). Also automatic build ordering in batch builds could also be of use
> here? In any case, I agree this would help a lot.

I don't see space for optimization here.  The maximum speed you can get for the
example of N packages, if the builds need to be serialized, is O(N).  You can
avoid cleaning mock caches, possibly, but I'm not sure this is what you want ->
and the "spawning time" as is is very small constant in the final time
complexity...

Pavel