Dist Git for Copr

[Adam Samalik]

Hello everyone,

I would like to make a proposal and start a discussion about Dist Git for Copr.

At the moment, Copr can build packages only from source rpms, which has been uploaded in
advance somewhere in the internet - so if you want to build something, you need your own
file server. As people often delete the sources from their server after the build process,
the builds are no longer easily reproducible. And there is also no history of the
sources.

We would like to use the Dist Git project[1] to store the sources. This would make all
builds reproducible, there would be no need of having a file server to build and we would
have a history of changes to each package.

At first, we would use the Dist Git server as a storage only. Users would not have direct
access. Instead, they would either:
  a) upload their source rpms somewhere and use Copr as before (no change in work-flow)
  b) submit the source rpms directly into Copr
In both cases, Copr would import ($ fedpkg import) the packages into the Dist Git and
build them from there.

The design of git repos would be:
  Repo name: 'user-name/copr-project/pkg-name'
  Branch name: 'epel-7', 'fedora-22'

At that moment, we would need a discussion on how to open the Dist Git server to users.
 - What would be the best web frontend?
 - Could we use Pagure? (used to be progit)
 - Which tool would they use?
 - How would they authenticate?

There has been some unfinished discussion on
https://fedorahosted.org/fedora-infrastructure/ticket/4564

The main purpouse of this is having a storage for sources, reproducible builds and history
of sources. Eventualy, we can extend it with bug tracking, easy forking and merging and
the other "Git-Hub-like" stuff.


Thanks!
Adam Samalik

[1] https://github.com/release-engineering/dist-git

[Vít Ondruch]

Dne 6.5.2015 v 12:31 Adam Samalik napsal(a):
>    Branch name: 'epel-7', 'fedora-22'
>
>  
Can we stick to Fedora's dist-git branch name conventions please?


Thanks


Vít

[Adam Samalik]

Sorry, the message I wanted to say was "git repos would represent packages and
branches would represent platforms". So yes, I agree, the branch names should be the
same as in Fedora.

Adam

----- Original Message -----
From: "Vít Ondruch" <vondruch(a)redhat.com&gt;
To: infrastructure(a)lists.fedoraproject.org
Sent: Wednesday, May 6, 2015 1:25:04 PM
Subject: Re: Dist Git for Copr

Dne 6.5.2015 v 12:31 Adam Samalik napsal(a):
>    Branch name: 'epel-7', 'fedora-22'
>
>  
Can we stick to Fedora's dist-git branch name conventions please?


Thanks


Vít
_______________________________________________
infrastructure mailing list
infrastructure(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

[Kevin Fenzi]

On Wed, 6 May 2015 06:31:03 -0400 (EDT)
Adam Samalik <asamalik(a)redhat.com&gt; wrote:

>  Hello everyone,
>  
>  I would like to make a proposal and start a discussion about Dist Git
>  for Copr. 
Sounds good. We had some discussion in tickets, but here is likely a
better place anyhow. ;) 

>  At the moment, Copr can build packages only from source rpms, which
>  has been uploaded in advance somewhere in the internet - so if you
>  want to build something, you need your own file server. As people
>  often delete the sources from their server after the build process,
>  the builds are no longer easily reproducible. And there is also no
>  history of the sources. 
Yeah.

...snip...

>  There has been some unfinished discussion on
>  https://fedorahosted.org/fedora-infrastructure/ticket/4564
>  
>  The main purpouse of this is having a storage for sources,
>  reproducible builds and history of sources. Eventualy, we can extend
>  it with bug tracking, easy forking and merging and the other
>  "Git-Hub-like" stuff. 
How about a short term and a longer term plan?

Short term: have copr download and store the src.rpm from build urls. 
This would at least make things reproducable and at least someone could
download the src.rpm and send a patch. Along with this a easy way to
mail the owner of a copr would be good. 

Longer term: use pagure. Have people setup their project there and work
with pagure folks to integrate copr. We could have an easy 'build this
in copr' type thing and on the copr side a 'visit this project on
pagure'. 

I think trying to do a standalone dist-get will just cause problems
when/if we want to move to something like pagure and has a lot of
issues without building up a bunch of infrastructure around it. 

Just my thoughts on it... 

kevin

[Miroslav Suchý]

Dne 6.5.2015 v 21:08 Kevin Fenzi napsal(a):
>  How about a short term and a longer term plan?
>  
>  Short term: have copr download and store the src.rpm from build urls. 
>  This would at least make things reproducable and at least someone could
>  download the src.rpm and send a patch. Along with this a easy way to
>  mail the owner of a copr would be good. 
Nope, short term is that user select SRPM and do "file upload" from form -- or
provide url.
And the result will be stored in dist-git.
So they do not need to host src.rpm on some public site.

>  Longer term: use pagure. Have people setup their project there and
> work
>  with pagure folks to integrate copr. We could have an easy 'build this
>  in copr' type thing and on the copr side a 'visit this project on
>  pagure'.  
I really dislike to store project in pagure. Unless it have git-annex or similar and we
store there just spec and
SOURCEX as in dist-git.
We could not force users to move their github project to pagure.

>  I think trying to do a standalone dist-get will just cause problems
>  when/if we want to move to something like pagure and has a lot of
>  issues without building up a bunch of infrastructure around it.  
Why it should cause trouble.
The goal is to have literally copy of fedora-dist. Just have it on different HW/Storage.
Then every change would be just carbon copy.


-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

[Adam Samalik]

Do we want to store the project, or packages?

To explain what I mean by "project" and "packages", let's have a
look at the Copr project.

The "project" is hosted here: https://git.fedorahosted.org/cgit/copr.git
This is the home of the project. It stores the source code and could also have some issue
tracker etc.
If we want to store "project", Pagure seems like a nice solution.

The "packages" (for fedora) are here: http://pkgs.fedoraproject.org/cgit/copr
This hosts the packages and is more focused on the spec file and the packaging side. The
project needs to live somewhere else.
If we want to store "packages", Dist Git might be great.

What we want to have? Or do we want to have some hybrid of those two above?

Adam



----- Original Message -----
From: "Miroslav Suchý" <msuchy(a)redhat.com&gt;
To: infrastructure(a)lists.fedoraproject.org
Sent: Friday, 15 May, 2015 2:41:07 PM
Subject: Re: Dist Git for Copr

Dne 6.5.2015 v 21:08 Kevin Fenzi napsal(a):
>  How about a short term and a longer term plan?
>  
>  Short term: have copr download and store the src.rpm from build urls. 
>  This would at least make things reproducable and at least someone could
>  download the src.rpm and send a patch. Along with this a easy way to
>  mail the owner of a copr would be good. 
Nope, short term is that user select SRPM and do "file upload" from form -- or
provide url.
And the result will be stored in dist-git.
So they do not need to host src.rpm on some public site.

>  Longer term: use pagure. Have people setup their project there and
> work
>  with pagure folks to integrate copr. We could have an easy 'build this
>  in copr' type thing and on the copr side a 'visit this project on
>  pagure'.  
I really dislike to store project in pagure. Unless it have git-annex or similar and we
store there just spec and
SOURCEX as in dist-git.
We could not force users to move their github project to pagure.

>  I think trying to do a standalone dist-get will just cause problems
>  when/if we want to move to something like pagure and has a lot of
>  issues without building up a bunch of infrastructure around it.  
Why it should cause trouble.
The goal is to have literally copy of fedora-dist. Just have it on different HW/Storage.
Then every change would be just carbon copy.


-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
_______________________________________________
infrastructure mailing list
infrastructure(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

[Mathieu Bridon]

On Fri, 2015-05-15 at 08:56 -0400, Adam Samalik wrote:
>  Do we want to store the project, or packages?
>  
>  To explain what I mean by "project" and "packages", let's have a
> look at the Copr project.
>  
>  The "project" is hosted here: https://git.fedorahosted.org/cgit/copr.git
>  This is the home of the project. It stores the source code and could also have some issue
> tracker etc.
>  If we want to store "project", Pagure seems like a nice solution.
>  
>  The "packages" (for fedora) are here: http://pkgs.fedoraproject.org/cgit/copr
>  This hosts the packages and is more focused on the spec file and the packaging side. The
> project needs to live somewhere else.
>  If we want to store "packages", Dist Git might be great. 
Pagure could also be great for the second.

It offers git hosting, the way dist-git needs it (to store specs and
patches), and it (could if it doesn't already) offers some space storage
to upload tarballs.

Then, maintainers of packages in coprs would gain a place for users to
report their issues with the copr packages, which doesn't sound like a
terrible thing.


-- 
Mathieu

[Adam Samalik]

But the second variant - to store "packages" - would mean that I have a git repo
for each package. For example Developer Toolset would need about 70 git projets because
it's got about 70 packages:
https://copr.fedoraproject.org/coprs/rhscl/devtoolset-3-el7/monitor/

That's fine for dist git, but not as a project. 

Ideal could be some kind of hybrid where we can have one project. This project could host
the upstream sources (pagure) or just have link to github, etc. Whatever we decide here.
We can even have both options to be more friendly :)
And this project would also include the packages. I can think about two variants here:
 a) include the packages in dist git or something like that
 b) have some easy description for Copr how to build it.

The example description for b) could look like this:
 1) have this project: https://git.fedorahosted.org/cgit/copr.git
 2) build from these directories: backend, selinux, cli, frontend, keygen
 3) get the sources by: tito build --srpm


Adam



----- Original Message -----
From: "Mathieu Bridon" <bochecha(a)fedoraproject.org&gt;
To: "Fedora Infrastructure" <infrastructure(a)lists.fedoraproject.org&gt;
Sent: Friday, 15 May, 2015 2:58:52 PM
Subject: Re: Dist Git for Copr

On Fri, 2015-05-15 at 08:56 -0400, Adam Samalik wrote:
>  Do we want to store the project, or packages?
>  
>  To explain what I mean by "project" and "packages", let's have a
> look at the Copr project.
>  
>  The "project" is hosted here: https://git.fedorahosted.org/cgit/copr.git
>  This is the home of the project. It stores the source code and could also have some issue
> tracker etc.
>  If we want to store "project", Pagure seems like a nice solution.
>  
>  The "packages" (for fedora) are here: http://pkgs.fedoraproject.org/cgit/copr
>  This hosts the packages and is more focused on the spec file and the packaging side. The
> project needs to live somewhere else.
>  If we want to store "packages", Dist Git might be great. 
Pagure could also be great for the second.

It offers git hosting, the way dist-git needs it (to store specs and
patches), and it (could if it doesn't already) offers some space storage
to upload tarballs.

Then, maintainers of packages in coprs would gain a place for users to
report their issues with the copr packages, which doesn't sound like a
terrible thing.


-- 
Mathieu

_______________________________________________
infrastructure mailing list
infrastructure(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

[Richard Marko]

On 05/15/2015 02:41 PM, Miroslav Suchý wrote:
>  Dne 6.5.2015 v 21:08 Kevin Fenzi napsal(a):
> > How about a short term and a longer term plan?
> >
> > Short term: have copr download and store the src.rpm from build urls. 
> > This would at least make things reproducable and at least someone could
> > download the src.rpm and send a patch. Along with this a easy way to
> > mail the owner of a copr would be good.
>  Nope, short term is that user select SRPM and do "file upload" from form -- or
> provide url.
>  And the result will be stored in dist-git.
>  So they do not need to host src.rpm on some public site.
>
> > Longer term: use pagure. Have people setup their project there and work
> > with pagure folks to integrate copr. We could have an easy 'build this
> > in copr' type thing and on the copr side a 'visit this project on
> > pagure'. 
>  I really dislike to store project in pagure. Unless it have git-annex or similar and we
> store there just spec and
>  SOURCEX as in dist-git. 
BTW I'm working on integrating git-annex to dist-git, see
https://github.com/release-engineering/dist-git/issues/1

-- 
Richard
irc: impure_hate