On Sat, 20 Nov 2021 at 15:05, Matthew Miller <mattdm(a)fedoraproject.org> wrote:
I mentioned on devel list that Sourcegraph is going to be indexing
source.fedoraproject.org.
So I guess, first — heads-up! if you see their traffic, it's not malicious.
(I asked them to provide us with the user agent they use, and I'll pass that
on.)
But then, I'd actually like to go a step further, and have them index _the
actual source for every build_. They're open to doing that, but what they
need is a git repo.
So...... how hard / ridiculous / bad would it be to have a step in the build
process in koji which, between the %prep and %build phases, push the
unpacked-and-prepped source tree to Gitlab, under, say,
https://gitlab.com/fedora/exploded-build-sources/package-name/?
0. When you mean %prep and %build you mean where koji builds a src.rpm
and then the builders get it versus the prep/build stage of mock?
1. I don't think the builders don't have access to the internet to do
that. Koji would have access but it is a limited resource.
2. I don't think you want any and every build to be broken because we
could not get to gitlab for a push due to timeouts, problems with
snips in the code etc.
You would want to plumb this into the build process as a completely
new tool. It would need a dedicated box which does this and it would
need to be able to
a) not stop builds and composes
b) be able to queue these actions so a mass rebuild doesn't take weeks
c) be able to resend/redo when we hit gitlab max actions per
second/hour. [Even paid accounts have quotas to keep overall service
working]
d) deal with MBS issues.
Otherwise you are going to run into various limitations that could
break the koji+mbs+pdc+bodhi+composer+etc which are all built with
certain tolerances and break when something slows things down.
I'm imaginging this would work something like this:
1. If remote repo doesn't exist, create it with the gitlab api
2. Do a shallow, no-checkout clone of that remote repo, using
--git-dir and --work-tree so that the .git directory isn't created inside
the working directory
3. copy the unpacked source tree from %prep into the work-tree dir
(are we building on btrfs? cp -l if not, to save io?)
4. git add --all
5. git commit -m "Build ID: ${buildID}
https://koji.fedoraproject.org/koji/buildinfo?buildID=${buildID}"
6. git push
With some details to be worked out. :) Like, repo tags as branches, maybe?
And, do this on all arches, or just one of them? Or maybe run up through
%prep as part of the src build rather than any of the binary builds? Make
the commit as the Fedora username of the person who did the build?
But those kinds of implementation details aside -- does this seem like
something we might be able to do?
Or, any ideas for an alternate approach? I mean, obviously, source-git would
be one such alternative, but getting that for _everything_ would require a
big rework of how everyone works. (I think we should get to that eventually,
but that's a long way off.)
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
_______________________________________________
infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
--
Stephen J Smoogen.
Let us be kind to one another, for most of us are fighting a hard
battle. -- Ian MacClaren