3:45 a.m.
Good morning, I'd like to announce the creation of Fedora Source-git SIG:
https://fedoraproject.org/wiki/SIGs/Source-git
Our main goal in the SIG right now is to establish a development
workflow for Fedora Linux packages using repositories with sources and
upstream history (this is what we call source-git), instead of just
distribution files with links to tarballs (dist-git).
Please head to the SIG wiki page to learn more about our proposed MVP.
We are looking for maintainers of Fedora Linux packages who'd be
interested in being early adopters and give us feedback during the
development process. You don't need to do any coding unless you want
to :)
With this email, I'd like to invite you to our first SIG meeting
hosted on freenode IRC next week. The details will be specified by
this Friday, please fill in "when is good" so we can pick a suitable
time:
https://whenisgood.net/bsyhsqq
Regards,
Tomas
7:39 a.m.
On 14.04.2021 10:45, Tomas Tomecek wrote:
Our main goal in the SIG right now is to establish a development
workflow for Fedora Linux packages using repositories with sources and
upstream history (this is what we call source-git), instead of just
distribution files with links to tarballs (dist-git).
Debian style with full local copies of repositories?
I don't think this is a good idea, because I don't want to clone
upstream repositories and store my SPECs in them.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
8:26 a.m.
On Wed, Apr 14, 2021 at 10:45:23AM +0200, Tomas Tomecek wrote:
Please head to the SIG wiki page to learn more about our proposed
MVP.
We are looking for maintainers of Fedora Linux packages who'd be
interested in being early adopters and give us feedback during the
development process. You don't need to do any coding unless you want
to :)
Yeah, I'm interested. I have a few packages I maintain that I think would
benefit from this workflow.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
8:55 a.m.
On Wed, Apr 14, 2021 at 10:45:23AM +0200, Tomas Tomecek wrote:
Good morning, I'd like to announce the creation of Fedora
Source-git SIG:
https://fedoraproject.org/wiki/SIGs/Source-git
Our main goal in the SIG right now is to establish a development
workflow for Fedora Linux packages using repositories with sources and
upstream history (this is what we call source-git), instead of just
distribution files with links to tarballs (dist-git).
Please head to the SIG wiki page to learn more about our proposed MVP.
We are looking for maintainers of Fedora Linux packages who'd be
interested in being early adopters and give us feedback during the
development process. You don't need to do any coding unless you want
to :)
We might be interested in trialling it with some of the virt packages.
I'm wondering about the scope of this statement from the wiki page
above:
"Whatever we produce here, it MUST NOT violate Fedora Packaging
Guidelines (we should strive to change them if needed)."
I can certainly understand the intent behind this when dealing with
legally restricted content. eg don't allow impls of patented algorithms
that are blocked from dist-git.
In terms of scope I can reasonably audit the source for current git
master or a specific git tag to ensure legal compliance.
I can't reasonably audit the entire source-git history of the project
back to day 1 though, to make sure the git repo has never had legally
restricted content at any point in the past 20 years of its life.
IOW, I'd hope that in terms of FPG compliance, we only need to consider
the specific tag/branch that's being used to populate dist-git and can
ignore history.
This could still potentially mean that source-git is a complication for
the packages where we have to re-create the tarballs after removing
patented crypto. Will legal allow it to remain in source git, but
require it to be purged when src-git syncs to dist-git or something
like that ?
Overall this does seem to imply though that if Fedora hosts src-git
repos with upstream history itself, then it is potentially opening
a new liability that it hasn't had before. If we're going to host
src-git on a Fedora namespace on gitlab.com though, then its someone
else's problem to worry about, and Fedora only needs to worry about
what's synced to dist-git for the actual RPMs builds.
Aside from legal, I also wonder about things like binary blobs or
bundled libraries. These are relatively common to see in upstream
git repos, even if they don't make it into the release tarballs that
Fedora traditionally consumes.
Hopefully the requirement to comply with Fedora Pakaging Guidelines
will only apply to files in src-git that actually get used for
Fedora builds, and not stuff that exists but is skipped/ignored ?
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
9:27 a.m.
On Wed, Apr 14, 2021 at 2:52 PM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
On 14.04.2021 10:45, Tomas Tomecek wrote:
> Our main goal in the SIG right now is to establish a development
> workflow for Fedora Linux packages using repositories with sources and
> upstream history (this is what we call source-git), instead of just
> distribution files with links to tarballs (dist-git).
Debian style with full local copies of repositories?
I don't think this is a good idea, because I don't want to clone
upstream repositories and store my SPECs in them.
Could you, please, be more constructive and say what the actual
problems are for you with such repositories?
I understand that upstream repositories can have a long history - we
could optimize and have shallow copies or only fetch recent upstream
history if needed. Also, one would ideally only clone once and kept
fetching new changes.
Thank you,
Tomas
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
9:32 a.m.
comments inline
On Wed, Apr 14, 2021 at 4:09 PM Daniel P. Berrangé <berrange(a)redhat.com> wrote:
On Wed, Apr 14, 2021 at 10:45:23AM +0200, Tomas Tomecek wrote:
> Good morning, I'd like to announce the creation of Fedora Source-git SIG:
>
> https://fedoraproject.org/wiki/SIGs/Source-git
>
> Our main goal in the SIG right now is to establish a development
> workflow for Fedora Linux packages using repositories with sources and
> upstream history (this is what we call source-git), instead of just
> distribution files with links to tarballs (dist-git).
>
> Please head to the SIG wiki page to learn more about our proposed MVP.
> We are looking for maintainers of Fedora Linux packages who'd be
> interested in being early adopters and give us feedback during the
> development process. You don't need to do any coding unless you want
> to :)
We might be interested in trialling it with some of the virt packages.
I'm wondering about the scope of this statement from the wiki page
above:
"Whatever we produce here, it MUST NOT violate Fedora Packaging
Guidelines (we should strive to change them if needed)."
I can certainly understand the intent behind this when dealing with
legally restricted content. eg don't allow impls of patented algorithms
that are blocked from dist-git.
In terms of scope I can reasonably audit the source for current git
master or a specific git tag to ensure legal compliance.
I can't reasonably audit the entire source-git history of the project
back to day 1 though, to make sure the git repo has never had legally
restricted content at any point in the past 20 years of its life.
IOW, I'd hope that in terms of FPG compliance, we only need to consider
the specific tag/branch that's being used to populate dist-git and can
ignore history.
This could still potentially mean that source-git is a complication for
the packages where we have to re-create the tarballs after removing
patented crypto. Will legal allow it to remain in source git, but
require it to be purged when src-git syncs to dist-git or something
like that ?
Overall this does seem to imply though that if Fedora hosts src-git
repos with upstream history itself, then it is potentially opening
a new liability that it hasn't had before. If we're going to host
src-git on a Fedora namespace on gitlab.com though, then its someone
else's problem to worry about, and Fedora only needs to worry about
what's synced to dist-git for the actual RPMs builds.
Thank you, Daniel, very good points! I'll add this to the agenda of
the first meeting to bring this to Fedora Legal. I also hope to make
the legal aspect of this someone else's problem - on the other hand,
if this workflow ever gets approved by FESCO and become official
(aside from the dist-git workflow) the repos would need to be hosted
on Fedora Infra.
Aside from legal, I also wonder about things like binary blobs or
bundled libraries. These are relatively common to see in upstream
git repos, even if they don't make it into the release tarballs that
Fedora traditionally consumes.
Hopefully the requirement to comply with Fedora Pakaging Guidelines
will only apply to files in src-git that actually get used for
Fedora builds, and not stuff that exists but is skipped/ignored ?
Yes, I'd say to files which are being put to dist-git and used for
official builds. Short term, I don't think it's feasible to create
production koji builds out of these source-git repositories. That
would be the long term plan :)
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
9:53 a.m.
On 14.04.2021 16:27, Tomas Tomecek wrote:
Could you, please, be more constructive and say what the actual
problems are for you with such repositories?
1. Some upstream repositories (Qt, Chromium, Linux kernel) are very huge
(more than 100 GiB). I don't want to download them from upstream and
then upload to Fedora.
2. Keeping such huge repositories will take up a lot of disk space on
the maintainer's computers.
3. Rebase problem. Maintainers will need do a manual rebase on every
upstream release/commit. Rebasing to the next major version will be a
serious problem for the projects with a lot of of downstream patches.
4. Some project have a weird git workflow between minor releases. I know
at least one project that uses Git tags with cherry-picked and manually
backported commits. Such detached tags cannot be merged into master
branch without resolving merge conflicts.
5. Force pushes must be enabled, which is too dangerous IMO.
I understand that upstream repositories can have a long history - we
could optimize and have shallow copies or only fetch recent upstream
history if needed. Also, one would ideally only clone once and kept
fetching new changes.
Do you want to force switch all Fedora packages to a new workflow?
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
10:19 a.m.
On Wed, Apr 14, 2021 at 04:53:06PM +0200, Vitaly Zaitsev via devel wrote:
On 14.04.2021 16:27, Tomas Tomecek wrote:
> Could you, please, be more constructive and say what the actual
> problems are for you with such repositories?
1. Some upstream repositories (Qt, Chromium, Linux kernel) are very huge
(more than 100 GiB). I don't want to download them from upstream and then
upload to Fedora.
2. Keeping such huge repositories will take up a lot of disk space on the
maintainer's computers.
Bear in mind that many maintainers already use a source-git workflow with
Fedora, and are also involved in the upstream project. IOW, they already
have these upstream repos present locally, and also be hosting it in some
remote git server outside normal Fedora dist-git.
source-git isn't proposed to be forced on all packages, and drive-by
contributors can also take a shallow clone instead of fetching all
history. Any possible "fedpkg" integration could potentially default
to a shallow clone.
3. Rebase problem. Maintainers will need do a manual rebase on every
upstream release/commit. Rebasing to the next major version will be a
serious problem for the projects with a lot of of downstream patches.
That's not my experiance. The cases where I know of maintainers are
using a source-git model with Fedora / RHEL already, are doing so
precisely because it makes ongoing maint and rebasing waaaaay easier
than with dist-git, especially when there are alot of downstream
patches (100's or even 1000's).
4. Some project have a weird git workflow between minor releases. I
know at
least one project that uses Git tags with cherry-picked and manually
backported commits. Such detached tags cannot be merged into master branch
without resolving merge conflicts.
I woudn't expect Fedora to track the git-master in most cases. You
generally still want Fedora to be base off releases, so you'd want
to track starting fron a release tag or branch.
5. Force pushes must be enabled, which is too dangerous IMO.
There are several ways you can do source git and they don't all
imply force pushes, so I think this is probably inventing a
problem where none yet exists.
One example approach to source-git I've used...
Rather than having source-git branch names matching dist-git,
use a different naming convention that is based off the upstream
version primarily.
eg if upstream has v1.0 and v1.2 tags, I might have a 'v1.0-f33'
branch, and if I rebase Fedora to v1.2, then I'd just switch to
using a v1.2-f33 branch instead. The v1.0-f33 history remains
intact forever, no force push required to rebase to new version.
If upstream has stable branches v1.0-maint and v1.2-maint, then
v1.0-f33 branch might be initialized with v1.0-maint content.
If upstream adds more commits to v1.0-maint branch, then you
wouldn't force push v1.0-f33, you'd just do a git merge to pull
them in.
> I understand that upstream repositories can have a long history
- we
> could optimize and have shallow copies or only fetch recent upstream
> history if needed. Also, one would ideally only clone once and kept
> fetching new changes.
Do you want to force switch all Fedora packages to a new workflow?
There's no mention of forcing everyone to switch to source-git. Some
upstreams don't even use git.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
1:46 p.m.
On 14.04.21 10:45, Tomas Tomecek wrote:
Good morning, I'd like to announce the creation of Fedora
Source-git SIG:
https://fedoraproject.org/wiki/SIGs/Source-git
Our main goal in the SIG right now is to establish a development
workflow for Fedora Linux packages using repositories with sources and
upstream history (this is what we call source-git), instead of just
distribution files with links to tarballs (dist-git).
Just wondering: will there be some coordination with the Fedora kernel
developers that are relying on a git based workflow since a few
weeks now (for rawhide even longer)?
To those who don't know: all the stuff in dist-git kernel is
generated from https://gitlab.com/cki-project/kernel-ark these days
afaik, so it might be wise to make sure the solution you work out works
for them as well. Especially as I find find some parts of how they do it
a bit questionable. The main tarball they use as Source0 for example
doesn't match the tarball downloadable from kernel.org(¹); all the
patches are stashed into one(²); patches for fedora specific stuff (like
the configs needed for building the kernel) are in the same branch as
the patches(³). I think that makes things quite confusing, especially
for outsiders. Sometimes I wonder if some of what the kernel people do
violates the Fedora Packaging Guidelines(⁴), but the kernel-ark people
ensured me it's fine.
CU, knurd
(¹)
https://src.fedoraproject.org/rpms/kernel/blob/f33/f/sources for example
states:
SHA512 (linux-5.11.14.tar.xz) =
ccf0eaf6df0dacd2984c361621f67a3d16cf7a7174155ebdf646f1acfec43e19ff942e6c17e5bc3b5dc7a300c32bdc6ee37877162c099f5bd9924244f9445467
But:
$ wget --quiet
https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.14.tar.xz
$ sha512sum linux-5.11.14.tar.xz
8dfc7ff184e5cb33fff74686071f1605f3a834669e201d272f3047aa00657339ec1a3cfd605d8761b8a0f335b8488c02c701e72ed30031856e9c154aa1ff2d88
linux-5.11.14.tar.xz
(²)
https://src.fedoraproject.org/rpms/kernel/blob/f33/f/patch-5.11-redhat.patch
FWIW, links to the individual patches can be found here:
https://src.fedoraproject.org/rpms/kernel/blob/f33/f/Patchlist.changelog
(³) for example
https://gitlab.com/cki-project/kernel-ark/-/commits/fedora-5.11
(⁴) like this part "sources used to build a package should be the
vanilla sources available from upstream. To help reviewers and QA
scripts verify this, the packager needs to indicate where a reviewer can
find the source that was used to make the rpm". The quote is from here:
https://docs.fedoraproject.org/en-US/packaging-guidelines/SourceURL/
3:02 p.m.
On Wed, Apr 14, 2021 at 04:19:29PM +0100, Daniel P. Berrangé wrote:
One example approach to source-git I've used...
Rather than having source-git branch names matching dist-git,
use a different naming convention that is based off the upstream
version primarily.
eg if upstream has v1.0 and v1.2 tags, I might have a 'v1.0-f33'
branch, and if I rebase Fedora to v1.2, then I'd just switch to
using a v1.2-f33 branch instead. The v1.0-f33 history remains
intact forever, no force push required to rebase to new version.
As a concrete example, this is how the Fedora OCaml repo works (with a
different naming convention):
https://pagure.io/fedora-ocaml/branches?branchname=master
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
4:52 a.m.
On 14.04.2021 17:19, Daniel P. Berrangé wrote:
That's not my experiance. The cases where I know of maintainers
are
using a source-git model with Fedora / RHEL already, are doing so
precisely because it makes ongoing maint and rebasing waaaaay easier
than with dist-git, especially when there are alot of downstream
patches (100's or even 1000's).
In some cases, yes.
I woudn't expect Fedora to track the git-master in most cases.
You
generally still want Fedora to be base off releases, so you'd want
to track starting fron a release tag or branch.
One more question. If the upstream ignores tags, can I create tags
myself in source-git?
There are several ways you can do source git and they don't all
imply force pushes, so I think this is probably inventing a
problem where none yet exists.
You need force pushes support in order to perform git rebases.
eg if upstream has v1.0 and v1.2 tags, I might have a
'v1.0-f33'
branch, and if I rebase Fedora to v1.2, then I'd just switch to
using a v1.2-f33 branch instead. The v1.0-f33 history remains
intact forever, no force push required to rebase to new version.
Looks like a dirty hack. Another pain for the maintainers.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
5:14 a.m.
On Thu, Apr 15, 2021 at 11:52:40AM +0200, Vitaly Zaitsev via devel wrote:
On 14.04.2021 17:19, Daniel P. Berrangé wrote:
> That's not my experiance. The cases where I know of maintainers are
> using a source-git model with Fedora / RHEL already, are doing so
> precisely because it makes ongoing maint and rebasing waaaaay easier
> than with dist-git, especially when there are alot of downstream
> patches (100's or even 1000's).
In some cases, yes.
> I woudn't expect Fedora to track the git-master in most cases. You
> generally still want Fedora to be base off releases, so you'd want
> to track starting fron a release tag or branch.
One more question. If the upstream ignores tags, can I create tags myself in
source-git?
Of course. Downstream source-git can have any additional git tags
it wants. It would be wise to pick a naming scheme for the downstream
tags that is unlikely to clash with potential future upstream tags.
> There are several ways you can do source git and they don't
all
> imply force pushes, so I think this is probably inventing a
> problem where none yet exists.
You need force pushes support in order to perform git rebases.
Err, I illustrated an example showing this is not the case, that
you've quoted right here:
> eg if upstream has v1.0 and v1.2 tags, I might have a
'v1.0-f33'
> branch, and if I rebase Fedora to v1.2, then I'd just switch to
> using a v1.2-f33 branch instead. The v1.0-f33 history remains
> intact forever, no force push required to rebase to new version.
Looks like a dirty hack. Another pain for the maintainers.
It isn't a dirty hack. This is a normal model for maintaining
maint branches based off releases.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
9:30 a.m.
Daniel, thank you for sharing your experience!
More comments inline.
On Wed, Apr 14, 2021 at 5:26 PM Daniel P. Berrangé <berrange(a)redhat.com> wrote:
On Wed, Apr 14, 2021 at 04:53:06PM +0200, Vitaly Zaitsev via devel wrote:
> On 14.04.2021 16:27, Tomas Tomecek wrote:
> > Could you, please, be more constructive and say what the actual
> > problems are for you with such repositories?
>
> 1. Some upstream repositories (Qt, Chromium, Linux kernel) are very huge
> (more than 100 GiB). I don't want to download them from upstream and then
> upload to Fedora.
Fedora kernel is already being maintained like this:
https://gitlab.com/cki-project/kernel-ark
> 2. Keeping such huge repositories will take up a lot of disk
space on the
> maintainer's computers.
Bear in mind that many maintainers already use a source-git workflow with
Fedora, and are also involved in the upstream project. IOW, they already
have these upstream repos present locally, and also be hosting it in some
remote git server outside normal Fedora dist-git.
source-git isn't proposed to be forced on all packages, and drive-by
contributors can also take a shallow clone instead of fetching all
history. Any possible "fedpkg" integration could potentially default
to a shallow clone.
Agreed, dist-git isn't going anywhere and with the source-git
workflow, we still want to preserve the rel-eng/proven-packager
workflow in dist-git. The source-git repos should make it way more
convenient for maintainers to track downstream patches, as you
describe.
> 3. Rebase problem. Maintainers will need do a manual rebase on
every
> upstream release/commit. Rebasing to the next major version will be a
> serious problem for the projects with a lot of of downstream patches.
That's not my experiance. The cases where I know of maintainers are
using a source-git model with Fedora / RHEL already, are doing so
precisely because it makes ongoing maint and rebasing waaaaay easier
than with dist-git, especially when there are alot of downstream
patches (100's or even 1000's).
+1, `git pull --rebase upstream $ref` is more convenient than trying
to fix downstream patches in whatever obscure way.
Shout out to rebase-helper which is trying to automate the patch files
rebase problem via a CLI utility:
https://github.com/rebase-helper/rebase-helper
> 4. Some project have a weird git workflow between minor
releases. I know at
> least one project that uses Git tags with cherry-picked and manually
> backported commits. Such detached tags cannot be merged into master branch
> without resolving merge conflicts.
I woudn't expect Fedora to track the git-master in most cases. You
generally still want Fedora to be base off releases, so you'd want
to track starting fron a release tag or branch.
Correct, the branches are meant to use upstream release git-tags as a base.
> 5. Force pushes must be enabled, which is too dangerous IMO.
There are several ways you can do source git and they don't all
imply force pushes, so I think this is probably inventing a
problem where none yet exists.
One example approach to source-git I've used...
Rather than having source-git branch names matching dist-git,
use a different naming convention that is based off the upstream
version primarily.
eg if upstream has v1.0 and v1.2 tags, I might have a 'v1.0-f33'
branch, and if I rebase Fedora to v1.2, then I'd just switch to
using a v1.2-f33 branch instead. The v1.0-f33 history remains
intact forever, no force push required to rebase to new version.
If upstream has stable branches v1.0-maint and v1.2-maint, then
v1.0-f33 branch might be initialized with v1.0-maint content.
If upstream adds more commits to v1.0-maint branch, then you
wouldn't force push v1.0-f33, you'd just do a git merge to pull
them in.
That's interesting: merge commits make it hard to work with the
git-history - how do you solve that? Daniel, does your tooling account
for a merge commit and is it able to still pick up commits and turn
them into patch files correctly?
Our current solution to rebases is to create new branches for every
upstream release.
> > I understand that upstream repositories can have a long
history - we
> > could optimize and have shallow copies or only fetch recent upstream
> > history if needed. Also, one would ideally only clone once and kept
> > fetching new changes.
>
> Do you want to force switch all Fedora packages to a new workflow?
There's no mention of forcing everyone to switch to source-git. Some
upstreams don't even use git.
100%, the workflow is optional, dist-git is staying and both workflows
should be compatible (to some extent, synchronizing downstream patches
b/w the two would be tricky to figure out)
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
9:43 a.m.
On Wed, Apr 14, 2021 at 8:52 PM Thorsten Leemhuis <fedora(a)leemhuis.info> wrote:
On 14.04.21 10:45, Tomas Tomecek wrote:
> Good morning, I'd like to announce the creation of Fedora Source-git SIG:
> https://fedoraproject.org/wiki/SIGs/Source-git
>
> Our main goal in the SIG right now is to establish a development
> workflow for Fedora Linux packages using repositories with sources and
> upstream history (this is what we call source-git), instead of just
> distribution files with links to tarballs (dist-git).
Just wondering: will there be some coordination with the Fedora kernel
developers that are relying on a git based workflow since a few
weeks now (for rawhide even longer)?
To those who don't know: all the stuff in dist-git kernel is
generated from https://gitlab.com/cki-project/kernel-ark these days
afaik, so it might be wise to make sure the solution you work out works
for them as well. Especially as I find find some parts of how they do it
a bit questionable. The main tarball they use as Source0 for example
doesn't match the tarball downloadable from kernel.org(¹); all the
patches are stashed into one(²); patches for fedora specific stuff (like
the configs needed for building the kernel) are in the same branch as
the patches(³). I think that makes things quite confusing, especially
for outsiders. Sometimes I wonder if some of what the kernel people do
violates the Fedora Packaging Guidelines(⁴), but the kernel-ark people
ensured me it's fine.
CU, knurd
(¹)
https://src.fedoraproject.org/rpms/kernel/blob/f33/f/sources for example
states:
> SHA512 (linux-5.11.14.tar.xz) =
ccf0eaf6df0dacd2984c361621f67a3d16cf7a7174155ebdf646f1acfec43e19ff942e6c17e5bc3b5dc7a300c32bdc6ee37877162c099f5bd9924244f9445467
But:
$ wget --quiet
https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.14.tar.xz
$ sha512sum linux-5.11.14.tar.xz
8dfc7ff184e5cb33fff74686071f1605f3a834669e201d272f3047aa00657339ec1a3cfd605d8761b8a0f335b8488c02c701e72ed30031856e9c154aa1ff2d88
linux-5.11.14.tar.xz
(²)
https://src.fedoraproject.org/rpms/kernel/blob/f33/f/patch-5.11-redhat.patch
FWIW, links to the individual patches can be found here:
https://src.fedoraproject.org/rpms/kernel/blob/f33/f/Patchlist.changelog
(³) for example
https://gitlab.com/cki-project/kernel-ark/-/commits/fedora-5.11
(⁴) like this part "sources used to build a package should be the
vanilla sources available from upstream. To help reviewers and QA
scripts verify this, the packager needs to indicate where a reviewer can
find the source that was used to make the rpm". The quote is from here:
https://docs.fedoraproject.org/en-US/packaging-guidelines/SourceURL/
Thorsten, thank you for pointing those out. I spoke with Don Zickus,
one of the leads behind the kernel-ark project (or probably the lead
:) and he confirmed to join the SIG. My hope is to create a platform
that would unify all the different source-git implementations being
used right now into one, which would have clear documentation, tooling
and infra support. Dreaming big :)
Tomas
9:47 a.m.
On Thu, Apr 15, 2021 at 04:30:26PM +0200, Tomas Tomecek wrote:
Daniel, thank you for sharing your experience!
More comments inline.
On Wed, Apr 14, 2021 at 5:26 PM Daniel P. Berrangé <berrange(a)redhat.com> wrote>
> > 4. Some project have a weird git workflow between minor releases. I know at
> > least one project that uses Git tags with cherry-picked and manually
> > backported commits. Such detached tags cannot be merged into master branch
> > without resolving merge conflicts.
>
> I woudn't expect Fedora to track the git-master in most cases. You
> generally still want Fedora to be base off releases, so you'd want
> to track starting fron a release tag or branch.
Correct, the branches are meant to use upstream release git-tags as a base.
> > 5. Force pushes must be enabled, which is too dangerous IMO.
>
> There are several ways you can do source git and they don't all
> imply force pushes, so I think this is probably inventing a
> problem where none yet exists.
>
> One example approach to source-git I've used...
>
> Rather than having source-git branch names matching dist-git,
> use a different naming convention that is based off the upstream
> version primarily.
>
> eg if upstream has v1.0 and v1.2 tags, I might have a 'v1.0-f33'
> branch, and if I rebase Fedora to v1.2, then I'd just switch to
> using a v1.2-f33 branch instead. The v1.0-f33 history remains
> intact forever, no force push required to rebase to new version.
>
> If upstream has stable branches v1.0-maint and v1.2-maint, then
> v1.0-f33 branch might be initialized with v1.0-maint content.
> If upstream adds more commits to v1.0-maint branch, then you
> wouldn't force push v1.0-f33, you'd just do a git merge to pull
> them in.
That's interesting: merge commits make it hard to work with the
git-history - how do you solve that? Daniel, does your tooling account
for a merge commit and is it able to still pick up commits and turn
them into patch files correctly?
I've not actually hit situations where I've needed merge commits
in general, except by my own mistake. My tooling merely consists
of running "git format-patch vN.M..." where "vN.M" is the tag
associated with the release the RPM is based off.
Our current solution to rebases is to create new branches for every
upstream release.
Yes, that's the good way IMHO. I was only mentioning merge commits
in the context of tracking changes within a release stream. eg some
projects may have stable branches in their upstream git, but don't
do releases off them very frequently. So you'd conceivably want to
periodically do a merge against upstream stable branche.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
12:52 p.m.
On Wed, Apr 14, 2021 at 1:47 PM Thorsten Leemhuis <fedora(a)leemhuis.info> wrote:
On 14.04.21 10:45, Tomas Tomecek wrote:
> Good morning, I'd like to announce the creation of Fedora Source-git SIG:
> https://fedoraproject.org/wiki/SIGs/Source-git
>
> Our main goal in the SIG right now is to establish a development
> workflow for Fedora Linux packages using repositories with sources and
> upstream history (this is what we call source-git), instead of just
> distribution files with links to tarballs (dist-git).
Just wondering: will there be some coordination with the Fedora kernel
developers that are relying on a git based workflow since a few
weeks now (for rawhide even longer)?
To those who don't know: all the stuff in dist-git kernel is
generated from https://gitlab.com/cki-project/kernel-ark these days
afaik, so it might be wise to make sure the solution you work out works
for them as well. Especially as I find find some parts of how they do it
a bit questionable. The main tarball they use as Source0 for example
doesn't match the tarball downloadable from kernel.org(¹); all the
patches are stashed into one(²); patches for fedora specific stuff (like
the configs needed for building the kernel) are in the same branch as
the patches(³). I think that makes things quite confusing, especially
for outsiders. Sometimes I wonder if some of what the kernel people do
violates the Fedora Packaging Guidelines(⁴), but the kernel-ark people
ensured me it's fine.
CU, knurd
(¹)
https://src.fedoraproject.org/rpms/kernel/blob/f33/f/sources for example
states:
> SHA512 (linux-5.11.14.tar.xz) =
ccf0eaf6df0dacd2984c361621f67a3d16cf7a7174155ebdf646f1acfec43e19ff942e6c17e5bc3b5dc7a300c32bdc6ee37877162c099f5bd9924244f9445467
But:
$ wget --quiet
https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.14.tar.xz
$ sha512sum linux-5.11.14.tar.xz
8dfc7ff184e5cb33fff74686071f1605f3a834669e201d272f3047aa00657339ec1a3cfd605d8761b8a0f335b8488c02c701e72ed30031856e9c154aa1ff2d88
linux-5.11.14.tar.xz
likely different compression options used by default, the upstream
tarball is just a bit smaller. We create the tarballourselves, but
based on greg's tag. If you explode and compare them, they are
identical:
fedora64:[~/srcchk]>diff -ruNp stable-5.11.14/ ark-5.11.14/
fedora64:[~/srcchk]>ls -l *5.11.14.tar.xz
-rw-rw-r--. 1 jforbes jforbes 123313416 Apr 15 12:38 ark-5.11.14.tar.xz
-rw-r--r--. 1 jforbes jforbes 117651976 Apr 15 12:42 linux-5.11.14.tar.xz
For what it's worth, until they changed compression, we used to
recompress the upstream tarball and end up with a similar result. As
to your (*) they are tagged in the git tree, you can verify that tag
in our clone of the upstream tree, or directly in the upstream tree if
you wish, both audit sources are available.
As to the single patch vs the broken out, it is a matter of git merge
vs git rebase. If we did a rebase with every release, they could be
broken out cleanly, but the trade off is any update becomes a forced
update, and any pending MR is invalidated. We do rebase the patches
on the creation of stable branches, so fedora-5.11 was created by
rebasing the rawhide branch over Greg's linux-5.11.y branch. It is
merged forward from there, but none of the patches should be too
different to quickly figure out.
>
> (²)
> https://src.fedoraproject.org/rpms/kernel/blob/f33/f/patch-5.11-redhat.patch
> FWIW, links to the individual patches can be found here:
> https://src.fedoraproject.org/rpms/kernel/blob/f33/f/Patchlist.changelog
>
> (³) for example
> https://gitlab.com/cki-project/kernel-ark/-/commits/fedora-5.11
>
> (⁴) like this part "sources used to build a package should be the
> vanilla sources available from upstream. To help reviewers and QA
> scripts verify this, the packager needs to indicate where a reviewer can
> find the source that was used to make the rpm". The quote is from here:
> https://docs.fedoraproject.org/en-US/packaging-guidelines/SourceURL/
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
4:05 p.m.
On Wed, Apr 14, 2021 at 04:53:06PM +0200, Vitaly Zaitsev via devel wrote:
On 14.04.2021 16:27, Tomas Tomecek wrote:
> Could you, please, be more constructive and say what the actual
> problems are for you with such repositories?
1. Some upstream repositories (Qt, Chromium, Linux kernel) are very huge
(more than 100 GiB). I don't want to download them from upstream and then
upload to Fedora.
Which one exactly is more than 100 GiB? The Linux kernel certainly
isn't (more like 2.5 GiB, last I checked.)
--b.
4:13 p.m.
On Thu, Apr 15, 2021 at 5:05 PM J. Bruce Fields <bfields(a)redhat.com> wrote:
On Wed, Apr 14, 2021 at 04:53:06PM +0200, Vitaly Zaitsev via devel wrote:
> On 14.04.2021 16:27, Tomas Tomecek wrote:
> > Could you, please, be more constructive and say what the actual
> > problems are for you with such repositories?
>
> 1. Some upstream repositories (Qt, Chromium, Linux kernel) are very huge
> (more than 100 GiB). I don't want to download them from upstream and then
> upload to Fedora.
Which one exactly is more than 100 GiB? The Linux kernel certainly
isn't (more like 2.5 GiB, last I checked.)
Chromium is almost 100GB on disk. It's the only repository where I use
Btrfs subvolumes and snapshots to roll back and forth through changes
instead of using Git commands, because it takes *so* long and can time
out and fail.
--
真実はいつも一つ！/ Always, there's only one truth!
6:24 p.m.
On Wed, 2021-04-14 at 10:45 +0200, Tomas Tomecek wrote:
...
We are looking for maintainers of Fedora Linux packages who'd
be
interested in being early adopters and give us feedback during the
development process. You don't need to do any coding unless you want
to :)
I'd like to register interest in this topic on behalf of myself and
Facebook. I'm the maintainer for one (very recently added) package. The
source-git workflow has appeal, especially in context of other SIGs -
namely CentOS Hyperscale which I'm part of. Looking forward to the
meeting!
Matthew.
1105
days inactive
1107
days old
19 comments
10 participants
participants (10)
-
Daniel P. Berrangé -
J. Bruce Fields -
Justin Forbes -
Matthew Almond -
Matthew Miller -
Neal Gompa -
Richard W.M. Jones -
Thorsten Leemhuis -
Tomas Tomecek -
Vitaly Zaitsev