4:56 p.m.
Createrepo_c in F30 has finally grown zchunk support and I've packaged
up some zdicts that we can use for F30/rawhide, so I'd love to see us
start building zchunk metadata for F30.
To enable zchunk metadata generation, whichever systems are running
createrepo_c/mergerepo_c for Rawhide would need createrepo_c-0.12 and
fedora-repo-zdicts installed. The call to createrepo_c or mergerepo_c
(whichever is run last to generate the final metadata) would need to be
run with the new zchunk arguments:
--zck --zck-dict-dir=/usr/share/fedora-repo-zdicts/f30
Mergerepo doesn't require zchunk metadata in the source repositories to
be able to generate zchunk metadata for the merged repository.
I'm not sure who to ask to turn these flags on, so if there's an
individual I need to ping, please point me in the right direction.
A huge thank you to Daniel Mach for reviewing and merging the
createrepo_c zchunk pull request, Jaroslav Mracek for building
createrepo_c with zchunk, Robert-André Mauchin for reviewing fedora-
repo-zdicts and Neal Gompa for keeping things moving forward with the
PR reviews.
Thanks,
Jonathan
5:12 p.m.
On 12/13/18 2:56 PM, Jonathan Dieter wrote:
Createrepo_c in F30 has finally grown zchunk support and I've
packaged
up some zdicts that we can use for F30/rawhide, so I'd love to see us
start building zchunk metadata for F30.
To enable zchunk metadata generation, whichever systems are running
createrepo_c/mergerepo_c for Rawhide would need createrepo_c-0.12 and
fedora-repo-zdicts installed. The call to createrepo_c or mergerepo_c
(whichever is run last to generate the final metadata) would need to be
run with the new zchunk arguments:
--zck --zck-dict-dir=/usr/share/fedora-repo-zdicts/f30
Mergerepo doesn't require zchunk metadata in the source repositories to
be able to generate zchunk metadata for the merged repository.
I'm not sure who to ask to turn these flags on, so if there's an
individual I need to ping, please point me in the right direction.
pungi calls createrepo_c for us (in both rawhide/branched and updates)
so we need a pungi patch (probibly with a config option?) to enable
this. If it's added as a optional thing we would need to add that
setting to our pungi-fedora config and set it to on.
Can you file a pungi issue on that?
kevin
5:34 p.m.
On Thu, 2018-12-13 at 15:12 -0800, Kevin Fenzi wrote:
pungi calls createrepo_c for us (in both rawhide/branched and
updates)
so we need a pungi patch (probibly with a config option?) to enable
this. If it's added as a optional thing we would need to add that
setting to our pungi-fedora config and set it to on.
Can you file a pungi issue on that?
I've just checked the pungi issues, and it looks like Lubomír took care
of this at Flock last summer by adding an arbitrary createrepo_c
commands option: createrepo_extra_args
I've done a PR for the pungi-fedora config here, but it's untested and
I'm not sure if I did the variable substitution correctly.
https://pagure.io/pungi-fedora/pull-request/678
Jonathan
6:42 p.m.
On 12/13/18 3:34 PM, Jonathan Dieter wrote:
On Thu, 2018-12-13 at 15:12 -0800, Kevin Fenzi wrote:
> pungi calls createrepo_c for us (in both rawhide/branched and updates)
> so we need a pungi patch (probibly with a config option?) to enable
> this. If it's added as a optional thing we would need to add that
> setting to our pungi-fedora config and set it to on.
>
> Can you file a pungi issue on that?
I've just checked the pungi issues, and it looks like Lubomír took care
of this at Flock last summer by adding an arbitrary createrepo_c
commands option: createrepo_extra_args
I've done a PR for the pungi-fedora config here, but it's untested and
I'm not sure if I did the variable substitution correctly.
https://pagure.io/pungi-fedora/pull-request/678
Cool.
I see the new createrepo_c only has a rawhide build... any chance for a
f29 update? Or should we look at building a newer in our infra repo?
kevin
10:20 a.m.
On Thu, 2018-12-13 at 22:56 +0000, Jonathan Dieter wrote:
The call to createrepo_c or mergerepo_c
(whichever is run last to generate the final metadata) would need to
be
run with the new zchunk arguments:
--zck --zck-dict-dir=/usr/share/fedora-repo-zdicts/f30
Hey Jonathan!
Bodhi uses createrepo_c both through pungi (to create the bulk of the
repository) and through the createrepo_c Python bindings (to generate
the updateinfo.xml file). Is there a way to ask the Python bindings to
do this? The Fedora 29 updateinfo.xml file looks like it's only about 1
MB for x86_64 right now, so maybe it's not as crucial. (F28 is 2.1 MB.)
12:52 p.m.
On Thu, 2018-12-13 at 16:42 -0800, Kevin Fenzi wrote:
On 12/13/18 3:34 PM, Jonathan Dieter wrote:
> On Thu, 2018-12-13 at 15:12 -0800, Kevin Fenzi wrote:
> > pungi calls createrepo_c for us (in both rawhide/branched and updates)
> > so we need a pungi patch (probibly with a config option?) to enable
> > this. If it's added as a optional thing we would need to add that
> > setting to our pungi-fedora config and set it to on.
> >
> > Can you file a pungi issue on that?
>
> I've just checked the pungi issues, and it looks like Lubomír took care
> of this at Flock last summer by adding an arbitrary createrepo_c
> commands option: createrepo_extra_args
>
> I've done a PR for the pungi-fedora config here, but it's untested and
> I'm not sure if I did the variable substitution correctly.
>
> https://pagure.io/pungi-fedora/pull-request/678
Cool.
I see the new createrepo_c only has a rawhide build... any chance for a
f29 update? Or should we look at building a newer in our infra repo?
I suspect that the maintainers would like to see this feature tested
more before pushing it to F29, but I can ask them, if you'd like.
If you do a F29 rebuild for the infra repo, you'll want to make sure to
pass --with zchunk to rpmbuild, as it defaults to off for anything F29
and below.
Jonathan
1:02 p.m.
On Fri, 2018-12-14 at 11:20 -0500, Randy Barlow wrote:
On Thu, 2018-12-13 at 22:56 +0000, Jonathan Dieter wrote:
> The call to createrepo_c or mergerepo_c
> (whichever is run last to generate the final metadata) would need to
> be
> run with the new zchunk arguments:
>
> --zck --zck-dict-dir=/usr/share/fedora-repo-zdicts/f30
Hey Jonathan!
Bodhi uses createrepo_c both through pungi (to create the bulk of the
repository) and through the createrepo_c Python bindings (to generate
the updateinfo.xml file). Is there a way to ask the Python bindings to
do this? The Fedora 29 updateinfo.xml file looks like it's only about 1
Hey Randy, at the moment the --zck option *only* applies to
primary.xml, filelists.xml and other.xml. It should be pretty
straightforward to add it to the others, but I wanted to get those
three working first.
As for python bindings, they can read zchunk metadata just fine, but I
don't think I hooked up creating the metadata. Where exactly does it
generate updateinfo in bodhi? I'd like to see how the function is used
so I can implement it.
Jonathan
1:13 p.m.
On 12/14/18 10:52 AM, Jonathan Dieter wrote:
On Thu, 2018-12-13 at 16:42 -0800, Kevin Fenzi wrote:
> On 12/13/18 3:34 PM, Jonathan Dieter wrote:
>> On Thu, 2018-12-13 at 15:12 -0800, Kevin Fenzi wrote:
>>> pungi calls createrepo_c for us (in both rawhide/branched and updates)
>>> so we need a pungi patch (probibly with a config option?) to enable
>>> this. If it's added as a optional thing we would need to add that
>>> setting to our pungi-fedora config and set it to on.
>>>
>>> Can you file a pungi issue on that?
>>
>> I've just checked the pungi issues, and it looks like Lubomír took care
>> of this at Flock last summer by adding an arbitrary createrepo_c
>> commands option: createrepo_extra_args
>>
>> I've done a PR for the pungi-fedora config here, but it's untested and
>> I'm not sure if I did the variable substitution correctly.
>>
>> https://pagure.io/pungi-fedora/pull-request/678
>
> Cool.
>
> I see the new createrepo_c only has a rawhide build... any chance for a
> f29 update? Or should we look at building a newer in our infra repo?
I suspect that the maintainers would like to see this feature tested
more before pushing it to F29, but I can ask them, if you'd like.
No, I am not suggesting we implement it now in F29, I am saying that the
rawhide-composer and bodhi-backend01 machines that run pungi are Fedora
29 hosts. The rawhide-composer runs in a chroot, so that should just
work, but the bodhi-backend01 updates pungi I don't think does, or if it
does it's a f29 chroot, not rawhide. So we will need a build for it.
If you do a F29 rebuild for the infra repo, you'll want to make
sure to
pass --with zchunk to rpmbuild, as it defaults to off for anything F29
and below.
Well, we cannot do that with a koji build, but I guess we could just
change the source.
Anyhow, perhaps we just target rawhide for now...
kevin
1:24 p.m.
On Fri, 2018-12-14 at 11:13 -0800, Kevin Fenzi wrote:
On 12/14/18 10:52 AM, Jonathan Dieter wrote:
> On Thu, 2018-12-13 at 16:42 -0800, Kevin Fenzi wrote:
> > Cool.
> >
> > I see the new createrepo_c only has a rawhide build... any chance for a
> > f29 update? Or should we look at building a newer in our infra repo?
>
> I suspect that the maintainers would like to see this feature tested
> more before pushing it to F29, but I can ask them, if you'd like.
No, I am not suggesting we implement it now in F29, I am saying that the
rawhide-composer and bodhi-backend01 machines that run pungi are Fedora
29 hosts. The rawhide-composer runs in a chroot, so that should just
work, but the bodhi-backend01 updates pungi I don't think does, or if it
does it's a f29 chroot, not rawhide. So we will need a build for it.
Ok, makes sense.
> If you do a F29 rebuild for the infra repo, you'll want to
make sure to
> pass --with zchunk to rpmbuild, as it defaults to off for anything F29
> and below.
Well, we cannot do that with a koji build, but I guess we could just
change the source.
It's just a conditional in the spec that's currently set to off for <=
F29, so easy enough to change.
Anyhow, perhaps we just target rawhide for now...
That works for me. Thanks so much! If there's anything else I can do
to help with this, please let me know.
Jonathan
1:40 p.m.
On Fri, 2018-12-14 at 19:24 +0000, Jonathan Dieter wrote:
On Fri, 2018-12-14 at 11:13 -0800, Kevin Fenzi wrote:
> On 12/14/18 10:52 AM, Jonathan Dieter wrote:
> > I suspect that the maintainers would like to see this feature tested
> > more before pushing it to F29, but I can ask them, if you'd like.
>
> No, I am not suggesting we implement it now in F29, I am saying that the
> rawhide-composer and bodhi-backend01 machines that run pungi are Fedora
> 29 hosts. The rawhide-composer runs in a chroot, so that should just
> work, but the bodhi-backend01 updates pungi I don't think does, or if it
> does it's a f29 chroot, not rawhide. So we will need a build for it.
Ok, makes sense.
While I'm thinking about it, fedora-repo-zdicts was just added to
Fedora yesterday and hasn't even been pushed to the testing
repositories yet. It should be available in Rawhide though...
Jonathan
2:15 p.m.
On Fri, 2018-12-14 at 19:02 +0000, Jonathan Dieter wrote:
Hey Randy, at the moment the --zck option *only* applies to
primary.xml, filelists.xml and other.xml. It should be pretty
straightforward to add it to the others, but I wanted to get those
three working first.
Cool, sounds good to me.
As for python bindings, they can read zchunk metadata just fine, but
I
don't think I hooked up creating the metadata. Where exactly does it
generate updateinfo in bodhi? I'd like to see how the function is
used
so I can implement it.
Bodhi's updateinfo code lives here:
https://github.com/fedora-infra/bodhi/blob/3.11.3/bodhi/server/metadata.py
3:13 p.m.
On Fri, 2018-12-14 at 15:15 -0500, Randy Barlow wrote:
On Fri, 2018-12-14 at 19:02 +0000, Jonathan Dieter wrote:
> Hey Randy, at the moment the --zck option *only* applies to
> primary.xml, filelists.xml and other.xml. It should be pretty
> straightforward to add it to the others, but I wanted to get those
> three working first.
Cool, sounds good to me.
> As for python bindings, they can read zchunk metadata just fine, but I
> don't think I hooked up creating the metadata. Where exactly does it
> generate updateinfo in bodhi? I'd like to see how the function is
> used so I can implement it.
Bodhi's updateinfo code lives here:
https://github.com/fedora-infra/bodhi/blob/3.11.3/bodhi/server/metadata.py
Thanks for this. I'll take a look at it and see what it will take to
make zchunk generation work in there.
Jonathan
1970
days inactive
1971
days old
infrastructure@lists.fedoraproject.org
11 comments
3 participants
participants (3)
-
Jonathan Dieter -
Kevin Fenzi -
Randy Barlow