This is a patch to run createrepo on the contents of the resultdir after a successful mock run. In theory this lets us suck those contents over for koji to have available.
-sv
This is a patch to run createrepo on the contents of the resultdir after a successful mock run. In theory this lets us suck those contents over for koji to have available.
Awesome. I've wanted for ages to have koji scratch/user/task_12345 dirs contain repodata.
Thanks, Roland
On Tue, 9 Feb 2010, Roland McGrath wrote:
This is a patch to run createrepo on the contents of the resultdir after a successful mock run. In theory this lets us suck those contents over for koji to have available.
Awesome. I've wanted for ages to have koji scratch/user/task_12345 dirs contain repodata.
exactly what we are driving at.
-sv
On 02/09/2010 08:56 PM, Seth Vidal wrote:
This is a patch to run createrepo on the contents of the resultdir after a successful mock run. In theory this lets us suck those contents over for koji to have available.
If the target is koji, then mock is the wrong place for this code.
...unless folks that use plain mock need this?
On Wed, 10 Feb 2010, Mike McLean wrote:
On 02/09/2010 08:56 PM, Seth Vidal wrote:
This is a patch to run createrepo on the contents of the resultdir after a successful mock run. In theory this lets us suck those contents over for koji to have available.
If the target is koji, then mock is the wrong place for this code.
why? You make the repo at end time then with ticket 167 (that I just filed for koji) it sucks up the repodata, too.
I talked to mbonnet about this on irc - he seemed fine w/it.
...unless folks that use plain mock need this?
sure. why not?
-sv
On 02/10/2010 03:28 PM, Seth Vidal wrote:
If the target is koji, then mock is the wrong place for this code.
why? You make the repo at end time then with ticket 167 (that I just filed for koji) it sucks up the repodata, too.
It's just as easy to run the createrepo from koji instead. Then, - no need to add the dep to mock - works in koji regardless of mock version
Furthermore, the repodata generated here will not mesh with the resulting layout on the hub. - for scratch builds all the rpms are placed in a single dir - for non-scratch builds the src rpms get split off and any noarch rpms generated from a non-noarch build will land in the noarch dir
...unless folks that use plain mock need this?
sure. why not?
I guess I've never understood how this is helpful. In order to use it, you'd first need to configure the repo, which means creating/editing a config file. And you go to that effort so that yum can resolve a handful of inter-subpackage deps for you?
I've just never had any trouble running yum localinstall against the resulting rpms.
All that being said, I'm not opposed to doing something -- I just want to do it right. In particular, I'd like to get a solid handle on the range of use cases that are behind this request.
On Wed, 10 Feb 2010, Mike McLean wrote:
Furthermore, the repodata generated here will not mesh with the resulting layout on the hub.
- for scratch builds all the rpms are placed in a single dir
- for non-scratch builds the src rpms get split off and any noarch
rpms generated from a non-noarch build will land in the noarch dir
...unless folks that use plain mock need this?
sure. why not?
I guess I've never understood how this is helpful. In order to use it, you'd first need to configure the repo, which means creating/editing a config file. And you go to that effort so that yum can resolve a handful of inter-subpackage deps for you?
look at the yum tmprepo plugin - it handles the above w/o config edits.
use cases:
1. have repodata immediately available for all koji scratch builds for easier testing 2. possibly speed up assembling repos from multiple koji builds 3. it's a pretty light-weight hit to go ahead and generate this data 4. it means if you know the taskid then you can find another good index to slurp down all the files related to it.
those are ones I can think of off the top of my head.
I don't have a strong preference toward this being in mock - when we were talking about it on irc yesterday it seemed the easiest place to put it.
-sv
-sv
On 02/10/2010 04:08 PM, Seth Vidal wrote:
- have repodata immediately available for all koji scratch builds for
easier testing 2. possibly speed up assembling repos from multiple koji builds
Now this is something I'm interested in but haven't had time to poke at. Certainly anything that speeds up repo generation is a win for koji.
I would also be interested in this from another angle -- if the main createrepo task in koji were to merely merge a ton of 1-build repos the we could probably eliminate the need for having /mnt/koji mounted on the builders that handle these tasks.
- it's a pretty light-weight hit to go ahead and generate this data
- it means if you know the taskid then you can find another good index to
slurp down all the files related to it.
those are ones I can think of off the top of my head.
I don't have a strong preference toward this being in mock - when we were talking about it on irc yesterday it seemed the easiest place to put it.
I'm curious at what level folks would like the repo created. By that I mean do we: a) make one repo for the entire build (all arches) b) make one repo for each arch within each build
The latter is mostly (apart from src and noarch subpackages) what the mock approach would get. However when this issue has come up in the past I definitely remember folks asking for the former.
Granted if we have an eye towards speeding up the generation of koji's main repos I suppose we should also add c) make one repo for each /canonical/ arch within each build ...which is a little different.
On Wed, 2010-02-10 at 16:24 -0500, Mike McLean wrote:
On 02/10/2010 04:08 PM, Seth Vidal wrote:
- have repodata immediately available for all koji scratch builds for
easier testing 2. possibly speed up assembling repos from multiple koji builds
Now this is something I'm interested in but haven't had time to poke at. Certainly anything that speeds up repo generation is a win for koji.
I would also be interested in this from another angle -- if the main createrepo task in koji were to merely merge a ton of 1-build repos the we could probably eliminate the need for having /mnt/koji mounted on the builders that handle these tasks.
Interesting indeed. Yes, I think this could work.
- it's a pretty light-weight hit to go ahead and generate this data
- it means if you know the taskid then you can find another good index to
slurp down all the files related to it.
those are ones I can think of off the top of my head.
I don't have a strong preference toward this being in mock - when we were talking about it on irc yesterday it seemed the easiest place to put it.
I'm curious at what level folks would like the repo created. By that I mean do we: a) make one repo for the entire build (all arches) b) make one repo for each arch within each build
The latter is mostly (apart from src and noarch subpackages) what the mock approach would get. However when this issue has come up in the past I definitely remember folks asking for the former.
Granted if we have an eye towards speeding up the generation of koji's main repos I suppose we should also add c) make one repo for each /canonical/ arch within each build ...which is a little different.
My preference would be (a). After that it be (c) then (b).
Cheers -- Dennis
I'm curious at what level folks would like the repo created. By that I mean do we: a) make one repo for the entire build (all arches) b) make one repo for each arch within each build
The latter is mostly (apart from src and noarch subpackages) what the mock approach would get. However when this issue has come up in the past I definitely remember folks asking for the former.
What I want is, "I can easily install it from a scratch build." The single repo makes this easy, but that is the only reason to care. So if it were per-arch repos in subdirectories, that would be fine if it's simple to have a '/$arch' on the end of a URL and it dtrt.
IOW, the innards really do not matter to the end user (me). If some different innards have benefits for feeding into the usual koji/newrepo/compose/whatnot stuff, rock on.
What I want is:
yum install --kojirepo=scratch/roland/task_1234567 foo yum upgrade --kojirepo=packages/kernel/2.6.37/92.9.f22
and Just Make It Happen. Plugins, scripts, one repo, arch repos, do what you want. Just make it easy to do the stuff that every package maintainer winds up wanting to do all the time.
Thanks, Roland
I guess I've never understood how this is helpful. In order to use it, you'd first need to configure the repo, which means creating/editing a config file. And you go to that effort so that yum can resolve a handful of inter-subpackage deps for you?
A large handful (build a gcc sometime), that you have to pick and download. And yum should have --repofrompath like repoquery does (I thought it did, actually). i.e.:
yum upgrade --repofrompath=test,http://koji.fedoraproject.org/scratch/roland/task_1965799
I've just never had any trouble running yum localinstall against the resulting rpms.
Nor have I, after either spending some time operating my mouse to download the right ones, and trying three different times to have it tell me I missed one I forgot was actually a dependency of the one I needed to test, or else spending the same 10 minutes again every time (I know, I should just write it down) to figure out how to make wget or curl download the whole directory full without filling my world with files called 'index.html?M=D.2' or following too many links and trying to download the entire koji site.
Thanks, Roland
On Wed, 2010-02-10 at 13:39 -0800, Roland McGrath wrote:
I guess I've never understood how this is helpful. In order to use it, you'd first need to configure the repo, which means creating/editing a config file. And you go to that effort so that yum can resolve a handful of inter-subpackage deps for you?
A large handful (build a gcc sometime), that you have to pick and download. And yum should have --repofrompath like repoquery does (I thought it did, actually). i.e.:
yum upgrade --repofrompath=test,http://koji.fedoraproject.org/scratch/roland/task_1965799
Currently you have to do:
yum upgrade --tmprepo=http://koji.fedoraproject.org/scratch/roland/task_1965799/repodata/repomd.xm...
...I'm looking at updating the tmprepo plugin so you could do something like:
yum upgrade --tmprepo=koji:roland,1965799
I've just never had any trouble running yum localinstall against the resulting rpms.
Nor have I, after either spending some time operating my mouse to download the right ones, and trying three different times to have it tell me I missed one I forgot was actually a dependency of the one I needed to test, or else spending the same 10 minutes again every time (I know, I should just write it down) to figure out how to make wget or curl download the whole directory full without filling my world with files called 'index.html?M=D.2' or following too many links and trying to download the entire koji site.
Note that in rawhide/3.2.26 yum you can just paste the http URLs to yum install/localinstall. Of course having real repos. will still be much nicer, IMNSHO.
On 02/10/2010 11:39 PM, Roland McGrath wrote:
I guess I've never understood how this is helpful. In order to use it, you'd first need to configure the repo, which means creating/editing a config file. And you go to that effort so that yum can resolve a handful of inter-subpackage deps for you?
A large handful (build a gcc sometime), that you have to pick and download. And yum should have --repofrompath like repoquery does (I thought it did, actually). i.e.:
yum upgrade --repofrompath=test,http://koji.fedoraproject.org/scratch/roland/task_1965799
I've just never had any trouble running yum localinstall against the resulting rpms.
Nor have I, after either spending some time operating my mouse to download the right ones, and trying three different times to have it tell me I missed one I forgot was actually a dependency of the one I needed to test, or else spending the same 10 minutes again every time (I know, I should just write it down) to figure out how to make wget or curl download the whole directory full without filling my world with files called 'index.html?M=D.2' or following too many links and trying to download the entire koji site.
Have you tried "koji download-build" (for official builds) or http://people.redhat.com/mikeb/scripts/download-scratch.py (for --scratch builds)?
On 02/09/2010 08:56 PM, Seth Vidal wrote:
This is a patch to run createrepo on the contents of the resultdir after a successful mock run. In theory this lets us suck those contents over for koji to have available.
If the target is koji, then mock is the wrong place for this code.
...unless folks that use plain mock need this?
I'd certainly like it to be de rigeur for all the canonical forms of producing a passel of rpms. A directory full of fresh built rpms that is not directly usable as a repo is just silly!
On 02/10/2010 03:41 PM, Roland McGrath wrote:
I'd certainly like it to be de rigeur for all the canonical forms of producing a passel of rpms. A directory full of fresh built rpms that is not directly usable as a repo is just silly!
Not every set of rpms makes sense as a repo.
If you really feel this way, perhaps you should request that yum support a dir of rpms as a repo. ;)
On Wed, 10 Feb 2010, Mike McLean wrote:
On 02/10/2010 03:41 PM, Roland McGrath wrote:
I'd certainly like it to be de rigeur for all the canonical forms of producing a passel of rpms. A directory full of fresh built rpms that is not directly usable as a repo is just silly!
Not every set of rpms makes sense as a repo.
If you really feel this way, perhaps you should request that yum support a dir of rpms as a repo. ;)
repoquery --repofrompath and the yum tmprepo plugin.
-sv
On Wed, 2010-02-10 at 16:09 -0500, Seth Vidal wrote:
On Wed, 10 Feb 2010, Mike McLean wrote:
On 02/10/2010 03:41 PM, Roland McGrath wrote:
I'd certainly like it to be de rigeur for all the canonical forms of producing a passel of rpms. A directory full of fresh built rpms that is not directly usable as a repo is just silly!
Not every set of rpms makes sense as a repo.
If you really feel this way, perhaps you should request that yum support a dir of rpms as a repo. ;)
repoquery --repofrompath and the yum tmprepo plugin.
-sv
I agree with Seth that there would be significant utility in having repodata readily available for brew builds. I suggest that for regular builds the repodata live at /data/repodata within the package dir. For scratch builds, the repodata directory could live right inside the task_123456 dir.
I think Mike is right that if we do add repodata creation, having it on the hub would make the most sense.
Cheers -- Dennis
On 02/10/2010 11:27 PM, Dennis Gregorovic wrote:
On Wed, 2010-02-10 at 16:09 -0500, Seth Vidal wrote:
On Wed, 10 Feb 2010, Mike McLean wrote:
On 02/10/2010 03:41 PM, Roland McGrath wrote:
I'd certainly like it to be de rigeur for all the canonical forms of producing a passel of rpms. A directory full of fresh built rpms that is not directly usable as a repo is just silly!
Not every set of rpms makes sense as a repo.
If you really feel this way, perhaps you should request that yum support a dir of rpms as a repo. ;)
repoquery --repofrompath and the yum tmprepo plugin.
-sv
I agree with Seth that there would be significant utility in having repodata readily available for brew builds. I suggest that for regular builds the repodata live at /data/repodata within the package dir. For scratch builds, the repodata directory could live right inside the task_123456 dir.
I think Mike is right that if we do add repodata creation, having it on the hub would make the most sense.
We have a Koji hub plugin system now, which makes implementing this kind of post-build operation pretty simple. I think the last time we talked about something like this, we were concerned about the load on the hub from constantly running createrepos. However, I guess running a createrepo on a handful of packages won't cause a huge increase in load, and if it's really useful it's worth it.
On Thu, 2010-02-11 at 10:12 +0200, Mike Bonnet wrote:
On 02/10/2010 11:27 PM, Dennis Gregorovic wrote:
On Wed, 2010-02-10 at 16:09 -0500, Seth Vidal wrote:
On Wed, 10 Feb 2010, Mike McLean wrote:
On 02/10/2010 03:41 PM, Roland McGrath wrote:
I'd certainly like it to be de rigeur for all the canonical forms of producing a passel of rpms. A directory full of fresh built rpms that is not directly usable as a repo is just silly!
Not every set of rpms makes sense as a repo.
If you really feel this way, perhaps you should request that yum support a dir of rpms as a repo. ;)
repoquery --repofrompath and the yum tmprepo plugin.
-sv
I agree with Seth that there would be significant utility in having repodata readily available for brew builds. I suggest that for regular builds the repodata live at /data/repodata within the package dir. For scratch builds, the repodata directory could live right inside the task_123456 dir.
I think Mike is right that if we do add repodata creation, having it on the hub would make the most sense.
We have a Koji hub plugin system now, which makes implementing this kind of post-build operation pretty simple. I think the last time we talked about something like this, we were concerned about the load on the hub from constantly running createrepos. However, I guess running a createrepo on a handful of packages won't cause a huge increase in load, and if it's really useful it's worth it.
Using a plugin would be handy if we want to store the repodata separate from the packages. Otherwise, we would need to add a mechanism to upload the repodata back to the hub (assuming that the box running createrepo doesn't have rw access to /mnt/koji). However, adding some generic routines for uploading content to the /data and /scratch directories could be useful down the line.
-- Dennis
On 02/11/2010 03:12 AM, Mike Bonnet wrote:
We have a Koji hub plugin system now, which makes implementing this kind of post-build operation pretty simple. I think the last time we talked about something like this, we were concerned about the load on the hub from constantly running createrepos. However, I guess running a createrepo on a handful of packages won't cause a huge increase in load, and if it's really useful it's worth it.
I'm not sure I'd want the createrepo to run on the hub, though I guess such small runs would be fairly quick and not overly frequent.
We can probably still farm it out to the builders, or just run it in kojid after the build. There are just a few issues with paths and urls to work out.
On Thu, 11 Feb 2010, Mike McLean wrote:
On 02/11/2010 03:12 AM, Mike Bonnet wrote:
We have a Koji hub plugin system now, which makes implementing this kind of post-build operation pretty simple. I think the last time we talked about something like this, we were concerned about the load on the hub from constantly running createrepos. However, I guess running a createrepo on a handful of packages won't cause a huge increase in load, and if it's really useful it's worth it.
I'm not sure I'd want the createrepo to run on the hub, though I guess such small runs would be fairly quick and not overly frequent.
We can probably still farm it out to the builders, or just run it in kojid after the build. There are just a few issues with paths and urls to work out.
I'd say run it on the builders, too. That way we can potentially be clever if we ever find ourselves with multiple versions of createrepo for specific trees. So we can build the right version for the process.
-sv
On Thu, 2010-02-11 at 17:03 -0500, Seth Vidal wrote:
On Thu, 11 Feb 2010, Mike McLean wrote:
On 02/11/2010 03:12 AM, Mike Bonnet wrote:
We have a Koji hub plugin system now, which makes implementing this kind of post-build operation pretty simple. I think the last time we talked about something like this, we were concerned about the load on the hub from constantly running createrepos. However, I guess running a createrepo on a handful of packages won't cause a huge increase in load, and if it's really useful it's worth it.
I'm not sure I'd want the createrepo to run on the hub, though I guess such small runs would be fairly quick and not overly frequent.
We can probably still farm it out to the builders, or just run it in kojid after the build. There are just a few issues with paths and urls to work out.
I'd say run it on the builders, too. That way we can potentially be clever if we ever find ourselves with multiple versions of createrepo for specific trees. So we can build the right version for the process.
Are you meaning doing it in the /chroot/ ? That'd mean adding createrepo as a chroot item for every build, or creating a chroot for every post-build createrepo call.
On Thu, 11 Feb 2010, Jesse Keating wrote:
I'd say run it on the builders, too. That way we can potentially be clever if we ever find ourselves with multiple versions of createrepo for specific trees. So we can build the right version for the process.
Are you meaning doing it in the /chroot/ ? That'd mean adding createrepo as a chroot item for every build, or creating a chroot for every post-build createrepo call.
I don't necessarily mean that - but potentially it would be worthwhile.
really all we need is host you're making the the chroot on to be using the same createrepo/metadata version.
just a zany thought for some potentially painful future.
-sv
On Thu, 11 Feb 2010, Seth Vidal wrote:
On Thu, 11 Feb 2010, Jesse Keating wrote:
I'd say run it on the builders, too. That way we can potentially be clever if we ever find ourselves with multiple versions of createrepo for specific trees. So we can build the right version for the process.
Are you meaning doing it in the /chroot/ ? That'd mean adding createrepo as a chroot item for every build, or creating a chroot for every post-build createrepo call.
I don't necessarily mean that - but potentially it would be worthwhile.
really all we need is host you're making the the chroot on to be using the same createrepo/metadata version.
just a zany thought for some potentially painful future.
this may just be muddying the waters - feel free to ignore it.
-sv
On Wed, 10 Feb 2010 12:41:34 -0800 (PST) Roland McGrath roland@redhat.com wrote:
On 02/09/2010 08:56 PM, Seth Vidal wrote:
This is a patch to run createrepo on the contents of the resultdir after a successful mock run. In theory this lets us suck those contents over for koji to have available.
If the target is koji, then mock is the wrong place for this code.
...unless folks that use plain mock need this?
I'd certainly like it to be de rigeur for all the canonical forms of producing a passel of rpms. A directory full of fresh built rpms that is not directly usable as a repo is just silly! -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
I just added Seth's patch to mock and spun up version 1.0.4. Should be pushed to testing in the morning
Clark
buildsys@lists.fedoraproject.org
-
Clark Williams -
Dennis Gregorovic -
James Antill -
Jesse Keating -
Mike Bonnet -
Mike McLean -
Roland McGrath -
Seth Vidal