Hi All!
The last discussion how to handle kernel-modules in the buildsys stuck at some point (for those who can't remember it see https://www.redhat.com/archives/fedora-buildsys-list/2005-August/msg00030.ht... for the start and https://www.redhat.com/archives/fedora-buildsys-list/2005-August/msg00036.ht... for one of the ends.
I really would like it if we could get this discussion to life again to solve this problem. I made a quick hack against mock 0.4 so it passes a "--define kernel\ some-kernel-version-here" to the rpmbuild-calls that build the pgk. This required a new mock command-line option ("--kernel put-some-kernel-version-here"). I included a quick check so this option only works with packages whose name starts with "kernel" so nobody sane can abuse it for other packages. Warning, I'm not programming very often, but the patch WORKSFORME. Seth, is something like that acceptable for you?
Dan, if something like that goes into mock -- what would be the best way to make this new mock-option usable in plague for kernel-modules in fedora-extras? Should plague automatically build the modules for all kernels available (the one that was shipped and those currently in updates-released)? Or only for the newest kernel available? Or should the packager say something like "make plague KERNEL=put-kernel-ver-here" when requesting build of a fdr-extras-kernel-module-pkg?
On Wed, 2005-09-28 at 15:20 +0200, Thorsten Leemhuis wrote:
Dan, if something like that goes into mock -- what would be the best way to make this new mock-option usable in plague for kernel-modules in fedora-extras? Should plague automatically build the modules for all kernels available (the one that was shipped and those currently in updates-released)? Or only for the newest kernel available? Or should the packager say something like "make plague KERNEL=put-kernel-ver-here" when requesting build of a fdr-extras-kernel-module-pkg?
I recognize the need for this, there are a few questions that need answering here though. First, methods.
1) have plague automatically build for all kernels
Requires a large amount of work with yum to essentially list all kernel-* packages in a given repository, then queue up builds for them. That's a chunk of work.
2) have plague build for latest kernel
Still requires work to ask yum about a certain repository to get latest kernel package. Slightly easier, but not very much so. Given that we are not guaranteed that the kernel-* package will be in the server's repository (it might be in the upstream "official" mirror instead) we pretty much have to use yum.
3) have the packager do it
Pretty much the status quo right now, not much of a win IMHO.
In any case, you're going to have to get some form of argument passing through Seth and into mock. I don't like the "allow packager to pass random args" thing, since that's a direct impact on the security of the build system. In the short term though, limiting such things to automatic rebuilds of kernel modules seems sane.
Ideally, yes, plague should be doing some depsolving on the server side. That's going to take a while to build in. In the mean time, we could build in some kernel-module specific stuff, I'm not against that. Most likely post 0.4.x work (1).
Some nice things that could go along with depsolving in the server:
- Store mock configs on the server and push them out to builders on-demand. Bonus: we use same yum configs on server & builders.
- Don't build packages until their deps are satisfied, waiting a max of 8 or 10 hours before failing the package.
Dan
(1) That doesn't mean "a year from now" and work could start on that fairly soon after 0.4.x comes out, which is "in a few weeks"
Hi!
Dan, thanks for taking the time for your reply.
Am Mittwoch, den 28.09.2005, 16:50 -0400 schrieb Dan Williams:
On Wed, 2005-09-28 at 15:20 +0200, Thorsten Leemhuis wrote:
Dan, if something like that goes into mock -- what would be the best way to make this new mock-option usable in plague for kernel-modules in fedora-extras? Should plague automatically build the modules for all kernels available (the one that was shipped and those currently in updates-released)? Or only for the newest kernel available? Or should the packager say something like "make plague KERNEL=put-kernel-ver-here" when requesting build of a fdr-extras-kernel-module-pkg?
I recognize the need for this, there are a few questions that need answering here though. First, methods.
- have plague automatically build for all kernels
[...]
Some people think that should be done. But building it for the latest kernel should be enough IMHO. In the past I also thought we should build the module for the kernel that was released with core, cause it seems a lot of people with slow modems or systems that are not directly connected to the internet don't update their kernels. If we build for more then one kernel this is the second important one.
- have plague build for latest kernel
Still requires work to ask yum about a certain repository to get latest kernel package. Slightly easier, but not very much so. Given that we are not guaranteed that the kernel-* package will be in the server's repository (it might be in the upstream "official" mirror instead) we pretty much have to use yum.
In the build system of another 3rd-party repo we had a file that listed for which kernel modules should be build. Could that be a short term solution for plague es well?
- have the packager do it
Pretty much the status quo right now, not much of a win IMHO.
I disagree. We have no "status quo" right now because
- plague developer(s) - mock developer(s) - people on fedora-packaging that want to specify kernel-module-packaging - fedora-extras packagers with packages like qemu, OpenAFS (and probably more) - yum developers
all wait for each other (since months) to standardize packaging and building of kernel-module-packages for extras. This is a real pity :| I really would like to get this sorted out somehow.
In any case, you're going to have to get some form of argument passing through Seth and into mock.
Seth, is something like the patch I posted yesterday acceptable for you? If you don't like part of it just tell me and I can see if I can fix it with my python knowledge (and improve that while at it)
I don't like the "allow packager to pass random args" thing,
For other packages I agree.
since that's a direct impact on the security of the build system. In the short term though, limiting such things to automatic rebuilds of kernel modules seems sane.
Just out of curiosity: What do you suggest in the long term?
On Wed, 2005-09-28 at 16:50 -0400, Dan Williams wrote:
On Wed, 2005-09-28 at 15:20 +0200, Thorsten Leemhuis wrote:
Dan, if something like that goes into mock -- what would be the best way to make this new mock-option usable in plague for kernel-modules in fedora-extras? Should plague automatically build the modules for all kernels available (the one that was shipped and those currently in updates-released)? Or only for the newest kernel available? Or should the packager say something like "make plague KERNEL=put-kernel-ver-here" when requesting build of a fdr-extras-kernel-module-pkg?
I recognize the need for this, there are a few questions that need answering here though. First, methods.
- have plague automatically build for all kernels
Requires a large amount of work with yum to essentially list all kernel-* packages in a given repository, then queue up builds for them. That's a chunk of work.
- have plague build for latest kernel
Still requires work to ask yum about a certain repository to get latest kernel package. Slightly easier, but not very much so. Given that we are not guaranteed that the kernel-* package will be in the server's repository (it might be in the upstream "official" mirror instead) we pretty much have to use yum.
- have the packager do it
Pretty much the status quo right now, not much of a win IMHO.
In any case, you're going to have to get some form of argument passing through Seth and into mock. I don't like the "allow packager to pass random args" thing, since that's a direct impact on the security of the build system. In the short term though, limiting such things to automatic rebuilds of kernel modules seems sane.
Ideally, yes, plague should be doing some depsolving on the server side. That's going to take a while to build in. In the mean time, we could build in some kernel-module specific stuff, I'm not against that. Most likely post 0.4.x work (1).
Some nice things that could go along with depsolving in the server:
- Store mock configs on the server and push them out to builders
on-demand. Bonus: we use same yum configs on server & builders.
- Don't build packages until their deps are satisfied, waiting a max of
8 or 10 hours before failing the package.
Dan
(1) That doesn't mean "a year from now" and work could start on that fairly soon after 0.4.x comes out, which is "in a few weeks"
sorry for being away for so long.
1. passing in kernel args is fine - but we're going to need to add a way to vet them out to make sure someone doesn't play silly buggers with the command line args passed to rpmbuild (ex: rpmbuild --define 'kernelver=something'; rm -rf /) as a dumb example. That's an obviously malicious example but I'm more concerned about the just silly mistakes that could hose up the buildsystem.
2. I don't like the idea of arbitrary defines being passed in for the buildsystem - it makes for hard-to-replicate builds, I think. I know a number of folks agree with me about that, too. With regard to the kernels I think we could reasonably write a couple of simple yum-utils that plague could use to query the chroot to find out what kernel to build for. It wouldn't need to run as root once the chroot was setup b/c the metadata would all be there.
So maybe this is a middle point but:
mock --defines=/path/to/some/file --other-mock-options-here
so then plague could just provide a defines file and mock would read those in and add them to the rpm defines in the .rpmmacros in the chrootbuilders homedir.
Or does that sound silly?
-sv
On Fri, 2005-10-07 at 01:42 -0400, seth vidal wrote:
So maybe this is a middle point but:
mock --defines=/path/to/some/file --other-mock-options-here
so then plague could just provide a defines file and mock would read those in and add them to the rpm defines in the .rpmmacros in the chrootbuilders homedir.
Or does that sound silly?
This was really the only thing I could think of to get around the malicious defines thing too... We avoid a _ton_ of command line quoting/scrubbing doing it this way.
Dan
Am Freitag, den 07.10.2005, 01:42 -0400 schrieb seth vidal:
On Wed, 2005-09-28 at 16:50 -0400, Dan Williams wrote:
On Wed, 2005-09-28 at 15:20 +0200, Thorsten Leemhuis wrote:
Dan, if something like that goes into mock -- what would be the best way to make this new mock-option usable in plague for kernel-modules in fedora-extras? Should plague automatically build the modules for all kernels available (the one that was shipped and those currently in updates-released)? Or only for the newest kernel available? Or should the packager say something like "make plague KERNEL=put-kernel-ver-here" when requesting build of a fdr-extras-kernel-module-pkg?
I recognize the need for this, there are a few questions that need answering here though. First, methods.
- have plague automatically build for all kernels
Requires a large amount of work with yum to essentially list all kernel-* packages in a given repository, then queue up builds for them. That's a chunk of work.
- have plague build for latest kernel
Still requires work to ask yum about a certain repository to get latest kernel package. Slightly easier, but not very much so. Given that we are not guaranteed that the kernel-* package will be in the server's repository (it might be in the upstream "official" mirror instead) we pretty much have to use yum.
- have the packager do it
Pretty much the status quo right now, not much of a win IMHO.
In any case, you're going to have to get some form of argument passing through Seth and into mock. I don't like the "allow packager to pass random args" thing, since that's a direct impact on the security of the build system. In the short term though, limiting such things to automatic rebuilds of kernel modules seems sane.
Ideally, yes, plague should be doing some depsolving on the server side. That's going to take a while to build in. In the mean time, we could build in some kernel-module specific stuff, I'm not against that. Most likely post 0.4.x work (1).
Some nice things that could go along with depsolving in the server:
- Store mock configs on the server and push them out to builders
on-demand. Bonus: we use same yum configs on server & builders.
- Don't build packages until their deps are satisfied, waiting a max of
8 or 10 hours before failing the package.
Dan
(1) That doesn't mean "a year from now" and work could start on that fairly soon after 0.4.x comes out, which is "in a few weeks"
sorry for being away for so long.
np
- passing in kernel args is fine - but we're going to need to add a way
to vet them out to make sure someone doesn't play silly buggers with the command line args passed to rpmbuild (ex: rpmbuild --define 'kernelver=something'; rm -rf /) as a dumb example. That's an obviously malicious example but I'm more concerned about the just silly mistakes that could hose up the buildsystem.
Sound reasonable
- I don't like the idea of arbitrary defines being passed in for the
buildsystem - it makes for hard-to-replicate builds, I think.
Is there really a big difference if the define is passed via the command line or via a defines file when it comes to "hard-to-replicate builds"?
I know a number of folks agree with me about that, too. With regard to the kernels I think we could reasonably write a couple of simple yum-utils that plague could use to query the chroot to find out what kernel to build for.
Questions 1 and 2 from the initial mail (quoted above) remain here: Build only for the latest kernel, for the initial shipped and the latest kernel or for all available? If one of the last two we might need a mechanism to exclude some kernels if they are know to fail build (or plague needs to ignore if a build fails). And what about kernels from updates-testing? Should we build for those, too, and but them in extras/testing while that kernel is in updates-testing?
It wouldn't need to run as root once the chroot was setup b/c the metadata would all be there.
So maybe this is a middle point but:
mock --defines=/path/to/some/file --other-mock-options-here
so then plague could just provide a defines file and mock would read those in and add them to the rpm defines in the .rpmmacros in the chrootbuilders homedir.
Or does that sound silly?
If you and Dan think doing it this way it's okay for me -- I don't really care how exactly the defines gets to the rpmbuild call.
buildsys@lists.fedoraproject.org
-
Dan Williams -
seth vidal -
Thorsten Leemhuis