Hello guys
I hope you're all doing fine, with joy in your hearts.
After a adventurous while, I have updated and QAed a new qemu srpm. I worked mostly on x86 and sparc QA, since they are the ones in which we have to build additional firmware.
Speaking of firware, for the very same reason as etherboot[1], I'm proposing some new packages. They are:
Bochs bios: http://glommer.fedorapeople.org/bochs-bios-2.3.8-0.1.git36989b0d2.fc11.src.r... http://koji.fedoraproject.org/koji/taskinfo?taskID=1121809
OpenBIOS: http://glommer.fedorapeople.org/openbios-1.0-0.1.svn450.fc11.src.rpm http://koji.fedoraproject.org/koji/taskinfo?taskID=1119906
VGABIOS: http://glommer.fedorapeople.org/vgabios-0.6-0.1beta.fc11.src.rpm http://koji.fedoraproject.org/koji/taskinfo?taskID=1120633
I'm not shouting the final review yet, since I'd like to reach agreement on whatever matters may arise, in here.
The qemu source package can be found at:
http://glommer.fedorapeople.org/qemu-0.9.2-0.1.svn6392.fc11.src.rpm
The x86 system emulator depends on bochs-bios and vgabios. The sparc system emulator depends on openbios
Openbios should be used for powerpc too, but upstream is not yet confident on that. So as I was having troubles building it, I just left them commented for this round.
If everybody is okay with that modulo minor comments to the packages, my next plan is:
1) Submmit new packages for review 2) Commit new qemu to CVS 3) Get KVM in the package. I prefer to do this in two steps, since it will get easier to spot anything that might go wrong 4) Have a bottle of wine.
[1] Most firmware is compiled for one specific architecture, but should run on qemu whichever your host architecture is. We don't have a cross compiler infra in place, so we have to compile it natively. The easiest would be to pick binaries directly from upstream, which usually provides it. But we have policies that say we have to build everything we ship. So we have a crazy bootstrap process in place, in which we first build natively, and then feed a tar.gz for the other architectures. It means every package should be built at least twice in the process of getting something out of the door.
Hi Glauber,
I'm glad you left the bottle of wine until last - this work requires a strong stomach! :-)
On Thu, 2009-02-12 at 13:29 -0200, Glauber Costa wrote:
Hello guys
I hope you're all doing fine, with joy in your hearts.
After a adventurous while, I have updated and QAed a new qemu srpm. I worked mostly on x86 and sparc QA, since they are the ones in which we have to build additional firmware.
Speaking of firware, for the very same reason as etherboot[1], I'm proposing some new packages. They are:
Bochs bios: http://glommer.fedorapeople.org/bochs-bios-2.3.8-0.1.git36989b0d2.fc11.src.r... http://koji.fedoraproject.org/koji/taskinfo?taskID=1121809
OpenBIOS: http://glommer.fedorapeople.org/openbios-1.0-0.1.svn450.fc11.src.rpm http://koji.fedoraproject.org/koji/taskinfo?taskID=1119906
VGABIOS: http://glommer.fedorapeople.org/vgabios-0.6-0.1beta.fc11.src.rpm http://koji.fedoraproject.org/koji/taskinfo?taskID=1120633
...
[1] Most firmware is compiled for one specific architecture, but should run on qemu whichever your host architecture is. We don't have a cross compiler infra in place, so we have to compile it natively. The easiest would be to pick binaries directly from upstream, which usually provides it. But we have policies that say we have to build everything we ship. So we have a crazy bootstrap process in place, in which we first build natively, and then feed a tar.gz for the other architectures. It means every package should be built at least twice in the process of getting something out of the door.
Okay, let me see if I understand this:
1) You need to fix or update something in the package
2) You make the change and build it in koji
3) The build finishes, you download the binary RPM, unpack with rpm2cpio and:
$> cd usr/share $> tar -cvjf etherboot-binaries-$(nvr).tar.bz etherboot/
4) Update the spec file with the new tarball name, upload the tarball to the sources repo, check in and build in Koji
Two suggestions:
1) The second build shouldn't rebuild the binaries on any arch - that way each arch has identical binaries
2) We add a "make update-binaries" the makefile - it would do:
$> koji download-build $(make verrel) $> rpm2cpio ... | cpio $> tar -cvjf ... $> grep -v etherboot-binaries sources > sources.tmp $> mv sources.tmp sources $> sed -i -e ... etherboot.spec $> make upload FILES=...
Cheers, Mark.
On Fri, Feb 13, 2009 at 09:50:06AM +0000, Mark McLoughlin wrote:
Hi Glauber,
I'm glad you left the bottle of wine until last - this work requires a strong stomach! :-)
On Thu, 2009-02-12 at 13:29 -0200, Glauber Costa wrote:
<snip>
[1] Most firmware is compiled for one specific architecture, but should run on qemu whichever your host architecture is. We don't have a cross compiler infra in place, so we have to compile it natively. The easiest would be to pick binaries directly from upstream, which usually provides it. But we have policies that say we have to build everything we ship. So we have a crazy bootstrap process in place, in which we first build natively, and then feed a tar.gz for the other architectures. It means every package should be built at least twice in the process of getting something out of the door.
Okay, let me see if I understand this:
You need to fix or update something in the package
You make the change and build it in koji
The build finishes, you download the binary RPM, unpack with rpm2cpio and:
$> cd usr/share $> tar -cvjf etherboot-binaries-$(nvr).tar.bz etherboot/
Update the spec file with the new tarball name, upload the tarball to the sources repo, check in and build in Koji
Two suggestions:
- The second build shouldn't rebuild the binaries on any arch - that way each arch has identical binaries
Currently (on etherboot, and I think that's the case on the bios packages made by Glauber), we use prebuilt binaries only on the arches we can't build the binaries. This lets us use exactly the same package for step 2 and 4. But maybe we can change this:
- First build: all prebuilt binaries tarballs removed from the spec, only native binaries built (using something like %define only_native 1) - Second build: added the prebuilt binaries from the first build, no binaries actually compiled, but uses only the prebuilt binaries
We have a problem with any of the above approaches: we don't want to make the first build a scratch build (we want to keep the build information stored somewhere on Koji), but we don't want this build to be available for the users, as the result is an incomplete package. I am thinking about asking a Koji tag to be created just to store those "intermediate build step" RPMs. What do you think?
We add a "make update-binaries" the makefile - it would do:
$> koji download-build $(make verrel) $> rpm2cpio ... | cpio $> tar -cvjf ... $> grep -v etherboot-binaries sources > sources.tmp $> mv sources.tmp sources $> sed -i -e ... etherboot.spec $> make upload FILES=...
+1. This should be automated as much as possible.
On Fri, Feb 13, 2009 at 10:49:12AM -0200, Eduardo Habkost wrote:
On Fri, Feb 13, 2009 at 09:50:06AM +0000, Mark McLoughlin wrote:
Hi Glauber,
I'm glad you left the bottle of wine until last - this work requires a strong stomach! :-)
On Thu, 2009-02-12 at 13:29 -0200, Glauber Costa wrote:
<snip> > > [1] Most firmware is compiled for one specific architecture, but > > should run on qemu whichever your host architecture is. We don't > > have a cross compiler infra in place, so we have to compile it natively. > > The easiest would be to pick binaries directly from upstream, which > > usually provides it. But we have policies that say we have to build > > everything we ship. So we have a crazy bootstrap process in place, > > in which we first build natively, and then feed a tar.gz for the > > other architectures. It means every package should be built at least > > twice in the process of getting something out of the door. > > Okay, let me see if I understand this: > > 1) You need to fix or update something in the package > > 2) You make the change and build it in koji > > 3) The build finishes, you download the binary RPM, unpack with > rpm2cpio and: > > $> cd usr/share > $> tar -cvjf etherboot-binaries-$(nvr).tar.bz etherboot/ > > 4) Update the spec file with the new tarball name, upload the tarball > to the sources repo, check in and build in Koji > > Two suggestions: > > 1) The second build shouldn't rebuild the binaries on any arch - that > way each arch has identical binaries
Currently (on etherboot, and I think that's the case on the bios packages made by Glauber), we use prebuilt binaries only on the arches we can't build the binaries. This lets us use exactly the same package for step 2 and 4. But maybe we can change this:
- First build: all prebuilt binaries tarballs removed from the spec, only native binaries built (using something like %define only_native 1)
- Second build: added the prebuilt binaries from the first build, no binaries actually compiled, but uses only the prebuilt binaries
We have a problem with any of the above approaches: we don't want to make the first build a scratch build (we want to keep the build information stored somewhere on Koji), but we don't want this build to be available for the users, as the result is an incomplete package. I am thinking about asking a Koji tag to be created just to store those "intermediate build step" RPMs. What do you think?
We add a "make update-binaries" the makefile - it would do:
$> koji download-build $(make verrel) $> rpm2cpio ... | cpio $> tar -cvjf ... $> grep -v etherboot-binaries sources > sources.tmp $> mv sources.tmp sources $> sed -i -e ... etherboot.spec $> make upload FILES=...
+1. This should be automated as much as possible.
I've talked to some people in #fedora-devel yesterday, for _hours_.
It turns out that they're planning on deploying some features in koji by Feb 28th, that _might_ help us a lot. Really, a lot.
The idea is that you can have your package built on an architecture exclusively with BuildArch directive. But your package can then have a subpackage with BuildArch: noarch. That way, your package will span all repositories. This seems like the perfect solution to us. I've helped them to proceed with some tests, and it seem to work. So what I'm plannin on doing, is post the rpms for review today, since it is unlikely that they will receive more updates in this mean time.
Getting them review and approved is the really important bits. We can set on the ideal solution a bit later.
On Fri, Feb 13, 2009 at 10:59:04AM -0200, Glauber Costa wrote: <snip>
It turns out that they're planning on deploying some features in koji by Feb 28th, that _might_ help us a lot. Really, a lot.
The idea is that you can have your package built on an architecture exclusively with BuildArch directive. But your package can then have a subpackage with BuildArch: noarch. That way, your package will span all repositories. This seems like the perfect solution to us. I've helped them to proceed with some tests, and it seem to work. So what I'm plannin on doing, is post the rpms for review today, since it is unlikely that they will receive more updates in this mean time.
That makes sense. From the host system point of view, the BIOS images are data, and should be noarch. I think this will solve our problems.
Is Feb 28th still in time to start actually building this stuff on Rawhide without breaking any deadline?
Getting them review and approved is the really important bits. We can set on the ideal solution a bit later.
On Fri, Feb 13, 2009 at 10:59:04AM -0200, Glauber Costa wrote:
We have a problem with any of the above approaches: we don't want to make the first build a scratch build (we want to keep the build information stored somewhere on Koji), but we don't want this build to be available for the users, as the result is an incomplete package. I am thinking about asking a Koji tag to be created just to store those "intermediate build step" RPMs. What do you think?
We add a "make update-binaries" the makefile - it would do:
$> koji download-build $(make verrel) $> rpm2cpio ... | cpio $> tar -cvjf ... $> grep -v etherboot-binaries sources > sources.tmp $> mv sources.tmp sources $> sed -i -e ... etherboot.spec $> make upload FILES=...
+1. This should be automated as much as possible.
I've talked to some people in #fedora-devel yesterday, for _hours_.
It turns out that they're planning on deploying some features in koji by Feb 28th, that _might_ help us a lot. Really, a lot.
The idea is that you can have your package built on an architecture exclusively with BuildArch directive. But your package can then have a subpackage with BuildArch: noarch. That way, your package will span all repositories. This seems like the perfect solution to us. I've helped them to proceed with some tests, and it seem to work. So what I'm plannin on doing, is post the rpms for review today, since it is unlikely that they will receive more updates in this mean time.
The main limitation with that approach is the architecture coverage, and the way it relates to secondary architectures. This will only let us easily do x86 & ppc. If we want sparc, arm or ia64 BIOS, the builds would happen in the 2nd arch build ssytem, and the result would not be available in our main Fedora YUM repos for primary archs. If we go with your current approach, we can do the build in the 2nd-ary arch, and then pull the result back into the packages on the primary arches
Daniel
On Fri, Feb 13, 2009 at 01:09:53PM +0000, Daniel P. Berrange wrote:
On Fri, Feb 13, 2009 at 10:59:04AM -0200, Glauber Costa wrote:
We have a problem with any of the above approaches: we don't want to make the first build a scratch build (we want to keep the build information stored somewhere on Koji), but we don't want this build to be available for the users, as the result is an incomplete package. I am thinking about asking a Koji tag to be created just to store those "intermediate build step" RPMs. What do you think?
We add a "make update-binaries" the makefile - it would do:
$> koji download-build $(make verrel) $> rpm2cpio ... | cpio $> tar -cvjf ... $> grep -v etherboot-binaries sources > sources.tmp $> mv sources.tmp sources $> sed -i -e ... etherboot.spec $> make upload FILES=...
+1. This should be automated as much as possible.
I've talked to some people in #fedora-devel yesterday, for _hours_.
It turns out that they're planning on deploying some features in koji by Feb 28th, that _might_ help us a lot. Really, a lot.
The idea is that you can have your package built on an architecture exclusively with BuildArch directive. But your package can then have a subpackage with BuildArch: noarch. That way, your package will span all repositories. This seems like the perfect solution to us. I've helped them to proceed with some tests, and it seem to work. So what I'm plannin on doing, is post the rpms for review today, since it is unlikely that they will receive more updates in this mean time.
The main limitation with that approach is the architecture coverage, and the way it relates to secondary architectures. This will only let us easily do x86 & ppc. If we want sparc, arm or ia64 BIOS, the builds would happen in the 2nd arch build ssytem, and the result would not be available in our main Fedora YUM repos for primary archs. If we go with your current approach, we can do the build in the 2nd-ary arch, and then pull the result back into the packages on the primary arches
I don't know if I understood correctly. This means that the new feature could solve our problems, but it won't solve all of them because some arches where we need some binaries to be built are second-class citzens on the build system?
On Fri, Feb 13, 2009 at 11:23:01AM -0200, Eduardo Habkost wrote:
On Fri, Feb 13, 2009 at 01:09:53PM +0000, Daniel P. Berrange wrote:
On Fri, Feb 13, 2009 at 10:59:04AM -0200, Glauber Costa wrote:
We have a problem with any of the above approaches: we don't want to make the first build a scratch build (we want to keep the build information stored somewhere on Koji), but we don't want this build to be available for the users, as the result is an incomplete package. I am thinking about asking a Koji tag to be created just to store those "intermediate build step" RPMs. What do you think?
We add a "make update-binaries" the makefile - it would do:
$> koji download-build $(make verrel) $> rpm2cpio ... | cpio $> tar -cvjf ... $> grep -v etherboot-binaries sources > sources.tmp $> mv sources.tmp sources $> sed -i -e ... etherboot.spec $> make upload FILES=...
+1. This should be automated as much as possible.
I've talked to some people in #fedora-devel yesterday, for _hours_.
It turns out that they're planning on deploying some features in koji by Feb 28th, that _might_ help us a lot. Really, a lot.
The idea is that you can have your package built on an architecture exclusively with BuildArch directive. But your package can then have a subpackage with BuildArch: noarch. That way, your package will span all repositories. This seems like the perfect solution to us. I've helped them to proceed with some tests, and it seem to work. So what I'm plannin on doing, is post the rpms for review today, since it is unlikely that they will receive more updates in this mean time.
The main limitation with that approach is the architecture coverage, and the way it relates to secondary architectures. This will only let us easily do x86 & ppc. If we want sparc, arm or ia64 BIOS, the builds would happen in the 2nd arch build ssytem, and the result would not be available in our main Fedora YUM repos for primary archs. If we go with your current approach, we can do the build in the 2nd-ary arch, and then pull the result back into the packages on the primary arches
I don't know if I understood correctly. This means that the new feature could solve our problems, but it won't solve all of them because some arches where we need some binaries to be built are second-class citzens on the build system?
Yes, that is essentially it. The 2nd-ary architectures have completely separate build system, and separate YUM repository composes. They're basically a downstream distro, that just gets triggered whenever the primary arch builds. Similarly the sparc 2ndary arch wouldn't have access to the '-noarch' x86 firmware we'd build. So for a complete solutuon I think the manual double build + tar.gz of just build blobs is probably the only way we'll get full coverage for all the QEMU arches.
Daniel
On Fri, Feb 13, 2009 at 01:09:53PM +0000, Daniel P. Berrange wrote:
On Fri, Feb 13, 2009 at 10:59:04AM -0200, Glauber Costa wrote:
We have a problem with any of the above approaches: we don't want to make the first build a scratch build (we want to keep the build information stored somewhere on Koji), but we don't want this build to be available for the users, as the result is an incomplete package. I am thinking about asking a Koji tag to be created just to store those "intermediate build step" RPMs. What do you think?
We add a "make update-binaries" the makefile - it would do:
$> koji download-build $(make verrel) $> rpm2cpio ... | cpio $> tar -cvjf ... $> grep -v etherboot-binaries sources > sources.tmp $> mv sources.tmp sources $> sed -i -e ... etherboot.spec $> make upload FILES=...
+1. This should be automated as much as possible.
I've talked to some people in #fedora-devel yesterday, for _hours_.
It turns out that they're planning on deploying some features in koji by Feb 28th, that _might_ help us a lot. Really, a lot.
The idea is that you can have your package built on an architecture exclusively with BuildArch directive. But your package can then have a subpackage with BuildArch: noarch. That way, your package will span all repositories. This seems like the perfect solution to us. I've helped them to proceed with some tests, and it seem to work. So what I'm plannin on doing, is post the rpms for review today, since it is unlikely that they will receive more updates in this mean time.
The main limitation with that approach is the architecture coverage, and the way it relates to secondary architectures. This will only let us easily do x86 & ppc. If we want sparc, arm or ia64 BIOS, the builds would happen in the 2nd arch build ssytem, and the result would not be available in our main Fedora YUM repos for primary archs. If we go with your current approach, we can do the build in the 2nd-ary arch, and then pull the result back into the packages on the primary arches
qemu currently does not have any kind of issue with arm. for ia64, there's no qemu, just kvm. So if we need, we'll just build locally.
The only real issue is sparc, and in this case, yeah, bummer, but we can still use the binaries approach.
Still leaves us with etherboot, bochs-bios and vga-bios. There are three packages in which we can do a much better job. Can't see why hold them because of the sparc package.
Daniel
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Fri, 2009-02-13 at 10:59 -0200, Glauber Costa wrote:
The idea is that you can have your package built on an architecture exclusively with BuildArch directive. But your package can then have a subpackage with BuildArch: noarch. That way, your package will span all repositories. This seems like the perfect solution to us.
Yep, that would be ideal ... as long as people don't take to throw things at us for shipping arch specific binaries in a noarch package :-)
Seriously, it sounds like it should work really well.
I've helped them to proceed with some tests, and it seem to work. So what I'm plannin on doing, is post the rpms for review today, since it is unlikely that they will receive more updates in this mean time.
Getting them review and approved is the really important bits. We can set on the ideal solution a bit later.
Yep, agree.
Cheers, Mark.
On Fri, Feb 13, 2009 at 01:46:49PM +0000, Mark McLoughlin wrote:
On Fri, 2009-02-13 at 10:59 -0200, Glauber Costa wrote:
The idea is that you can have your package built on an architecture exclusively with BuildArch directive. But your package can then have a subpackage with BuildArch: noarch. That way, your package will span all repositories. This seems like the perfect solution to us.
Yep, that would be ideal ... as long as people don't take to throw things at us for shipping arch specific binaries in a noarch package :-)
Seriously, it sounds like it should work really well.
I've helped them to proceed with some tests, and it seem to work. So what I'm plannin on doing, is post the rpms for review today, since it is unlikely that they will receive more updates in this mean time.
Getting them review and approved is the really important bits. We can set on the ideal solution a bit later.
Yep, agree.
Please people: https://bugzilla.redhat.com/show_bug.cgi?id=485417 https://bugzilla.redhat.com/show_bug.cgi?id=485418 https://bugzilla.redhat.com/show_bug.cgi?id=485420
You can help me out in this enterprise by reviewing the aforementioned BZs.
On Fri, 2009-02-13 at 10:49 -0200, Eduardo Habkost wrote:
- First build: all prebuilt binaries tarballs removed from the spec,
only native binaries built (using something like %define only_native 1)
- Second build: added the prebuilt binaries from the first build, no binaries actually compiled, but uses only the prebuilt binaries
Yes, that's what I was thinking.
We have a problem with any of the above approaches: we don't want to make the first build a scratch build (we want to keep the build information stored somewhere on Koji),
Agreed.
but we don't want this build to be available for the users, as the result is an incomplete package.
If you automate it so that the builds happen one after the other, it's going to be highly unlikely that they e.g. get pulled in by a rawhide compose.
I am thinking about asking a Koji tag to be created just to store those "intermediate build step" RPMs. What do you think?
Perhaps, I'd only worry about it if we ever actually hit problems, though.
Cheers, Mark.
-
Daniel P. Berrange -
Eduardo Habkost -
Glauber Costa -
Mark McLoughlin