Is there a way to deal with the following situation in Koji?
· Build tool B has a build-time dependency on itself. · B is linked to library L version 1. · L gets upgraded to version 2, which changes its soname.
B needs to be rebuilt to link to libL.so.2, but building B requires a working B, which requires libL.so.1 because it hasn't been rebuilt yet.
As I understand it, when the L-2 package goes into the buildroot it immediately replaces L-1. Is there a way to keep L-1 available until B has been rebuilt?
Is the answer to link B statically?
Björn Persson
On Sun, 29 Nov 2015 00:35:37 +0100 Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
Is there a way to deal with the following situation in Koji?
· Build tool B has a build-time dependency on itself. · B is linked to library L version 1. · L gets upgraded to version 2, which changes its soname.
B needs to be rebuilt to link to libL.so.2, but building B requires a working B, which requires libL.so.1 because it hasn't been rebuilt yet.
As I understand it, when the L-2 package goes into the buildroot it immediately replaces L-1. Is there a way to keep L-1 available until B has been rebuilt?
Nope.
Is the answer to link B statically?
Nope.
The usual way to handle this is to add a compat version of the library (so.1) to L, rebuild B, then drop the compat library, ie just keep it for one build to rebuild B.
I think for example libpng has used this in the past if you want an example.
kevin
Kevin Fenzi wrote:
On Sun, 29 Nov 2015 00:35:37 +0100 Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
Is there a way to deal with the following situation in Koji?
· Build tool B has a build-time dependency on itself. · B is linked to library L version 1. · L gets upgraded to version 2, which changes its soname.
B needs to be rebuilt to link to libL.so.2, but building B requires a working B, which requires libL.so.1 because it hasn't been rebuilt yet.
As I understand it, when the L-2 package goes into the buildroot it immediately replaces L-1. Is there a way to keep L-1 available until B has been rebuilt?
Nope.
Is the answer to link B statically?
Nope.
The usual way to handle this is to add a compat version of the library (so.1) to L, rebuild B, then drop the compat library, ie just keep it for one build to rebuild B.
So the L source package shall temporarily contain two versions of the source tarball and build them both, and then install one as usual and extract only the binary library from the other? That would complicate the spec quite a bit even in an otherwise simple package, and when the spec is already big and complex ...
Time to get specific. L is libgnat, a subpackage of GCC. B is GPRbuild, the builder that builds the Ada packages. GPRbuild is linked to XMLada, and both GPRbuild and XMLada are linked to libgnat, as are all other Ada programs and libraries.
With every major GCC upgrade the soname of libgnat changes, and all the Ada packages stop working until they get rebuilt. But rebuilding requires a working GPRbuild.
Previously a catch-22 was avoided because XMLada and GPRbuild were built with Gnatmake, which is built as a part of GCC. Now they're both built with GPRbuild instead, and Gnatmake is emitting warnings that it's going to lose support for the project files that control the build. The next GCC upgrade will make the problem acute.
So now you're saying that the GCC package, whose spec is already 3000 lines long, needs to contain an entire additional GCC and build large parts of it, just to work around a limitation in Koji? I can of course ask the GCC maintainer, but I fully expect that he'll refuse.
And then XMLada must be rebuilt before GPRbuild can be rebuilt. This XMLada build will have to provide a compatibility instance of XMLada linked to the old libgnat, because otherwise both versions of libgnat would be loaded into GPRbuild the next time it runs.
For that to be possible, the compatibility version of libgnat must include not only the binary library, but also the old version of the entire content of libgnat-devel, so that the compatibility instance of XMLada can be linked. That in turn means that the old version of the compiler itself must be provided, because GNAT performs consistency checks and won't link code compiled with one version of GNAT to a library compiled with another version of GNAT.
In short, this seem like an enormous amount of trouble, mostly in the GCC pacakage. It would be so much easier if the already existing libraries could remain available for a limited time.
Björn Persson
On Tue, 1 Dec 2015 15:46:26 +0100 Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
So the L source package shall temporarily contain two versions of the source tarball and build them both, and then install one as usual and extract only the binary library from the other? That would complicate the spec quite a bit even in an otherwise simple package, and when the spec is already big and complex ...
Time to get specific. L is libgnat, a subpackage of GCC. B is GPRbuild, the builder that builds the Ada packages. GPRbuild is linked to XMLada, and both GPRbuild and XMLada are linked to libgnat, as are all other Ada programs and libraries.
With every major GCC upgrade the soname of libgnat changes, and all the Ada packages stop working until they get rebuilt. But rebuilding requires a working GPRbuild.
Previously a catch-22 was avoided because XMLada and GPRbuild were built with Gnatmake, which is built as a part of GCC. Now they're both built with GPRbuild instead, and Gnatmake is emitting warnings that it's going to lose support for the project files that control the build. The next GCC upgrade will make the problem acute.
How are others supposed to handle this?
So now you're saying that the GCC package, whose spec is already 3000 lines long, needs to contain an entire additional GCC and build large parts of it, just to work around a limitation in Koji? I can of course ask the GCC maintainer, but I fully expect that he'll refuse.
I'm just explaining how koji works here, don't shoot me. ;)
So, ideally here, there's a new gcc upgrade, the gcc maintainer would build with a compat-libgnat subpackage that installs libgnat in another place. Then they build the new gcc without it. You then can buildrequire that compat-libgnat so you can rebuild other things, then finally do another rebuild without compat-libgnat to bring everything up on the current gcc.
And then XMLada must be rebuilt before GPRbuild can be rebuilt. This XMLada build will have to provide a compatibility instance of XMLada linked to the old libgnat, because otherwise both versions of libgnat would be loaded into GPRbuild the next time it runs.
For that to be possible, the compatibility version of libgnat must include not only the binary library, but also the old version of the entire content of libgnat-devel, so that the compatibility instance of XMLada can be linked. That in turn means that the old version of the compiler itself must be provided, because GNAT performs consistency checks and won't link code compiled with one version of GNAT to a library compiled with another version of GNAT.
In short, this seem like an enormous amount of trouble, mostly in the GCC pacakage. It would be so much easier if the already existing libraries could remain available for a limited time.
I don't see any way to do that with the current setup.
kevin
Kevin Fenzi kevin@scrye.com wrote:
On Tue, 1 Dec 2015 15:46:26 +0100 Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
GPRbuild is linked to XMLada, and both GPRbuild and XMLada are linked to libgnat, as are all other Ada programs and libraries.
With every major GCC upgrade the soname of libgnat changes, and all the Ada packages stop working until they get rebuilt. But rebuilding requires a working GPRbuild.
Previously a catch-22 was avoided because XMLada and GPRbuild were built with Gnatmake, which is built as a part of GCC. Now they're both built with GPRbuild instead, and Gnatmake is emitting warnings that it's going to lose support for the project files that control the build. The next GCC upgrade will make the problem acute.
How are others supposed to handle this?
Other users of the GNAT toolchain? Well, if they don't use Koji then they can simply keep the old libgnat around, alongside the new libgnat and the rest of the new GCC, until XMLada and GPRbuild have been rebuilt. On the file level there is no problem. The library filenames are versioned so they can coexist. The soname ensures that the right one will be loaded when GPRbuild runs, and the symlink "libgnat.so" ensures that linking will be done with the newest library.
This works on the RPM level too by the way. Two libgnat packages can be installed at the same time.
Handling XMLada is slightly trickier. The rebuilt XMLada will have the same soname as the old one (unless they upgrade both GCC and XMLada at the same time), so they need to install it to a staging directory and add that directory to the search path when rebuilding GPRbuild. Then the old XMLada and the old libgnat will be loaded when the old GPRbuild runs, and the rebuilt GPRbuild will be linked to the rebuilt XMLada, which is linked to the new libgnat. Then they can copy the rebuilt GPRbuild and the rebuilt XMLada into place together, and remove the old libraries.
With a little patch to a project file it would also be possible to add a suffix to the soname of XMLada. Then the two instances of XMLada could coexist the same way as the two versions of libgnat, and the staging directory wouldn't be needed.
They can also get rid of the whole problem by linking GPRbuild statically. This seems to be what Adacore themselves do. The compiled GPRbuild they distribute requires only libpthread.so.0, librt.so.1 and libc.so.6, so libgnat and XMLada must have been statically linked in.
So now you're saying that the GCC package, whose spec is already 3000 lines long, needs to contain an entire additional GCC and build large parts of it, just to work around a limitation in Koji? I can of course ask the GCC maintainer, but I fully expect that he'll refuse.
I'm just explaining how koji works here, don't shoot me. ;)
So, ideally here, there's a new gcc upgrade, the gcc maintainer would build with a compat-libgnat subpackage that installs libgnat in another place. Then they build the new gcc without it. You then can buildrequire that compat-libgnat so you can rebuild other things, then finally do another rebuild without compat-libgnat to bring everything up on the current gcc.
If I understand this correctly, you mean that the compat-libgnat subpackage would remain in the buildroot even after the other subpackages were replaced with the later build. Is that right? I always thought that when packages are tagged into buildroots, side tags and stuff, then all the subpackages from a single source package are added or removed together. If it's done with subpackage granularity, then that makes things a bit easier.
Then I have some questions about that method:
Why should libgnat be installed in another place? It seems to me that it could be in /usr/lib64 as usual, thanks to the versioned filenames.
Would an explicit "BuildRequires: compat-libgnat" be needed? Wouldn't it automatically provide the soname as usual?
Would someone need to manually remove the compat package from the buildroot afterwards? Is that what usually happens when a package drops a subpackage, that the subpackage lingers in the buildroot unless someone removes it?
Björn Persson
On Mon, 7 Dec 2015 11:32:44 +0100 Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
Kevin Fenzi kevin@scrye.com wrote:
On Tue, 1 Dec 2015 15:46:26 +0100 Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
GPRbuild is linked to XMLada, and both GPRbuild and XMLada are linked to libgnat, as are all other Ada programs and libraries.
With every major GCC upgrade the soname of libgnat changes, and all the Ada packages stop working until they get rebuilt. But rebuilding requires a working GPRbuild.
Previously a catch-22 was avoided because XMLada and GPRbuild were built with Gnatmake, which is built as a part of GCC. Now they're both built with GPRbuild instead, and Gnatmake is emitting warnings that it's going to lose support for the project files that control the build. The next GCC upgrade will make the problem acute.
How are others supposed to handle this?
Other users of the GNAT toolchain? Well, if they don't use Koji then
maybe Kevin meant other distros - Debian, OpenSUSE - they all use a buildsystem
they can simply keep the old libgnat around, alongside the new libgnat and the rest of the new GCC, until XMLada and GPRbuild have been rebuilt. On the file level there is no problem. The library filenames are versioned so they can coexist. The soname ensures that the right one will be loaded when GPRbuild runs, and the symlink "libgnat.so" ensures that linking will be done with the newest library.
This works on the RPM level too by the way. Two libgnat packages can be installed at the same time.
Handling XMLada is slightly trickier. The rebuilt XMLada will have the same soname as the old one (unless they upgrade both GCC and XMLada at the same time), so they need to install it to a staging directory and add that directory to the search path when rebuilding GPRbuild. Then the old XMLada and the old libgnat will be loaded when the old GPRbuild runs, and the rebuilt GPRbuild will be linked to the rebuilt XMLada, which is linked to the new libgnat. Then they can copy the rebuilt GPRbuild and the rebuilt XMLada into place together, and remove the old libraries.
With a little patch to a project file it would also be possible to add a suffix to the soname of XMLada. Then the two instances of XMLada could coexist the same way as the two versions of libgnat, and the staging directory wouldn't be needed.
They can also get rid of the whole problem by linking GPRbuild statically. This seems to be what Adacore themselves do. The compiled GPRbuild they distribute requires only libpthread.so.0, librt.so.1 and libc.so.6, so libgnat and XMLada must have been statically linked in.
I would consider static linking for GPRbuild in this situation acceptable
So now you're saying that the GCC package, whose spec is already 3000 lines long, needs to contain an entire additional GCC and build large parts of it, just to work around a limitation in Koji? I can of course ask the GCC maintainer, but I fully expect that he'll refuse.
I'm just explaining how koji works here, don't shoot me. ;)
So, ideally here, there's a new gcc upgrade, the gcc maintainer would build with a compat-libgnat subpackage that installs libgnat in another place. Then they build the new gcc without it. You then can buildrequire that compat-libgnat so you can rebuild other things, then finally do another rebuild without compat-libgnat to bring everything up on the current gcc.
If I understand this correctly, you mean that the compat-libgnat subpackage would remain in the buildroot even after the other subpackages were replaced with the later build. Is that right? I always thought that when packages are tagged into buildroots, side tags and stuff, then all the subpackages from a single source package are added or removed together. If it's done with subpackage granularity, then that makes things a bit easier.
Then I have some questions about that method:
Why should libgnat be installed in another place? It seems to me that it could be in /usr/lib64 as usual, thanks to the versioned filenames.
Would an explicit "BuildRequires: compat-libgnat" be needed? Wouldn't it automatically provide the soname as usual?
no, the binary will bring the compat library via the automaticaly managed soname Requires/Provides
Would someone need to manually remove the compat package from the buildroot afterwards? Is that what usually happens when a package drops a subpackage, that the subpackage lingers in the buildroot unless someone removes it?
no, the buildroots as created by resolving Requires and Provides, if nothing (no binary) will use the library with the old soname, it won't be be dragged into the buildroot
Dan
Dan Horák wrote:
On Mon, 7 Dec 2015 11:32:44 +0100 Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
Kevin Fenzi kevin@scrye.com wrote:
On Tue, 1 Dec 2015 15:46:26 +0100 Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
GPRbuild is linked to XMLada, and both GPRbuild and XMLada are linked to libgnat, as are all other Ada programs and libraries.
With every major GCC upgrade the soname of libgnat changes, and all the Ada packages stop working until they get rebuilt. But rebuilding requires a working GPRbuild.
Previously a catch-22 was avoided because XMLada and GPRbuild were built with Gnatmake, which is built as a part of GCC. Now they're both built with GPRbuild instead, and Gnatmake is emitting warnings that it's going to lose support for the project files that control the build. The next GCC upgrade will make the problem acute.
How are others supposed to handle this?
Other users of the GNAT toolchain? Well, if they don't use Koji then
maybe Kevin meant other distros - Debian, OpenSUSE - they all use a buildsystem
OK, if I read the question as "How does Adacore expect Free Software distributions to handle this?", then the answer is that I think they don't care much.
Adacore's paying customers are big corporations and institutions. Those generally handle tools and libraries in the form of relocatable packages that they unpack in some directory in their networked filesystem. They can have multiple versions of each package available, and manipulate environment variables to point to the one they want to run or link to. That's the use case that Adacore care about.
Adacore do set their software free, but if you don't pay for a support contract then you pretty much have to take it or leave it. They may accept patches, but only if the proposed changes don't inconvenience them too much. If they have decided that maintaining duplicated code for project file support in both Gnatmake and GPRbuild is too much trouble, then they won't change their minds just to make things easier for Free Software distributions.
If I would complain, then they would probably say that they'll ensure that each version of GPRbuild can be built by the previous version, and if Fedora's build system can't handle that, then that's Fedora's problem. And they would be right.
Björn Persson
Björn Persson wrote:
Kevin Fenzi kevin@scrye.com wrote:
So, ideally here, there's a new gcc upgrade, the gcc maintainer would build with a compat-libgnat subpackage that installs libgnat in another place. Then they build the new gcc without it. You then can buildrequire that compat-libgnat so you can rebuild other things, then finally do another rebuild without compat-libgnat to bring everything up on the current gcc.
If I understand this correctly, you mean that the compat-libgnat subpackage would remain in the buildroot even after the other subpackages were replaced with the later build. Is that right? I always thought that when packages are tagged into buildroots, side tags and stuff, then all the subpackages from a single source package are added or removed together. If it's done with subpackage granularity, then that makes things a bit easier.
I made an experiment to test this. I dropped a subpackage and rebuilt. Once that build was available in Rawhide I tried to rebuild another package that had a build-time dependency on the dropped subpackage. The build failed because the subpackage wasn't found. So it is as I thought: Each new build replaces all subpackages from the previous build, even those that weren't produced in the new build.
Thus it seems that linking GPRbuild statically is the only solution that is even close to practical, given the limitations of Koji. I'll coordinate with Pavel and get to work on it.
Björn Persson
-
Björn Persson -
Dan Horák -
Kevin Fenzi