Panu Matilainen schreef op vr 26-11-2010 om 13:20 [+0200]:
In particular, I'm interested in feedback on the new, pluggable and enhanced dependency extration system. Documentation is scarce at the moment but some background and examples can be found here: http://laiskiainen.org/blog/?p=35
All mingw32 packages in Fedora contain these set of instructions in the .spec files:
%global _use_internal_dependency_generator 0 %global __find_requires %{_mingw32_findrequires} %global __find_provides %{_mingw32_findprovides}
Does this new dependency extraction system make these kind of instructions obsolete?
If I understand your blog entry correctly then we (the Fedora MinGW SIG) are recommended to use something like this:
%__mingw32_provides %{_mingw32_findprovides} %__mingw32_requires %{_mingw32_findrequires}
Is this correct or do you recommend something different?
The macros %{_mingw32_findrequires} and %{_mingw32_findprovides} are mentioned in the file /etc/rpm/macros.mingw32 which is part of the mingw32-filesystem package. Both refer to a small shell script which uses the i686-pc-mingw32-objdump tool to extract dependency information.
Kind regards,
Erik van Pienbroek
On Tue, 30 Nov 2010, Erik van Pienbroek wrote:
Panu Matilainen schreef op vr 26-11-2010 om 13:20 [+0200]:
In particular, I'm interested in feedback on the new, pluggable and enhanced dependency extration system. Documentation is scarce at the moment but some background and examples can be found here: http://laiskiainen.org/blog/?p=35
All mingw32 packages in Fedora contain these set of instructions in the .spec files:
%global _use_internal_dependency_generator 0 %global __find_requires %{_mingw32_findrequires} %global __find_provides %{_mingw32_findprovides}
Does this new dependency extraction system make these kind of instructions obsolete?
The above still works with the new rpm version.
If I understand your blog entry correctly then we (the Fedora MinGW SIG) are recommended to use something like this:
%__mingw32_provides %{_mingw32_findprovides} %__mingw32_requires %{_mingw32_findrequires}
Is this correct or do you recommend something different?
That alone wont do anything at all, to create a new "file attribute" called mingw32 you'd add a file like this to a suitable package, mingw32-filesystem probably:
/usr/lib/rpm/fileattrs/mingw32.attr:
%__mingw32_requires /usr/lib/rpm/mingw32-find-requires.sh %__mingw32_provides /usr/lib/rpm/mingw32-find-provides.sh %__mingw32_magic ^PE32 executable for MS Windows.* 80386 32-bit$
The magic rule is based on what 'file -b <file>' outputs for mingw32 executables and dll's - the above includes both, but you can make it tighter to only include DLL's or have different extractors for DLL's and EXE's by creating two rules instead of just one etc. Or if libmagic strings aren't good ("fakedll" binaries from Wine would match the above rule), path based regexes can be used too. It all depends on what makes sense in a given scenario.
You could also easily have a mingw32-specific pkg-config dependency extractor which uses a different namespace than the regular pkgconfig(foo) and only activates on .pc files from the mingw32 sys-root directory.
And with necessary mingw32-specific .attr files in place through buildrequires, there's no need for override kludges in each and every mingw32 spec.
- Panu -
Panu Matilainen schreef op di 30-11-2010 om 22:10 [+0200]:
On Tue, 30 Nov 2010, Erik van Pienbroek wrote:
If I understand your blog entry correctly then we (the Fedora MinGW SIG) are recommended to use something like this:
%__mingw32_provides %{_mingw32_findprovides} %__mingw32_requires %{_mingw32_findrequires}
Is this correct or do you recommend something different?
That alone wont do anything at all, to create a new "file attribute" called mingw32 you'd add a file like this to a suitable package, mingw32-filesystem probably:
/usr/lib/rpm/fileattrs/mingw32.attr:
%__mingw32_requires /usr/lib/rpm/mingw32-find-requires.sh %__mingw32_provides /usr/lib/rpm/mingw32-find-provides.sh %__mingw32_magic ^PE32 executable for MS Windows.* 80386 32-bit$
The magic rule is based on what 'file -b <file>' outputs for mingw32 executables and dll's - the above includes both, but you can make it tighter to only include DLL's or have different extractors for DLL's and EXE's by creating two rules instead of just one etc. Or if libmagic strings aren't good ("fakedll" binaries from Wine would match the above rule), path based regexes can be used too. It all depends on what makes sense in a given scenario.
You could also easily have a mingw32-specific pkg-config dependency extractor which uses a different namespace than the regular pkgconfig(foo) and only activates on .pc files from the mingw32 sys-root directory.
And with necessary mingw32-specific .attr files in place through buildrequires, there's no need for override kludges in each and every mingw32 spec.
Ah yes, thanks for the detailed information. This sure looks interesting for us! I'll try to play around a bit with this in a mock environment and see if it's possible to update our packaging guidelines to make use of it.
Regards,
Erik van Pienbroek
On Tue, 30 Nov 2010, Erik van Pienbroek wrote:
Panu Matilainen schreef op di 30-11-2010 om 22:10 [+0200]:
On Tue, 30 Nov 2010, Erik van Pienbroek wrote:
If I understand your blog entry correctly then we (the Fedora MinGW SIG) are recommended to use something like this:
%__mingw32_provides %{_mingw32_findprovides} %__mingw32_requires %{_mingw32_findrequires}
Is this correct or do you recommend something different?
That alone wont do anything at all, to create a new "file attribute" called mingw32 you'd add a file like this to a suitable package, mingw32-filesystem probably:
/usr/lib/rpm/fileattrs/mingw32.attr:
%__mingw32_requires /usr/lib/rpm/mingw32-find-requires.sh %__mingw32_provides /usr/lib/rpm/mingw32-find-provides.sh %__mingw32_magic ^PE32 executable for MS Windows.* 80386 32-bit$
The magic rule is based on what 'file -b <file>' outputs for mingw32 executables and dll's - the above includes both, but you can make it tighter to only include DLL's or have different extractors for DLL's and EXE's by creating two rules instead of just one etc. Or if libmagic strings aren't good ("fakedll" binaries from Wine would match the above rule), path based regexes can be used too. It all depends on what makes sense in a given scenario.
You could also easily have a mingw32-specific pkg-config dependency extractor which uses a different namespace than the regular pkgconfig(foo) and only activates on .pc files from the mingw32 sys-root directory.
And with necessary mingw32-specific .attr files in place through buildrequires, there's no need for override kludges in each and every mingw32 spec.
Ah yes, thanks for the detailed information. This sure looks interesting for us! I'll try to play around a bit with this in a mock environment and see if it's possible to update our packaging guidelines to make use of it.
Do play around with it if you can, but I'd suggest leaving the packaging guideline considerations until the new rpm has actually landed AND gotten past the final feature acceptance line. The mechanism is likely to grow some enhancements before that, and also more than just mingw32 guidelines should be revisited + revised then (and might well be F16 material)
- Panu -
Hello,
I poked a bit at the new RPM 4.9 dependency extraction system and it looks really nice. Right now we have the following boilerplate macros at the top of each and every mingw32- spec file:
%global _use_internal_dependency_generator 0 %global __find_requires %{_mingw32_findrequires} %global __find_provides %{_mingw32_findprovides}
With RPM 4.9 we can, finally, get rid of them.
The way the new dependency extraction system works is that we would drop a single mingw32.attr file in /usr/lib/rpm/fileattrs/, which defines a new "file attribute" definition. There are both file magic and path constraints, and if both match, our custom provides and requires extraction scripts are automatically called. This is the contents of the mingw32.attr file: %__mingw32_provides %{_rpmconfigdir}/mingw32-find-provides.sh %__mingw32_requires %{_rpmconfigdir}/mingw32-find-requires.sh %__mingw32_magic ^PE32 executable.* Intel 80386.*, for MS Windows$ %__mingw32_path ^%{_mingw32_prefix}/.*$
It should also be possible to use the new dependency extraction system in the mingw32-gcc package where we previously had to resort to manual mingw32(*) provides. The reason we previously couldn't define '_use_internal_dependency_generator 0' in that package is that the mingw32-gcc binary packages mixed both native binaries and cross compiled binaries; turning off dependency generator and replacing it with a mingw32 one would have resulted in no dependency generation for native binaries. But this should all work automagically now.
I am planning to add the new mingw32.attr file to the mingw32-filesystem package in rawhide soon. Any objections?
It might also make sense to mass rebuild all the mingw32 packages in rawhide with the boilerplate macros removed. This is also something I could do; would it be a good idea?
If it all works out nicely, we could also backport the mingw32-filesystem related changes to F15 (but not rebuild any other packages). The rationale for doing so is that packagers could then keep the F15 and rawhide spec files in sync, if they would want to do so. I would also like to note that supporting the new dependency extraction system in mingw32-filesystem shouldn't break existing packages that are still using the boilerplate macros at the top of the spec files.
Comments?
Oh, and thanks go to Panu Matilainen for the new RPM feature.
On 05/19/2011 01:49 PM, Kalev Lember wrote:
Hello,
I poked a bit at the new RPM 4.9 dependency extraction system and it looks really nice. Right now we have the following boilerplate macros at the top of each and every mingw32- spec file:
%global _use_internal_dependency_generator 0 %global __find_requires %{_mingw32_findrequires} %global __find_provides %{_mingw32_findprovides}
With RPM 4.9 we can, finally, get rid of them.
The way the new dependency extraction system works is that we would drop a single mingw32.attr file in /usr/lib/rpm/fileattrs/, which defines a new "file attribute" definition. There are both file magic and path constraints, and if both match, our custom provides and requires extraction scripts are automatically called. This is the contents of the mingw32.attr file: %__mingw32_provides %{_rpmconfigdir}/mingw32-find-provides.sh %__mingw32_requires %{_rpmconfigdir}/mingw32-find-requires.sh %__mingw32_magic ^PE32 executable.* Intel 80386.*, for MS Windows$ %__mingw32_path ^%{_mingw32_prefix}/.*$
Note that this doesn't work quite as you want it to, just yet: in rpm 4.9.0 path and magic are handled as an inclusive or, whereas this (and various other things) needs an and-rule. Adding support for this is technically trivial, just the naming is under consideration: http://lists.rpm.org/pipermail/rpm-maint/2011-May/003022.html
It should also be possible to use the new dependency extraction system in the mingw32-gcc package where we previously had to resort to manual mingw32(*) provides. The reason we previously couldn't define '_use_internal_dependency_generator 0' in that package is that the mingw32-gcc binary packages mixed both native binaries and cross compiled binaries; turning off dependency generator and replacing it with a mingw32 one would have resulted in no dependency generation for native binaries. But this should all work automagically now.
I am planning to add the new mingw32.attr file to the mingw32-filesystem package in rawhide soon. Any objections?
Depends on how soon your "soon" is :) - you'll probably want to wait until the and-rule support lands in rawhide rpm. OTOH if you're in hurry, I guess you could get away with just defining mingw32_magic and leaving the path out for now, Wine is probably the only thing besides mingw32 having Windows PE32 executables (and you could have an exclude-rule for the wine-paths just to be on the safe side).
It might also make sense to mass rebuild all the mingw32 packages in rawhide with the boilerplate macros removed. This is also something I could do; would it be a good idea?
If it all works out nicely, we could also backport the mingw32-filesystem related changes to F15 (but not rebuild any other packages). The rationale for doing so is that packagers could then keep the F15 and rawhide spec files in sync, if they would want to do so. I would also like to note that supporting the new dependency extraction system in mingw32-filesystem shouldn't break existing packages that are still using the boilerplate macros at the top of the spec files.
Comments?
Oh, and thanks go to Panu Matilainen for the new RPM feature.
Nice to see folks finding use for it :)
- Panu -
On 05/19/2011 02:19 PM, Panu Matilainen wrote:
On 05/19/2011 01:49 PM, Kalev Lember wrote:
This is the contents of the mingw32.attr file: %__mingw32_provides %{_rpmconfigdir}/mingw32-find-provides.sh %__mingw32_requires %{_rpmconfigdir}/mingw32-find-requires.sh %__mingw32_magic ^PE32 executable.* Intel 80386.*, for MS Windows$ %__mingw32_path ^%{_mingw32_prefix}/.*$
Note that this doesn't work quite as you want it to, just yet: in rpm 4.9.0 path and magic are handled as an inclusive or, whereas this (and various other things) needs an and-rule. Adding support for this is technically trivial, just the naming is under consideration: http://lists.rpm.org/pipermail/rpm-maint/2011-May/003022.html
Ah, good catch. I would love to see the "and-rule" in RPM, it would certainly make things easier.
I am planning to add the new mingw32.attr file to the mingw32-filesystem package in rawhide soon. Any objections?
Depends on how soon your "soon" is :) - you'll probably want to wait until the and-rule support lands in rawhide rpm. OTOH if you're in hurry, I guess you could get away with just defining mingw32_magic and leaving the path out for now, Wine is probably the only thing besides mingw32 having Windows PE32 executables (and you could have an exclude-rule for the wine-paths just to be on the safe side).
Mono assemblies also have a very similar magic, which is somewhat hard to discern from the usual PE32 magic.
mingw32 cross compiled zlib1.dll: PE32 executable (DLL) (console) Intel 80386 (stripped to external PDB), for MS Windows Mono assembly nunit.core.dll: PE32 executable (DLL) (console) Intel 80386 Mono/.Net assembly, for MS Windows
If we currently don't have the "and-rule", I would go for a simple path based regex that matches all the .dll and .exe files in the mingw32 sysroot and forget about the libmagic matches for now. Once the "and-rule" makes it in rawhide, we can switch over to using that.
The advantages with plain path based regexes are: - It's similar to how we are matching the files right now, so it would be harder to regress something; - It would work with RPM 4.9.0 and we would be able to backport the mingw32-filesystem changes to F15 to provide compatibility for maintainers who want to keep F15 and rawhide spec files in sync; - We wouldn't have to worry about excluding strange paths where some .dll files might be, like /usr/share/.
Depends on how soon your "soon" is :)
I would actually like to get the fileattr dependency extraction in rather sooner than later. There's no guarantee I would have time to work on it in a month or two and I'd rather finish it now when I have the time.
On 05/19/2011 08:25 PM, Kalev Lember wrote:
On 05/19/2011 02:19 PM, Panu Matilainen wrote:
On 05/19/2011 01:49 PM, Kalev Lember wrote:
This is the contents of the mingw32.attr file: %__mingw32_provides %{_rpmconfigdir}/mingw32-find-provides.sh %__mingw32_requires %{_rpmconfigdir}/mingw32-find-requires.sh %__mingw32_magic ^PE32 executable.* Intel 80386.*, for MS Windows$ %__mingw32_path ^%{_mingw32_prefix}/.*$
Note that this doesn't work quite as you want it to, just yet: in rpm 4.9.0 path and magic are handled as an inclusive or, whereas this (and various other things) needs an and-rule. Adding support for this is technically trivial, just the naming is under consideration: http://lists.rpm.org/pipermail/rpm-maint/2011-May/003022.html
Ah, good catch. I would love to see the "and-rule" in RPM, it would certainly make things easier.
I am planning to add the new mingw32.attr file to the mingw32-filesystem package in rawhide soon. Any objections?
Depends on how soon your "soon" is :) - you'll probably want to wait until the and-rule support lands in rawhide rpm. OTOH if you're in hurry, I guess you could get away with just defining mingw32_magic and leaving the path out for now, Wine is probably the only thing besides mingw32 having Windows PE32 executables (and you could have an exclude-rule for the wine-paths just to be on the safe side).
Mono assemblies also have a very similar magic, which is somewhat hard to discern from the usual PE32 magic.
mingw32 cross compiled zlib1.dll: PE32 executable (DLL) (console) Intel 80386 (stripped to external PDB), for MS Windows Mono assembly nunit.core.dll: PE32 executable (DLL) (console) Intel 80386 Mono/.Net assembly, for MS Windows
Ah, mono too. Those could be filtered out with an exclude_magic rule on Mono/.Net but...
If we currently don't have the "and-rule", I would go for a simple path based regex that matches all the .dll and .exe files in the mingw32 sysroot and forget about the libmagic matches for now. Once the "and-rule" makes it in rawhide, we can switch over to using that.
The advantages with plain path based regexes are:
- It's similar to how we are matching the files right now, so it would be harder to regress something;
- It would work with RPM 4.9.0 and we would be able to backport the mingw32-filesystem changes to F15 to provide compatibility for maintainers who want to keep F15 and rawhide spec files in sync;
- We wouldn't have to worry about excluding strange paths where some .dll files might be, like /usr/share/.
Indeed, for something like mingw32 where everything lives in a distinct directory structure with easily recognizable suffixes, additional magic-rules adds little value over plain path-based regexes.
Depends on how soon your "soon" is :)
I would actually like to get the fileattr dependency extraction in rather sooner than later. There's no guarantee I would have time to work on it in a month or two and I'd rather finish it now when I have the time.
Ok. Well, if only path based regexes are used then that will work right now in both rawhide and F15.
- Panu -
On 05/19/2011 08:52 PM, Panu Matilainen wrote:
Indeed, for something like mingw32 where everything lives in a distinct directory structure with easily recognizable suffixes, additional magic-rules adds little value over plain path-based regexes.
Thanks Panu.
After discussing it with Erik van Pienbroek and Richard W.M. Jones on IRC, I've built mingw32-filesystem with the new dep extraction support in rawhide.
This is how the mingw32.attr file now looks:
%__mingw32_provides %{_rpmconfigdir}/mingw32-find-provides.sh %__mingw32_requires %{_rpmconfigdir}/mingw32-find-requires.sh %__mingw32_path ^%{_mingw32_prefix}/.*.([Dd][Ll][Ll]|[Ee][Xx][Ee])$
On 05/19/2011 08:25 PM, Kalev Lember wrote:
On 05/19/2011 02:19 PM, Panu Matilainen wrote:
On 05/19/2011 01:49 PM, Kalev Lember wrote:
This is the contents of the mingw32.attr file: %__mingw32_provides %{_rpmconfigdir}/mingw32-find-provides.sh %__mingw32_requires %{_rpmconfigdir}/mingw32-find-requires.sh %__mingw32_magic ^PE32 executable.* Intel 80386.*, for MS Windows$ %__mingw32_path ^%{_mingw32_prefix}/.*$
Note that this doesn't work quite as you want it to, just yet: in rpm 4.9.0 path and magic are handled as an inclusive or, whereas this (and various other things) needs an and-rule. Adding support for this is technically trivial, just the naming is under consideration: http://lists.rpm.org/pipermail/rpm-maint/2011-May/003022.html
Ah, good catch. I would love to see the "and-rule" in RPM, it would certainly make things easier.
Just FWIW, rpm >= 4.9.1.1 (currently in rawhide, updates submitted for F15 and F16) supports the magic_and_path flag. Whether you need it or not is a another question and up to you, but just thought to let you know now that I happened to remember this discussion :)
- Panu -
Kalev Lember schreef op do 19-05-2011 om 13:49 [+0300]:
I am planning to add the new mingw32.attr file to the mingw32-filesystem package in rawhide soon. Any objections?
Hi Kalev,
Thanks for looking into this feature.
While this feature sure looks interesting for the future I think it's too soon to switch to this yet. As Panu indicated there are some things which still need to be implemented (the and-rule) so this makes it tricky to backport to old Fedora releases. Another issue is that we will lose EL6 support if we're going to implement this in all mingw packages. EL6 support is expected soon (as far as I've heard about it, various mingw packages have popped up recently in the RHEL6 bugzilla) so it wouldn't be really nice to break EL6 support so soon.
I think it would be more sane to wait with implementing this future until it has matured in RPM and is available in all current Fedora releases.
Kind regards,
Erik van Pienbroek
On 05/19/2011 03:11 PM, Erik van Pienbroek wrote:
Hi Kalev,
Thanks for looking into this feature.
While this feature sure looks interesting for the future I think it's too soon to switch to this yet. As Panu indicated there are some things which still need to be implemented (the and-rule) so this makes it tricky to backport to old Fedora releases.
The missing "and-rule" would certainly be nice to have, but it is not a showstopper. A lot of different packages are working just fine without it in F15 and it's easy to make the mingw32 fileattr extractor also work without needing the "and-rule". As to exactly how, I have details in my reply to Panu.
Most (if not all) "internal" dependency generators were converted over to the new fileattr based generators in RPM 4.9. Also several other high profile packages like cups, maven, and gstreamer have switched over in F15, which makes the mingw32 packages an exception, rather than a rule. We are the ones lagging behind.
See for yourself, the following are all new fileattr generator types in use in F15 (yum provides '/usr/lib/rpm/fileattrs/*.attr'): desktop.attr elf.attr font.attr gstreamer.attr libsymlink.attr libtool.attr maven.attr mono.attr ocaml.attr perl.attr perllib.attr pkgconfig.attr psdriver.attr python.attr script.attr
Another issue is that we will lose EL6 support if we're going to implement this in all mingw packages. EL6 support is expected soon (as far as I've heard about it, various mingw packages have popped up recently in the RHEL6 bugzilla) so it wouldn't be really nice to break EL6 support so soon.
Right, RHEL 6.1 was released today with a bunch of mingw32 packages. They are all, however, from the F13 era and if you want to build something mingw32 related in EPEL, it would also make sense to branch these packages from F13. For example, the mingw32-gtk2 package we have in rawhide / F15 would not work in EPEL 6 anyway, because the mingw32-glib2 in RHEL is too old.
I am talking about changing the dependency generation in Fedora Rawhide (and also later backporting it to F15 to keep full srpm compatibility); the RHEL packages are free to keep carrying the boilerplate macros, nobody is breaking these. We are also retaining full compatibility with all old spec files.
I think it would be more sane to wait with implementing this future until it has matured in RPM and is available in all current Fedora releases.
Sorry, but I disagree. Fedora Rawhide is exactly the place to innovate and not wait for RHEL to catch up.
Also, keep in mind that the new dependency generation is fully compatible with all the old spec files. Older spec files that manually disable the "internal" dependency generator in the spec file are still going to work fine.
-
Erik van Pienbroek -
Kalev Lember -
Panu Matilainen