smp_flags considered dangerous?
by Axel Thimm
Well, we know smp_flags lead to undeterministic builds that lead to
o uncomparable build logs
o difficult to trace bugs in the the build output
o undeterministic triggering of build system bugs
I had vtk [1] in the queue for a year, and one of the reviewers'
demand was to add smp_flags, since "it worked that well". I added this
for the sake of getting it reviewed and once the package was reviewed
it wouldn't build on Fedora builders. And it was quite
non-reproducible on my 4-way system, probably due to different number
of make threads, different timings etc. It was also non-reproducable
on the Fedora builders - it would fail but at different build stages,
so I started to check what BR changed between invocations and the
like.
The benefits of smp_flags is that some very big builds are taking less
time. So if say openoffice uses them (I have no idea) it would make no
sense to forbid them for these packages. But for most packages that
usually build under 15 minutes on a serial invocation the drawbacks
are higher than the benefits (like package rebuilds simply breaking
for no apparent reason like vtk did).
But as *recommendation* we should switch from endorsing it to
questioning it and recommending not to use it.
[1] https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=199405
--
Axel.Thimm at ATrpms.net
16 years, 9 months
build ID -debuginfo rpm convention
by Roland McGrath
Hi folks. Please see http://fedoraproject.org/wiki/Releases/FeatureBuildId
for background (and read through all that before asking any technical
questions on the details aside from the rpm packaging convention).
The upshot is that with F8 tools each binary linked will contain something
akin to a UUID, and core dumps will include the IDs of the executable and
DSOs in the crashed process. Married with some conventions about storing
and looking up ID bits, this enables nice automagic debuggability things
including e.g. yum install foo-debuginfo for all the foos in your core file.
For rpm builds, the final ID bits are chosen by debugedit. BZ 246404
tracks the debugedit change, which has not gone in yet (just went into
rpm5.org though ;-)
http://people.redhat.com/roland/build-id/ has all the patches.
Here I want to talk about those conventions. This will be an upstream
convention that the consuming tools (debuggers et al) will have as default
behavior, not just a convention for Fedora or rpm. But I want to reach
consensus on the integrated plan for Fedora repos before proposing a
particular convention upstream.
See http://fedoraproject.org/wiki/RolandMcGrath/BuildID#symlinks for the
details of my initial proposal. In short, the debuginfo rpm covering
/usr/bin/debug would include two symlinks like these:
/usr/lib/debug/.build-id/7c/763b567137492087bc1e50809b1218 -> ../../../bin/foo
/usr/lib/debug/.build-id/7c/763b567137492087bc1e50809b1218.debug -> ../../usr/bin/foo
Consumers will use the .build-id/xx/x... convention to find the binaries
with ID bits xxx... Fancy consumers will do:
yum install /usr/lib/debug/.build-id/7c/763b567137492087bc1e50809b1218.debug
The basic filesystem part of this is simple and efficient to have as the
new standard convention in upstream consumers. This will wind up with two
symlinks per binary installed in /usr/{bin,sbin,lib} etc., which seems
reasonable enough for the maximum size of /usr/lib/debug/.build-id/
directories in an installed system.
Is this reasonable for the packaging in the whole-distro perspective?
The issues I can imagine might be problems are:
* adds two (long) file names per installed binary to debuginfo repo
Is this going to be an unreasonable bloat on the filelist metadata?
* danglink symlink from foo-debuginfo rpm to file in foo rpm
Is this kosher?
The big benefit of this particular convention is that it requires no new
infrastructure work. The extent of the implementation work in the
packaging sphere is a patch to find-debuginfo.sh (below).
Please let me know ASAP what you think about this plan.
Thanks,
Roland
diff -r 4178c07c98c8 scripts/find-debuginfo.sh
--- a/scripts/find-debuginfo.sh Fri Jun 29 14:12:44 2007 +0300
+++ b/scripts/find-debuginfo.sh Sun Jul 01 16:11:35 2007 -0700
@@ -11,16 +11,30 @@ SOURCEFILE=$BUILDDIR/debugsources.list
debugdir="${RPM_BUILD_ROOT}/usr/lib/debug"
-echo -n > $SOURCEFILE
+> $SOURCEFILE
+> $LISTFILE
strip_to_debug()
{
eu-strip --remove-comment -f "$1" "$2" || :
}
+debug_link()
+{
+ tgt="$1"
+ tgt="${tgt/#..\/..\/..\/..\/..\/usr\//../../../../}"
+ tgt="${tgt/#..\/..\/..\/..\/lib\//../../../}"
+ tgt="${1/#..\/..\/..\/debug\//../../}"
+ ln -snf "$tgt" "${debugdir}/$2"
+ echo >> $LISTFILE "/usr/lib/debug/$2"
+}
+
# Strip ELF binaries
-for f in `find $RPM_BUILD_ROOT ! -path "${debugdir}/*.debug" -type f \( -perm -0100 -or -perm -0010 -or -perm -0001 \) -exec file {} \; | \
- sed -n -e 's/^\(.*\):[ ]*.*ELF.*, not stripped/\1/p'`
+find $RPM_BUILD_ROOT ! -path "${debugdir}/*.debug" -type f \
+ \( -perm -0100 -or -perm -0010 -or -perm -0001 \) \
+ -print |
+file -N -f - | sed -n -e 's/^\(.*\):[ ]*.*ELF.*, not stripped/\1/p' |
+while read f
do
dn=$(dirname $f | sed -n -e "s#^$RPM_BUILD_ROOT##p")
bn=$(basename $f .debug).debug
@@ -30,12 +44,23 @@ do
[ -f "${debugfn}" ] && continue
echo extracting debug info from $f
- /usr/lib/rpm/debugedit -b "$RPM_BUILD_DIR" -d /usr/src/debug -l "$SOURCEFILE" "$f"
+ id=`/usr/lib/rpm/debugedit -b "$RPM_BUILD_DIR" -d /usr/src/debug \
+ -i -l "$SOURCEFILE" "$f"` || exit
+ if [ -z "$id" ]; then
+ echo >&2 "No build ID note found in $f"
+ # Demand build IDs in all binaries for Fedora rebuild.
+ # Comment out this exit to be less anal about it.
+ exit 2
+ else
+ id="${id:0:2}/${id:2}"
+ debug_link "../../../../..$dn/$(basename $f)" ".build-id/${id}"
+ fi
# A binary already copied into /usr/lib/debug doesn't get stripped,
# just has its file names collected and adjusted.
case "$dn" in
- /usr/lib/debug/*) continue ;;
+ /usr/lib/debug/*)
+ continue ;;
esac
mkdir -p "${debugdn}"
@@ -46,12 +71,15 @@ do
strip_to_debug "${debugfn}" "$f"
chmod u-w "$f"
fi
+
+ [ -z "$id" ] || debug_link "../..$dn/$bn" ".build-id/${id}.debug"
done
mkdir -p ${RPM_BUILD_ROOT}/usr/src/debug
-cat $SOURCEFILE | (cd $RPM_BUILD_DIR; LANG=C sort -z -u | cpio -pd0mL ${RPM_BUILD_ROOT}/usr/src/debug)
+LANG=C sort -z -u $SOURCEFILE |
+(cd $RPM_BUILD_DIR; cpio -pd0mL ${RPM_BUILD_ROOT}/usr/src/debug)
# stupid cpio creates new directories in mode 0700, fixup
find ${RPM_BUILD_ROOT}/usr/src/debug -type d -print0 | xargs -0 chmod a+rx
-find ${RPM_BUILD_ROOT}/usr/lib/debug -type f | sed -n -e "s#^$RPM_BUILD_ROOT##p" > $LISTFILE
+find ${RPM_BUILD_ROOT}/usr/lib/debug -type f | sed -n -e "s#^$RPM_BUILD_ROOT##p" >> $LISTFILE
find ${RPM_BUILD_ROOT}/usr/src/debug -mindepth 1 -maxdepth 1 | sed -n -e "s#^$RPM_BUILD_ROOT##p" >> $LISTFILE
16 years, 10 months
Multiple version naming overly restrictive?
by Toshio Kuratomi
Hey all, I just noticed this section of the packaging Guidelines:
http://fedoraproject.org/wiki/Packaging/NamingGuidelines#MultiplePackages
says:
'''
For many reasons, it is sometimes advantageous to keep multiple versions
of a package in Fedora to be installed simultaneously. When doing so,
the package name should reflect this fact. The most recent version of a
package should use the base name with no versions, and all other addons
should note their version in the name.
'''
I think specifying that the latest package version has [basename] and
the others have [basename][version] may be overly restrictive. We have
precedent in gtk+ vs gtk2 (long term), gcc (3.x) vs gcc4 (short term),
gtkhtml vs gtkhtml2 vs gtkhtml3 (long term) for doing things the other
way. Do we really want to restrict this?
If not, we could amend the text like this:
'''
One package should use the base name with no versions and all other
addons should note their version in the name.
'''
-Toshio
16 years, 10 months
Meeting today
by Axel Thimm
I will probably not make it to the meeting, but I'd like to propose to
reverse the recommendation to use smp_flags with the opposite (make
the opt-out an opt-in) for the reasons outlined. Count my +1 for that.
--
Axel.Thimm at ATrpms.net
16 years, 10 months