Hi,
below discussion from Fedora Packaging FYI, in case the space saving
discussed there are of interest for you.
Original thread starts at:
https://www.redhat.com/archives/fedora-packaging/2007-April/msg00003.html
CU
thl
------- Original-Nachricht --------
Betreff: [Fedora-packaging] Hardlinking *.pyc and *.pyo
Datum: Tue, 3 Apr 2007 13:42:24 +0300
Von: Ville Skyttä <ville.skytta(a)iki.fi>
Antwort an: Discussion of RPM packaging standards and practices for
Fedora Extras <fedora-packaging(a)redhat.com>
An: fedora-packaging Mailing List <fedora-packaging(a)redhat.com>
Hello,
Related to recent space saving discussions, I came across PLD's
rpm-build-macros package recently, and found that they hardlink
identical *.pyc and *.pyo. In a lot of cases, they're the same,
and there's some potential for saving some MB "for free",
on my FC6 x86_64 box:
$ /usr/sbin/hardlink -ncv /usr/lib*/python2.4 2>&1 | tail -n 1
Would save 11116544
The PLD implementation looks like this:
# Hardlink binary identical .pyc and .pyo files
# (idea by glen <at> pld-linux <dot> org)
%__spec_install_post_py_hardlink {\
%{!?no_install_post_py_hardlink: __spec_install_post_py_hardlink() { \
[ ! -d "$RPM_BUILD_ROOT" ] || find "$RPM_BUILD_ROOT" -name
'*.pyc' |
while read a; do \
b="$(echo $a|sed -e 's/.pyc$/.pyo/')"; \
if cmp -s "$a" "$b"; then \
ln -f "$a" "$b"; \
fi; \
done \
}; __spec_install_post_py_hardlink } }
The use of "cmp" would require diffutils installed. Or the above
could be converted to use hardlink instead (which would have to be made
sure to be around) or maybe sha1sum (in coreutils, pretty much always
around in buildroots).
I suppose something like the above could be easily added to
redhat-rpm-config or rpm, eg. embedded in brp-python-bytecompile
or run after it in %__os_install_post.
Worth it? Other comments?
-------- Original-Nachricht --------
Return-Path: <Thorsten_Leemhuis(a)gmx.de>
X-Original-To: rpmbuild@localhost
Delivered-To: rpmbuild(a)localhost.thl.home
Received: by truhe.thl.home (Postfix, from userid 500) id 67F1B8110;
Tue, 3 Apr 2007 13:24:27 +0200 (CEST)
X-Original-To: thl@localhost
Delivered-To: thl(a)localhost.thl.home
Received: from truhe.thl.home (localhost [127.0.0.1]) by truhe.thl.home
(Postfix) with ESMTP id 510D280AD for <thl@localhost>; Tue, 3 Apr 2007
13:24:27 +0200 (CEST)
X-Original-To: fedora(a)leemhuis.info
Delivered-To: web550p1(a)basicbox7.server-home.net
Received: from leemhuis.info [195.137.212.29] by truhe.thl.home with
POP3 (fetchmail-6.3.6) for <thl@localhost> (single-drop); Tue, 03 Apr
2007 13:24:27 +0200 (CEST)
Received: from
hormel.redhat.com (
hormel.redhat.com [209.132.177.30]) by
basicbox7.server-home.net (Postfix) with ESMTP id 1B0C63741D9 for
<fedora(a)leemhuis.info>; Tue, 3 Apr 2007 13:17:29 +0200 (CEST)
Received: from
listman.util.phx.redhat.com
(
listman.util.phx.redhat.com [10.8.4.110]) by
hormel.redhat.com
(Postfix) with ESMTP id B188E72E40; Tue, 3 Apr 2007 07:17:30 -0400 (EDT)
Received: from
int-mx1.corp.redhat.com (
int-mx1.corp.redhat.com
[172.16.52.254]) by
listman.util.phx.redhat.com (8.13.1/8.13.1) with
ESMTP id l33BHSHq025579 for
<fedora-packaging(a)listman.util.phx.redhat.com>; Tue, 3 Apr 2007 07:17:28
-0400
Received: from
mx1.redhat.com (
mx1.redhat.com [172.16.48.31]) by
int-mx1.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l33BHSBL023395 for
<fedora-packaging(a)redhat.com>; Tue, 3 Apr 2007 07:17:28 -0400
Received: from TZMXR01.htp-tel.de (tzmxr01.htp-tel.de [81.14.243.17]) by
mx1.redhat.com (8.13.1/8.13.1) with ESMTP id l33BHM8I029053 for
<fedora-packaging(a)redhat.com>; Tue, 3 Apr 2007 07:17:23 -0400
Received: from TZMXR01.htp-tel.de (localhost [127.0.0.1]) by
TZMXR01.htp-tel.de with ESMTP id l33BHGYK029546 for
<fedora-packaging(a)redhat.com>; Tue, 3 Apr 2007 13:17:16 +0200 (CEST)
Received: from truhe.thl.home (a81-14-184-169.net-htp.de
[81.14.184.169]) by TZMXR01.htp-tel.de with ESMTP id l33BHF0E029531 for
<fedora-packaging(a)redhat.com>; Tue, 3 Apr 2007 13:17:16 +0200 (CEST)
Received: from thl.ct.heise.de (localhost [127.0.0.1]) by truhe.thl.home
(Postfix) with ESMTP id 70DC580AD for <fedora-packaging(a)redhat.com>;
Tue, 3 Apr 2007 13:17:15 +0200 (CEST)
Message-ID: <461237BB.5050700(a)leemhuis.info>
Date: Tue, 03 Apr 2007 13:17:15 +0200
From: Thorsten Leemhuis <fedora(a)leemhuis.info>
User-Agent: Thunderbird 2.0b2 (X11/20070308)
MIME-Version: 1.0
To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging(a)redhat.com>
Subject: Re: [Fedora-packaging] Hardlinking *.pyc and *.pyo
References: <200704031342.25238.ville.skytta(a)iki.fi>
In-Reply-To: <200704031342.25238.ville.skytta(a)iki.fi>
X-Enigmail-Version: 0.94.3.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-RedHat-Spam-Score: 0
X-loop: fedora-packaging(a)redhat.com
X-BeenThere: fedora-packaging(a)redhat.com
X-Mailman-Version: 2.1.5
Precedence: junk
Reply-To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging(a)redhat.com>
List-Id: Discussion of RPM packaging standards and practices for
Fedora Extras <fedora-packaging.redhat.com>
List-Unsubscribe:
<
https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request@redhat.com?subject=unsubscribe>
List-Archive: <
https://www.redhat.com/archives/fedora-packaging>
List-Post: <mailto:fedora-packaging@redhat.com>
List-Help: <mailto:fedora-packaging-request@redhat.com?subject=help>
List-Subscribe:
<
https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request@redhat.com?subject=subscribe>
Sender: fedora-packaging-bounces(a)redhat.com
Errors-To: fedora-packaging-bounces(a)redhat.com
X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on
basicbox7.server-home.net
X-Spam-Level:
X-Spam-Status: No, score=-0.1 required=5.0 tests=AWL
autolearn=disabled version=3.0.3
X-UIDL: dg:!!WW-!!BIp"!23Y"!
On 03.04.2007 12:42, Ville Skyttä wrote:
Worth it? Other comments?
Hehe, sounds interesting. Did you tell the Live-CD guys about it? I'd
say they will probably very interested in something like this (they
could run hardlink directly for now until we decide what we want to do).
CU
thl
-------- Original-Nachricht --------
Return-Path: <Thorsten_Leemhuis(a)gmx.de>
X-Original-To: rpmbuild@localhost
Delivered-To: rpmbuild(a)localhost.thl.home
Received: by truhe.thl.home (Postfix, from userid 500) id DB9EB8110;
Tue, 3 Apr 2007 13:54:57 +0200 (CEST)
X-Original-To: thl@localhost
Delivered-To: thl(a)localhost.thl.home
Received: from truhe.thl.home (localhost [127.0.0.1]) by truhe.thl.home
(Postfix) with ESMTP id C928480AD for <thl@localhost>; Tue, 3 Apr 2007
13:54:57 +0200 (CEST)
X-Original-To: fedora(a)leemhuis.info
Delivered-To: web550p1(a)basicbox7.server-home.net
Received: from leemhuis.info [195.137.212.29] by truhe.thl.home with
POP3 (fetchmail-6.3.6) for <thl@localhost> (single-drop); Tue, 03 Apr
2007 13:54:57 +0200 (CEST)
Received: from
hormel.redhat.com (
hormel.redhat.com [209.132.177.30]) by
basicbox7.server-home.net (Postfix) with ESMTP id 1EF8C374D37 for
<fedora(a)leemhuis.info>; Tue, 3 Apr 2007 13:45:41 +0200 (CEST)
Received: from
listman.util.phx.redhat.com
(
listman.util.phx.redhat.com [10.8.4.110]) by
hormel.redhat.com
(Postfix) with ESMTP id AAF3D73009; Tue, 3 Apr 2007 07:45:42 -0400 (EDT)
Received: from
int-mx2.corp.redhat.com (
nat-pool.util.phx.redhat.com
[10.8.5.200]) by
listman.util.phx.redhat.com (8.13.1/8.13.1) with
ESMTP id l33Bjfnb029939 for
<fedora-packaging(a)listman.util.phx.redhat.com>; Tue, 3 Apr 2007 07:45:41
-0400
Received: from
mx2.redhat.com (
mx2.redhat.com [10.255.15.25]) by
int-mx2.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l33Bje9H030402 for
<fedora-packaging(a)redhat.com>; Tue, 3 Apr 2007 07:45:40 -0400
Received: from smtp5.pp.htv.fi (smtp5.pp.htv.fi [213.243.153.39]) by
mx2.redhat.com (8.13.1/8.13.1) with ESMTP id l33BjcSD019557 for
<fedora-packaging(a)redhat.com>; Tue, 3 Apr 2007 07:45:39 -0400
Received: from viper.localdomain (cs181043142.pp.htv.fi
[82.181.43.142]) by smtp5.pp.htv.fi (Postfix) with ESMTP id
B4D425BC1E9 for <fedora-packaging(a)redhat.com>; Tue, 3 Apr 2007 14:45:37
+0300 (EEST)
From: Ville Skyttä <ville.skytta(a)iki.fi>
To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging(a)redhat.com>
Subject: Re: [Fedora-packaging] Hardlinking *.pyc and *.pyo
Date: Tue, 3 Apr 2007 14:45:37 +0300
User-Agent: KMail/1.9.6
References: <200704031342.25238.ville.skytta(a)iki.fi>
<461237BB.5050700(a)leemhuis.info>
In-Reply-To: <461237BB.5050700(a)leemhuis.info>
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Disposition: inline
Message-Id: <200704031445.37318.ville.skytta(a)iki.fi>
X-RedHat-Spam-Score: 0
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by
listman.util.phx.redhat.com id l33Bjfnb029939
X-loop: fedora-packaging(a)redhat.com
X-BeenThere: fedora-packaging(a)redhat.com
X-Mailman-Version: 2.1.5
Precedence: junk
Reply-To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging(a)redhat.com>
List-Id: Discussion of RPM packaging standards and practices for
Fedora Extras <fedora-packaging.redhat.com>
List-Unsubscribe:
<
https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request@redhat.com?subject=unsubscribe>
List-Archive: <
https://www.redhat.com/archives/fedora-packaging>
List-Post: <mailto:fedora-packaging@redhat.com>
List-Help: <mailto:fedora-packaging-request@redhat.com?subject=help>
List-Subscribe:
<
https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request@redhat.com?subject=subscribe>
Sender: fedora-packaging-bounces(a)redhat.com
Errors-To: fedora-packaging-bounces(a)redhat.com
X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on
basicbox7.server-home.net
X-Spam-Level:
X-Spam-Status: No, score=-0.3 required=5.0 tests=AWL
autolearn=disabled version=3.0.3
X-UIDL: GSK"!`f=!!:Wi"!p1+"!
On Tuesday 03 April 2007, Thorsten Leemhuis wrote:
> On 03.04.2007 12:42, Ville Skyttä wrote:
>
Worth it? Other comments?
>
> Hehe, sounds interesting. Did you tell the Live-CD guys about it? I'd
> say they will probably very interested in something like this (they
> could run hardlink directly for now until we decide what we want to do).
Nope, just brought it up in last week's packaging meeting and now posted
here.
Feel free to forward.
Another thing they could be interested in (again on my FC6 x86_64):
$ /usr/sbin/hardlink -ncv /usr/share/doc 2>&1 | tail -n 1
Would save 6692864 (of which COPYING's share is about 3.2M)
-------- Original-Nachricht --------
Return-Path: <Thorsten_Leemhuis(a)gmx.de>
X-Original-To: rpmbuild@localhost
Delivered-To: rpmbuild(a)localhost.thl.home
Received: by truhe.thl.home (Postfix, from userid 500) id EF7088117;
Tue, 3 Apr 2007 14:05:05 +0200 (CEST)
X-Original-To: thl@localhost
Delivered-To: thl(a)localhost.thl.home
Received: from truhe.thl.home (localhost [127.0.0.1]) by truhe.thl.home
(Postfix) with ESMTP id D99CE80DE for <thl@localhost>; Tue, 3 Apr 2007
14:05:05 +0200 (CEST)
X-Original-To: fedora(a)leemhuis.info
Delivered-To: web550p1(a)basicbox7.server-home.net
Received: from leemhuis.info [195.137.212.29] by truhe.thl.home with
POP3 (fetchmail-6.3.6) for <thl@localhost> (single-drop); Tue, 03 Apr
2007 14:05:05 +0200 (CEST)
Received: from
hormel.redhat.com (
hormel.redhat.com [209.132.177.30]) by
basicbox7.server-home.net (Postfix) with ESMTP id BE818374D4B for
<fedora(a)leemhuis.info>; Tue, 3 Apr 2007 14:01:32 +0200 (CEST)
Received: from
listman.util.phx.redhat.com
(
listman.util.phx.redhat.com [10.8.4.110]) by
hormel.redhat.com
(Postfix) with ESMTP id 93CE072FB4; Tue, 3 Apr 2007 08:01:33 -0400 (EDT)
Received: from
int-mx1.corp.redhat.com (
int-mx1.corp.redhat.com
[172.16.52.254]) by
listman.util.phx.redhat.com (8.13.1/8.13.1) with
ESMTP id l33C1VoW032671 for
<fedora-packaging(a)listman.util.phx.redhat.com>; Tue, 3 Apr 2007 08:01:32
-0400
Received: from
mx1.redhat.com (
mx1.redhat.com [172.16.48.31]) by
int-mx1.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l33C1UEY004042 for
<fedora-packaging(a)redhat.com>; Tue, 3 Apr 2007 08:01:30 -0400
Received: from junior.physik.fu-berlin.de
(IDENT:U2FsdGVkX184gQ/KwtoT7+xBQgH3zjb63w3ZGzlWGD4@junior.physik.fu-berlin.de
[130.133.35.30]) by
mx1.redhat.com (8.13.1/8.13.1) with ESMTP id
l33C1RN5026158 for <fedora-packaging(a)redhat.com>; Tue, 3 Apr 2007
08:01:27 -0400
Received: from neu-vpn.physik.fu-berlin.de ([160.45.32.209]
helo=neu.nirvana) by
mail2.atrpms.net with esmtps (TLSv1:AES256-SHA:256)
(Exim 4.66) id 1HYhhE-0001uV-06; Tue, 03 Apr 2007 14:01:25 +0200
Received: from thimm by neu.nirvana with local (Exim 4.66)
(envelope-from <Axel.Thimm(a)ATrpms.net>) id 1HYhhC-0001x3-TQ; Tue, 03 Apr
2007 14:01:06 +0200
Date: Tue, 3 Apr 2007 14:01:06 +0200
From: Axel Thimm <Axel.Thimm(a)ATrpms.net>
To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging(a)redhat.com>
Message-ID: <20070403120106.GD29856(a)neu.nirvana>
References: <200704031342.25238.ville.skytta(a)iki.fi>
MIME-Version: 1.0
In-Reply-To: <200704031342.25238.ville.skytta(a)iki.fi>
User-Agent: Mutt/1.5.13 (2006-08-11)
X-Scanned: No viruses found.
X-Scan-Signature: fe522d7d72b3ce68d16a3279c0f8f490
X-RedHat-Spam-Score: 0
X-loop: fedora-packaging(a)redhat.com
Subject: [Fedora-packaging] Re: Hardlinking *.pyc and *.pyo
X-BeenThere: fedora-packaging(a)redhat.com
X-Mailman-Version: 2.1.5
Precedence: junk
Reply-To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging(a)redhat.com>
List-Id: Discussion of RPM packaging standards and practices for
Fedora Extras <fedora-packaging.redhat.com>
List-Unsubscribe:
<
https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request@redhat.com?subject=unsubscribe>
List-Archive: <
https://www.redhat.com/archives/fedora-packaging>
List-Post: <mailto:fedora-packaging@redhat.com>
List-Help: <mailto:fedora-packaging-request@redhat.com?subject=help>
List-Subscribe:
<
https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request@redhat.com?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============1626666452=="
Sender: fedora-packaging-bounces(a)redhat.com
Errors-To: fedora-packaging-bounces(a)redhat.com
X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on
basicbox7.server-home.net
X-Spam-Level:
X-Spam-Status: No, score=-0.2 required=5.0 tests=AWL
autolearn=disabled version=3.0.3
X-UIDL: ff="!/9`"!@MP!!34]!!
On Tue, Apr 03, 2007 at 01:42:24PM +0300, Ville Skyttä wrote:
Related to recent space saving discussions, I came across PLD's
rpm-build-macros package recently, and found that they hardlink
identical *.pyc and *.pyo. In a lot of cases, they're the same,
and there's some potential for saving some MB "for free",
on my FC6 x86_64 box:
$ /usr/sbin/hardlink -ncv /usr/lib*/python2.4 2>&1 | tail -n 1
Would save 11116544
I get more than twice as much on a typical FC6/x86_64 system:
27275264. That's 26 MB on 166MB total, e.g. saving 16%.
# du -sc /usr/lib*/python2.4| tail -n 1
170200 total
On another system I get 21MB of 144MB total, e.g about 15%.
The PLD implementation looks like this:
# Hardlink binary identical .pyc and .pyo files
# (idea by glen <at> pld-linux <dot> org)
%__spec_install_post_py_hardlink {\
%{!?no_install_post_py_hardlink: __spec_install_post_py_hardlink() { \
[ ! -d "$RPM_BUILD_ROOT" ] || find "$RPM_BUILD_ROOT" -name
'*.pyc' |
while read a; do \
b="$(echo $a|sed -e 's/.pyc$/.pyo/')"; \
if cmp -s "$a" "$b"; then \
ln -f "$a" "$b"; \
fi; \
done \
}; __spec_install_post_py_hardlink } }
The use of "cmp" would require diffutils installed. Or the above
could be converted to use hardlink instead (which would have to be made
sure to be around) or maybe sha1sum (in coreutils, pretty much always
around in buildroots).
I suppose something like the above could be easily added to
redhat-rpm-config or rpm, eg. embedded in brp-python-bytecompile
or run after it in %__os_install_post.
brp-python-bytecompile sounds like the best spot since the pyc/pyos
are created there. Maybe we should ship an improved
brp-python-bytecompile in redhat-rpm-config while lobbying rpm
upstream to adopt it?
Worth it? Other comments?
A 15% space gain (under python) w/o any drawbacks? Always worth it. :)