[Fedora-livecd-list] Space savings discussed on fedora-packaging-list
Thorsten Leemhuis
fedora at leemhuis.info
Tue Apr 3 13:03:20 UTC 2007
Hi,
below discussion from Fedora Packaging FYI, in case the space saving
discussed there are of interest for you.
Original thread starts at:
https://www.redhat.com/archives/fedora-packaging/2007-April/msg00003.html
CU
thl
------- Original-Nachricht --------
Betreff: [Fedora-packaging] Hardlinking *.pyc and *.pyo
Datum: Tue, 3 Apr 2007 13:42:24 +0300
Von: Ville Skyttä <ville.skytta at iki.fi>
Antwort an: Discussion of RPM packaging standards and practices for
Fedora Extras <fedora-packaging at redhat.com>
An: fedora-packaging Mailing List <fedora-packaging at redhat.com>
Hello,
Related to recent space saving discussions, I came across PLD's
rpm-build-macros package recently, and found that they hardlink
identical *.pyc and *.pyo. In a lot of cases, they're the same,
and there's some potential for saving some MB "for free",
on my FC6 x86_64 box:
$ /usr/sbin/hardlink -ncv /usr/lib*/python2.4 2>&1 | tail -n 1
Would save 11116544
The PLD implementation looks like this:
# Hardlink binary identical .pyc and .pyo files
# (idea by glen <at> pld-linux <dot> org)
%__spec_install_post_py_hardlink {\
%{!?no_install_post_py_hardlink: __spec_install_post_py_hardlink() { \
[ ! -d "$RPM_BUILD_ROOT" ] || find "$RPM_BUILD_ROOT" -name '*.pyc' |
while read a; do \
b="$(echo $a|sed -e 's/.pyc$/.pyo/')"; \
if cmp -s "$a" "$b"; then \
ln -f "$a" "$b"; \
fi; \
done \
}; __spec_install_post_py_hardlink } }
The use of "cmp" would require diffutils installed. Or the above
could be converted to use hardlink instead (which would have to be made
sure to be around) or maybe sha1sum (in coreutils, pretty much always
around in buildroots).
I suppose something like the above could be easily added to
redhat-rpm-config or rpm, eg. embedded in brp-python-bytecompile
or run after it in %__os_install_post.
Worth it? Other comments?
-------- Original-Nachricht --------
Return-Path: <Thorsten_Leemhuis at gmx.de>
X-Original-To: rpmbuild at localhost
Delivered-To: rpmbuild at localhost.thl.home
Received: by truhe.thl.home (Postfix, from userid 500) id 67F1B8110;
Tue, 3 Apr 2007 13:24:27 +0200 (CEST)
X-Original-To: thl at localhost
Delivered-To: thl at localhost.thl.home
Received: from truhe.thl.home (localhost [127.0.0.1]) by truhe.thl.home
(Postfix) with ESMTP id 510D280AD for <thl at localhost>; Tue, 3 Apr 2007
13:24:27 +0200 (CEST)
X-Original-To: fedora at leemhuis.info
Delivered-To: web550p1 at basicbox7.server-home.net
Received: from leemhuis.info [195.137.212.29] by truhe.thl.home with
POP3 (fetchmail-6.3.6) for <thl at localhost> (single-drop); Tue, 03 Apr
2007 13:24:27 +0200 (CEST)
Received: from hormel.redhat.com (hormel.redhat.com [209.132.177.30]) by
basicbox7.server-home.net (Postfix) with ESMTP id 1B0C63741D9 for
<fedora at leemhuis.info>; Tue, 3 Apr 2007 13:17:29 +0200 (CEST)
Received: from listman.util.phx.redhat.com
(listman.util.phx.redhat.com [10.8.4.110]) by hormel.redhat.com
(Postfix) with ESMTP id B188E72E40; Tue, 3 Apr 2007 07:17:30 -0400 (EDT)
Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com
[172.16.52.254]) by listman.util.phx.redhat.com (8.13.1/8.13.1) with
ESMTP id l33BHSHq025579 for
<fedora-packaging at listman.util.phx.redhat.com>; Tue, 3 Apr 2007 07:17:28
-0400
Received: from mx1.redhat.com (mx1.redhat.com [172.16.48.31]) by
int-mx1.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l33BHSBL023395 for
<fedora-packaging at redhat.com>; Tue, 3 Apr 2007 07:17:28 -0400
Received: from TZMXR01.htp-tel.de (tzmxr01.htp-tel.de [81.14.243.17]) by
mx1.redhat.com (8.13.1/8.13.1) with ESMTP id l33BHM8I029053 for
<fedora-packaging at redhat.com>; Tue, 3 Apr 2007 07:17:23 -0400
Received: from TZMXR01.htp-tel.de (localhost [127.0.0.1]) by
TZMXR01.htp-tel.de with ESMTP id l33BHGYK029546 for
<fedora-packaging at redhat.com>; Tue, 3 Apr 2007 13:17:16 +0200 (CEST)
Received: from truhe.thl.home (a81-14-184-169.net-htp.de
[81.14.184.169]) by TZMXR01.htp-tel.de with ESMTP id l33BHF0E029531 for
<fedora-packaging at redhat.com>; Tue, 3 Apr 2007 13:17:16 +0200 (CEST)
Received: from thl.ct.heise.de (localhost [127.0.0.1]) by truhe.thl.home
(Postfix) with ESMTP id 70DC580AD for <fedora-packaging at redhat.com>;
Tue, 3 Apr 2007 13:17:15 +0200 (CEST)
Message-ID: <461237BB.5050700 at leemhuis.info>
Date: Tue, 03 Apr 2007 13:17:15 +0200
From: Thorsten Leemhuis <fedora at leemhuis.info>
User-Agent: Thunderbird 2.0b2 (X11/20070308)
MIME-Version: 1.0
To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging at redhat.com>
Subject: Re: [Fedora-packaging] Hardlinking *.pyc and *.pyo
References: <200704031342.25238.ville.skytta at iki.fi>
In-Reply-To: <200704031342.25238.ville.skytta at iki.fi>
X-Enigmail-Version: 0.94.3.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-RedHat-Spam-Score: 0
X-loop: fedora-packaging at redhat.com
X-BeenThere: fedora-packaging at redhat.com
X-Mailman-Version: 2.1.5
Precedence: junk
Reply-To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging at redhat.com>
List-Id: Discussion of RPM packaging standards and practices for
Fedora Extras <fedora-packaging.redhat.com>
List-Unsubscribe:
<https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request at redhat.com?subject=unsubscribe>
List-Archive: <https://www.redhat.com/archives/fedora-packaging>
List-Post: <mailto:fedora-packaging at redhat.com>
List-Help: <mailto:fedora-packaging-request at redhat.com?subject=help>
List-Subscribe:
<https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request at redhat.com?subject=subscribe>
Sender: fedora-packaging-bounces at redhat.com
Errors-To: fedora-packaging-bounces at redhat.com
X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on
basicbox7.server-home.net
X-Spam-Level:
X-Spam-Status: No, score=-0.1 required=5.0 tests=AWL
autolearn=disabled version=3.0.3
X-UIDL: dg:!!WW-!!BIp"!23Y"!
On 03.04.2007 12:42, Ville Skyttä wrote:
> Worth it? Other comments?
Hehe, sounds interesting. Did you tell the Live-CD guys about it? I'd
say they will probably very interested in something like this (they
could run hardlink directly for now until we decide what we want to do).
CU
thl
-------- Original-Nachricht --------
Return-Path: <Thorsten_Leemhuis at gmx.de>
X-Original-To: rpmbuild at localhost
Delivered-To: rpmbuild at localhost.thl.home
Received: by truhe.thl.home (Postfix, from userid 500) id DB9EB8110;
Tue, 3 Apr 2007 13:54:57 +0200 (CEST)
X-Original-To: thl at localhost
Delivered-To: thl at localhost.thl.home
Received: from truhe.thl.home (localhost [127.0.0.1]) by truhe.thl.home
(Postfix) with ESMTP id C928480AD for <thl at localhost>; Tue, 3 Apr 2007
13:54:57 +0200 (CEST)
X-Original-To: fedora at leemhuis.info
Delivered-To: web550p1 at basicbox7.server-home.net
Received: from leemhuis.info [195.137.212.29] by truhe.thl.home with
POP3 (fetchmail-6.3.6) for <thl at localhost> (single-drop); Tue, 03 Apr
2007 13:54:57 +0200 (CEST)
Received: from hormel.redhat.com (hormel.redhat.com [209.132.177.30]) by
basicbox7.server-home.net (Postfix) with ESMTP id 1EF8C374D37 for
<fedora at leemhuis.info>; Tue, 3 Apr 2007 13:45:41 +0200 (CEST)
Received: from listman.util.phx.redhat.com
(listman.util.phx.redhat.com [10.8.4.110]) by hormel.redhat.com
(Postfix) with ESMTP id AAF3D73009; Tue, 3 Apr 2007 07:45:42 -0400 (EDT)
Received: from int-mx2.corp.redhat.com (nat-pool.util.phx.redhat.com
[10.8.5.200]) by listman.util.phx.redhat.com (8.13.1/8.13.1) with
ESMTP id l33Bjfnb029939 for
<fedora-packaging at listman.util.phx.redhat.com>; Tue, 3 Apr 2007 07:45:41
-0400
Received: from mx2.redhat.com (mx2.redhat.com [10.255.15.25]) by
int-mx2.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l33Bje9H030402 for
<fedora-packaging at redhat.com>; Tue, 3 Apr 2007 07:45:40 -0400
Received: from smtp5.pp.htv.fi (smtp5.pp.htv.fi [213.243.153.39]) by
mx2.redhat.com (8.13.1/8.13.1) with ESMTP id l33BjcSD019557 for
<fedora-packaging at redhat.com>; Tue, 3 Apr 2007 07:45:39 -0400
Received: from viper.localdomain (cs181043142.pp.htv.fi
[82.181.43.142]) by smtp5.pp.htv.fi (Postfix) with ESMTP id
B4D425BC1E9 for <fedora-packaging at redhat.com>; Tue, 3 Apr 2007 14:45:37
+0300 (EEST)
From: Ville Skyttä <ville.skytta at iki.fi>
To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging at redhat.com>
Subject: Re: [Fedora-packaging] Hardlinking *.pyc and *.pyo
Date: Tue, 3 Apr 2007 14:45:37 +0300
User-Agent: KMail/1.9.6
References: <200704031342.25238.ville.skytta at iki.fi>
<461237BB.5050700 at leemhuis.info>
In-Reply-To: <461237BB.5050700 at leemhuis.info>
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Disposition: inline
Message-Id: <200704031445.37318.ville.skytta at iki.fi>
X-RedHat-Spam-Score: 0
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by
listman.util.phx.redhat.com id l33Bjfnb029939
X-loop: fedora-packaging at redhat.com
X-BeenThere: fedora-packaging at redhat.com
X-Mailman-Version: 2.1.5
Precedence: junk
Reply-To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging at redhat.com>
List-Id: Discussion of RPM packaging standards and practices for
Fedora Extras <fedora-packaging.redhat.com>
List-Unsubscribe:
<https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request at redhat.com?subject=unsubscribe>
List-Archive: <https://www.redhat.com/archives/fedora-packaging>
List-Post: <mailto:fedora-packaging at redhat.com>
List-Help: <mailto:fedora-packaging-request at redhat.com?subject=help>
List-Subscribe:
<https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request at redhat.com?subject=subscribe>
Sender: fedora-packaging-bounces at redhat.com
Errors-To: fedora-packaging-bounces at redhat.com
X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on
basicbox7.server-home.net
X-Spam-Level:
X-Spam-Status: No, score=-0.3 required=5.0 tests=AWL
autolearn=disabled version=3.0.3
X-UIDL: GSK"!`f=!!:Wi"!p1+"!
On Tuesday 03 April 2007, Thorsten Leemhuis wrote:
> On 03.04.2007 12:42, Ville Skyttä wrote:
> > Worth it? Other comments?
>
> Hehe, sounds interesting. Did you tell the Live-CD guys about it? I'd
> say they will probably very interested in something like this (they
> could run hardlink directly for now until we decide what we want to do).
Nope, just brought it up in last week's packaging meeting and now posted
here.
Feel free to forward.
Another thing they could be interested in (again on my FC6 x86_64):
$ /usr/sbin/hardlink -ncv /usr/share/doc 2>&1 | tail -n 1
Would save 6692864 (of which COPYING's share is about 3.2M)
-------- Original-Nachricht --------
Return-Path: <Thorsten_Leemhuis at gmx.de>
X-Original-To: rpmbuild at localhost
Delivered-To: rpmbuild at localhost.thl.home
Received: by truhe.thl.home (Postfix, from userid 500) id EF7088117;
Tue, 3 Apr 2007 14:05:05 +0200 (CEST)
X-Original-To: thl at localhost
Delivered-To: thl at localhost.thl.home
Received: from truhe.thl.home (localhost [127.0.0.1]) by truhe.thl.home
(Postfix) with ESMTP id D99CE80DE for <thl at localhost>; Tue, 3 Apr 2007
14:05:05 +0200 (CEST)
X-Original-To: fedora at leemhuis.info
Delivered-To: web550p1 at basicbox7.server-home.net
Received: from leemhuis.info [195.137.212.29] by truhe.thl.home with
POP3 (fetchmail-6.3.6) for <thl at localhost> (single-drop); Tue, 03 Apr
2007 14:05:05 +0200 (CEST)
Received: from hormel.redhat.com (hormel.redhat.com [209.132.177.30]) by
basicbox7.server-home.net (Postfix) with ESMTP id BE818374D4B for
<fedora at leemhuis.info>; Tue, 3 Apr 2007 14:01:32 +0200 (CEST)
Received: from listman.util.phx.redhat.com
(listman.util.phx.redhat.com [10.8.4.110]) by hormel.redhat.com
(Postfix) with ESMTP id 93CE072FB4; Tue, 3 Apr 2007 08:01:33 -0400 (EDT)
Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com
[172.16.52.254]) by listman.util.phx.redhat.com (8.13.1/8.13.1) with
ESMTP id l33C1VoW032671 for
<fedora-packaging at listman.util.phx.redhat.com>; Tue, 3 Apr 2007 08:01:32
-0400
Received: from mx1.redhat.com (mx1.redhat.com [172.16.48.31]) by
int-mx1.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l33C1UEY004042 for
<fedora-packaging at redhat.com>; Tue, 3 Apr 2007 08:01:30 -0400
Received: from junior.physik.fu-berlin.de
(IDENT:U2FsdGVkX184gQ/KwtoT7+xBQgH3zjb63w3ZGzlWGD4 at junior.physik.fu-berlin.de
[130.133.35.30]) by mx1.redhat.com (8.13.1/8.13.1) with ESMTP id
l33C1RN5026158 for <fedora-packaging at redhat.com>; Tue, 3 Apr 2007
08:01:27 -0400
Received: from neu-vpn.physik.fu-berlin.de ([160.45.32.209]
helo=neu.nirvana) by mail2.atrpms.net with esmtps (TLSv1:AES256-SHA:256)
(Exim 4.66) id 1HYhhE-0001uV-06; Tue, 03 Apr 2007 14:01:25 +0200
Received: from thimm by neu.nirvana with local (Exim 4.66)
(envelope-from <Axel.Thimm at ATrpms.net>) id 1HYhhC-0001x3-TQ; Tue, 03 Apr
2007 14:01:06 +0200
Date: Tue, 3 Apr 2007 14:01:06 +0200
From: Axel Thimm <Axel.Thimm at ATrpms.net>
To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging at redhat.com>
Message-ID: <20070403120106.GD29856 at neu.nirvana>
References: <200704031342.25238.ville.skytta at iki.fi>
MIME-Version: 1.0
In-Reply-To: <200704031342.25238.ville.skytta at iki.fi>
User-Agent: Mutt/1.5.13 (2006-08-11)
X-Scanned: No viruses found.
X-Scan-Signature: fe522d7d72b3ce68d16a3279c0f8f490
X-RedHat-Spam-Score: 0
X-loop: fedora-packaging at redhat.com
Subject: [Fedora-packaging] Re: Hardlinking *.pyc and *.pyo
X-BeenThere: fedora-packaging at redhat.com
X-Mailman-Version: 2.1.5
Precedence: junk
Reply-To: Discussion of RPM packaging standards and practices for Fedora
Extras <fedora-packaging at redhat.com>
List-Id: Discussion of RPM packaging standards and practices for
Fedora Extras <fedora-packaging.redhat.com>
List-Unsubscribe:
<https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request at redhat.com?subject=unsubscribe>
List-Archive: <https://www.redhat.com/archives/fedora-packaging>
List-Post: <mailto:fedora-packaging at redhat.com>
List-Help: <mailto:fedora-packaging-request at redhat.com?subject=help>
List-Subscribe:
<https://www.redhat.com/mailman/listinfo/fedora-packaging>,
<mailto:fedora-packaging-request at redhat.com?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============1626666452=="
Sender: fedora-packaging-bounces at redhat.com
Errors-To: fedora-packaging-bounces at redhat.com
X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on
basicbox7.server-home.net
X-Spam-Level:
X-Spam-Status: No, score=-0.2 required=5.0 tests=AWL
autolearn=disabled version=3.0.3
X-UIDL: ff="!/9`"!@MP!!34]!!
On Tue, Apr 03, 2007 at 01:42:24PM +0300, Ville Skyttä wrote:
> Related to recent space saving discussions, I came across PLD's
> rpm-build-macros package recently, and found that they hardlink
> identical *.pyc and *.pyo. In a lot of cases, they're the same,
> and there's some potential for saving some MB "for free",
> on my FC6 x86_64 box:
>
> $ /usr/sbin/hardlink -ncv /usr/lib*/python2.4 2>&1 | tail -n 1
> Would save 11116544
I get more than twice as much on a typical FC6/x86_64 system:
27275264. That's 26 MB on 166MB total, e.g. saving 16%.
# du -sc /usr/lib*/python2.4| tail -n 1
170200 total
On another system I get 21MB of 144MB total, e.g about 15%.
> The PLD implementation looks like this:
>
> # Hardlink binary identical .pyc and .pyo files
> # (idea by glen <at> pld-linux <dot> org)
> %__spec_install_post_py_hardlink {\
> %{!?no_install_post_py_hardlink: __spec_install_post_py_hardlink() { \
> [ ! -d "$RPM_BUILD_ROOT" ] || find "$RPM_BUILD_ROOT" -name '*.pyc' |
while read a; do \
> b="$(echo $a|sed -e 's/.pyc$/.pyo/')"; \
> if cmp -s "$a" "$b"; then \
> ln -f "$a" "$b"; \
> fi; \
> done \
> }; __spec_install_post_py_hardlink } }
>
> The use of "cmp" would require diffutils installed. Or the above
> could be converted to use hardlink instead (which would have to be made
> sure to be around) or maybe sha1sum (in coreutils, pretty much always
> around in buildroots).
>
> I suppose something like the above could be easily added to
> redhat-rpm-config or rpm, eg. embedded in brp-python-bytecompile
> or run after it in %__os_install_post.
brp-python-bytecompile sounds like the best spot since the pyc/pyos
are created there. Maybe we should ship an improved
brp-python-bytecompile in redhat-rpm-config while lobbying rpm
upstream to adopt it?
> Worth it? Other comments?
A 15% space gain (under python) w/o any drawbacks? Always worth it. :)
More information about the livecd
mailing list