8:43 a.m.
Dear Fedora community,
several months ago, at the Developer conference in Brno, Software Management
team received a whole bunch of proposals for new functionality in RPM and
related software stack.
We acknowledge the need for some changes in Software Management stack in
Fedora but we don't want to make changes just by guessing what our
users want. Therefore I call to you, consumers of our products (dnf, yum and
rpm): what are the changes that you would like to see in the foreseeable
future (say 2-3 years) and why would you like to see them (what would they
help you with)?
There is already a list of some RFEs on rpm.org wiki, you can use it as an
inspiration, to see what RFEs we have already received:
http://rpm.org/wiki/FeaturePlanning
The only limitation for your requests is our manifest which defines the scope
of SW management stack for the future. It is attached to this email (note that
it's quite extensive but the first part should give you a good image of what is
the planned scope of SW management stack).
Please send your requests as replies to this email so they can be properly
discussed.
After your proposals are filed and discussed, all will be evaluated by our
team and a roadmap with priorities will be created with those selected as
doable and meaningful.
Thank you in advance for your participation
Jan
9:10 a.m.
I'd definitely add a big +1 to "Dependency cleaner"!
Thanks,
Sandro
On Wed, May 22, 2013 at 3:43 PM, Jan Zelený <jzeleny(a)redhat.com> wrote:
Dear Fedora community,
several months ago, at the Developer conference in Brno, Software
Management
team received a whole bunch of proposals for new functionality in RPM and
related software stack.
We acknowledge the need for some changes in Software Management stack in
Fedora but we don't want to make changes just by guessing what our
users want. Therefore I call to you, consumers of our products (dnf, yum
and
rpm): what are the changes that you would like to see in the foreseeable
future (say 2-3 years) and why would you like to see them (what would they
help you with)?
There is already a list of some RFEs on rpm.org wiki, you can use it as an
inspiration, to see what RFEs we have already received:
http://rpm.org/wiki/FeaturePlanning
The only limitation for your requests is our manifest which defines the
scope
of SW management stack for the future. It is attached to this email (note
that
it's quite extensive but the first part should give you a good image of
what is
the planned scope of SW management stack).
Please send your requests as replies to this email so they can be properly
discussed.
After your proposals are filed and discussed, all will be evaluated by our
team and a roadmap with priorities will be created with those selected as
doable and meaningful.
Thank you in advance for your participation
Jan
--
devel mailing list
devel(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
11:35 a.m.
On 05/22/2013 11:16 AM, Christopher Meng wrote:
What about speeding up the yum running?
It's a bit slow now.
+1, and what's worse it tends to significantly degrade as it collects
more history in its databases. Most of my systems have multiple repos:
fedora, rpmfusion-{free,nonfree} and their -updates version, plus
-debuginfo---just getting the new metadata takes significant time. Maybe
redesigning the metadata so that rsync-like protocol would work better?
6:08 a.m.
On 22. 5. 2013 at 23:16:16, Christopher Meng wrote:
What about speeding up the yum running?
It's a bit slow now.
Have you tried using dnf instead of yum? It is much faster.
To be perfectly honest we don't plan to invest much effort in developing new
things for yum, it will more and more shift towards maintenance mode and the
focus will be on dnf.
Thanks
Jan
2:20 p.m.
On 05/23/2013 07:08 AM, Jan Zelený wrote:
Have you tried using dnf instead of yum? It is much faster.
To be perfectly honest we don't plan to invest much effort in developing new
things for yum, it will more and more shift towards maintenance mode and the
focus will be on dnf.
What does the "we" stand for here? If this is Red Hat on the whole, a
broader announcement of the plan would be helpful. I will note that
while dnf is faster overall, it doesn't have many of the important
features of yum and I still ran into bugs in some trivial tests from
time to time.
https://fedoranext.wordpress.com/2013/05/18/status-of-dnf-experimental-fo...
Rahul
2:46 a.m.
On 24. 5. 2013 at 15:20:24, Rahul Sundaram wrote:
On 05/23/2013 07:08 AM, Jan Zelený wrote:
> Have you tried using dnf instead of yum? It is much faster.
>
> To be perfectly honest we don't plan to invest much effort in developing
> new things for yum, it will more and more shift towards maintenance mode
> and the focus will be on dnf.
What does the "we" stand for here? If this is Red Hat on the whole, a
broader announcement of the plan would be helpful. I will note that
while dnf is faster overall, it doesn't have many of the important
features of yum and I still ran into bugs in some trivial tests from
time to time.
https://fedoranext.wordpress.com/2013/05/18/status-of-dnf-experimental-fo... of-yum/
The "we" means the Software management team. Obviously the dnf is still
nowhere near the state of yum but it's continuously getting there. If we don't
stop development of features in yum at some point, it will be impossible to
replace yum with dnf in the future.
If you feel that there are some important features missing, please let us know
which ones. Although we won't automatically prioritize all of them, we will
take your input into consideration. Note that everyone can have his own set of
features that he considers to be important.
Thanks
Jan
8:30 a.m.
On 05/24/2013 09:20 PM, Rahul Sundaram wrote:
On 05/23/2013 07:08 AM, Jan Zelený wrote:
> Have you tried using dnf instead of yum? It is much faster.
>
> To be perfectly honest we don't plan to invest much effort in
> developing new
> things for yum, it will more and more shift towards maintenance mode
> and the
> focus will be on dnf.
What does the "we" stand for here? If this is Red Hat on the whole, a
broader announcement of the plan would be helpful. I will note that
while dnf is faster overall, it doesn't have many of the important
features of yum and I still ran into bugs in some trivial tests from
time to time.
https://fedoranext.wordpress.com/2013/05/18/status-of-dnf-experimental-fo...
Rahul
The "one weird bug" (dnf remove wants to install new packages) mentioned
by the article just got fixed:
https://bugzilla.redhat.com/show_bug.cgi?id=916662
Ales
10:38 a.m.
Therefore I call to you, consumers of our products (dnf, yum and
rpm): what are the changes that you would like to see in the foreseeable
future (say 2-3 years) and why would you like to see them (what would they
help you with)?
Feature: facility for atomic upgrade of an entire set of packages.
Either all succeed, or none succeed; and in bounded time.
This would help with maintaining consistent development environments:
gcc+binutils+gdb, etc. In practice it matters (especially for
cross-platform development, and reproducing bugs reported by remote machines),
even if in theory it shouldn't.
Feature: faster performance.
rpm install/upgrade is 2X too slow. yum install/upgrade is 3X too slow.
"The disk" should be 100% busy from start to finish, and at all times
each CPU core should be either busy or waiting for "the disk".
Specific goal: a full install of 1200 packages totaling 1.5GB of .rpm
expanding to 3.6GB on the target takes five minutes on a common desktop/laptop.
This averages 5MB/s input ("4X" DVD, "32X" CD), and 12MB/s output.
Speed rpm install/upgrade by parallel and pipeline:
Topological sort into tiers. tier 0: no dependencies; tier K+1: dependencies
only in tiers 0..K.
Within a tier: all packages are independent, thus can be done in parallel.
Within a package: all of the reads that occur before the first write
can be done in parallel with *ANY* other package, regardless of dependencies.
Thus parsing and decompression of .rpm into memory (or a unique staging
directory within a tmpfs) can be parallel regardless of dependencies.
Perhaps writes to database must be restricted to one package at a time (despite
logical parallelism within a tier), but this can be pipelined with the read phase.
Speed yum install/upgrade by speeding rpm, plus distribute "Verifying ..."
and symlink data-basing out of a separate pass and into the rpm pipeline.
--
6:14 a.m.
On 22. 5. 2013 at 08:38:35, John Reiser wrote:
> Therefore I call to you, consumers of our products (dnf, yum
and
> rpm): what are the changes that you would like to see in the foreseeable
> future (say 2-3 years) and why would you like to see them (what would they
> help you with)?
Feature: facility for atomic upgrade of an entire set of packages.
Either all succeed, or none succeed; and in bounded time.
This would help with maintaining consistent development environments:
gcc+binutils+gdb, etc. In practice it matters (especially for
cross-platform development, and reproducing bugs reported by remote
machines), even if in theory it shouldn't.
I can recommend you the yum-fs-snapshot plugin for that. Doing snapshots is
currently the only feasible way to ensure atomicity of multi-package
transactions. It's important to note that restrictions for this don't come
from rpm but rather from kernel (e.g. filesystem features).
Feature: faster performance.
rpm install/upgrade is 2X too slow. yum install/upgrade is 3X too slow.
"The disk" should be 100% busy from start to finish, and at all times
each CPU core should be either busy or waiting for "the disk".
Specific goal: a full install of 1200 packages totaling 1.5GB of .rpm
expanding to 3.6GB on the target takes five minutes on a common
desktop/laptop. This averages 5MB/s input ("4X" DVD, "32X" CD), and
12MB/s
output.
Speed rpm install/upgrade by parallel and pipeline:
Topological sort into tiers. tier 0: no dependencies; tier K+1:
dependencies only in tiers 0..K.
Within a tier: all packages are independent, thus can be done in parallel.
Within a package: all of the reads that occur before the first write can be
done in parallel with *ANY* other package, regardless of dependencies. Thus
parsing and decompression of .rpm into memory (or a unique staging
directory within a tmpfs) can be parallel regardless of dependencies.
Perhaps writes to database must be restricted to one package at a time
(despite logical parallelism within a tier), but this can be pipelined with
the read phase.
Speed yum install/upgrade by speeding rpm, plus distribute "Verifying ..."
and symlink data-basing out of a separate pass and into the rpm pipeline.
I'm pretty sure we don't have the manpower for this sort of optimization but
we'll take it into consideration.
Thank you
Jan
10:47 a.m.
2013/5/22 Jan Zelený <jzeleny(a)redhat.com>:
Dear Fedora community,
several months ago, at the Developer conference in Brno, Software Management
team received a whole bunch of proposals for new functionality in RPM and
related software stack.
As a packager, some way to transparently handle an upgrade when a
directory changes to a symlink or vice-versa.
A minor, extra checking one that I was hit recently was lack of checking
a package that have a symlink to /builddir/some-where, and would only work on
my computer because the symlink did point to /home/pcpa/rpmbuild/BUILD/...
As an user, some way to, but mostly adding some policy, to have multiple
sonames of a library installed. Usually only useful to avoid a window of time
with broken dependencies, but sometimes useful to have some package
that only works with an older version of some library functional, without
blocking an update.
Thank you in advance for your participation
Jan
Paulo
1:33 p.m.
Paulo César Pereira de Andrade wrote:
As an user, some way to, but mostly adding some policy, to have
multiple
sonames of a library installed. Usually only useful to avoid a window of time
with broken dependencies, but sometimes useful to have some package
that only works with an older version of some library functional, without
blocking an update.
Seconded. Although old and possibly insecure libraries shouldn't be kept
around indefinitely, it really ought to be possible to keep the
previous version available for some time. We already do it with kernels,
so doing it with libraries can't be too difficult. There must be a
better way to handle soname bumps like the one in LibPNG that is being
discussed right now.
If automatic cleanup of dependencies gets implemented, then an old
version of a library should be removed when no installed packages
require it anymore.
Björn Persson
2:06 p.m.
On Wed, 2013-05-22 at 20:33 +0200, Björn Persson wrote:
Paulo César Pereira de Andrade wrote:
> As an user, some way to, but mostly adding some policy, to have multiple
> sonames of a library installed. Usually only useful to avoid a window of time
> with broken dependencies, but sometimes useful to have some package
> that only works with an older version of some library functional, without
> blocking an update.
Seconded. Although old and possibly insecure libraries shouldn't be kept
around indefinitely, it really ought to be possible to keep the
previous version available for some time. We already do it with kernels,
so doing it with libraries can't be too difficult. There must be a
better way to handle soname bumps like the one in LibPNG that is being
discussed right now.
If automatic cleanup of dependencies gets implemented, then an old
version of a library should be removed when no installed packages
require it anymore.
I don't see why handling this on the package management side seems
advisable. We already have a perfectly reasonable distro mechanism for
shipping multiple sonames of a library if we want to do that. We just
try to avoid it wherever possible, because it's not a good idea.
I'm not a genius, but I can't see a way to do this on the package
management side which wouldn't be a horrible hack job.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
6:18 a.m.
On 22. 5. 2013 at 12:47:58, Paulo César Pereira de Andrade wrote:
2013/5/22 Jan Zelený <jzeleny(a)redhat.com>:
> Dear Fedora community,
> several months ago, at the Developer conference in Brno, Software
> Management team received a whole bunch of proposals for new functionality
> in RPM and related software stack.
As a packager, some way to transparently handle an upgrade when a
directory changes to a symlink or vice-versa.
A minor, extra checking one that I was hit recently was lack of checking
a package that have a symlink to /builddir/some-where, and would only work
on my computer because the symlink did point to
/home/pcpa/rpmbuild/BUILD/...
As an user, some way to, but mostly adding some policy, to have multiple
sonames of a library installed. Usually only useful to avoid a window of
time with broken dependencies, but sometimes useful to have some package
that only works with an older version of some library functional, without
blocking an update.
I'm not sure I understand you correctly. Do you actually ask for multiple
versions of a single package to be installed simultaneously? If that's the
case, I can say no right away ;-)
The other thing I sense you might have on your mind is to remove from update
transaction those packages that are held back by some requirements and
continue with the transaction. Is that correct?
Thanks
Jan
10:55 a.m.
Performance improvement: improve scaling to 5K+ installed packages.
Since the TeXLive repackaging, my laptop several thousand packages,
about half of which are TeX-related (I like to have a fairly full TeX
install with all the docs). There are two noticable problems: yum slows
down considerably (esp. in transaction check operations), and PackageKit
can't handle upgrades of more than 2500 packages at a time.
- Michael
6:21 a.m.
On 22. 5. 2013 at 10:55:14, Michael Ekstrand wrote:
Performance improvement: improve scaling to 5K+ installed packages.
Since the TeXLive repackaging, my laptop several thousand packages,
about half of which are TeX-related (I like to have a fairly full TeX
install with all the docs). There are two noticable problems: yum slows
down considerably (esp. in transaction check operations), and PackageKit
can't handle upgrades of more than 2500 packages at a time.
Well, PackageKit is already out of our scope but I will talk to Richard about
this.
On the rpm side - we are already working on that
On the yum side - have you tried dnf? It should handle this much better.
Thanks
Jan
8:17 a.m.
On 05/23/2013 06:21 AM, Jan Zelený wrote:
On 22. 5. 2013 at 10:55:14, Michael Ekstrand wrote:
> Performance improvement: improve scaling to 5K+ installed packages.
> Since the TeXLive repackaging, my laptop several thousand packages,
> about half of which are TeX-related (I like to have a fairly full TeX
> install with all the docs). There are two noticable problems: yum slows
> down considerably (esp. in transaction check operations), and PackageKit
> can't handle upgrades of more than 2500 packages at a time.
Well, PackageKit is already out of our scope but I will talk to Richard about
this.
OK. Relevant bug report: https://bugzilla.redhat.com/show_bug.cgi?id=894731
On the rpm side - we are already working on that
On the yum side - have you tried dnf? It should handle this much better.
Not yet. I plan to upgrade my laptop to F19 sometime in the beta or RC
time frame, and was thinking of trying dnf then.
- Michael
8:34 a.m.
On Wed, May 22, 2013 at 11:55 AM, Michael Ekstrand <michael(a)elehack.net> wrote:
Performance improvement: improve scaling to 5K+ installed packages.
* Amen. This is particularly compounded by poor caching default
behavior, so that a few yum commands in a row each wind up reaching
out to downloading metadata again, and again, and again.
I think this can be addressed by moving the metadata updates to a
different function, and calling it *separately* only as needed. The
Debian "apt" tool does this quite effectively.
* Script parseable output without extraneous labeling.
The output of "yum list is cluttered with unnecessary headers, and
whitespace handling that winds up trimming package names or inserting
extraneous new lines. I'd fiind it invaluable if "yum list --tsv" gave
a tab separated variables list of:
name version release arch reponame
And leave *OFF* the "Installed Packages" and "Available Packages"
entries for such tab separated variable output. I loathe having to
sort those out for scriptable operations.
4:10 a.m.
On Sat, May 25, 2013 at 09:34:32AM -0400, Nico Kadel-Garcia wrote:
I think this can be addressed by moving the metadata updates to a
different function, and calling it *separately* only as needed. The
Debian "apt" tool does this quite effectively.
You can kind of simulate that by forcing yum to run from cache only. Use
the -C flag for that.
Lars
4:12 a.m.
Le Dim 26 mai 2013 11:10, Lars Seipel a écrit :
On Sat, May 25, 2013 at 09:34:32AM -0400, Nico Kadel-Garcia wrote:
> I think this can be addressed by moving the metadata updates to a
> different function, and calling it *separately* only as needed. The
> Debian "apt" tool does this quite effectively.
You can kind of simulate that by forcing yum to run from cache only. Use
the -C flag for that.
But the basic reason stands: yum only handles well perfectly synced repos,
and workarounds this by trying to get the freshest metadata all the time.
If yum's error handling was better, it would not fail as soon as there is
the slightest sync error, a semi-stale repo, a caching proxy, etc, etc
--
Nicolas Mailhot
2:32 a.m.
On 25. 5. 2013 at 09:34:32, Nico Kadel-Garcia wrote:
On Wed, May 22, 2013 at 11:55 AM, Michael Ekstrand
<michael(a)elehack.net>
wrote:
> Performance improvement: improve scaling to 5K+ installed
packages.
* Amen. This is particularly compounded by poor caching default
behavior, so that a few yum commands in a row each wind up reaching
out to downloading metadata again, and again, and again.
I think this can be addressed by moving the metadata updates to a
different function, and calling it *separately* only as needed. The
Debian "apt" tool does this quite effectively.
Unfortunately there is not much we can do about this. Debian has completely
different repository policy - they keep all versions of packages in the repo so
there is no need to update metadata on client machines every time.
* Script parseable output without extraneous labeling.
The output of "yum list is cluttered with unnecessary headers, and
whitespace handling that winds up trimming package names or inserting
extraneous new lines. I'd fiind it invaluable if "yum list --tsv" gave
a tab separated variables list of:
name version release arch reponame
And leave *OFF* the "Installed Packages" and "Available Packages"
entries for such tab separated variable output. I loathe having to
sort those out for scriptable operations.
Interesting idea, might be worth looking into.
Thanks
Jan
2:43 a.m.
On 05/27/2013 09:32 AM, Jan Zelený wrote:
On 25. 5. 2013 at 09:34:32, Nico Kadel-Garcia wrote:
> On Wed, May 22, 2013 at 11:55 AM, Michael Ekstrand <michael(a)elehack.net>
wrote:
>> Performance improvement: improve scaling to 5K+ installed packages.
>
> * Amen. This is particularly compounded by poor caching default
> behavior, so that a few yum commands in a row each wind up reaching
> out to downloading metadata again, and again, and again.
>
> I think this can be addressed by moving the metadata updates to a
> different function, and calling it *separately* only as needed. The
> Debian "apt" tool does this quite effectively.
Unfortunately there is not much we can do about this. Debian has completely
different repository policy - they keep all versions of packages in the repo so
there is no need to update metadata on client machines every time.
I don't quite understand this comment.
Debian repository policy varies quite a bit. Some repositories keep old
versions, some don't. Mostly the latter, actually, because not all
repository managers (there a couple of implementations) can deal with
multiple versions for a single package/architecture combination.
As far as I can tell, the main difference is that apt-get and apt-cache
read very few, relatively large files at the beginning, so they don't
block on disk reads early.
dpkg, on the other hand, uses a database scatter across many small files
on disk, so you get the delay only when you actually install or remove
any packages. At the beginning, this is quite fast, but eventually, the
files will be scattered quite badly, and there is a considerable delay
at this step.
--
Florian Weimer / Red Hat Product Security Team
3:24 a.m.
On 27. 5. 2013 at 09:43:15, Florian Weimer wrote:
On 05/27/2013 09:32 AM, Jan Zelený wrote:
> On 25. 5. 2013 at 09:34:32, Nico Kadel-Garcia wrote:
>> On Wed, May 22, 2013 at 11:55 AM, Michael Ekstrand <michael(a)elehack.net>
>
> wrote:
>>> Performance improvement: improve scaling to 5K+ installed packages.
>>
>> * Amen. This is particularly compounded by poor caching default
>> behavior, so that a few yum commands in a row each wind up reaching
>> out to downloading metadata again, and again, and again.
>>
>> I think this can be addressed by moving the metadata updates to a
>> different function, and calling it *separately* only as needed. The
>> Debian "apt" tool does this quite effectively.
>
> Unfortunately there is not much we can do about this. Debian has
> completely
> different repository policy - they keep all versions of packages in the
> repo so there is no need to update metadata on client machines every
> time.
I don't quite understand this comment.
Debian repository policy varies quite a bit. Some repositories keep old
versions, some don't. Mostly the latter, actually, because not all
repository managers (there a couple of implementations) can deal with
multiple versions for a single package/architecture combination.
I'm sorry but the Debian repositories say otherwise, see Iceweasel for
instance:
http://ftp.cz.debian.org/debian/pool/main/i/iceweasel/
Multiple old versions are kept in there. That's why they don't need to update
their metadata every single time - the old ones are simply still valid.
As far as I can tell, the main difference is that apt-get and
apt-cache
read very few, relatively large files at the beginning, so they don't
block on disk reads early.
dpkg, on the other hand, uses a database scatter across many small files
on disk, so you get the delay only when you actually install or remove
any packages. At the beginning, this is quite fast, but eventually, the
files will be scattered quite badly, and there is a considerable delay
at this step.
This part is about disk read-write but that was not what I was writing about.
From my experience users mostly complain about the metadata download
which is
explained above.
Thanks
Jan
3:53 a.m.
On 05/27/2013 10:24 AM, Jan Zelený wrote:
> Debian repository policy varies quite a bit. Some repositories
keep old
> versions, some don't. Mostly the latter, actually, because not all
> repository managers (there a couple of implementations) can deal with
> multiple versions for a single package/architecture combination.
I'm sorry but the Debian repositories say otherwise, see Iceweasel for
instance:
http://ftp.cz.debian.org/debian/pool/main/i/iceweasel/
Multiple old versions are kept in there. That's why they don't need to update
their metadata every single time - the old ones are simply still valid.
Ah! That's the physical directory structure on the mirrors. Package
pools where introduced to allow physical sharing of
.deb/.diff.gz/.orig.tar.gz files across architectures and between
different versions of the distribution. I've checked a few of the
examples in the directory, and all the multiple versions I checked are a
result of that. For example, 17.0.6esr-1 has not yet been built on
armhf and mips, so unstable still contains the previous 17.0.5esr-1
versions, and both are listed in the directory. Some older versions are
only present in experimental, which is not subject to autobuilding on
all architectures, and that's why they haven't been superseded by newer
versions yet.
They key point is that none of this triggers listing of an outdated
package version in the distribution-specific package manifest, so it's
completely invisible to clients which just look at the manifest. For
example,
<http://ftp.de.debian.org/debian/dists/unstable/main/binary-amd64/Packages...
contains just one record for the iceweasel package:
Package: iceweasel
Version: 17.0.6esr-1
Directory listing is not consistent across mirrors because it depends on
web server configuration, so this really isn't something apt-get could
use anyway.
Debian might eventually start shipping outdated versions if they were
used as a build dependency for another package, for strict copyleft
compliance, but I don't think this has yet been implemented.
This part is about disk read-write but that was not what I was
writing about.
From my experience users mostly complain about the metadata download which is
explained above.
Metadata download is mainly accelerated by two things: First of all,
it's not implicit you have to manually requested it with "apt-get
update". And there package diffs, which are ed-style diffs of the
Packages file I mentioned above. This approach would work quite well
for primary.xml because it doesn't contain cross-references between
packages using non-natural keys. It doesn't work for the SQLite
database, either in binary or SQL dump format, because of the reliance
on artificial primary keys (such as package IDs).
However, for many users that follow unstable or testing, package diffs
are currently slower than downloading the full Packages file because the
diffs are incremental (i.e., they contain the changes from file version
N to N+1, and you have to apply all of them to get to the current
version) and apt-get can easily write 100 MB or more because the
Packages file is rewritten locally multiple times.
--
Florian Weimer / Red Hat Product Security Team
4:48 a.m.
And there package diffs, which are ed-style diffs of the
Packages file I mentioned above. This approach would work quite well
for primary.xml because it doesn't contain cross-references between
packages using non-natural keys. It doesn't work for the SQLite
database, either in binary or SQL dump format, because of the reliance
on artificial primary keys (such as package IDs).
I've once tried this. With about 10k packages in fedora-updates, the delta
over 2-3 days was +491 -479. Assuming deletions are cheap, the delta should
ideally be 5%. As expected, binary bsddiff yields much bigger (~29%) delta.
Very roughly, it's 5% that really describe new packages, plus an almost
constant 24% overhead to fix up the inevitable changes in surrogate keys.
Not as bad as I was afraid, but still not worth it (IMO).
So, we need *.xml deltas. Yum can rebuild xml => .sqlite locally, but
this needs quite a lot of memory and takes TENS of seconds. Add the time
needed to patch the quite large uncompressed xml file, and suddenly the
fact that you're downloading just 1/10th of data hardly pays off
(ignoring very specific use cases, like mobile data for a moment)
For DNF, it's different. It has to rebuild xml => .solv anyway, so this
comes for free.
However, for many users that follow unstable or testing, package
diffs
are currently slower than downloading the full Packages file because the
diffs are incremental (i.e., they contain the changes from file version
N to N+1, and you have to apply all of them to get to the current
version) and apt-get can easily write 100 MB or more because the
Packages file is rewritten locally multiple times.
Yes, patch chaining should be avoided. I'd like to use N => 1 deltas,
that could be applied to many recent snapshots.
5:17 a.m.
On 05/27/2013 11:48 AM, Zdenek Pavlas wrote:
> And there package diffs, which are ed-style diffs of the
> Packages file I mentioned above. This approach would work quite well
> for primary.xml because it doesn't contain cross-references between
> packages using non-natural keys. It doesn't work for the SQLite
> database, either in binary or SQL dump format, because of the reliance
> on artificial primary keys (such as package IDs).
I've once tried this. With about 10k packages in fedora-updates, the delta
over 2-3 days was +491 -479. Assuming deletions are cheap, the delta should
ideally be 5%. As expected, binary bsddiff yields much bigger (~29%) delta.
A line-wise diff is much smaller because dependencies and package
descriptions mostly stay the same. (This assumes consistent sorting of
the primary.xml file.)
Can you point me to the primary.xml -> SQLite translation in yum? I've
got a fairly efficient primary.xml parser. It might be interesting to
see if it's possible to reduce the latency introduced by the SQLite
conversion to close to zero. (Decompression and INSERTs can be
interleaved with downloading, and maybe the index creation improvements
in SQLite are sufficient these days.)
> However, for many users that follow unstable or testing, package
diffs
> are currently slower than downloading the full Packages file because the
> diffs are incremental (i.e., they contain the changes from file version
> N to N+1, and you have to apply all of them to get to the current
> version) and apt-get can easily write 100 MB or more because the
> Packages file is rewritten locally multiple times.
Yes, patch chaining should be avoided. I'd like to use N => 1 deltas,
that could be applied to many recent snapshots.
The Debian package diffs could be combined efficiently in the client
because it's possible to combine diffs for two adjacent versions without
actually knowing what the old or new versions look like. But this
hasn't been implemented in APT because ABI impact (which is a bit
puzzling, but anyway). Instead, the diffs should soon be combined on
the archive side.
--
Florian Weimer / Red Hat Product Security Team
7:53 a.m.
On Mon, May 27, 2013 at 6:17 AM, Florian Weimer <fweimer(a)redhat.com> wrote:
On 05/27/2013 11:48 AM, Zdenek Pavlas wrote:
>>
>> And there package diffs, which are ed-style diffs of the
>> Packages file I mentioned above. This approach would work quite well
>> for primary.xml because it doesn't contain cross-references between
>> packages using non-natural keys. It doesn't work for the SQLite
>> database, either in binary or SQL dump format, because of the reliance
>> on artificial primary keys (such as package IDs).
>
>
> I've once tried this. With about 10k packages in fedora-updates, the delta
> over 2-3 days was +491 -479. Assuming deletions are cheap, the delta
> should
> ideally be 5%. As expected, binary bsddiff yields much bigger (~29%)
> delta.
A line-wise diff is much smaller because dependencies and package
descriptions mostly stay the same. (This assumes consistent sorting of the
primary.xml file.)
The "diffs" are not the problem. The problem is hte excessively
frequent downloads of the repodata, which are compressed binaries with
checksums, not published as deltas or diffs. The result is a grossly
inefficient and far-too-frequent download of upstream repository
information. It's not the local SQLite database operations in the yum
cache that are killing me, at least, it's the short "metadata_expire"
in /etc/yum.conf.
Very few of us really need our metadata expired between our first cup
of coffee in the morning, and luncthtime. And very few of us need
yum-updatesd and other auto-magic update tools grinding our host and
our proxies bandwidth for several 20 Megabyte files every few hours,
nor grinding our local disk with the uncompressed *80 Megabyte* file
of primary.xml sitting in /var/cache/yum/.
This is an inherent problem with the "let's store more, and more, and
more data in the database". The databases for yum have gotten bulky
and cumbersome, and the automatic churn involved in updating them with
fresh repodata has become quite large.
Can you point me to the primary.xml -> SQLite translation in yum?
I've got
a fairly efficient primary.xml parser. It might be interesting to see if
it's possible to reduce the latency introduced by the SQLite conversion to
close to zero. (Decompression and INSERTs can be interleaved with
downloading, and maybe the index creation improvements in SQLite are
sufficient these days.)
Good luck with that! It's not what I, personally, was just looking
for, but improving that would be nice. But the SQLite files are
getting larger, and larger, and larger. At 80 Meg and counting for the
Fedora primary.sqlie, it's getting out of hand.
>> However, for many users that follow unstable or testing,
package diffs
>> are currently slower than downloading the full Packages file because the
>> diffs are incremental (i.e., they contain the changes from file version
>> N to N+1, and you have to apply all of them to get to the current
>> version) and apt-get can easily write 100 MB or more because the
>> Packages file is rewritten locally multiple times.
>
>
> Yes, patch chaining should be avoided. I'd like to use N => 1 deltas,
> that could be applied to many recent snapshots.
And it has little to do with the yum issue I've raised, which is
entirely about the metadata. My personal experience is that the use of
"deltarpms" is a bandwidth and resource issue completely overshadowed
by the constant grinding of disk and bandwidth for hte metadata.
10:29 a.m.
Can you point me to the primary.xml -> SQLite translation in yum?
I've
got a fairly efficient primary.xml parser.
Just set mddownloadpolicy=xml in yum.conf. It should work, but
since downloading sqlite.bz2 is much better, very few use this.
Yum uses fairly efficient parser, written in C, using libxml2
(the yum-metadata-parser package). It's always bundled, because
Yum has to support xml-only repositories anyway.
Oh, there's a typo in yum.conf.5 .. fixed.
It might be interesting to
see if it's possible to reduce the latency introduced by the SQLite
conversion to close to zero. (Decompression and INSERTs can be
interleaved with downloading, and maybe the index creation improvements
in SQLite are sufficient these days.)
We have to checksum the downloaded data before processing, and this
kills pipelining. Also, when updating primary_db with a bunch of INSERTs
and DELETEs, your database differs from the one on server:
- different *.sqlite checksum
- different pkgId <=> PkgKey mapping
- different order of packages from SELECTs
For speed, Yum joins primary_db and filelists_db via pkgKey,
so #2 breaks Yum, unless you always download/delta-update both-
so this kills the win in "we don't need filelists" case.
1:33 p.m.
On 05/27/2013 10:24 AM, Jan Zelený wrote:
> As far as I can tell, the main difference is that apt-get and
apt-cache
> read very few, relatively large files at the beginning, so they don't
> block on disk reads early.
>
> dpkg, on the other hand, uses a database scatter across many small files
> on disk, so you get the delay only when you actually install or remove
> any packages. At the beginning, this is quite fast, but eventually, the
> files will be scattered quite badly, and there is a considerable delay
> at this step.
This part is about disk read-write but that was not what I was writing about.
From my experience users mostly complain about the metadata download which is
explained above.
Users sometimes misdiagnose issues, *especially* when complaining. 8-)
I did some tests and cold cache performance tests on an old Debian
installation. Performance with cold caches is more than adequate.
Full-text searches take about two seconds. Package installation reaches
the confirmation prompt (after dependency resolution) in less than three
seconds, even for ridiculously complex tasks such as installing the
entire KDE desktop (365 additional packages on my test system).
In contrast, on Fedora or RHEL systems, 30 seconds for dependency
resolution with a cold cache are common, plus around 6 seconds for
loading all the Python code for yum. /usr/bin/time reports much more
I/O than it does non Debian (about ten times as much, as reflected in
the wall clock time).
Regarding network traffic, in addition to the explicit "apt-get update"
step (which avoids time-consuming downloads in the first place), Debian
also forces users to pick a single mirror close to them. The Fedora
master mirror list instructs yum to pick a mirror from a larger list,
which causes much greater variance in performance.
--
Florian Weimer / Red Hat Product Security Team
4:06 p.m.
On Wed, 2013-05-29 at 20:33 +0200, Florian Weimer wrote:
I did some tests and cold cache performance tests on an old Debian
installation. Performance with cold caches is more than adequate.
Full-text searches take about two seconds. Package installation reaches
the confirmation prompt (after dependency resolution) in less than three
seconds, even for ridiculously complex tasks such as installing the
entire KDE desktop (365 additional packages on my test system).
In contrast, on Fedora or RHEL systems, 30 seconds for dependency
resolution with a cold cache are common, plus around 6 seconds for
loading all the Python code for yum. /usr/bin/time reports much more
I/O than it does non Debian (about ten times as much, as reflected in
the wall clock time).
Yeh, it's almost like yum is dealing with more data:
http://yum.baseurl.org/wiki/apt2yum#Generalpoints
4:44 a.m.
On 05/29/2013 11:06 PM, James Antill wrote:
On Wed, 2013-05-29 at 20:33 +0200, Florian Weimer wrote:
> I did some tests and cold cache performance tests on an old Debian
> installation. Performance with cold caches is more than adequate.
> Full-text searches take about two seconds. Package installation reaches
> the confirmation prompt (after dependency resolution) in less than three
> seconds, even for ridiculously complex tasks such as installing the
> entire KDE desktop (365 additional packages on my test system).
>
> In contrast, on Fedora or RHEL systems, 30 seconds for dependency
> resolution with a cold cache are common, plus around 6 seconds for
> loading all the Python code for yum. /usr/bin/time reports much more
> I/O than it does non Debian (about ten times as much, as reflected in
> the wall clock time).
Yeh, it's almost like yum is dealing with more data:
http://yum.baseurl.org/wiki/apt2yum#Generalpoints
Are the yum numbers current? I don't think filelists are processed by
current yum if that can be avoided.
Anyway, let's hope that DNF will make all that history. 8-)
--
Florian Weimer / Red Hat Product Security Team
4:23 a.m.
On Wed, May 29, 2013 at 08:33:16PM +0200, Florian Weimer wrote:
Users sometimes misdiagnose issues, *especially* when complaining.
8-)
I did some tests and cold cache performance tests on an old Debian
installation. Performance with cold caches is more than adequate.
Full-text searches take about two seconds. Package installation reaches
the confirmation prompt (after dependency resolution) in less than three
seconds, even for ridiculously complex tasks such as installing the entire
KDE desktop (365 additional packages on my test system).
In contrast, on Fedora or RHEL systems, 30 seconds for dependency
resolution with a cold cache are common, plus around 6 seconds for loading
all the Python code for yum. /usr/bin/time reports much more I/O than it
does non Debian (about ten times as much, as reflected in the wall clock
time).
If you want to do some measurements for depsolving without the python
overhead, you can try the "libsolv-demo" package (it's a subpackage
of libsolv, so it should be available for Fedora). It consists of
a single binary, "/usr/bin/solv", that is a little package manager
on top of libsolv.
Cheers,
Michael.
--
Michael Schroeder mls(a)suse.de
SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg
main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}
3:31 a.m.
Am 27.05.2013 09:32, schrieb Jan Zelený:
On 25. 5. 2013 at 09:34:32, Nico Kadel-Garcia wrote:
> On Wed, May 22, 2013 at 11:55 AM, Michael Ekstrand <michael(a)elehack.net>
wrote:
>> Performance improvement: improve scaling to 5K+ installed packages.
>
> * Amen. This is particularly compounded by poor caching default
> behavior, so that a few yum commands in a row each wind up reaching
> out to downloading metadata again, and again, and again.
>
> I think this can be addressed by moving the metadata updates to a
> different function, and calling it *separately* only as needed. The
> Debian "apt" tool does this quite effectively.
Unfortunately there is not much we can do about this. Debian has completely
different repository policy - they keep all versions of packages in the repo so
there is no need to update metadata on client machines every time
what does keep old versions or not change besides you need
to do "apt-get update" if you want to find "apt-get upgrade"
to find new packages?
the real problem is that the metadata are *way too fat* in Fedora
after a "yum clean metadata && yum update" on a slow line you
have to wait a very long time and even the download of the
presto-metadata often is larger and takes longer as the
packages which are updated in reality
hey on my 100 Mbit all is nice and fine but on a machine behind
DSL with around 100 KB/Second it is way too slow and large and
i refuse to imagine how this feels on a 56kbit modem
1:51 a.m.
On 27. 5. 2013 at 10:31:29, Reindl Harald wrote:
Am 27.05.2013 09:32, schrieb Jan Zelený:
> On 25. 5. 2013 at 09:34:32, Nico Kadel-Garcia wrote:
>> On Wed, May 22, 2013 at 11:55 AM, Michael Ekstrand <michael(a)elehack.net>
>
> wrote:
>>> Performance improvement: improve scaling to 5K+ installed packages.
>>
>> * Amen. This is particularly compounded by poor caching default
>> behavior, so that a few yum commands in a row each wind up reaching
>> out to downloading metadata again, and again, and again.
>>
>> I think this can be addressed by moving the metadata updates to a
>> different function, and calling it *separately* only as needed. The
>> Debian "apt" tool does this quite effectively.
>
> Unfortunately there is not much we can do about this. Debian has
> completely
> different repository policy - they keep all versions of packages in the
> repo so there is no need to update metadata on client machines every time
what does keep old versions or not change besides you need
to do "apt-get update" if you want to find "apt-get upgrade"
to find new packages?
Something like that, yes. If you don't update your metadata you won't get
updates but installation will still work, even if the installed version is not
the latest available.
the real problem is that the metadata are *way too fat* in Fedora
That's not "the real problem". It's just one of the problems which in
combination cause this pain.
after a "yum clean metadata && yum update" on a
slow line you
have to wait a very long time and even the download of the
presto-metadata often is larger and takes longer as the
packages which are updated in reality
hey on my 100 Mbit all is nice and fine but on a machine behind
DSL with around 100 KB/Second it is way too slow and large and
i refuse to imagine how this feels on a 56kbit modem
I couldn't agree more. But as I have said, we need to find the most simple and
unintrusive things that can be done to improve this. For instance: file lists
take a considerable portion of the entire metadata size. But if we were to
remove them, things like "yum install /usr/bin/vim" would not work any more.
And you get similar scenario with almost all the metadata that we store - we
store them for a reason and without them some things that people use will not
work.
Thanks
Jan
3:39 a.m.
On 28.05.2013 09:51, Jan Zelený wrote:
I couldn't agree more. But as I have said, we need to find the most simple and
unintrusive things that can be done to improve this. For instance: file lists
take a considerable portion of the entire metadata size. But if we were to
remove them, things like "yum install /usr/bin/vim" would not work any more.
And you get similar scenario with almost all the metadata that we store - we
store them for a reason and without them some things that people use will not
work.
Not so unintrusive, but would it be acceptable if we merge all .sqlite
DBs from all repos in a single .sqlite with tree-like schema? Let say we
achieve overall size reduction by factor of 4, at the price of more
complicated but faster SQL queries. [I hope that such a change will also
make the incremental by the XML sources easier]
I am going to experiment this, if make sense ...
5:54 a.m.
On 28. 5. 2013 at 11:39:35, Alek Paunov wrote:
On 28.05.2013 09:51, Jan Zelený wrote:
> I couldn't agree more. But as I have said, we need to find the most simple
> and unintrusive things that can be done to improve this. For instance:
> file lists take a considerable portion of the entire metadata size. But
> if we were to remove them, things like "yum install /usr/bin/vim" would
> not work any more. And you get similar scenario with almost all the
> metadata that we store - we store them for a reason and without them some
> things that people use will not work.
Not so unintrusive, but would it be acceptable if we merge all .sqlite
DBs from all repos in a single .sqlite with tree-like schema? Let say we
achieve overall size reduction by factor of 4, at the price of more
complicated but faster SQL queries. [I hope that such a change will also
make the incremental by the XML sources easier]
I am going to experiment this, if make sense ...
Perhaps it's just that I don't fully understand your vision but I'm not sure
that's a feasible solution. How exactly would you solve the fact that users
have different repos enabled on their machines?
Or did you mean merging them on the client side? That would not solve the
issue of data download.
Thanks
Jan
6:59 a.m.
On 28.05.2013 13:54, Jan Zelený wrote:
On 28. 5. 2013 at 11:39:35, Alek Paunov wrote:
> On 28.05.2013 09:51, Jan Zelený wrote:
>> I couldn't agree more. But as I have said, we need to find the most simple
>> and unintrusive things that can be done to improve this. For instance:
>> file lists take a considerable portion of the entire metadata size. But
>> if we were to remove them, things like "yum install /usr/bin/vim"
would
>> not work any more. And you get similar scenario with almost all the
>> metadata that we store - we store them for a reason and without them some
>> things that people use will not work.
>
> Not so unintrusive, but would it be acceptable if we merge all .sqlite
> DBs from all repos in a single .sqlite with tree-like schema? Let say we
> achieve overall size reduction by factor of 4, at the price of more
> complicated but faster SQL queries. [I hope that such a change will also
> make the incremental by the XML sources easier]
>
> I am going to experiment this, if make sense ...
Perhaps it's just that I don't fully understand your vision but I'm not sure
that's a feasible solution. How exactly would you solve the fact that users
have different repos enabled on their machines?
Or did you mean merging them on the client side? That would not solve the
issue of data download.
Oh sorry. On the client side of course - these which are under the
/var/cache/yum tree. My question was more targeted on the complains
about 1) metadata local size, 2) speed and 3) capabilities of the local
yum queries.
[Furthermore you currently reformulating "Package Management" as
"Software Lifecycle Management", so it would be normal for me to expect
that the data backend will have to be enhanced accordingly. Or will
libsolv stores be enough for all?]
Regarding the metadata download speedup, I completely agree with Florian
and others on the thread, that current bulk downloads should be replaced
with XML only downloads and incremental update of the local DB as a
first step and introducing *optional* metadata services (just XML git or
something smarter) for the mirrors which are willing to upgrade.
Thank you,
Alek
1:38 p.m.
Dne Út 28. května 2013 14:59:20, Alek Paunov napsal(a):
On 28.05.2013 13:54, Jan Zelený wrote:
> On 28. 5. 2013 at 11:39:35, Alek Paunov wrote:
>> On 28.05.2013 09:51, Jan Zelený wrote:
>>> I couldn't agree more. But as I have said, we need to find the most
>>> simple
>>> and unintrusive things that can be done to improve this. For instance:
>>> file lists take a considerable portion of the entire metadata size. But
>>> if we were to remove them, things like "yum install /usr/bin/vim"
would
>>> not work any more. And you get similar scenario with almost all the
>>> metadata that we store - we store them for a reason and without them
>>> some
>>> things that people use will not work.
>>
>> Not so unintrusive, but would it be acceptable if we merge all .sqlite
>> DBs from all repos in a single .sqlite with tree-like schema? Let say we
>> achieve overall size reduction by factor of 4, at the price of more
>> complicated but faster SQL queries. [I hope that such a change will also
>> make the incremental by the XML sources easier]
>>
>> I am going to experiment this, if make sense ...
>
> Perhaps it's just that I don't fully understand your vision but I'm not
> sure that's a feasible solution. How exactly would you solve the fact
> that users have different repos enabled on their machines?
>
> Or did you mean merging them on the client side? That would not solve the
> issue of data download.
Oh sorry. On the client side of course - these which are under the
/var/cache/yum tree. My question was more targeted on the complains
about 1) metadata local size, 2) speed and 3) capabilities of the local
yum queries.
I'm not sure merging database would help in any of these areas. But then again
I can be wrong. If you plan to do the testing, I am very curious about the
results.
[Furthermore you currently reformulating "Package
Management" as
"Software Lifecycle Management", so it would be normal for me to expect
that the data backend will have to be enhanced accordingly. Or will
libsolv stores be enough for all?]
This has to be discussed along the way, at this point it is too early to say
what will happen to metadata.
Regarding the metadata download speedup, I completely agree with
Florian
and others on the thread, that current bulk downloads should be replaced
with XML only downloads and incremental update of the local DB as a
first step and introducing *optional* metadata services (just XML git or
something smarter) for the mirrors which are willing to upgrade.
We are performing some tests and so far it seems that for yum this is not the
way to go, as the speed gain would be compensated by the local yum db rebuild.
We will continue to work on this but so far it seems that dnf is the way to go
so the optimization will most likely happen there.
Thanks
Jan
10:51 a.m.
On Tue, 28 May 2013 08:51:03 +0200
Jan Zelený <jzeleny(a)redhat.com> wrote:
> after a "yum clean metadata && yum update" on a slow line you
> have to wait a very long time and even the download of the
> presto-metadata often is larger and takes longer as the
> packages which are updated in reality
>
> hey on my 100 Mbit all is nice and fine but on a machine behind
> DSL with around 100 KB/Second it is way too slow and large and
> i refuse to imagine how this feels on a 56kbit modem
I couldn't agree more. But as I have said, we need to find the most
simple and unintrusive things that can be done to improve this. For
instance: file lists take a considerable portion of the entire
metadata size. But if we were to remove them, things like "yum
install /usr/bin/vim" would not work any more. And you get similar
scenario with almost all the metadata that we store - we store them
for a reason and without them some things that people use will not
work.
Jan,
the above is not correct.
Files in *bin/* are in the primary metadata - not in the filelists.
That was specifically designed to handle the 90% of file-deps which are
*bin/* or /etc/*. It's not accidental.
so if you nuked filelists entirely you'd only lose people who have
filedeps on something outside of those wildcards above. I've spent
HERCULEAN amounts of effort to whittle down the set of filedeps outside
of that area. I filed hundreds of bugs on the subject in years passed.
I simply got tired of tilting at that particular windmill when
confronted with some particularly egregious cases (see libguestfs
sometime).
But it is absolutely NOT TRUE that removing filelists will cause 'yum
install /usr/bin/vim' to not work.
If you have further questions about how the metadata works or was
designed please feel free to ask me, directly. I believe I and Adrian
Likins are the only current Red Hat Employees or Fedora Contributors
who were present/involved in its original 'design'. Such as it was.
It might prove helpful to know that background to avoid walking down
blindalleys in DNF.
Thanks!
-sv
12:42 p.m.
On 28.05.2013 18:51, seth vidal wrote:
On Tue, 28 May 2013 08:51:03 +0200
Jan Zelený <jzeleny(a)redhat.com> wrote:
>
>> after a "yum clean metadata && yum update" on a slow line you
>> have to wait a very long time and even the download of the
>> presto-metadata often is larger and takes longer as the
>> packages which are updated in reality
>>
>> hey on my 100 Mbit all is nice and fine but on a machine behind
>> DSL with around 100 KB/Second it is way too slow and large and
>> i refuse to imagine how this feels on a 56kbit modem
>
> I couldn't agree more. But as I have said, we need to find the most
> simple and unintrusive things that can be done to improve this. For
> instance: file lists take a considerable portion of the entire
> metadata size. But if we were to remove them, things like "yum
> install /usr/bin/vim" would not work any more. And you get similar
> scenario with almost all the metadata that we store - we store them
> for a reason and without them some things that people use will not
> work.
Jan,
the above is not correct.
Files in *bin/* are in the primary metadata - not in the filelists.
That was specifically designed to handle the 90% of file-deps which are
*bin/* or /etc/*. It's not accidental.
so if you nuked filelists entirely you'd only lose people who have
filedeps on something outside of those wildcards above. I've spent
HERCULEAN amounts of effort to whittle down the set of filedeps outside
of that area. I filed hundreds of bugs on the subject in years passed.
I simply got tired of tilting at that particular windmill when
confronted with some particularly egregious cases (see libguestfs
sometime).
I just tested on a F18 box the following:
yum erase -y datalog
yum clean all
yum install sqlite-datalog #3 non-existing package
yum install -y /usr/bin/datalog
In the above sequence, yum do not downloaded the filelists at all.
yum erase -y datalog
yum clean all
yum install sqlite-datalog #3 non-existing package
yum install -y /usr/share/lua/5.1/datalog.lua #4
In the second sequence, yum started the filelists downloading at #4.
So, it seems that yum already have the "filelists on demand"
optimization implemented. Why you are asking for removing a feature,
which do not make the things worse ... ?
If you have further questions about how the metadata works or was
designed please feel free to ask me, directly. I believe I and Adrian
Likins are the only current Red Hat Employees or Fedora Contributors
who were present/involved in its original 'design'. Such as it was.
I have a few questions:
* What is the reasoning behind the splitting of the database across
many .sqlite files?
* Why the sql schema is so denormalized (IMO, leads to both bandwidth
and disk overspending without speed benefits)?. For example: Why
provides and requires tables do not use the common domain table?
* Why the incremental update mechanism (eg. applying xml diffs to the
sqlite database) was not been considered from the very beginning?
Thanks!
Alek
1:18 p.m.
On Tue, 28 May 2013 20:42:13 +0300
Alek Paunov <alex(a)declera.com> wrote:
So, it seems that yum already have the "filelists on demand"
optimization implemented. Why you are asking for removing a feature,
which do not make the things worse ... ?
I'm not.
But when you download the filelists - it is A LOT of data.
I'd rather not have filedeps so it doesn't get pulled in for other
things in depsolving.
I have a few questions:
* What is the reasoning behind the splitting of the database across
many .sqlite files?
many? it's 3 afaik. primary, filelists, other.
how do you mean 'many?
* Why the sql schema is so denormalized (IMO, leads to both
bandwidth and disk overspending without speed benefits)?. For
example: Why provides and requires tables do not use the common
domain table?
B/c it was designed 8yrs ago and we were going for compressable space
and making it as quick as possible to search?
* Why the incremental update mechanism (eg. applying xml diffs to
the sqlite database) was not been considered from the very beginning?
It wasn't necessary? There was a massively smaller number of pkgs to
consider.
-sv
3:04 p.m.
On 28.05.2013 21:18, seth vidal wrote:
On Tue, 28 May 2013 20:42:13 +0300
Alek Paunov <alex(a)declera.com> wrote:
> So, it seems that yum already have the "filelists on demand"
> optimization implemented. Why you are asking for removing a feature,
> which do not make the things worse ... ?
I'm not.
But when you download the filelists - it is A LOT of data.
It is of course :-). It is big and slow now, but it implements one more
distinguishing and convenient Fedora feature ... and under careful
schema and encoding, can be scaled down several times in both space and
query time.
Actually, every "positive" (install, update) yum operation implies
access to the repos. Repos contain everything. If our software was
perfectly optimized, not only filelists but all other parts of the
database (including primary.files, which you have cited initially)
should be lazily synced, right?
I'd rather not have filedeps so it doesn't get pulled in for other
things in depsolving.
Sorry, I do not know how this amount of data will impact libsolv in the
future. IMO, for yum (I mean in the sqlite based solution) it is a
matter of optimizations.
> I have a few questions:
>
> * What is the reasoning behind the splitting of the database across
> many .sqlite files?
many? it's 3 afaik. primary, filelists, other.
how do you mean 'many?
Multiplied by the number of the repos. That is what I am trying to
understand - Why not just single .sqlite file for the whole yum database?
> * Why the sql schema is so denormalized (IMO, leads to both
> bandwidth and disk overspending without speed benefits)?. For
> example: Why provides and requires tables do not use the common
> domain table?
B/c it was designed 8yrs ago and we were going for compressable space
and making it as quick as possible to search?
In the provides and requires example, we do not have any space/speed
benefits achieved by the missing common domain (dependency +
dependency_evr tables). In the current situation we have fat and slow
text duplication and indexes instead of integer references to the domain
subnodes (dependencies is the biggest domain in the primary). Yes, in
bunch of cases a little denormalization is inevitable when we fight for
speed, but IMO, this and few other space flaws are with negative impact
on the speed too.
> * Why the incremental update mechanism (eg. applying xml diffs to
> the sqlite database) was not been considered from the very beginning?
It wasn't necessary? There was a massively smaller number of pkgs to
consider.
Indeed. Also, 8 years ago the possibilities and the number of ideas to
reuse were definitely different :-)
Thank you,
Alek
4:16 p.m.
On Tue, 2013-05-28 at 14:18 -0400, seth vidal wrote:
> * Why the sql schema is so denormalized (IMO, leads to both
> bandwidth and disk overspending without speed benefits)?. For
> example: Why provides and requires tables do not use the common
> domain table?
B/c it was designed 8yrs ago and we were going for compressable space
and making it as quick as possible to search?
It wasn't just that, I did the work to normalize it ~5 years ago:
http://james.fedorapeople.org/yum/patches/yum-metadata-parser-ids.patch
...but the download savings were so minimal (IIRC ~1% saving on the .bz2
file) it just didn't seem worth the effort, and any downsides.
Ofc. anyone else has been free to run with it if they want to.
1:42 p.m.
Dne Út 28. května 2013 11:51:21, seth vidal napsal(a):
On Tue, 28 May 2013 08:51:03 +0200
Jan Zelený <jzeleny(a)redhat.com> wrote:
> > after a "yum clean metadata && yum update" on a slow line
you
> > have to wait a very long time and even the download of the
> > presto-metadata often is larger and takes longer as the
> > packages which are updated in reality
> >
> > hey on my 100 Mbit all is nice and fine but on a machine behind
> > DSL with around 100 KB/Second it is way too slow and large and
> > i refuse to imagine how this feels on a 56kbit modem
>
> I couldn't agree more. But as I have said, we need to find the most
> simple and unintrusive things that can be done to improve this. For
> instance: file lists take a considerable portion of the entire
> metadata size. But if we were to remove them, things like "yum
> install /usr/bin/vim" would not work any more. And you get similar
> scenario with almost all the metadata that we store - we store them
> for a reason and without them some things that people use will not
> work.
Jan,
the above is not correct.
Files in *bin/* are in the primary metadata - not in the filelists.
That was specifically designed to handle the 90% of file-deps which are
*bin/* or /etc/*. It's not accidental.
so if you nuked filelists entirely you'd only lose people who have
filedeps on something outside of those wildcards above. I've spent
HERCULEAN amounts of effort to whittle down the set of filedeps outside
of that area. I filed hundreds of bugs on the subject in years passed.
I simply got tired of tilting at that particular windmill when
confronted with some particularly egregious cases (see libguestfs
sometime).
But it is absolutely NOT TRUE that removing filelists will cause 'yum
install /usr/bin/vim' to not work.
If you have further questions about how the metadata works or was
designed please feel free to ask me, directly. I believe I and Adrian
Likins are the only current Red Hat Employees or Fedora Contributors
who were present/involved in its original 'design'. Such as it was.
It might prove helpful to know that background to avoid walking down
blindalleys in DNF.
I knew there were some optimizations in the filelist case, I just wasn't
completely sure what exactly would cause downloads of the filelist.
Anyway, thanks for the explanation
Jan
5:48 a.m.
On Tue, May 28, 2013 at 11:51:21AM -0400, seth vidal wrote:
I simply got tired of tilting at that particular windmill when
confronted with some particularly egregious cases (see libguestfs
sometime).
$ rpm -qR libguestfs|grep ^/
/sbin/ldconfig
/sbin/ldconfig
/lib64/rtkaio/librt.so.1
/usr/lib64/sasl2/libanonymous.so.2
/usr/lib64/sasl2/libsasldb.so.2
To resolve 3 strings we have to download 26 MB of data.
Getting rid of filelists seems like a bad idea because they are so
useful. Implementing them better on the other hand ...
At the moment they are stored in a sqlite database which is bzip2
compressed. The filelists DB for Fedora 18 is 26 MB compressed or 143 MB
uncompressed.
The sqlite database just stores basically the strings as-is. There
are some structures which are better for storing strings that have a
lot of common prefixes, such as tries and suffix trees.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
5:52 a.m.
Also do we know how many mirrors support byte ranges?
We could go all the way and have a relatively large uncompressed
database stored on the mirrors, but have the client only access small
byte ranges from it.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
8:13 a.m.
On Wed, 29 May 2013 11:52:04 +0100
"Richard W.M. Jones" <rjones(a)redhat.com> wrote:
Also do we know how many mirrors support byte ranges?
We could go all the way and have a relatively large uncompressed
database stored on the mirrors, but have the client only access small
byte ranges from it.
We used to use byte-ranges but what we discovered is how many proxies
do NOT support byte-ranges and how quickly that falls apart.
-sv
8:23 a.m.
On Wed, 29 May 2013 11:48:14 +0100
"Richard W.M. Jones" <rjones(a)redhat.com> wrote:
On Tue, May 28, 2013 at 11:51:21AM -0400, seth vidal wrote:
> I simply got tired of tilting at that particular windmill when
> confronted with some particularly egregious cases (see libguestfs
> sometime).
$ rpm -qR libguestfs|grep ^/
/sbin/ldconfig
/sbin/ldconfig
/lib64/rtkaio/librt.so.1
/usr/lib64/sasl2/libanonymous.so.2
/usr/lib64/sasl2/libsasldb.so.2
this must be in f19 - in f18 I see:
/lib/i686/nosegneg/libc.so.6
/lib/i686/nosegneg/libm.so.6
/lib/i686/nosegneg/libpthread.so.0
/lib/i686/nosegneg/librt.so.1
/lib/i686/nosegneg/libthread_db.so.1
/lib/rtkaio/i686/nosegneg/librt.so.1
/lib/rtkaio/librt.so.1
/sbin/ldconfig
/usr/lib/sasl2/libanonymous.so.2
/usr/lib/sasl2/libsasldb.so.2
/usr/lib/sse2/libgmp.so.10
/usr/lib/sse2/libgmpxx.so.4
/usr/lib/sse2/libmp.so.3
/lib64/rtkaio/librt.so.1
/sbin/ldconfig
/usr/lib64/sasl2/libanonymous.so.2
/usr/lib64/sasl2/libsasldb.so.2
and it was much much worse in the past.
To resolve 3 strings we have to download 26 MB of data.
Getting rid of filelists seems like a bad idea because they are so
useful. Implementing them better on the other hand ...
At the moment they are stored in a sqlite database which is bzip2
compressed. The filelists DB for Fedora 18 is 26 MB compressed or
143 MB uncompressed.
The sqlite database just stores basically the strings as-is. There
are some structures which are better for storing strings that have a
lot of common prefixes, such as tries and suffix trees.
Actually the sqlite db doesn't just store the strings it stores a table
which has a pkg id (a number) then all the files in a specific dir for
each row.
like:
13960|/usr/share/locale/pt/LC_MESSAGES|gnokii.mo|f
13960|/usr/share/man/man8|mgnokiidev.8.gz/gnokiid.8.gz|ff
13960|/usr/share/doc/gnokii-0.6.31|sample/protocol/ringtones.txt/logos.txt/gnokii.nol/gnokii-ir-howto/gnokii-hackers-howto/gnokii-IrDA-Linux/gettext-howto/TODO/README.libsms/README-siemens/README-ericsson/README-dancall/README-WINDOWS/README-Symbian/README-PCSC/README-MacOSX/README-DKU2/README-7110/README-6510/README-6110/README-3810/README-2110/README/MAINTAINERS/KNOWN_BUGS/FAQ/DataCalls-QuickStart/CodingStyle/ChangeLog/CREDITS/COPYRIGHT/COPYING/Bugs
just as an example.
-sv
3:11 a.m.
On 05/27/2013 09:32 AM, Jan Zelený wrote:
Unfortunately there is not much we can do about this. Debian has
completely
different repository policy - they keep all versions of packages in the repo so
there is no need to update metadata on client machines every time.
Actually we can do something. Have diffs on metadata.
Let imagine that yesterday I done full upgrade.
Today PackageKit is saying that I have one package for upgrade. Avarage
package has 800kB. Because of those few kB I have to download all those
metadata again. On my machine it is 28 MB (!). But the diff on those
metadata is actually just few kB.
We have presto plugin for diffing rpm packages, but I would actually
save more time and more bandwidth if we would have diff on metadatas.
--
Miroslav Suchy
Red Hat Systems Management Engineering
1:29 p.m.
Dne Út 28. května 2013 10:11:55, Miroslav Suchý napsal(a):
On 05/27/2013 09:32 AM, Jan Zelený wrote:
> Unfortunately there is not much we can do about this. Debian has
> completely
> different repository policy - they keep all versions of packages in the
> repo so there is no need to update metadata on client machines every
> time.
Actually we can do something. Have diffs on metadata.
Let imagine that yesterday I done full upgrade.
Today PackageKit is saying that I have one package for upgrade. Avarage
package has 800kB. Because of those few kB I have to download all those
metadata again. On my machine it is 28 MB (!). But the diff on those
metadata is actually just few kB.
We have presto plugin for diffing rpm packages, but I would actually
save more time and more bandwidth if we would have diff on metadatas.
We have been working on this for some time now. However there are some other
consequent problems that we need to figure out first. There are already some
proposals that came up from our last week meeting with Richard Hughes.
Thanks
Jan
1:17 p.m.
I don't know if this feature already exists, so forgive my
ignorance if it is already there.
I think having an RPM equivalent of Systemd's
"ConditionVirtualization" will be very useful
for controlling packages that are intended for
virtualized environments. It could gracefully
warn users about unsupported environments. It
can also be overridden using some switch to
force ignore the warning.
For example, it would be very clean to say if I
can specify "Requires: ConditionVirtualization=vmware"
to disallow my RPM installation on non-VMware
environments and at the same time have a switch
to force install in case that is what I really
want to do because I'm building an image for
different runtime environment.
Thanks,
Ravindra
1:34 p.m.
Am 22.05.2013 20:17, schrieb Ravindra Kumar:
I don't know if this feature already exists, so forgive my
ignorance if it is already there.
I think having an RPM equivalent of Systemd's
"ConditionVirtualization" will be very useful
for controlling packages that are intended for
virtualized environments. It could gracefully
warn users about unsupported environments. It
can also be overridden using some switch to
force ignore the warning
or skip manpages/docfiles as default or at least
controlled by a option in "yum.conf"
that would greatly reduce the space on virtualized
clusters with 20, 30, 40... fedora setups and in
such environemnts you typically need no man command
on any of the machines because you maintain them
remote from your workstation
11:05 a.m.
or skip manpages/docfiles as default or at least
controlled by a option in "yum.conf"
tsflags=nodocs in yum.conf should do the job. Though apparently, if enabled
after packages already installed files in doc, the files in doc won't be
removed anymore when uninstalling the package.
12:32 p.m.
Am 23.05.2013 18:05, schrieb Sandro Mani:
or skip manpages/docfiles as default or at least
controlled by a option in "yum.conf"
tsflags=nodocs in yum.conf should do the job. Though apparently, if enabled after
packages already installed files
in doc, the files in doc won't be removed anymore when uninstalling the package
and apparently also not at updates and also not by "yum reinstall"
which leaves no clean way to get rid if it and additionally
i am not sure if "tsflags=nodocs" also avoids /usr/share/man
and not only /usr/share/doc
12:48 p.m.
On 05/23/2013 08:32 PM, Reindl Harald wrote:
Am 23.05.2013 18:05, schrieb Sandro Mani:
> or skip manpages/docfiles as default or at least
> controlled by a option in "yum.conf"
>
>
> tsflags=nodocs in yum.conf should do the job. Though apparently, if enabled after
packages already installed files
> in doc, the files in doc won't be removed anymore when uninstalling the package
and apparently also not at updates and also not by "yum reinstall"
which leaves no clean way to get rid if it and additionally
i am not sure if "tsflags=nodocs" also avoids /usr/share/man
and not only /usr/share/doc
Yum's tsflags=nodocs aka --excludedocs on rpm cli only applies to
package installation. I fail to see how it could cause files to be left
behind on erasure/update (reinstall might be a bit, uh, special though),
but if it does then please file a bug on rpm with exact reproducer steps.
- Panu -
1:08 p.m.
Am 23.05.2013 19:48, schrieb Panu Matilainen:
On 05/23/2013 08:32 PM, Reindl Harald wrote:
>
> Am 23.05.2013 18:05, schrieb Sandro Mani:
>> or skip manpages/docfiles as default or at least
>> controlled by a option in "yum.conf"
>>
>> tsflags=nodocs in yum.conf should do the job. Though apparently, if enabled after
packages already installed files
>> in doc, the files in doc won't be removed anymore when uninstalling the
package
>
> and apparently also not at updates and also not by "yum reinstall"
> which leaves no clean way to get rid if it and additionally
> i am not sure if "tsflags=nodocs" also avoids /usr/share/man
> and not only /usr/share/doc
Yum's tsflags=nodocs aka --excludedocs on rpm cli only applies to package
installation. I fail to see how it could
cause files to be left behind on erasure/update (reinstall might be a bit, uh, special
though), but if it does then
please file a bug on rpm with exact reproducer steps
the much cleaner solution would be as said split packages to <packagename>
and <packagename-manuals> and "tsflags=nodocs" is only a hack and if we
speak about "Software Management call for RFEs" it would make a lot of sense
if someone uninstalled the man-packages itself not pull "-manuals" as trigger
1:32 p.m.
Yum's tsflags=nodocs aka --excludedocs on rpm cli only applies to
package installation. I fail to see how it could cause files to be
left behind on erasure/update (reinstall might be a bit, uh, special
though), but if it does then please file a bug on rpm with exact
reproducer steps.
- Panu -
Bug filed: https://bugzilla.redhat.com/show_bug.cgi?id=966715
1:42 p.m.
New subject: Software Management call for RFEs (tsflags=nodocs)
Am 23.05.2013 20:32, schrieb Sandro Mani:
> Yum's tsflags=nodocs aka --excludedocs on rpm cli only applies to package
installation. I fail to see how it
> could cause files to be left behind on erasure/update (reinstall might be a bit, uh,
special though), but if it
> does then please file a bug on rpm with exact reproducer steps.
>
Bug filed: https://bugzilla.redhat.com/show_bug.cgi?id=966715
thanks, i added my comments
if i understand this correctly "tsflags=nodocs" in /etc/yum.conf has the
potential to make much more
harm by never get rid of docs even if packages are uninstalled after get orphaned or
removed or
do i get something wrong here?
1:51 p.m.
and apparently also not at updates and also not by "yum
reinstall"
which leaves no clean way to get rid if it and additionally
i am not sure if "tsflags=nodocs" also avoids /usr/share/man
and not only /usr/share/doc
No it does not avoid man, but you could mount a null-filesystem such as
[1] to /usr/share/man ;)
[1] https://github.com/xrgtn/nullfs
1:53 p.m.
Am 23.05.2013 20:51, schrieb Sandro Mani:
> and apparently also not at updates and also not by "yum reinstall"
> which leaves no clean way to get rid if it and additionally
> i am not sure if "tsflags=nodocs" also avoids /usr/share/man
> and not only /usr/share/doc
>
No it does not avoid man, but you could mount a null-filesystem such as [1] to
/usr/share/man ;)
sorry but this is a bad bad hack and off-topic in case of
"Software Management call for RFEs" because nobody knows
to what this would lead after some dist-upgrades in case of
consistency
1:59 p.m.
On 05/23/2013 09:51 PM, Sandro Mani wrote:
> and apparently also not at updates and also not by "yum reinstall"
> which leaves no clean way to get rid if it and additionally
> i am not sure if "tsflags=nodocs" also avoids /usr/share/man
> and not only /usr/share/doc
>
No it does not avoid man, but you could mount a null-filesystem such as
[1] to /usr/share/man ;)
Yum's tsflags=nodocs aka rpm's --excludedocs applies to all files
flagged as documentation (try 'rpm -qd <package>' to see them), this
happens automatically for man pages as long as they live in normal
paths. For example:
[pmatilai@mursu ~]$ rpm -qd telnet
/usr/share/doc/telnet-0.17/README
/usr/share/man/man1/telnet.1.gz
[pmatilai@mursu ~]$
- Panu -
2:01 p.m.
On 23.05.2013 20:59, Panu Matilainen wrote:
On 05/23/2013 09:51 PM, Sandro Mani wrote:
>
>> and apparently also not at updates and also not by "yum reinstall"
>> which leaves no clean way to get rid if it and additionally
>> i am not sure if "tsflags=nodocs" also avoids /usr/share/man
>> and not only /usr/share/doc
>>
> No it does not avoid man, but you could mount a null-filesystem such as
> [1] to /usr/share/man ;)
Yum's tsflags=nodocs aka rpm's --excludedocs applies to all files
flagged as documentation (try 'rpm -qd <package>' to see them), this
happens automatically for man pages as long as they live in normal
paths. For example:
[pmatilai@mursu ~]$ rpm -qd telnet
/usr/share/doc/telnet-0.17/README
/usr/share/man/man1/telnet.1.gz
[pmatilai@mursu ~]$
- Panu -
Whoops, I guess I tried by reinstalling a package, which has the
reported side effect... But thanks for clarifying!
4:49 p.m.
Le mercredi 22 mai 2013 à 11:17 -0700, Ravindra Kumar a écrit :
I don't know if this feature already exists, so forgive my
ignorance if it is already there.
I think having an RPM equivalent of Systemd's
"ConditionVirtualization" will be very useful
for controlling packages that are intended for
virtualized environments. It could gracefully
warn users about unsupported environments. It
can also be overridden using some switch to
force ignore the warning.
libsolv/yast does it ( if I understood correctly ).You can have some
virtual provides that exist as fake packages in the db, and then have a
package have a requires on it. So it cannot be installed if the hardware
is not here.
--
Michael Scherer
5:16 p.m.
libsolv/yast does it ( if I understood correctly ).You can have some
virtual provides that exist as fake packages in the db, and then have a
package have a requires on it. So it cannot be installed if the hardware
is not here.
Having a fake package in DB makes it very static. I think a
dynamic (evaluated each time rpm commands are run) implementation
will be more useful for the cases like P2V and V2V.
Thanks,
Ravindra
5:31 p.m.
On Wed, May 22, 2013 at 03:16:55PM -0700, Ravindra Kumar wrote:
> libsolv/yast does it ( if I understood correctly ).You can have
some
> virtual provides that exist as fake packages in the db, and then have a
> package have a requires on it. So it cannot be installed if the hardware
> is not here.
Having a fake package in DB makes it very static. I think a
dynamic (evaluated each time rpm commands are run) implementation
will be more useful for the cases like P2V and V2V.
The problem I see here is that you can boot the same OS on different
hypervisors or (with P2V) transfer it from physical to virtual.
Should RPM start (un-)installing things when this happens? Could a
reboot result in broken dependencies?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
6:28 p.m.
> Having a fake package in DB makes it very static. I think a
> dynamic (evaluated each time rpm commands are run) implementation
> will be more useful for the cases like P2V and V2V.
The problem I see here is that you can boot the same OS on different
hypervisors or (with P2V) transfer it from physical to virtual.
Should RPM start (un-)installing things when this happens? Could a
reboot result in broken dependencies?
No, I was not thinking of reboot/RPM changing anything already
installed. I was referring to DB solution as static because
it would stick one configuration forever. Instead, I was
thinking of RPM to always base its decision on the environment
where it is running at that point of time and provide a way
to override RPM behavior so that advanced users can make a
choice.
Thanks,
Ravindra
8:44 a.m.
Le Jeu 23 mai 2013 01:28, Ravindra Kumar a écrit :
>> Having a fake package in DB makes it very static. I think a
>> dynamic (evaluated each time rpm commands are run) implementation
>> will be more useful for the cases like P2V and V2V.
> The problem I see here is that you can boot the same OS on different
> hypervisors or (with P2V) transfer it from physical to virtual.
> Should RPM start (un-)installing things when this happens? Could a
> reboot result in broken dependencies?
No, I was not thinking of reboot/RPM changing anything already
installed. I was referring to DB solution as static because
it would stick one configuration forever. Instead, I was
thinking of RPM to always base its decision on the environment
where it is running at that point of time and provide a way
to override RPM behavior so that advanced users can make a
choice.
This is the logic windows installers use but I don't think it's the
correct one. The installer should never change behaviour based on a system
state that can change over time (unless this state is under its control
which is not the case here)
--
Nicolas Mailhot
12:38 p.m.
> No, I was not thinking of reboot/RPM changing anything already
> installed. I was referring to DB solution as static because
> it would stick one configuration forever. Instead, I was
> thinking of RPM to always base its decision on the environment
> where it is running at that point of time and provide a way
> to override RPM behavior so that advanced users can make a
> choice.
This is the logic windows installers use but I don't think it's the
correct one. The installer should never change behaviour based on a system
state that can change over time (unless this state is under its control
which is not the case here)
I think of this like two categories of machine:
1. Systems that will not change the environment for a very long
time and continue to run in either a physical or a virtual
environment.
2. Systems that will change the runtime environment occasionally.
DB approach works for #1 and fails miserably for #2 (example below).
At the same time runtime detection approach works for #1 and works
almost right for #2 with few exceptions. Exceptions can be addressed
through an override switch.
Example: Fedora image running in 'X' virtual environment is moved
to 'Y' virtual environment. If 'X' is sticking in the RPM DB,
packages intended for 'Y' can't be installed because RPM DB
thinks otherwise.
Thanks,
Ravindra
6:24 a.m.
On 22. 5. 2013 at 11:17:16, Ravindra Kumar wrote:
I don't know if this feature already exists, so forgive my
ignorance if it is already there.
I think having an RPM equivalent of Systemd's
"ConditionVirtualization" will be very useful
for controlling packages that are intended for
virtualized environments. It could gracefully
warn users about unsupported environments. It
can also be overridden using some switch to
force ignore the warning.
For example, it would be very clean to say if I
can specify "Requires: ConditionVirtualization=vmware"
to disallow my RPM installation on non-VMware
environments and at the same time have a switch
to force install in case that is what I really
want to do because I'm building an image for
different runtime environment.
Conditional dependencies is one of the original requests we discussed on the
DevConf and it will most likely be addressed in the future. We realize the
great potential this feature set would bring, we just want to do it right and
that will take some time to design and implement.
Thanks
Jan
2:06 p.m.
Jan Zelený wrote:
what are the changes that you would like to see in the foreseeable
future (say 2-3 years) and why would you like to see them (what would they
help you with)?
Dare I say ... (puts on a helmet) ... Recommends and Suggests?
We really should have a way for Yum and Packagekit to say "Hey, if
you're installing that, maybe you also want this?". A -devel package
and its corresponding -doc package should recommend each other for
example. They shouldn't require each other, because then they could
just as well be the same package, but usually you want to install them
together and it would be helpful to at least notify users who install
the -devel package that the -doc package also exists.
Another example is a database server and its command line client, which
you often but not always want to install together. The server should
recommend the client, while the client might only suggest the server.
A third example is graphical administration tools for some daemon that
are in a separate package so that the daemon can be installed without
pulling in half a desktop environment. In this case the daemon should
suggest the tools, but perhaps not recommend them.
Björn Persson
4:55 p.m.
Le mercredi 22 mai 2013 à 21:06 +0200, Björn Persson a écrit :
Jan Zelený wrote:
> what are the changes that you would like to see in the foreseeable
> future (say 2-3 years) and why would you like to see them (what would they
> help you with)?
Dare I say ... (puts on a helmet) ... Recommends and Suggests?
It would take more then 2 to 3 years to make sure everybody understand
the same thing for Suggests and Recommends.
Without explaining why a package is suggested or recommended, people
cannot make informed choice on it. So in the end, people will either get
too much, or they will not get enough. And if you had suggest/recommend
by default, you bloat the system, and if you don't, then that's useless.
I am not against the idea, but that's not a technical issue.
Technically, Suggest can be added to rpm quite fast, there is patch
floating around.
--
Michael Scherer
5:33 p.m.
On Wed, May 22, 2013 at 11:55:43PM +0200, Michael Scherer wrote:
Le mercredi 22 mai 2013 à 21:06 +0200, Björn Persson a écrit :
> Jan Zelený wrote:
> > what are the changes that you would like to see in the foreseeable
> > future (say 2-3 years) and why would you like to see them (what would they
> > help you with)?
>
> Dare I say ... (puts on a helmet) ... Recommends and Suggests?
It would take more then 2 to 3 years to make sure everybody understand
the same thing for Suggests and Recommends.
Without explaining why a package is suggested or recommended, people
cannot make informed choice on it. So in the end, people will either get
too much, or they will not get enough. And if you had suggest/recommend
by default, you bloat the system, and if you don't, then that's useless.
I am not against the idea, but that's not a technical issue.
Technically, Suggest can be added to rpm quite fast, there is patch
floating around.
TBH I don't think we need Suggests AND Recommends. I can never
remember the difference on Debian. Wouldn't most people would be
satisfied with a single way to suggest useful packages?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
6:20 p.m.
On Wednesday 22 May 2013 23:33:05 Richard W.M. Jones wrote:
TBH I don't think we need Suggests AND Recommends. I can never
remember the difference on Debian. Wouldn't most people would be
satisfied with a single way to suggest useful packages?
Thinking about it, the terminology adopted by "comps" is clearer
and provides a generalization of this -- if someone select something
they get:
- Mandatory packages (cannot be deselected)
- Default packages (selected, but the user may deselect)
- Optional packages (deselected, but the user may select)
Borrowing similar logic for rpm we could have in the spec file:
Name: acme
Requires: foo, foo-utils
InstallDefault: bar, perl-bar, python-bar
InstallOptional: baz, baz-ldap
Now it would be classic to use "--with/--without" as command line flags,
but it's already taken :-(
--
Oron Peled Voice: +972-4-8228492
oron(a)actcom.co.il http://users.actcom.co.il/~oron
Boycott Microsoft -- http://www.vcnet.com/bms
6:26 p.m.
On Thu, 2013-05-23 at 02:20 +0300, Oron Peled wrote:
On Wednesday 22 May 2013 23:33:05 Richard W.M. Jones wrote:
> TBH I don't think we need Suggests AND Recommends. I can never
> remember the difference on Debian. Wouldn't most people would be
> satisfied with a single way to suggest useful packages?
Thinking about it, the terminology adopted by "comps" is clearer
and provides a generalization of this -- if someone select something
they get:
- Mandatory packages (cannot be deselected)
- Default packages (selected, but the user may deselect)
- Optional packages (deselected, but the user may select)
Borrowing similar logic for rpm we could have in the spec file:
Name: acme
Requires: foo, foo-utils
InstallDefault: bar, perl-bar, python-bar
InstallOptional: baz, baz-ldap
Now it would be classic to use "--with/--without" as command line flags,
but it's already taken :-(
I'm pretty sure that's precisely the distinction expressed by Suggests
and Recommends, FWIW.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
12:27 a.m.
Adam Williamson wrote:
On Thu, 2013-05-23 at 02:20 +0300, Oron Peled wrote:
> Thinking about it, the terminology adopted by "comps" is clearer
> and provides a generalization of this -- if someone select something
> they get:
> - Mandatory packages (cannot be deselected)
> - Default packages (selected, but the user may deselect)
> - Optional packages (deselected, but the user may select)
>
>
> Borrowing similar logic for rpm we could have in the spec file:
> Name: acme
> Requires: foo, foo-utils
> InstallDefault: bar, perl-bar, python-bar
> InstallOptional: baz, baz-ldap
>
> Now it would be classic to use "--with/--without" as command line flags,
> but it's already taken :-(
I'm pretty sure that's precisely the distinction expressed by Suggests
and Recommends, FWIW.
For interactive operations that's how I envision it, yes. When yum
lists the packages it's going to install and asks for confirmation, it
would list recommended packages and their requirements under
"Installing because of recommendations", and suggested packages under
"Suggested packages not being installed". The user can then choose to
abort the operation and run the command again, adding some suggested
packages to the command line or using --exclude on some of the
recommended ones.
In batch operations like Kickstart and Mock I'm thinking recommendations
and suggestions would be ignored and only required packages would be
pulled in. I don't know how Kickstart handles Comps groups but "default"
sounds to me like it would be installed even in batch operations.
Björn Persson
11:24 a.m.
On 23 May 2013, at 01:27, Björn Persson wrote:
Adam Williamson wrote:
> On Thu, 2013-05-23 at 02:20 +0300, Oron Peled wrote:
>> Thinking about it, the terminology adopted by "comps" is clearer
>> and provides a generalization of this -- if someone select something
>> they get:
>> - Mandatory packages (cannot be deselected)
>> - Default packages (selected, but the user may deselect)
>> - Optional packages (deselected, but the user may select)
>>
>>
>> Borrowing similar logic for rpm we could have in the spec file:
>> Name: acme
>> Requires: foo, foo-utils
>> InstallDefault: bar, perl-bar, python-bar
>> InstallOptional: baz, baz-ldap
>>
>> Now it would be classic to use "--with/--without" as command line
>> flags,
>> but it's already taken :-(
>
> I'm pretty sure that's precisely the distinction expressed by
> Suggests
> and Recommends, FWIW.
For interactive operations that's how I envision it, yes. When yum
lists the packages it's going to install and asks for confirmation, it
would list recommended packages and their requirements under
"Installing because of recommendations", and suggested packages under
"Suggested packages not being installed". The user can then choose to
abort the operation and run the command again, adding some suggested
packages to the command line or using --exclude on some of the
recommended ones.
Because this is an interactive operation, rather than aborting, why
not change "(y/N)" to "(y/N/m)". If the user chooses "m"
for modify,
then yum could ask whether the user wanted to delete a recommended
package and, if response is "y", show the recommended packages one
line at a time for a "y" or "n" response. Then yum could ask whether
the user wanted to add a suggested package and, if response is "y",
show the suggested packages one line at a time for a "y" or "n"
response. Dealing with the modifications interactively would be less
annoying than aborting and typing a new yum command full of --
excludes or additional package names.
--
Mike
12:28 a.m.
Michael Scherer wrote:
Without explaining why a package is suggested or recommended, people
cannot make informed choice on it.
In all of the examples I mentioned the reason would be pretty obvious
once you know that the recommended package exists. If you're not sure
whether you want a recommended package you can read the description to
see if it seems useful to you.
In less obvious cases it would of course be helpful to explain in the
package description why the other package would be useful together with
this one, but users might never see that explanation unless Yum or
Packagekit or whatever notifies them about the recommendation.
And if you had suggest/recommend
by default, you bloat the system, and if you don't, then that's useless.
Getting notified about relevant add-ons that I was unaware of is hardly
useless.
As for bloat, that's where the distinction between recommendations and
suggestions comes into play. If the optional package is very likely to
be useful in most cases and has modest requirements, then it should be
recommended and get pulled in by default at least in interactive
operations. If it's likely to be useful only to some users, or if it
pulls in lots of additional stuff, then it should be suggested and get
installed only on the user's explicit request. This minimizes bloat.
Björn Persson
6:26 a.m.
On 22. 5. 2013 at 21:06:53, Björn Persson wrote:
Jan Zelený wrote:
> what are the changes that you would like to see in the foreseeable
> future (say 2-3 years) and why would you like to see them (what would they
> help you with)?
Dare I say ... (puts on a helmet) ... Recommends and Suggests?
We really should have a way for Yum and Packagekit to say "Hey, if
you're installing that, maybe you also want this?". A -devel package
and its corresponding -doc package should recommend each other for
example. They shouldn't require each other, because then they could
just as well be the same package, but usually you want to install them
together and it would be helpful to at least notify users who install
the -devel package that the -doc package also exists.
Another example is a database server and its command line client, which
you often but not always want to install together. The server should
recommend the client, while the client might only suggest the server.
A third example is graphical administration tools for some daemon that
are in a separate package so that the daemon can be installed without
pulling in half a desktop environment. In this case the daemon should
suggest the tools, but perhaps not recommend them.
See my reply above in the thread about dynamic dependencies. Short version:
already being discussed :-)
Thanks
Jan
1:36 p.m.
2013/5/22 Björn Persson <bjorn(a)xn--rombobjrn-67a.se>:
Jan Zelený wrote:
> what are the changes that you would like to see in the foreseeable
> future (say 2-3 years) and why would you like to see them (what would they
> help you with)?
Dare I say ... (puts on a helmet) ... Recommends and Suggests?
We really should have a way for Yum and Packagekit to say "Hey, if
you're installing that, maybe you also want this?". A -devel package
and its corresponding -doc package should recommend each other for
example. They shouldn't require each other, because then they could
just as well be the same package, but usually you want to install them
together and it would be helpful to at least notify users who install
the -devel package that the -doc package also exists.
Another example is a database server and its command line client, which
you often but not always want to install together. The server should
recommend the client, while the client might only suggest the server.
A third example is graphical administration tools for some daemon that
are in a separate package so that the daemon can be installed without
pulling in half a desktop environment. In this case the daemon should
suggest the tools, but perhaps not recommend them.
The only few times I used Suggests (in Mandriva) was to "suggest"
optional, but better experience/functionality if installed, packages in
non-free, but I think I never fully understood what it is supposed to
mean, just that it is a Requires that does not break dependencies.
Björn Persson
Paulo
12:09 p.m.
On Thu, 2013-05-23 at 15:36 -0300, Paulo César Pereira de Andrade wrote:
2013/5/22 Björn Persson <bjorn(a)xn--rombobjrn-67a.se>:
> Jan Zelený wrote:
>> what are the changes that you would like to see in the foreseeable
>> future (say 2-3 years) and why would you like to see them (what would they
>> help you with)?
>
> Dare I say ... (puts on a helmet) ... Recommends and Suggests?
>
> We really should have a way for Yum and Packagekit to say "Hey, if
> you're installing that, maybe you also want this?". A -devel package
> and its corresponding -doc package should recommend each other for
> example. They shouldn't require each other, because then they could
> just as well be the same package, but usually you want to install them
> together and it would be helpful to at least notify users who install
> the -devel package that the -doc package also exists.
>
> Another example is a database server and its command line client, which
> you often but not always want to install together. The server should
> recommend the client, while the client might only suggest the server.
>
> A third example is graphical administration tools for some daemon that
> are in a separate package so that the daemon can be installed without
> pulling in half a desktop environment. In this case the daemon should
> suggest the tools, but perhaps not recommend them.
The only few times I used Suggests (in Mandriva) was to "suggest"
optional, but better experience/functionality if installed, packages in
non-free, but I think I never fully understood what it is supposed to
mean, just that it is a Requires that does not break dependencies.
Usually softer dependencies don't have a single specific 'meaning'
exactly. Even Requires doesn't _quite_, because you can override
Requires with rpm --nodeps .
It's more useful (I've found, anyway) to think of this as metadata that
it's the package manager's responsibility to interpret, and that you
take all the possible ways the package manager can interpret that
metadata into account when you create it.
Usually, package managers are configurable to do various things with
soft dependencies. IIRC, Mandriva was configured to install the soft
dependencies of packages by default. However, you could then remove them
without the 'requiring' package being removed. So if foo Suggests: bar,
'urpmi foo' would install bar, but 'urpme bar' would not require the
removal of foo.
However, you could configure urpmi not to install Suggested packages, if
you wanted to keep a very minimal system.
So the semantics of Mandriva's system are fairly simple, and the
executive takeaway for a packager is pretty simple too: use Suggests:
for things that a package doesn't strictly require to work, but which
might make the experience of using it better for many people.
With a more complex three-level system like Debian's there are likely to
be more configuration options for the package manager's behaviour with
regards to each form of dependency, and hence more possible semantics,
but again once you understand the various 'supported' options for how to
handle the different types of dependency at the package manager level,
you should be able to understand which form of dependency to express in
your package.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
4:08 p.m.
Hi
On Wed, May 22, 2013 at 9:43 AM, Jan Zelený wrote:
We acknowledge the need for some changes in Software Management stack in
Fedora but we don't want to make changes just by guessing what our
users want. Therefore I call to you, consumers of our products (dnf, yum
and
rpm): what are the changes that you would like to see in the foreseeable
future (say 2-3 years) and why would you like to see them (what would they
help you with)?
Thanks for taking on this effort. What I would like to see is solid git
integration. Git has become the standard distributed vcs and github and
google code etc have stopped hosting tarballs and/or discouraging it and
GNOME is planning to do that as well. If Source URL could point directly
to a git url with a hash or git tag, we would benefit.
Also, if would could make it easy to easier to do things like updating the
icon cache etc only once per transaction instead of per package would help
with the speed of the initial Fedora installation or any large updates
Rahul
6:35 a.m.
On 22. 5. 2013 at 17:08:33, Rahul Sundaram wrote:
Hi
On Wed, May 22, 2013 at 9:43 AM, Jan Zelený wrote:
> We acknowledge the need for some changes in Software Management stack in
> Fedora but we don't want to make changes just by guessing what our
> users want. Therefore I call to you, consumers of our products (dnf, yum
> and
> rpm): what are the changes that you would like to see in the foreseeable
> future (say 2-3 years) and why would you like to see them (what would they
> help you with)?
Thanks for taking on this effort. What I would like to see is solid git
integration. Git has become the standard distributed vcs and github and
google code etc have stopped hosting tarballs and/or discouraging it and
GNOME is planning to do that as well. If Source URL could point directly
to a git url with a hash or git tag, we would benefit.
Interesting idea however I'm not sure if this shouldn't be targeted somewhere
else. Anyway, thanks for the suggestion, we will consider it.
Also, if would could make it easy to easier to do things like
updating the
icon cache etc only once per transaction instead of per package would help
with the speed of the initial Fedora installation or any large updates
Honestly, I'm not completely sure what do you mean by icon cache.
Thanks
Jan
8:52 a.m.
From: Rahul Sundaram <metherid(a)gmail.com>
What I would like to see is
solid git integration. Git has become the standard distributed vcs
and github and google code etc have stopped hosting tarballs and/or
discouraging it and GNOME is planning to do that as well. If Source
URL could point directly to a git url with a hash or git tag, we
would benefit.
Amen to that! I roll my own rpms daily from locally developed sources
where we have no policy of pushing tarballs. From everything I've ever
been able to figure out, it's necessary for me to make temporary tarballs
just to feed rpmbuild. It always seems such a huge waste of time,
especially for very large packages.
--
John Florian
9:47 a.m.
John.Florian(a)dart.biz wrote:
> From: Rahul Sundaram <metherid(a)gmail.com>
> What I would like to see is
> solid git integration. Git has become the standard distributed vcs
> and github and google code etc have stopped hosting tarballs and/or
> discouraging it and GNOME is planning to do that as well. If Source
> URL could point directly to a git url with a hash or git tag, we
> would benefit.
Amen to that! I roll my own rpms daily from locally developed sources
where we have no policy of pushing tarballs. From everything I've ever
been able to figure out, it's necessary for me to make temporary tarballs
just to feed rpmbuild. It always seems such a huge waste of time,
especially for very large packages.
RPM would still be making tarballs behind the scenes, even with better
integration with git, wouldn't it? -- you still need the ability to make
SRPMs.
9:54 a.m.
On 05/23/2013 04:47 PM, Paul Flo Williams wrote:
John.Florian(a)dart.biz wrote:
>> From: Rahul Sundaram <metherid(a)gmail.com>
>> What I would like to see is
>> solid git integration. Git has become the standard distributed vcs
>> and github and google code etc have stopped hosting tarballs and/or
>> discouraging it and GNOME is planning to do that as well. If Source
>> URL could point directly to a git url with a hash or git tag, we
>> would benefit.
>
> Amen to that! I roll my own rpms daily from locally developed sources
> where we have no policy of pushing tarballs. From everything I've ever
> been able to figure out, it's necessary for me to make temporary tarballs
> just to feed rpmbuild. It always seems such a huge waste of time,
> especially for very large packages.
RPM would still be making tarballs behind the scenes, even with better
integration with git, wouldn't it? -- you still need the ability to make
SRPMs.
But rpm could just do a git-tar-tree behind the scenes, which sounds
easy enough.
Thanks & regards, Phil
--
Philipp Knirsch | Tel.: +49-711-96437-470
Manager Core Services | Fax.: +49-711-96437-111
Red Hat GmbH | Email: Phil Knirsch <pknirsch(a)redhat.com>
Wankelstrasse 5 | Web: http://www.redhat.com/
D-70563 Stuttgart, Germany
10:09 a.m.
From: pknirsch(a)redhat.com
On 05/23/2013 04:47 PM, Paul Flo Williams wrote:
> John.Florian(a)dart.biz wrote:
>>> From: Rahul Sundaram <metherid(a)gmail.com>
>>> What I would like to see is
>>> solid git integration. Git has become the standard distributed vcs
>>> and github and google code etc have stopped hosting tarballs and/or
>>> discouraging it and GNOME is planning to do that as well. If Source
>>> URL could point directly to a git url with a hash or git tag, we
>>> would benefit.
>>
>> Amen to that! I roll my own rpms daily from locally developed
sources
>> where we have no policy of pushing tarballs. From
everything I've
ever
>> been able to figure out, it's necessary for me to make
temporary
tarballs
>> just to feed rpmbuild. It always seems such a huge waste of
time,
>> especially for very large packages.
>
> RPM would still be making tarballs behind the scenes, even with better
> integration with git, wouldn't it? -- you still need the ability to
make
> SRPMs.
>
But rpm could just do a git-tar-tree behind the scenes, which sounds
easy enough.
Exactly. And even though I have to give rpmbuild a tarball, I don't
believe it ever reuses it "as is". My understanding is that the content
gets extracted, processed and tarballed again.
I'd like to see it behave more the way I expected it to when I naively
first started rolling my own packages. Specifically, it would be nice if
the %Source URI was processed intelligently to automatically retrieve the
content via HTTP, FTP, GIT, FILE or whatever (within reason) happens to be
specified there.
--
John Florian
10:29 a.m.
On 05/23/2013 06:09 PM, John.Florian(a)dart.biz wrote:
> From: pknirsch(a)redhat.com
> On 05/23/2013 04:47 PM, Paul Flo Williams wrote:
> > John.Florian(a)dart.biz wrote:
> >>> From: Rahul Sundaram <metherid(a)gmail.com>
> >>> What I would like to see is
> >>> solid git integration. Git has become the standard distributed vcs
> >>> and github and google code etc have stopped hosting tarballs and/or
> >>> discouraging it and GNOME is planning to do that as well. If Source
> >>> URL could point directly to a git url with a hash or git tag, we
> >>> would benefit.
> >>
> >> Amen to that! I roll my own rpms daily from locally developed sources
> >> where we have no policy of pushing tarballs. From everything I've
ever
> >> been able to figure out, it's necessary for me to make temporary
tarballs
> >> just to feed rpmbuild. It always seems such a huge waste of time,
> >> especially for very large packages.
> >
> > RPM would still be making tarballs behind the scenes, even with better
> > integration with git, wouldn't it? -- you still need the ability to
make
> > SRPMs.
> >
>
> But rpm could just do a git-tar-tree behind the scenes, which sounds
> easy enough.
Exactly. And even though I have to give rpmbuild a tarball, I don't
believe it ever reuses it "as is". My understanding is that the content
gets extracted, processed and tarballed again.
I dont know what gave you such an idea, rpm certainly does nothing of
the sort. The tarball is obviously extracted for building, but what ends
up in the src.rpm is the original tarball and the patches defined in the
spec - this is the "pristine sources" principle:
http://rpm.org/max-rpm-snapshot/ch-rpm-philosophy.html#S1-RPM-PHILOSOPHY-...
I'd like to see it behave more the way I expected it to when I
naively
first started rolling my own packages. Specifically, it would be nice
if the %Source URI was processed intelligently to automatically retrieve
the content via HTTP, FTP, GIT, FILE or whatever (within reason) happens
to be specified there.
Rpm >= 4.10 can automatically download remote sources and patches over
http and ftp, but since there's (currently) no way to verify downloaded
content the feature is disabled by default as its quite a security risk
to download arbitrary content from the internet without checking
checksums at least.
- Panu -
--
John Florian
10:54 a.m.
From: pmatilai(a)laiskiainen.org
On 05/23/2013 06:09 PM, John.Florian(a)dart.biz wrote:
> And even though I have to give rpmbuild a tarball, I don't
> believe it ever reuses it "as is". My understanding is that the
content
> gets extracted, processed and tarballed again.
I dont know what gave you such an idea
Me neither. Perhaps the apparent slowness with large packages and/or a
lack of coffee this morning.
, rpm certainly does nothing of
the sort. The tarball is obviously extracted for building, but what
ends
up in the src.rpm is the original tarball and the patches defined in
the
spec - this is the "pristine sources" principle:
http://rpm.org/max-rpm-snapshot/ch-rpm-philosophy.html#S1-RPM-
PHILOSOPHY-PRISTINE-SOURCES
You are, of course, right. I actually knew that for srpms. I just tend
not to think about the srpms much since for nearly all the builds I do, *I
am the upstream* source so I'm really only interested in the binary rpms.
> I'd like to see it behave more the way I expected it to when
I naively
> first started rolling my own packages. Specifically, it would be nice
> if the %Source URI was processed intelligently to automatically
retrieve
> the content via HTTP, FTP, GIT, FILE or whatever (within reason)
happens
> to be specified there.
Rpm >= 4.10 can automatically download remote sources and patches over
http and ftp, but since there's (currently) no way to verify downloaded
content the feature is disabled by default as its quite a security risk
to download arbitrary content from the internet without checking
checksums at least.
That seems sensible. I guess my biggest wish here in this sub-thread is
that building rpms was more streamlined/efficient for use case such as
mine where I author and package and only distribute via rpm. The current
model imposes a need to self-publish tarballs, even if they only have very
short life span in /tmp. If I've got a git clone with everything needed,
including a spec file there, I'd like to build an rpm directly from that.
I've scripted to meet my use case but it always seems quite hackish for
some reason. I guess with the above knowledge about srpms, it's not as
bad as first thought.
RPM is a great container/delivery system, but it definitely feels like it
might be generalized more to handle both the common FOSS model and
internal private uses.
PS. When I speak of large packages, I don't joke. One terrifying example
is something I call "mayflower" (think early American settlers) which
ships an entire livecd-tools generated image along with some magic glue
which when this package is installed causes a special initrd to be
generated with a dracut module for doing a swap of two livecd images.
Install the mayflower rpm, reboot and you get an in-place upgrade just
like that. (This is unbelievably useful if you have hundreds or more of
nodes running such images in an embedded hardware.) Yeah, I forced a
round peg into a very square hole, but it works beautifully. I'm both
embarrassed and proud! :-)
--
John Florian
3:41 a.m.
Dne Čt 23. května 2013 11:54:04, John.Florian(a)dart.biz napsal(a):
> From: pmatilai(a)laiskiainen.org
>
> On 05/23/2013 06:09 PM, John.Florian(a)dart.biz wrote:
> > And even though I have to give rpmbuild a tarball, I don't
> > believe it ever reuses it "as is". My understanding is that the
content
> > gets extracted, processed and tarballed again.
>
> I dont know what gave you such an idea
Me neither. Perhaps the apparent slowness with large packages and/or a
lack of coffee this morning.
, rpm certainly does nothing of
> the sort. The tarball is obviously extracted for building, but what ends
>
> up in the src.rpm is the original tarball and the patches defined in the
>
> spec - this is the "pristine sources" principle:
> http://rpm.org/max-rpm-snapshot/ch-rpm-philosophy.html#S1-RPM-
> PHILOSOPHY-PRISTINE-SOURCES
You are, of course, right. I actually knew that for srpms. I just tend
not to think about the srpms much since for nearly all the builds I do, *I
am the upstream* source so I'm really only interested in the binary rpms.
> > I'd like to see it behave more the way I expected it to when I naively
> > first started rolling my own packages. Specifically, it would be nice
> > if the %Source URI was processed intelligently to automatically
retrieve
> > the content via HTTP, FTP, GIT, FILE or whatever (within reason)
happens
> > to be specified there.
>
> Rpm >= 4.10 can automatically download remote sources and patches over
> http and ftp, but since there's (currently) no way to verify downloaded
> content the feature is disabled by default as its quite a security risk
> to download arbitrary content from the internet without checking
> checksums at least.
This sounds more like something that should be a part of rpmdev-tools (if it
isn't already. I recommend getting in touch with the maintainer of those and
see if there is something that can be done there.
Thanks
Jan
12:11 p.m.
On Thu, 2013-05-23 at 18:29 +0300, Panu Matilainen wrote:
Rpm >= 4.10 can automatically download remote sources and patches
over
http and ftp, but since there's (currently) no way to verify downloaded
content the feature is disabled by default as its quite a security risk
to download arbitrary content from the internet without checking
checksums at least.
And note that it's as much Fedora policy as packaging stack capabilities
that prevents this happening at present: as discussed in another thread
it's a fundamental part of the Fedora packaging system's design that the
builders have no outside access, and it's the package maintainer's
explicit responsibility to provide the source files to the build system.
(The implication of this is that it is the package maintainer's
responsibility to provide, and verify that they are providing, the
_correct_ sources.)
We could of course build the smarts into the fedpkg layer - have some
fedpkg commands for checking out and building tarballs of SCM-hosted
content - but then you've just moved the security risk Panu mentioned to
that layer; if we do that it kind of sends a bad implication that it's
fine to just trust whatever you get from the SCM URL.
Thinking about this, it's one reason the style of doing 'git snapshot'
builds where you have Source0 be the last stable tarball and then
include the full patch series to master as generated by 'git
format-patch' as Patches could be considered superior to simply
including a git master snapshot tarball: at least if someone's concerned
about some kind of breach, they have an easily-verifiable base to work
off - as there should be an official checksum for the last release
tarball - and only have to check the patches for problems, rather than
checking the entire tree.
I think, to be honest, a lot of us as packagers slip some way short of
the 'ideals' here in day-to-day life, but that's probably no excuse for
making it _easier_ to avoid our responsibilities :)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
3:54 p.m.
On Fri, 2013-05-24 at 10:11 -0700, Adam Williamson wrote:
We could of course build the smarts into the fedpkg layer - have
some
fedpkg commands for checking out and building tarballs of SCM-hosted
content - but then you've just moved the security risk Panu mentioned to
that layer; if we do that it kind of sends a bad implication that it's
fine to just trust whatever you get from the SCM URL.
I'm not going to debate this extensively, because unless someone who can
actually change things is planning to do so, it's just pointless noise.
But basically there are two threats:
1) MITM attacks by third parties. Answer: SSL. Yes, it's not perfect,
but it's good enough for online banking. Yes, governments and
affiliated groups have wildcard certificates, but there are defenses.
Manual human signoff on new root CAs would be pretty good.
2) Corrupted repository server side: The answer to this is to have a
system that actually *encourages* people to look at the source code.
If you truly wanted to be serious about that, we could have a UI
that actually you know, unpacks the source by default, diffs it
from the previous, and requires human signoff before building it.
But at the moment, all the crappy package metadata sadly is what's
front and center, not the actual source code.
10:12 a.m.
On Thu, 2013-05-23 at 16:54 +0200, Phil Knirsch wrote:
But rpm could just do a git-tar-tree behind the scenes, which sounds
easy enough.
It's not quite that easy, given the possible presence of git
submodules.
http://stackoverflow.com/questions/1591387/need-to-handle-git-submodules-...
There are scripts out there though, it's not a huge amount of
engineering.
However if you take the next step and ask yourself, "why not actually
just use git repositories instead of the lookaside pile of tarballs",
then you get into the part I found the most tricky with doing continuous
integration (and complying with the GPL by mirroring the source you're
building), which is recursively mirroring the submodules, and changing
the submodule references to point to your local mirror:
https://git.gnome.org/browse/gnome-ostree/tree/src/js/vcs.js#n213
Even the Jenkins git plugin doesn't really handle this quite correctly,
but the above code has survived a lot of real-world testing.
9:04 a.m.
On Thu, May 23, 2013 at 9:52 AM, <John.Florian(a)dart.biz> wrote:
> From: Rahul Sundaram <metherid(a)gmail.com>
> What I would like to see is
> solid git integration. Git has become the standard distributed vcs
> and github and google code etc have stopped hosting tarballs and/or
> discouraging it and GNOME is planning to do that as well. If Source
> URL could point directly to a git url with a hash or git tag, we
> would benefit.
Amen to that! I roll my own rpms daily from locally developed sources where
we have no policy of pushing tarballs. From everything I've ever been able
to figure out, it's necessary for me to make temporary tarballs just to feed
rpmbuild. It always seems such a huge waste of time, especially for very
large packages.
Nahh, you can work around this. Manipulate %setup" to build a local
compilation directory, but not unload tarballs, and then in the
"%build" step do a git pull froom your git repo. And add
"BuildRequires: /usr/bin/git".
But this means that your SRPM's will not have the available build
materials, and will require an outside web access to download source
code. This is *tremendous* source control problem.
10:56 a.m.
Hey,
On Wed, May 22, 2013 at 05:08:33PM -0400, Rahul Sundaram wrote:
On Wed, May 22, 2013 at 9:43 AM, Jan Zelený wrote:
>
> We acknowledge the need for some changes in Software Management stack in
> Fedora but we don't want to make changes just by guessing what our
> users want. Therefore I call to you, consumers of our products (dnf, yum
> and
> rpm): what are the changes that you would like to see in the foreseeable
> future (say 2-3 years) and why would you like to see them (what would they
> help you with)?
>
Thanks for taking on this effort. What I would like to see is solid git
integration. Git has become the standard distributed vcs and github and
google code etc have stopped hosting tarballs and/or discouraging it and
GNOME is planning to do that as well. If Source URL could point directly
to a git url with a hash or git tag, we would benefit.
I'd add to that an optional GPG key to check the git tag against.
Christophe
4:18 p.m.
[This is a copy of an email I sent to an internal Red Hat list last month]
In no particular order:
(1) Yum should not be so slow. In particular "yum install" takes ages
compared to "apt-get install". I don't want to argue about how yum
has to download metadata or whatever, the fact of the matter is the
yum experience is slow and the apt experience is faster. Play with
Debian some time to see what I mean.
(2) DNF's dependency resolver should be the default.
(3) RPM's spec file format needs to be redone using a Real Parser. At
the moment it has all sorts of strange corner cases (for example, how
to define a macro containing an arch-dependent list?). It'd be a good
opportunity to fix brokenness such as "global" meaning "define", lack
of direct support for configuration flags, writing "0%{?rhel}",
complexity of non-trivial %setup's, etc.
(4) Get rid of %{bindir} etc. There's no need for it. It didn't even
help during UsrMove, the one time ever that it plausibly might have helped.
(5) Almost all %pre/%post scripts need to be eliminated. There's no
reason that RPM can't detect when a shared library is being installed.
Related to this, things like users/groups/services should be written
declaratively, not imperatively:
%need-user kvm.kvm
%need-service nfsd.service
and have RPM work out how to implement it most efficiently.
(6) It should be possible to install into a chroot, and this should be
a supported configuration. Needed by LXC.
(7) %changelog section of RPM should be abandoned. Instead, you
should only have to write a changelog once (ideally in the upstream
VCS, which is then propagated everywhere it is needed).
Developers or packagers may also need to write summary notes for each
release, but again they should only have to be written once.
(8) Base packages like 'filesystem', 'setup' should include
everything. At the moment mock contains extra rules for device files
and things like that, for no reason I can understand. (See
mock/backend.py '_setupDev' function).
(9) Get rid of yum's pretend "transactions". Either implement real,
ACID transactions where the filesystem allows it, or don't pretend.
(10) Get rid of multilib, /usr/lib64 etc and copy what Debian/Ubuntu
are doing.
(11) Make it easier for yum to be consumed from other programs. DNF
fixes this.
(12) Suggests/Recommends.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
4:35 p.m.
On Wed, 2013-05-22 at 22:18 +0100, Richard W.M. Jones wrote:
[...snip various points I agree with...]
(3) RPM's spec file format needs to be redone using a Real
Parser. At
the moment it has all sorts of strange corner cases (for example, how
to define a macro containing an arch-dependent list?). It'd be a good
opportunity to fix brokenness such as "global" meaning "define",
lack
of direct support for configuration flags, writing "0%{?rhel}",
complexity of non-trivial %setup's, etc.
Another wart: IIRC, in many cases the parse won't let you have
whitespace before "%if" in a line. This makes it more difficult to use
indentation to show hierarchical structure of nested conditionals e.g.
%if 0%{some_cond}
%if 0%{some_other_cond}
%else
%endif
%else
%endif
if you see what I mean. It would be great if the parser could cope with
this.
[...snip various other points I agree with...]
Hope this is constructive
Dave
5 p.m.
Le mercredi 22 mai 2013 à 22:18 +0100, Richard W.M. Jones a écrit :
(3) RPM's spec file format needs to be redone using a Real
Parser. At
the moment it has all sorts of strange corner cases (for example, how
to define a macro containing an arch-dependent list?). It'd be a good
opportunity to fix brokenness such as "global" meaning "define",
lack
of direct support for configuration flags, writing "0%{?rhel}",
complexity of non-trivial %setup's, etc.
in 2/3 years :) Doubtful.
But then, add "have the spec format to be easy to parse by a software so
we can write better QA tools like rpmlint or fedora-review".
(5) Almost all %pre/%post scripts need to be eliminated. There's
no
reason that RPM can't detect when a shared library is being installed.
+1 for file triggers
Mandriva did have that, and that greatly help to enforce a consistent
policy and let packager focus on the others part rather than cut and
paste the same stuff.
And if we do have less %post script, we increase compatibility with
other distributions if they do the same.
--
Michael Scherer
4:22 a.m.
On 05/22/2013 11:18 PM, Richard W.M. Jones wrote:
(5) Almost all %pre/%post scripts need to be eliminated. There's
no
reason that RPM can't detect when a shared library is being installed.
I'd like to see this for the configuration of the source tree during the
build process (up to and including the %prep stage). Right now, just
applying the patches can require a system image which provides the build
dependencies. Many %prep scripts also leave behind random stuff for
developer convenience (think "patch -b"), and this interferes with
things like generating diffs or source tree indexing. And right now,
accessing the actual source code is computationally expensive.
--
Florian Weimer / Red Hat Product Security Team
6:33 a.m.
Hello,
On 22 May 2013 23:18, Richard W.M. Jones <rjones(a)redhat.com> wrote:
(10) Get rid of multilib, /usr/lib64 etc and copy what Debian/Ubuntu
are doing.
might I ask the reasoning behind this? I found the current RHEL/Fedora
approach much better.
For example; at work we use IBM Lotus Notes, which is a 32 bit package. To
install it in a 32 bit or 64 bit environment the command is the same (i.e.
yum localinstall) as it will pull in the correct 32 bit dependencies.
Basically hassle free.
The guys that run Debian/Ubuntu notebooks, have to go through a series of
dependency problems just to install the package in a 64 bit environment
with getlibs and the like:
http://usablesoftware.wordpress.com/2012/11/09/install-lotus-notes-8-5-3-...
http://www-10.lotus.com/ldd/nd85forum.nsf/DateAllThreadedWeb/69b50f2db7bb...
Regards,
--Simone
--
You cannot discover new oceans unless you have the courage to lose sight of
the shore (R. W. Emerson).
http://xkcd.com/229/
6:44 a.m.
On 23. 5. 2013 at 13:33:37, Simone Caronni wrote:
Hello,
On 22 May 2013 23:18, Richard W.M. Jones <rjones(a)redhat.com> wrote:
> (10) Get rid of multilib, /usr/lib64 etc and copy what Debian/Ubuntu
> are doing.
might I ask the reasoning behind this? I found the current RHEL/Fedora
approach much better.
For example; at work we use IBM Lotus Notes, which is a 32 bit package. To
install it in a 32 bit or 64 bit environment the command is the same (i.e.
yum localinstall) as it will pull in the correct 32 bit dependencies.
Basically hassle free.
The guys that run Debian/Ubuntu notebooks, have to go through a series of
dependency problems just to install the package in a 64 bit environment
with getlibs and the like:
+1 for this, the dependency hell for 32 bit applications is really a major
pain in the new versions of Ubuntu. I have dealt with that multiple times and
I wasn't able to resolve all the problems (e.g. Google Earth still doesn't
work on Ubuntu for me)
.
Thanks
Jan
6:59 a.m.
On 23 May 2013 13:44, Jan Zelený <jzeleny(a)redhat.com> wrote:
+1 for this, the dependency hell for 32 bit applications is really a
major
pain in the new versions of Ubuntu. I have dealt with that multiple times
and
I wasn't able to resolve all the problems (e.g. Google Earth still doesn't
work on Ubuntu for me)
Actually the first time I stepped onto this (Notes installation, in fact;
which we could never get to work) it reminded me a lot of the Windows
SysWOW64 mess.
For those that do not know:
32 bit: C:\WINDOWS\system32
64 bit: C:\WINDOWS\system32
32 on 64 bit: C:\WINDOWS\SysWOW64 + the 32 bit software sees that folder as
C:\WINDOWS\system32.
Argh.
--Simone
--
You cannot discover new oceans unless you have the courage to lose sight of
the shore (R. W. Emerson).
http://xkcd.com/229/
7:23 a.m.
On 05/23/2013 01:44 PM, Jan Zelený wrote:
+1 for this, the dependency hell for 32 bit applications is really a
major
pain in the new versions of Ubuntu. I have dealt with that multiple times and
I wasn't able to resolve all the problems (e.g. Google Earth still doesn't
work on Ubuntu for me)
Isn't that simply because deb packages do not express library
dependencies in as detailed way as rpm packages do?
I'm sure Richard was not suggesting to get rid of that rpm feature.
Michal
8:10 a.m.
On 23. 5. 2013 at 14:23:30, Michal Schmidt wrote:
On 05/23/2013 01:44 PM, Jan Zelený wrote:
> +1 for this, the dependency hell for 32 bit applications is really a major
> pain in the new versions of Ubuntu. I have dealt with that multiple times
> and I wasn't able to resolve all the problems (e.g. Google Earth still
> doesn't work on Ubuntu for me)
Isn't that simply because deb packages do not express library
dependencies in as detailed way as rpm packages do?
I'm sure Richard was not suggesting to get rid of that rpm feature.
TBH I'm not perfectly sure what causes the problem but since the new
"multilib" in Ubuntu started I am dealing with a whole bunch of issues like
the one I described. So to sum up, I personally think we are doing much better
with the way how things work in Fedora.
Thanks
Jan
8:25 a.m.
On 05/23/2013 03:10 PM, Jan Zelený wrote:
On 23. 5. 2013 at 14:23:30, Michal Schmidt wrote:
> On 05/23/2013 01:44 PM, Jan Zelený wrote:
>> +1 for this, the dependency hell for 32 bit applications is really a major
>> pain in the new versions of Ubuntu. I have dealt with that multiple times
>> and I wasn't able to resolve all the problems (e.g. Google Earth still
>> doesn't work on Ubuntu for me)
>
> Isn't that simply because deb packages do not express library
> dependencies in as detailed way as rpm packages do?
>
> I'm sure Richard was not suggesting to get rid of that rpm feature.
TBH I'm not perfectly sure what causes the problem but since the new
"multilib" in Ubuntu started I am dealing with a whole bunch of issues like
the one I described. So to sum up, I personally think we are doing much better
with the way how things work in Fedora.
In some ways we are, in other aspects we are not. Drawing conclusions
from your experience with Google Earth without real understanding is a
mistake.
Michal
8:28 a.m.
On 23. 5. 2013 at 15:25:23, Michal Schmidt wrote:
On 05/23/2013 03:10 PM, Jan Zelený wrote:
> On 23. 5. 2013 at 14:23:30, Michal Schmidt wrote:
>> On 05/23/2013 01:44 PM, Jan Zelený wrote:
>>> +1 for this, the dependency hell for 32 bit applications is really a
>>> major
>>> pain in the new versions of Ubuntu. I have dealt with that multiple
>>> times
>>> and I wasn't able to resolve all the problems (e.g. Google Earth still
>>> doesn't work on Ubuntu for me)
>>
>> Isn't that simply because deb packages do not express library
>> dependencies in as detailed way as rpm packages do?
>>
>> I'm sure Richard was not suggesting to get rid of that rpm feature.
>
> TBH I'm not perfectly sure what causes the problem but since the new
> "multilib" in Ubuntu started I am dealing with a whole bunch of issues
> like
> the one I described. So to sum up, I personally think we are doing much
> better with the way how things work in Fedora.
In some ways we are, in other aspects we are not. Drawing conclusions
from your experience with Google Earth without real understanding is a
mistake.
That's completely true. But again I'm not even sure this is something that:
a) can be fixed on rpm level
b) is other than a policy issue
All in all I think this is something that should be brought to FESCO (maybe
proposed as a Fedora Feature?) before anything else.
Thanks
Jan
9 a.m.
AIUI it would mainly involve *removing* the big hack that is
multilib from rpm.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
2:15 p.m.
On 05/23/2013 09:10 AM, Jan Zelený wrote:
TBH I'm not perfectly sure what causes the problem but since the
new
"multilib" in Ubuntu started I am dealing with a whole bunch of issues like
the one I described. So to sum up, I personally think we are doing much better
with the way how things work in Fedora.
Not really. We are focusing only on the x86_64/x86 case and ignoring
the broader problem which Debian has tackled. Jumping to the conclusion
that because you had some multi-lib issues in Ubuntu, we are doing
better is premature. Considering the fact that ARM is going to become
primary architecture for Fedora, we really need to solve the broader
problem one way or the other.
Rahul
2:41 p.m.
On Fri, 24 May 2013 15:15:30 -0400
Rahul Sundaram <metherid(a)gmail.com> wrote:
On 05/23/2013 09:10 AM, Jan Zelený wrote:
> TBH I'm not perfectly sure what causes the problem but since the new
> "multilib" in Ubuntu started I am dealing with a whole bunch of
> issues like the one I described. So to sum up, I personally think
> we are doing much better with the way how things work in Fedora.
Not really. We are focusing only on the x86_64/x86 case and ignoring
the broader problem which Debian has tackled. Jumping to the
conclusion that because you had some multi-lib issues in Ubuntu, we
are doing better is premature. Considering the fact that ARM is
going to become primary architecture for Fedora, we really need to
solve the broader problem one way or the other.
It was my understanding that arm is not going to do any multilib.
aarch64 cannot run other stuff, so you cannot run armv7 or whatever on
a aarch64 box, it's just completely different.
(I sure hope this is the case)
kevin
3:48 p.m.
On Fri, May 24, 2013 at 01:41:41PM -0600, Kevin Fenzi wrote:
It was my understanding that arm is not going to do any multilib.
aarch64 cannot run other stuff, so you cannot run armv7 or whatever on
a aarch64 box, it's just completely different.
(I sure hope this is the case)
Looks like you're right:
https://fedoraproject.org/wiki/Architectures/ARM/AArch64
(I should note that the new 64 bit ARM processors *can* run 32 bit
code, it's just that Fedora won't be supporting that mode)
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
5:42 p.m.
On Friday 24 May 2013 13:41:41 Kevin Fenzi wrote:
On Fri, 24 May 2013 15:15:30 -0400 Rahul Sundaram
<metherid(a)gmail.com> wrote:
> Not really. We are focusing only on the x86_64/x86 case and ignoring
> the broader problem which Debian has tackled. Jumping to the
> conclusion that because you had some multi-lib issues in Ubuntu, we
> are doing better is premature. Considering the fact that ARM is
> going to become primary architecture for Fedora, we really need to
> solve the broader problem one way or the other.
It was my understanding that arm is not going to do any multilib.
aarch64 cannot run other stuff, so you cannot run armv7 or whatever on
a aarch64 box, it's just completely different.
I think you missed the whole point of Debian's multi-arch -- instead of
special handling for "sister" architectures (e.g: x86/x86_64), or proving
there aren't (e.g: aarch64/armv7) -- it creates a symmetric world.
The *huge* benefit of multi-arch is to people that *cross-compile*.
Example (without multi-arch -- like in Fedora today):
* You cross compile libfoo (e.g: on x86 you compile for arm)
* You can install the resulting libfoo rpm on the arm device
* Now you want to cross-compile an application that use this
library, so you need to install libfoo-devel on your workstation -- BUT!
* The libfoo-devel you have is for arm (not installable on your x86)
* Furthermore, even if you manage to install it, it will put the files
in the wrong location (/usr/lib) and not where the cross-compiler
will search them (e.g: /usr/arm-unknown-linux/.../lib)
* So you have to extract the arm libraries from libfoo-devel and
manually put them into the correct location.
(btw: Debian embedded developers has automatic tool to "convert"
these packages -- dpkg-cross).
* If you maintain an embedded arm based product, multiply this
scenario by the number of libraries your developers maintain and
their daily commits.... does it scale? How does it affect your
whole build environment, continuous integration tools, etc.
Same example (but with multi-arch):
* Every compiler/linker/dynamic-linker for every architecture search first
in /usr/lib/<arch> and falls back to /usr/lib (to cover legacy libraries
which haven't been migrated yet).
* Under this scheme, *both* the native and cross toolchains search for
libraries in the same location -- /usr/lib/<arch>
* With careful package design, there should be no problem installing
the arm based libfoo-devel on our x86 system (no conflicting files).
* Please note, that it doesn't matter if libfoo-devel was built on an arm
machine (native compilation) or on x86 with a cross-compiler -- in both
cases the resulting library files would be on /usr/lib/arm-unknown-linux
* This means cross-toolchains becomes first class citizens.
Bye,
--
Oron Peled Voice: +972-4-8228492
oron(a)actcom.co.il http://users.actcom.co.il/~oron
Gratis is nice, Libre is an inalienable right.
12:07 p.m.
On Sat, 25 May 2013 01:42:44 +0300
Oron Peled <oron(a)actcom.co.il> wrote:
I think you missed the whole point of Debian's multi-arch --
instead
of special handling for "sister" architectures (e.g: x86/x86_64), or
proving there aren't (e.g: aarch64/armv7) -- it creates a symmetric
world.
The *huge* benefit of multi-arch is to people that *cross-compile*.
...snip...
* This means cross-toolchains becomes first class citizens.
I understand that some people cross compile, and that's nice, but I'm
not convinced it's important enough a use case to rearrange lots of
things with all the pain involved.
Since I've not actually used debian in years, I'll bow out of this
discussion now. ;)
kevin
7:21 a.m.
On 05/23/2013 01:33 PM, Simone Caronni wrote:
On 22 May 2013 23:18, Richard W.M. Jones wrote:
> (10) Get rid of multilib, /usr/lib64 etc and copy what Debian/Ubuntu
> are doing.
might I ask the reasoning behind this? I found the current RHEL/Fedora
approach much better.
For example; at work we use IBM Lotus Notes, which is a 32 bit package.
To install it in a 32 bit or 64 bit environment the command is the same
(i.e. yum localinstall) as it will pull in the correct 32 bit
dependencies. Basically hassle free.
The guys that run Debian/Ubuntu notebooks, have to go through a series
of dependency problems just to install the package in a 64 bit
environment with getlibs and the like:
http://usablesoftware.wordpress.com/2012/11/09/install-lotus-notes-8-5-3-...
http://www-10.lotus.com/ldd/nd85forum.nsf/DateAllThreadedWeb/69b50f2db7bb...
These examples both suggest the use of the "ia32-libs" package. The
package was an ugly workaround for the missing multilib support in
Debian, but it should be obsolete with full multiarch in Debian 7.0.
See http://wiki.debian.org/Multiarch
Michal
7:58 a.m.
On Thu, May 23, 2013 at 01:33:37PM +0200, Simone Caronni wrote:
might I ask the reasoning behind this? I found the current
RHEL/Fedora
approach much better.
Debian (since 7?) has settled on using subdirectories of /usr/lib for
different architectures. See:
http://wiki.debian.org/Multiarch
It supports more than just "64 bit or not", such as different kernels,
different endianness, and different instruction set extensions.
There's a link which explains better than I can why they did it:
http://wiki.debian.org/Multiarch/TheCaseForMultiarch
The reason I specifically said:
On 22 May 2013 23:18, Richard W.M. Jones <rjones(a)redhat.com> wrote:
(10) Get rid of multilib, /usr/lib64 etc and copy what Debian/Ubuntu
are doing.
is that Debian and derivatives are more popular than Fedora and the
Linux community ought to settle on one way of doing this, which just
by virtue of popularity and completeness of the proposal means doing
it the Debian way in this case.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
8:29 a.m.
On 23 May 2013 14:58, Richard W.M. Jones <rjones(a)redhat.com> wrote:
Debian (since 7?) has settled on using subdirectories of /usr/lib
for
different architectures. See:
http://wiki.debian.org/Multiarch
It supports more than just "64 bit or not", such as different kernels,
different endianness, and different instruction set extensions.
There's a link which explains better than I can why they did it:
http://wiki.debian.org/Multiarch/TheCaseForMultiarch
Thanks for the links.
I'm not the best person to judge, but it looks overcomplicated to me. For
sure existing commercial binary packages shipped in RPM format will have a
lot of problems.
Thanks,
--Simone
--
You cannot discover new oceans unless you have the courage to lose sight of
the shore (R. W. Emerson).
http://xkcd.com/229/
8:54 a.m.
On Thu, May 23, 2013 at 03:29:16PM +0200, Simone Caronni wrote:
I'm not the best person to judge, but it looks overcomplicated to
me. For
sure existing commercial binary packages shipped in RPM format will have a
lot of problems.
For the vast majority of packages, it simply means the library
moves from /usr/lib64 to /usr/lib/x86_64-linux-gnu. That's not
complicated.
It can certainly be made complicated -- eg. by having a library
that is compiled multiple times with different SSE extensions.
But that's something that cannot be done at all on Fedora
without using package-specific hacking.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
9:24 a.m.
On 05/23/2013 02:58 PM, Richard W.M. Jones wrote:
The reason I specifically said:
On 22 May 2013 23:18, Richard W.M. Jones <rjones(a)redhat.com> wrote:
> (10) Get rid of multilib, /usr/lib64 etc and copy what Debian/Ubuntu
> are doing.
is that Debian and derivatives are more popular than Fedora and the
Linux community ought to settle on one way of doing this, which just
by virtue of popularity and completeness of the proposal means doing
it the Debian way in this case.
The "Debian way" (not sure if it's actually the "Ubuntu way")
hasn't
been widely upstreamed. GCC didn't build for a year or two, and many
upstream configure-type scripts still fail. The lib64 approach appears
to be less invasive (or Fedora was just better at upstreaming support
for it 8-).
Some of the rationale for multiarch doesn't make much sense anymore.
For example, you can now use IFUNCs to select specialized functions at
run time and don't have to ship separate DSOs anymore.
--
Florian Weimer / Red Hat Product Security Team
9:38 a.m.
On Thu, May 23, 2013 at 04:24:01PM +0200, Florian Weimer wrote:
Some of the rationale for multiarch doesn't make much sense
anymore.
For example, you can now use IFUNCs to select specialized functions
at run time and don't have to ship separate DSOs anymore.
IFUNCs are indeed interesting. An explanation of them is here:
https://wiki.linaro.org/WorkingGroups/ToolChain/Outputs/IFunc
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
6:37 a.m.
Since you have already sent the email, all those requests that make sense are
in one way or another on the list of RFEs I referred to. If you are missing
something there, feel free to let me know.
Thanks
Jan
On 22. 5. 2013 at 22:18:58, Richard W.M. Jones wrote:
[This is a copy of an email I sent to an internal Red Hat list last
month]
In no particular order:
(1) Yum should not be so slow. In particular "yum install" takes ages
compared to "apt-get install". I don't want to argue about how yum
has to download metadata or whatever, the fact of the matter is the
yum experience is slow and the apt experience is faster. Play with
Debian some time to see what I mean.
(2) DNF's dependency resolver should be the default.
(3) RPM's spec file format needs to be redone using a Real Parser. At
the moment it has all sorts of strange corner cases (for example, how
to define a macro containing an arch-dependent list?). It'd be a good
opportunity to fix brokenness such as "global" meaning "define",
lack
of direct support for configuration flags, writing "0%{?rhel}",
complexity of non-trivial %setup's, etc.
(4) Get rid of %{bindir} etc. There's no need for it. It didn't even
help during UsrMove, the one time ever that it plausibly might have helped.
(5) Almost all %pre/%post scripts need to be eliminated. There's no
reason that RPM can't detect when a shared library is being installed.
Related to this, things like users/groups/services should be written
declaratively, not imperatively:
%need-user kvm.kvm
%need-service nfsd.service
and have RPM work out how to implement it most efficiently.
(6) It should be possible to install into a chroot, and this should be
a supported configuration. Needed by LXC.
(7) %changelog section of RPM should be abandoned. Instead, you
should only have to write a changelog once (ideally in the upstream
VCS, which is then propagated everywhere it is needed).
Developers or packagers may also need to write summary notes for each
release, but again they should only have to be written once.
(8) Base packages like 'filesystem', 'setup' should include
everything. At the moment mock contains extra rules for device files
and things like that, for no reason I can understand. (See
mock/backend.py '_setupDev' function).
(9) Get rid of yum's pretend "transactions". Either implement real,
ACID transactions where the filesystem allows it, or don't pretend.
(10) Get rid of multilib, /usr/lib64 etc and copy what Debian/Ubuntu
are doing.
(11) Make it easier for yum to be consumed from other programs. DNF
fixes this.
(12) Suggests/Recommends.
8:59 a.m.
Hi,
I'd also like support for tuple provides, to expose font font facets to
the software updater.
For example, a font file can provide a 'bold' variant with support for a
list of locales.
Right now if I ask rpm to install a bold sans-serif font for russian for
example, it will happily install a package with an italic serif russian
font, a bold japanese font and some other unrelated sans-serif stuff.
There is no way to tell it I want both properties in the same provides.
(I'm sure there are other use cases)
--
Nicolas Mailhot
4:52 p.m.
On 05/22/2013 07:43 AM, Jan Zelený wrote:
Dear Fedora community,
several months ago, at the Developer conference in Brno, Software Management
team received a whole bunch of proposals for new functionality in RPM and
related software stack.
We acknowledge the need for some changes in Software Management stack in
Fedora but we don't want to make changes just by guessing what our
users want. Therefore I call to you, consumers of our products (dnf, yum and
rpm): what are the changes that you would like to see in the foreseeable
future (say 2-3 years) and why would you like to see them (what would they
help you with)?
There is already a list of some RFEs on rpm.org wiki, you can use it as an
inspiration, to see what RFEs we have already received:
http://rpm.org/wiki/FeaturePlanning
The only limitation for your requests is our manifest which defines the scope
of SW management stack for the future. It is attached to this email (note that
it's quite extensive but the first part should give you a good image of what is
the planned scope of SW management stack).
Please send your requests as replies to this email so they can be properly
discussed.
After your proposals are filed and discussed, all will be evaluated by our
team and a roadmap with priorities will be created with those selected as
doable and meaningful.
Thank you in advance for your participation
Jan
Something I'm just now running into - I have a package that can make use of
one of two different backends, but it definitely needs one of them. I don't
want to pick which one in the package. Also, it is explicitly referencing
specific implementations, not a generic interface, so a generic Provides in
the backend packages is not appropriate. But something like:
Requires: ( pkgA || pkgB )
might do the trick.
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
5:07 p.m.
Reverting changes to files handled by RPM (or installing a single file out
of the package), for instance:
rpm -qp some-rpm.rpm --revert/--extract /etc/some-rpm.conf
/etc/another-file.conf
I know it can be done with rpm2cpio, just a suggestion to implement it
natively and extract the files to their correct location.
On Thu, May 23, 2013 at 7:52 AM, Orion Poplawski <orion(a)cora.nwra.com>wrote:
On 05/22/2013 07:43 AM, Jan Zelený wrote:
> Dear Fedora community,
> several months ago, at the Developer conference in Brno, Software
> Management
> team received a whole bunch of proposals for new functionality in RPM and
> related software stack.
>
> We acknowledge the need for some changes in Software Management stack in
> Fedora but we don't want to make changes just by guessing what our
> users want. Therefore I call to you, consumers of our products (dnf, yum
> and
> rpm): what are the changes that you would like to see in the foreseeable
> future (say 2-3 years) and why would you like to see them (what would they
> help you with)?
>
> There is already a list of some RFEs on rpm.org wiki, you can use it as
> an
> inspiration, to see what RFEs we have already received:
> http://rpm.org/wiki/**FeaturePlanning<http://rpm.org/wiki/FeaturePlann...
>
> The only limitation for your requests is our manifest which defines the
> scope
> of SW management stack for the future. It is attached to this email (note
> that
> it's quite extensive but the first part should give you a good image of
> what is
> the planned scope of SW management stack).
>
> Please send your requests as replies to this email so they can be properly
> discussed.
> After your proposals are filed and discussed, all will be evaluated by our
> team and a roadmap with priorities will be created with those selected as
> doable and meaningful.
>
> Thank you in advance for your participation
> Jan
>
Something I'm just now running into - I have a package that can make use
of one of two different backends, but it definitely needs one of them. I
don't want to pick which one in the package. Also, it is explicitly
referencing specific implementations, not a generic interface, so a generic
Provides in the backend packages is not appropriate. But something like:
Requires: ( pkgA || pkgB )
might do the trick.
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
--
devel mailing list
devel(a)lists.fedoraproject.org
https://admin.fedoraproject.**org/mailman/listinfo/devel<https://admin...
6:47 a.m.
May I ask what is the use case for this? I'm not sure why would you need to
deal with individual files instead of the entire packages.
Thanks
Jan
On 23. 5. 2013 at 08:07:21, Dan Fruehauf wrote:
Reverting changes to files handled by RPM (or installing a single
file out
of the package), for instance:
rpm -qp some-rpm.rpm --revert/--extract /etc/some-rpm.conf
/etc/another-file.conf
I know it can be done with rpm2cpio, just a suggestion to implement it
natively and extract the files to their correct location.
On Thu, May 23, 2013 at 7:52 AM, Orion Poplawski <orion(a)cora.nwra.com>wrote:
> On 05/22/2013 07:43 AM, Jan Zelený wrote:
>> Dear Fedora community,
>> several months ago, at the Developer conference in Brno, Software
>> Management
>> team received a whole bunch of proposals for new functionality in RPM and
>> related software stack.
>>
>> We acknowledge the need for some changes in Software Management stack in
>> Fedora but we don't want to make changes just by guessing what our
>> users want. Therefore I call to you, consumers of our products (dnf, yum
>> and
>> rpm): what are the changes that you would like to see in the foreseeable
>> future (say 2-3 years) and why would you like to see them (what would
>> they
>> help you with)?
>>
>> There is already a list of some RFEs on rpm.org wiki, you can use it as
>> an
>> inspiration, to see what RFEs we have already received:
>> http://rpm.org/wiki/**FeaturePlanning<http://rpm.org/wiki/FeaturePlanning
>> >
>>
>> The only limitation for your requests is our manifest which defines the
>> scope
>> of SW management stack for the future. It is attached to this email (note
>> that
>> it's quite extensive but the first part should give you a good image of
>> what is
>> the planned scope of SW management stack).
>>
>> Please send your requests as replies to this email so they can be
>> properly
>> discussed.
>> After your proposals are filed and discussed, all will be evaluated by
>> our
>> team and a roadmap with priorities will be created with those selected as
>> doable and meaningful.
>>
>> Thank you in advance for your participation
>> Jan
>
> Something I'm just now running into - I have a package that can make use
> of one of two different backends, but it definitely needs one of them. I
> don't want to pick which one in the package. Also, it is explicitly
> referencing specific implementations, not a generic interface, so a
> generic
> Provides in the backend packages is not appropriate. But something like:
>
> Requires: ( pkgA || pkgB )
>
> might do the trick.
6:52 a.m.
On 23 May 2013 13:47, Jan Zelený <jzeleny(a)redhat.com> wrote:
May I ask what is the use case for this? I'm not sure why would
you need to
deal with individual files instead of the entire packages.
Maybe to reinstall one default config file out of a package that contains
some? I found it useful.
Example:
I fiddle around with a new Nagios installation, then something stops
working. I'm pretty sure it is some modifications in /etc/nagios/nagios.cfg
but I cannot track it down.
As an example I could do:
mv /etc/nagios/nagios.cfg /etc/nagios/nagios.cfg.mine
yum reinstall nagios --onlyfile /etc/nagios/nagios.cfg
Thanks,
--Simone
8:13 a.m.
On 23. 5. 2013 at 13:52:39, Simone Caronni wrote:
On 23 May 2013 13:47, Jan Zelený <jzeleny(a)redhat.com> wrote:
> May I ask what is the use case for this? I'm not sure why would you need
> to
> deal with individual files instead of the entire packages.
Maybe to reinstall one default config file out of a package that contains
some? I found it useful.
Example:
I fiddle around with a new Nagios installation, then something stops
working. I'm pretty sure it is some modifications in /etc/nagios/nagios.cfg
but I cannot track it down.
As an example I could do:
mv /etc/nagios/nagios.cfg /etc/nagios/nagios.cfg.mine
yum reinstall nagios --onlyfile /etc/nagios/nagios.cfg
Thanks for the explanation. I will mark the email and make sure to put the use
case to the updated list of RFEs.
Thanks
Jan
11:05 a.m.
On Thu, May 23, 2013 at 03:13:30PM +0200, Jan Zelený wrote:
On 23. 5. 2013 at 13:52:39, Simone Caronni wrote:
> I fiddle around with a new Nagios installation, then something
stops
> working. I'm pretty sure it is some modifications in /etc/nagios/nagios.cfg
> but I cannot track it down.
> As an example I could do:
>
> mv /etc/nagios/nagios.cfg /etc/nagios/nagios.cfg.mine
> yum reinstall nagios --onlyfile /etc/nagios/nagios.cfg
Thanks for the explanation. I will mark the email and make sure to put the use
case to the updated list of RFEs.
I would like it even more if RPM keeps a backup copy of every %config
file somewhere protected to allow to restore config files even without
having the original RPM available, because the RPM might not be
available anymore.
Regards
Till
3:48 a.m.
Dne Čt 23. května 2013 18:05:40, Till Maas napsal(a):
On Thu, May 23, 2013 at 03:13:30PM +0200, Jan Zelený wrote:
> On 23. 5. 2013 at 13:52:39, Simone Caronni wrote:
> > I fiddle around with a new Nagios installation, then something stops
> > working. I'm pretty sure it is some modifications in
> > /etc/nagios/nagios.cfg
> > but I cannot track it down.
> > As an example I could do:
> >
> > mv /etc/nagios/nagios.cfg /etc/nagios/nagios.cfg.mine
> > yum reinstall nagios --onlyfile /etc/nagios/nagios.cfg
>
> Thanks for the explanation. I will mark the email and make sure to put the
> use case to the updated list of RFEs.
I would like it even more if RPM keeps a backup copy of every %config
file somewhere protected to allow to restore config files even without
having the original RPM available, because the RPM might not be
available anymore.
Already on the list (see the very first RFE).
Thanks
Jan
10:38 a.m.
On Thu, 2013-05-23 at 13:52 +0200, Simone Caronni wrote:
On 23 May 2013 13:47, Jan Zelený <jzeleny(a)redhat.com> wrote:
> May I ask what is the use case for this? I'm not sure why would you need to
> deal with individual files instead of the entire packages.
Maybe to reinstall one default config file out of a package that contains
some? I found it useful.
Example:
I fiddle around with a new Nagios installation, then something stops
working. I'm pretty sure it is some modifications in /etc/nagios/nagios.cfg
but I cannot track it down.
As an example I could do:
mv /etc/nagios/nagios.cfg /etc/nagios/nagios.cfg.mine
yum reinstall nagios --onlyfile /etc/nagios/nagios.cfg
How is this functionally different from "yum reinstall nagios" ?
Yes, yum/rpm will currently replace files with identical copies but
functionally the extra copies are "just wasting reinstall time".
11:05 a.m.
On 23 May 2013 17:38, James Antill <james(a)fedoraproject.org> wrote:
On Thu, 2013-05-23 at 13:52 +0200, Simone Caronni wrote:
> mv /etc/nagios/nagios.cfg /etc/nagios/nagios.cfg.mine
> yum reinstall nagios --onlyfile /etc/nagios/nagios.cfg
How is this functionally different from "yum reinstall nagios" ?
Yes, yum/rpm will currently replace files with identical copies but
functionally the extra copies are "just wasting reinstall time".
By doing "yum reinstall nagios" I don't have the the default config files
(i.e. nagios.cfg.rpmnew is not created). This happens only on upgrades.
--
You cannot discover new oceans unless you have the courage to lose sight of
the shore (R. W. Emerson).
http://xkcd.com/229/
12:08 a.m.
On 05/23/2013 06:05 PM, Simone Caronni wrote:
On 23 May 2013 17:38, James Antill <james(a)fedoraproject.org
<mailto:james@fedoraproject.org>> wrote:
On Thu, 2013-05-23 at 13:52 +0200, Simone Caronni wrote:
> mv /etc/nagios/nagios.cfg /etc/nagios/nagios.cfg.mine
> yum reinstall nagios --onlyfile /etc/nagios/nagios.cfg
How is this functionally different from "yum reinstall nagios" ?
Yes, yum/rpm will currently replace files with identical copies but
functionally the extra copies are "just wasting reinstall time".
By doing "yum reinstall nagios" I don't have the the default config
files (i.e. nagios.cfg.rpmnew is not created). This happens only on
upgrades.
yum remove <pkg>
yum install <pkg>
should do what you want (backups of modified config files).
Ralf
2:26 a.m.
Le vendredi 24 mai 2013 à 07:08 +0200, Ralf Corsepius a écrit :
On 05/23/2013 06:05 PM, Simone Caronni wrote:
> On 23 May 2013 17:38, James Antill <james(a)fedoraproject.org
> <mailto:james@fedoraproject.org>> wrote:
>
> On Thu, 2013-05-23 at 13:52 +0200, Simone Caronni wrote:
> > mv /etc/nagios/nagios.cfg /etc/nagios/nagios.cfg.mine
> > yum reinstall nagios --onlyfile /etc/nagios/nagios.cfg
>
> How is this functionally different from "yum reinstall nagios" ?
> Yes, yum/rpm will currently replace files with identical copies but
> functionally the extra copies are "just wasting reinstall time".
>
>
> By doing "yum reinstall nagios" I don't have the the default config
> files (i.e. nagios.cfg.rpmnew is not created). This happens only on
> upgrades.
yum remove <pkg>
yum install <pkg>
should do what you want (backups of modified config files).
But that's unfortunately not always convenient if you need to remove
dependent packages for "yum remove" ?
I guess yum-shell may help in that case ?
--
Michael Scherer
11:58 a.m.
On 05/24/2013 09:26 AM, Michael Scherer wrote:
Le vendredi 24 mai 2013 à 07:08 +0200, Ralf Corsepius a écrit :
> On 05/23/2013 06:05 PM, Simone Caronni wrote:
>> On 23 May 2013 17:38, James Antill <james(a)fedoraproject.org
>> <mailto:james@fedoraproject.org>> wrote:
>>
>> On Thu, 2013-05-23 at 13:52 +0200, Simone Caronni wrote:
>> > mv /etc/nagios/nagios.cfg /etc/nagios/nagios.cfg.mine
>> > yum reinstall nagios --onlyfile /etc/nagios/nagios.cfg
>>
>> How is this functionally different from "yum reinstall nagios"
?
>> Yes, yum/rpm will currently replace files with identical copies but
>> functionally the extra copies are "just wasting reinstall time".
>>
>>
>> By doing "yum reinstall nagios" I don't have the the default
config
>> files (i.e. nagios.cfg.rpmnew is not created). This happens only on
>> upgrades.
>
> yum remove <pkg>
> yum install <pkg>
>
> should do what you want (backups of modified config files).
But that's unfortunately not always convenient if you need to remove
dependent packages for "yum remove" ?
I guess yum-shell may help in that case ?
c.f. what I wrote in
http://lists.fedoraproject.org/pipermail/devel/2013-May/183321.html
This works in most cases (I haven't seen it fails in any real world
case, it's imaginable it fails in some stateful packages).
Ralf
11:26 p.m.
Well, another thought is that if some software has a lot of optional
dependencies, how to handle them from the view of end-users?
Should we list these optionals or add a option for install all with deps?
Thanks.
3:02 p.m.
On 05/23/2013 07:52 AM, Simone Caronni wrote:
I fiddle around with a new Nagios installation, then something stops
working. I'm pretty sure it is some modifications in
/etc/nagios/nagios.cfg but I cannot track it down.
As an example I could do:
mv /etc/nagios/nagios.cfg /etc/nagios/nagios.cfg.mine
yum reinstall nagios --onlyfile /etc/nagios/nagios.cfg
What's the advantage over just yum reinstall nagios, given that modified
config files are preserved? A known baseline is a huge advantage of RPM,
and allowing retail installation seems to go against that.
9:52 a.m.
On 24 May 2013 22:02, Przemek Klosowski <przemek.klosowski(a)nist.gov> wrote:
On 05/23/2013 07:52 AM, Simone Caronni wrote:
I fiddle around with a new Nagios installation, then something stops
> working. I'm pretty sure it is some modifications in
> /etc/nagios/nagios.cfg but I cannot track it down.
> As an example I could do:
>
> mv /etc/nagios/nagios.cfg /etc/nagios/nagios.cfg.mine
> yum reinstall nagios --onlyfile /etc/nagios/nagios.cfg
>
What's the advantage over just yum reinstall nagios, given that modified
config files are preserved? A known baseline is a huge advantage of RPM,
and allowing retail installation seems to go against that.
Well, that's true; I was thinking of a more obscure case that happened to
me with some config files and manipulated files under /usr/share but that's
really a corner case. If I just save the file I get the same behaviour.
Forget it.
Regards,
--Simone
--
You cannot discover new oceans unless you have the courage to lose sight of
the shore (R. W. Emerson).
http://xkcd.com/229/
5:30 p.m.
On Wed, May 22, 2013 at 03:52:22PM -0600, Orion Poplawski wrote:
Something I'm just now running into - I have a package that can
make
use of one of two different backends, but it definitely needs one of
them. I don't want to pick which one in the package. Also, it is
explicitly referencing specific implementations, not a generic
interface, so a generic Provides in the backend packages is not
appropriate. But something like:
Requires: ( pkgA || pkgB )
might do the trick.
FWIW dpkg can express this kind of dependency, eg:
Depends: exim | mail-transport-agent
[from: http://www.debian.org/doc/debian-policy/ch-relationships.html]
If implemented, this would have implications for Fedora. You could
have two people who have installed the same package, with a very
different set of dependent packages, leading to some sort of
combinatorial explosion of QA.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
5:44 p.m.
On Wed, 2013-05-22 at 23:30 +0100, Richard W.M. Jones wrote:
different set of dependent packages, leading to some sort of
combinatorial explosion of QA.
Right now we have approximately seven zillion combinatorial explosions
of QA, so one more on the pile is not going to make much of an
appreciable difference...though it could spell all kinds of fun for
image composes, I suspect.
Still, I don't see how that's really worth doing instead of just using a
virtual provide. The OP's reasoning for not using a virtual provide
sounded rather specious to me. The two things are functionally
equivalent so far as I can see. Just different syntax.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
5:55 p.m.
On 05/22/2013 04:44 PM, Adam Williamson wrote:
On Wed, 2013-05-22 at 23:30 +0100, Richard W.M. Jones wrote:
> different set of dependent packages, leading to some sort of
> combinatorial explosion of QA.
Right now we have approximately seven zillion combinatorial explosions
of QA, so one more on the pile is not going to make much of an
appreciable difference...though it could spell all kinds of fun for
image composes, I suspect.
Still, I don't see how that's really worth doing instead of just using a
virtual provide. The OP's reasoning for not using a virtual provide
sounded rather specious to me. The two things are functionally
equivalent so far as I can see. Just different syntax.
I'll get more specific then:
python-pyface can use two different graphics backends - either wxPython or
pyQt4. In no way do these two packages provide the same thing in any
meaningful way other than to pyface. So, while one could go the provides
route, it doesn't seem to make sense to me. I suppose I could ask for:
Provides: pyface-backend
in both PyQt4 and wxPython. Think that would make sense?
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
6:19 p.m.
On 23.05.2013 00:55, Orion Poplawski wrote:
I'll get more specific then:
python-pyface can use two different graphics backends - either
wxPython or pyQt4. In no way do these two packages provide the same
thing in any meaningful way other than to pyface. So, while one could
go the provides route, it doesn't seem to make sense to me. I suppose
I could ask for:
Provides: pyface-backend
in both PyQt4 and wxPython. Think that would make sense?
From my two-second look at the pyface codebase, it looks like backends
are treated as modular components. So why not just split out the backend
specific modules in python-pyface-qt and python-pyface-wx packages, put
the rest in -common, and have the user decide which to install? Just an
idea.
- Sandro
6:49 a.m.
On 22. 5. 2013 at 16:55:50, Orion Poplawski wrote:
On 05/22/2013 04:44 PM, Adam Williamson wrote:
> On Wed, 2013-05-22 at 23:30 +0100, Richard W.M. Jones wrote:
>> different set of dependent packages, leading to some sort of
>> combinatorial explosion of QA.
>
> Right now we have approximately seven zillion combinatorial explosions
> of QA, so one more on the pile is not going to make much of an
> appreciable difference...though it could spell all kinds of fun for
> image composes, I suspect.
>
> Still, I don't see how that's really worth doing instead of just using a
> virtual provide. The OP's reasoning for not using a virtual provide
> sounded rather specious to me. The two things are functionally
> equivalent so far as I can see. Just different syntax.
I'll get more specific then:
python-pyface can use two different graphics backends - either wxPython or
pyQt4. In no way do these two packages provide the same thing in any
meaningful way other than to pyface. So, while one could go the provides
route, it doesn't seem to make sense to me. I suppose I could ask for:
Provides: pyface-backend
in both PyQt4 and wxPython. Think that would make sense?
Yes :-)
Thanks
Jan
10:34 a.m.
On Wed, 2013-05-22 at 16:55 -0600, Orion Poplawski wrote:
I'll get more specific then:
python-pyface can use two different graphics backends - either wxPython or
pyQt4. In no way do these two packages provide the same thing in any
meaningful way other than to pyface. So, while one could go the provides
route, it doesn't seem to make sense to me. I suppose I could ask for:
Provides: pyface-backend
in both PyQt4 and wxPython. Think that would make sense?
Add a layer, and it works perfectly:
python-pyface:
Requires: python-pyface-backend
python-pyface-backend-wxPython:
Provides: python-pyface-backend
Requires: wxPython
python-pyface-backend-pyQt4:
Provides: python-pyface-backend
Requires: pyQt4
3:18 p.m.
On 05/23/2013 09:34 AM, James Antill wrote:
On Wed, 2013-05-22 at 16:55 -0600, Orion Poplawski wrote:
> I'll get more specific then:
>
> python-pyface can use two different graphics backends - either wxPython or
> pyQt4. In no way do these two packages provide the same thing in any
> meaningful way other than to pyface. So, while one could go the provides
> route, it doesn't seem to make sense to me. I suppose I could ask for:
>
> Provides: pyface-backend
>
> in both PyQt4 and wxPython. Think that would make sense?
Add a layer, and it works perfectly:
python-pyface:
Requires: python-pyface-backend
python-pyface-backend-wxPython:
Provides: python-pyface-backend
Requires: wxPython
python-pyface-backend-pyQt4:
Provides: python-pyface-backend
Requires: pyQt4
Took me a little bit to realize that no actual code needed to be split out to
do this (Sandro's suggestion too much in my mind). But yes, this does the
trick nicely, at the expense of some extra dummy packages.
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
3:03 a.m.
On 2013-05-23, Orion Poplawski <orion(a)cora.nwra.com> wrote:
On 05/23/2013 09:34 AM, James Antill wrote:
> On Wed, 2013-05-22 at 16:55 -0600, Orion Poplawski wrote:
>
>> I'll get more specific then:
>>
>> python-pyface can use two different graphics backends - either wxPython or
>> pyQt4. In no way do these two packages provide the same thing in any
>> meaningful way other than to pyface. So, while one could go the provides
>> route, it doesn't seem to make sense to me. I suppose I could ask for:
>>
>> Provides: pyface-backend
>>
>> in both PyQt4 and wxPython. Think that would make sense?
>
> Add a layer, and it works perfectly:
>
> python-pyface:
> Requires: python-pyface-backend
>
> python-pyface-backend-wxPython:
> Provides: python-pyface-backend
> Requires: wxPython
>
> python-pyface-backend-pyQt4:
> Provides: python-pyface-backend
> Requires: pyQt4
>
Took me a little bit to realize that no actual code needed to be split out to
do this (Sandro's suggestion too much in my mind). But yes, this does the
trick nicely, at the expense of some extra dummy packages.
Do you know virtual packages are forbidden in Fedora?
-- Petr
5:37 a.m.
On 2013-05-24, Jonathan Dieter <jdieter(a)lesbg.com> wrote:
On Fri, 2013-05-24 at 08:03 +0000, Petr Pisar wrote:
> Do you know virtual packages are forbidden in Fedora?
Sorry, I just scanned through the guidelines and didn't see this
anywhere. Do you mind citing a reference, please?
Well it's probably not forbidden by guidelines, but I've heard this rule
many times. There exists a thread on fedora-packaging mailing list
<http://lists.fedoraproject.org/pipermail/packaging/2009-January/005481.ht...;.
I also think rpmlint considers empty package as an error and I remember
I saw an rpmbuild run that did not produce an sub-package just because
there was no %files section.
Despite all of that I consider this solution with meta sub-packages as
the best option until RPM will support disjunction.
-- Petr
6:27 a.m.
On 05/24/2013 01:37 PM, Petr Pisar wrote:
On 2013-05-24, Jonathan Dieter <jdieter(a)lesbg.com> wrote:
> On Fri, 2013-05-24 at 08:03 +0000, Petr Pisar wrote:
>> Do you know virtual packages are forbidden in Fedora?
>
> Sorry, I just scanned through the guidelines and didn't see this
> anywhere. Do you mind citing a reference, please?
>
Well it's probably not forbidden by guidelines, but I've heard this rule
many times. There exists a thread on fedora-packaging mailing list
<http://lists.fedoraproject.org/pipermail/packaging/2009-January/005481.ht...;.
What's generally frowned upon is the "task-foo" style gigantic
metapackages, but metapackages are used in Fedora for more specific
purposes. Take xorg-x11-drivers or wine for example.
I also think rpmlint considers empty package as an error and I
remember
I saw an rpmbuild run that did not produce an sub-package just because
there was no %files section.
Missing %files section is different from an empty %files section: a
non-existing %files section indeed causes no package to be generated
(which various people are taking advantage of for simplifying
conditional builds), but empty %files section is perfectly legal, always
been.
- Panu -
12:08 p.m.
On Fri, 2013-05-24 at 10:37 +0000, Petr Pisar wrote:
On 2013-05-24, Jonathan Dieter <jdieter(a)lesbg.com> wrote:
> On Fri, 2013-05-24 at 08:03 +0000, Petr Pisar wrote:
>> Do you know virtual packages are forbidden in Fedora?
>
> Sorry, I just scanned through the guidelines and didn't see this
> anywhere. Do you mind citing a reference, please?
>
Well it's probably not forbidden by guidelines, but I've heard this rule
many times. There exists a thread on fedora-packaging mailing list
<http://lists.fedoraproject.org/pipermail/packaging/2009-January/005481.ht...;.
Well, I think there's a bit of conflation going on here. 'Virtual
packages', 'metapackages', and 'virtual provides' are somewhat
different
things, and what has historically been discouraged in Fedora is one
specific form. Let's call this form specifically a 'metapackage', to be
clear.
A 'metapackage' is an actual package shipped in the repositories which
contains no files, and whose raison d'etre is to express some
dependencies. There are a few of these in Fedora, xorg-x11-drivers being
the classic example, but they are generally strongly discouraged. The
idea is that Fedora uses comps groups to express the concept 'this group
of packages forms some kind of cohesive set and can be installed
together', not metapackages.
What was being discussed in this sub-thread was not a metapackage at
all, only the use of virtual Provides:. A virtual Provide is kind of a
slithery concept when you sit down and start thinking about it, but I
guess I'd say it's when a package Provides: something that is a kind of
abstract concept (in order that multiple packages can provide the same
thing, and other packages can express 'I need some capability that all
of these packages provide'). Really, almost any Provides: statement that
actually has to be written into the spec file by the packager is a
'virtual provide'.
There's certainly nothing in the guidelines that restricts the use of
virtual provides, and it even specifically requires them in one
scenario:
"Static libraries only. When a package only provides static libraries
you can place all the static library files in the *-devel subpackage.
When doing this you also must have a virtual Provide for the *-static
package:
%package devel
Provides: foo-static = %{version}-%{release}"
I don't think there's any problem with the use of 'virtual provides' in
Fedora policy or tradition, it's 'metapackages' that are usually
discouraged.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
1:36 a.m.
Hi,
On Fri, May 24, 2013 at 10:08:06AM -0700, Adam Williamson wrote:
A 'metapackage' is an actual package shipped in the
repositories which
contains no files, and whose raison d'etre is to express some
dependencies. There are a few of these in Fedora, xorg-x11-drivers being
the classic example, but they are generally strongly discouraged. The
idea is that Fedora uses comps groups to express the concept 'this group
of packages forms some kind of cohesive set and can be installed
together', not metapackages.
Some things in favor of metapackages:
1. The packager has to know that such thing as comps exists. (I know
about it. I have even seen patches for it, so I know it is some sort of
XML. But I have no idea where to look for it. Is there a repo for it?
"fedpkg clone -B comps" was unsuccessful.) On the other side, the
packager already knows how to make a (sub-)package that depends on other
package(s).
2. A metapackage uses rpm syntax for specifying the dependencies. No
need to learn anything new.
3. A metapackage is under the packager's control. A comps group is not.
(I doubt packagers have commit access to to comps repo (if there is a
repo). That means creating a patch, opening a bug, waiting for reaction
(probably for days),... tadda yadda... Or I can create a subpackage and
be done in a minute.)
D.
1:40 a.m.
On Sat, 2013-05-25 at 08:36 +0200, David Tardon wrote:
Hi,
On Fri, May 24, 2013 at 10:08:06AM -0700, Adam Williamson wrote:
> A 'metapackage' is an actual package shipped in the repositories which
> contains no files, and whose raison d'etre is to express some
> dependencies. There are a few of these in Fedora, xorg-x11-drivers being
> the classic example, but they are generally strongly discouraged. The
> idea is that Fedora uses comps groups to express the concept 'this group
> of packages forms some kind of cohesive set and can be installed
> together', not metapackages.
Some things in favor of metapackages:
1. The packager has to know that such thing as comps exists. (I know
about it. I have even seen patches for it, so I know it is some sort of
XML. But I have no idea where to look for it. Is there a repo for it?
"fedpkg clone -B comps" was unsuccessful.) On the other side, the
First google result for 'fedora comps' is:
https://fedoraproject.org/wiki/How_to_use_and_edit_comps.xml_for_package_...
It has a 'how to edit comps' section which gives the repo URL.
3. A metapackage is under the packager's control. A comps group
is not.
(I doubt packagers have commit access to to comps repo (if there is a
repo).
Guess again. All packagers can commit to comps, I believe. But please,
please, try to avoid committing to it during release freezes without
checking in with notting/qa/releng, especially if your change adds
packages to release media. Or is completely broken, as has happened. ;)
That means creating a patch, opening a bug, waiting for reaction
(probably for days),... tadda yadda... Or I can create a subpackage and
be done in a minute.)
Or you could not make assumptions :)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
3:17 a.m.
On 2013-05-22, Orion Poplawski <orion(a)cora.nwra.com> wrote:
On 05/22/2013 04:44 PM, Adam Williamson wrote:
> On Wed, 2013-05-22 at 23:30 +0100, Richard W.M. Jones wrote:
>
> Still, I don't see how that's really worth doing instead of just using a
> virtual provide. The OP's reasoning for not using a virtual provide
> sounded rather specious to me. The two things are functionally
> equivalent so far as I can see. Just different syntax.
>
I'll get more specific then:
python-pyface can use two different graphics backends - either wxPython or
pyQt4. In no way do these two packages provide the same thing in any
meaningful way other than to pyface. So, while one could go the provides
route, it doesn't seem to make sense to me. I suppose I could ask for:
Provides: pyface-backend
in both PyQt4 and wxPython. Think that would make sense?
In my option this is wrong becuse:
(1) You have to touch foreign package for that. You have to persuade
maintainer of that packages to put to provide there and bump release
just for that purpose.
(2) The feaure to use PyQt4 as a back-end is ability of pyface, not
ability of PyQt4. So you should declare the feature in pyface, not in
the PyQt4.
(3) What will happen when pyface looses ability to use PyQt4 as a
back-end? Will there keep stray Provides in the PyQt4 forever?
In short, interchangable requirement list (the OR in Requires) is more
natural and efficent way how to express it than putting yet another
layer of virtual provides.
-- Petr
11:14 a.m.
2013-05-23 00:30, Richard W.M. Jones skrev:
On Wed, May 22, 2013 at 03:52:22PM -0600, Orion Poplawski wrote:
> Something I'm just now running into - I have a package that can make
> use of one of two different backends, but it definitely needs one of
> them. I don't want to pick which one in the package. Also, it is
> explicitly referencing specific implementations, not a generic
> interface, so a generic Provides in the backend packages is not
> appropriate. But something like:
>
> Requires: ( pkgA || pkgB )
>
> might do the trick.
FWIW dpkg can express this kind of dependency, eg:
Depends: exim | mail-transport-agent
[from: http://www.debian.org/doc/debian-policy/ch-relationships.html]
If I remember this right, Debian's implementation uses the
short-circuiting OR operator. This makes it possible to express
dependencies with an order of preference. For instance, with the example
from above:
Depends: exim | mail-transport-agent
(note that exim has virtual 'mail-transport-agent' provides)
If no mail-transport-agent is installed, it drags in exim; otherwise it
is happy with any package that has virtual mail-transport-agent provides.
This is a case that is not possible to solely express with rpm's virtual
provides.
If implemented, this would have implications for Fedora. You could
have two people who have installed the same package, with a very
different set of dependent packages, leading to some sort of
combinatorial explosion of QA.
yum's depsolver already has some logic that resolves Provides
differently depending on what packages are already installed, which
makes it highly unpredictable. I don't think QA would get any harder if
we get the ability to explicitly express this in the spec files.
From http://yum.baseurl.org/wiki/CompareProviders :
1) Candidates from the same srpm as the requiring package are
preferred
2) Candidates with names that start the same as the requiring package
are favored
3a) Candidates with the same (or similar) arch as the requiring
package have more pull than those with a less similar arch
3b) Candidates with the same (or similar) arch as the system have
more pulls than those with a less similar arch
3c) Candidates with close relationships to the already installed
system (and transaction/requestor) have more pull than those that
would require more changes to the installed system
4) In the event of a tie, candidates with very long names will lose
the tie
5) In the event they are still tied, they are sorted alphabetically
and "highest" wins (zpkg is picked over apkg).
--
Kalev
3:53 a.m.
Hi,
I would like better integration with domain-specific package managers.
By which I mean npm (for node.js), gem (for ruby), pip (for python),
cpan (for perl), pecl/pear (for PHP), CRAN (for R), CTAN (for TeX), and
many more I'm sure.
By integrating RPM with these package managers, I feel it would be
possible to provide a consistent view of the system, as well as a
consistent management interface for sysadmins as opposed to application
developers. The latter I might expect to continue to use the domain
specific package managers, simply because they add value to domain
experts -- but for the common usecase "install this app on the server"
it would be nice to use RPM only.
Another advantage that I see is that it saves Fedora packager manpower
-- if the "translation" is good enough, it should be possible to work
with upstream packages and simply automate the fedora rpm process as
much as possible. Current examples are R2spec and the TeXLive package
scripts.
And maybe, in the future, it might even be possible for new domains to
rely on RPM itself instead of writing Yet-Another-Package-Manager etc.
But again, that is not the goal itself.
To succesfully integrate RPM implies (to me) at least the following,
some of which are already on the feature page:
- convert the basic package metadata from $domain to RPM (it is OK to
go back to $domain package manager for exotic use cases, but
list/update/remove packages should be doable from RPM)
- reproducible environment (Gemfile / pip requirements.txt / ...)
- both system- and per-user installation tracked and supported
- more I can't think of right now?
Regards,
--Stijn
7:02 a.m.
On 23. 5. 2013 at 10:53:10, Stijn Hoop wrote:
Hi,
I would like better integration with domain-specific package managers.
By which I mean npm (for node.js), gem (for ruby), pip (for python),
cpan (for perl), pecl/pear (for PHP), CRAN (for R), CTAN (for TeX), and
many more I'm sure.
The problem is that some of these languages have fundamentally different
philosophy than Fedora and unfortunatelly it's not a mix-and-match situation.
That being said, there already are different tools to create spec files from
those upstream representations (gem2spec, cpan2spec, ...)
By integrating RPM with these package managers, I feel it would be
possible to provide a consistent view of the system, as well as a
consistent management interface for sysadmins as opposed to application
developers. The latter I might expect to continue to use the domain
specific package managers, simply because they add value to domain
experts -- but for the common usecase "install this app on the server"
it would be nice to use RPM only.
Another advantage that I see is that it saves Fedora packager manpower
-- if the "translation" is good enough, it should be possible to work
with upstream packages and simply automate the fedora rpm process as
much as possible. Current examples are R2spec and the TeXLive package
scripts.
You can't really do this because of all the Fedora packaging policies. It
might be feasible in some private repositories though.
Thanks
Jan
8:30 a.m.
On Thu, 23 May 2013 14:02:34 +0200
Jan Zelený <jzeleny(a)redhat.com> wrote:
On 23. 5. 2013 at 10:53:10, Stijn Hoop wrote:
> I would like better integration with domain-specific package
> managers. By which I mean npm (for node.js), gem (for ruby), pip
> (for python), cpan (for perl), pecl/pear (for PHP), CRAN (for R),
> CTAN (for TeX), and many more I'm sure.
The problem is that some of these languages have fundamentally
different philosophy than Fedora and unfortunatelly it's not a
mix-and-match situation. That being said, there already are different
tools to create spec files from those upstream representations
(gem2spec, cpan2spec, ...)
Yes, it is true that there sometimes is a different philosophy, but
fundamentally is "in the eye of the beholder". If there are domain2spec
tools available NOW, why would it not be possible technically? And if
the non-technical philosophical differences are too big, maybe it is
also a sign that Fedora needs to change the requirements? After all, an
OS that does not help developers with development might not be a good
environment to keep.
> By integrating RPM with these package managers, I feel it would
be
> possible to provide a consistent view of the system, as well as a
> consistent management interface for sysadmins as opposed to
> application developers. The latter I might expect to continue to
> use the domain specific package managers, simply because they add
> value to domain experts -- but for the common usecase "install this
> app on the server" it would be nice to use RPM only.
>
> Another advantage that I see is that it saves Fedora packager
> manpower -- if the "translation" is good enough, it should be
> possible to work with upstream packages and simply automate the
> fedora rpm process as much as possible. Current examples are R2spec
> and the TeXLive package scripts.
You can't really do this because of all the Fedora packaging
policies. It might be feasible in some private repositories though.
I disagree. Why are the domain2spec and TeXLive approaches in the
repository then? I know that some domain2spec tools do need
hand-editing, but that is a problem that can be worked on. Either by
integrating more knowledge in the tools, or working with upstream on
providing more metadata for the package.
I do acknowledge that all of this will not be EASY. But then, I did not
see that in your list of requirements.
If it is really impossible, can you give an practical example why?
(Sidenote: I really do appreciate the call for features and the
considered feature page. Thank you for that!)
Regards,
--Stijn
8:59 a.m.
On 23. 5. 2013 at 15:30:55, Stijn Hoop wrote:
On Thu, 23 May 2013 14:02:34 +0200
Jan Zelený <jzeleny(a)redhat.com> wrote:
> On 23. 5. 2013 at 10:53:10, Stijn Hoop wrote:
> > I would like better integration with domain-specific package
> > managers. By which I mean npm (for node.js), gem (for ruby), pip
> > (for python), cpan (for perl), pecl/pear (for PHP), CRAN (for R),
> > CTAN (for TeX), and many more I'm sure.
>
> The problem is that some of these languages have fundamentally
> different philosophy than Fedora and unfortunatelly it's not a
> mix-and-match situation. That being said, there already are different
> tools to create spec files from those upstream representations
> (gem2spec, cpan2spec, ...)
Yes, it is true that there sometimes is a different philosophy, but
fundamentally is "in the eye of the beholder". If there are domain2spec
tools available NOW, why would it not be possible technically? And if
the non-technical philosophical differences are too big, maybe it is
also a sign that Fedora needs to change the requirements? After all, an
OS that does not help developers with development might not be a good
environment to keep.
I will give a simple example: Fedora - Ruby. Those are two completely different
worlds that will probably never synchronize their philosophies. And you can't
say which one should take a step back because they both have valid points in
their philosophies.
> > By integrating RPM with these package managers, I feel it
would be
> > possible to provide a consistent view of the system, as well as a
> > consistent management interface for sysadmins as opposed to
> > application developers. The latter I might expect to continue to
> > use the domain specific package managers, simply because they add
> > value to domain experts -- but for the common usecase "install this
> > app on the server" it would be nice to use RPM only.
> >
> > Another advantage that I see is that it saves Fedora packager
> > manpower -- if the "translation" is good enough, it should be
> > possible to work with upstream packages and simply automate the
> > fedora rpm process as much as possible. Current examples are R2spec
> > and the TeXLive package scripts.
>
> You can't really do this because of all the Fedora packaging
> policies. It might be feasible in some private repositories though.
I disagree. Why are the domain2spec and TeXLive approaches in the
repository then? I know that some domain2spec tools do need
hand-editing, but that is a problem that can be worked on. Either by
integrating more knowledge in the tools, or working with upstream on
providing more metadata for the package.
I do acknowledge that all of this will not be EASY. But then, I did not
see that in your list of requirements.
Well, technically it might be possible. It's just that I have strong doubts
that the domain2spec tools can be improved to the degree where they can handle
*any* domain-based package and produce rpm package that can be accepted to
Fedora. Again, with private repositories, this won't be a concern and we can
happily go for it, as there is no such thing as policy there ;-)
(Sidenote: I really do appreciate the call for features and the
considered feature page. Thank you for that!)
Don't thank me yet, we still don't have neither the roadmap nor the list of
accepted RFEs ;-)
Thanks
Jan
9:23 a.m.
Dne 23.5.2013 15:30, Stijn Hoop napsal(a):
On Thu, 23 May 2013 14:02:34 +0200
Jan Zelený <jzeleny(a)redhat.com> wrote:
> On 23. 5. 2013 at 10:53:10, Stijn Hoop wrote:
>> I would like better integration with domain-specific package
>> managers. By which I mean npm (for node.js), gem (for ruby), pip
>> (for python), cpan (for perl), pecl/pear (for PHP), CRAN (for R),
>> CTAN (for TeX), and many more I'm sure.
> The problem is that some of these languages have fundamentally
> different philosophy than Fedora and unfortunatelly it's not a
> mix-and-match situation. That being said, there already are different
> tools to create spec files from those upstream representations
> (gem2spec, cpan2spec, ...)
Yes, it is true that there sometimes is a different philosophy, but
fundamentally is "in the eye of the beholder". If there are domain2spec
tools available NOW, why would it not be possible technically? And if
the non-technical philosophical differences are too big, maybe it is
also a sign that Fedora needs to change the requirements? After all, an
OS that does not help developers with development might not be a good
environment to keep.
*It is not possible to convert the packages technically nor philosophically*
You might think million times that the sentence is not truth, but that
is as it is. I'll give you several examples:
* Gems cannot express dependencies on system libraries such sqlite3,
libxml2, etc.
* Gems does not undergoing legal review, i.e. you have to trust to the
author of the gem, that the license is correct, which sadly may not true.
* Bundling, quite common phenomenon.
Just this short list of issues should be enough. It might work for you,
when you decide to neglect all the issue mentioned above and probably
several else, but it does not work for distribution. Sorry.
Vít
9:29 a.m.
On Thu, May 23, 2013 at 4:23 PM, Vít Ondruch <vondruch(a)redhat.com> wrote:
*It is not possible to convert the packages technically nor
philosophically*
You might think million times that the sentence is not truth, but that is
as it is. I'll give you several examples:
* Gems cannot express dependencies on system libraries such sqlite3,
libxml2, etc.
Doesn't matter for the system administrator who has already installed the
gem.
* Gems does not undergoing legal review, i.e. you have to trust to
the
author of the gem, that the license is correct, which sadly may not true.
Doesn't matter for the system administrator who has already installed the
gem.
* Bundling, quite common phenomenon.
Doesn't matter for the system administrator who has already installed the
gem.
Just this short list of issues should be enough. It might work for you,
when you decide to neglect all the issue mentioned above and
probably
several else, but it does not work for distribution. Sorry.
I don't think this is necessarily targeted at changing how _Fedora_
distributes things; just giving system administrators a single command that
works on an installed system might be an useful improvement.
Mirek
9:41 a.m.
Dne 23.5.2013 16:29, Miloslav Trmač napsal(a):
On Thu, May 23, 2013 at 4:23 PM, Vít Ondruch <vondruch(a)redhat.com
<mailto:vondruch@redhat.com>> wrote:
*It is not possible to convert the packages technically nor
philosophically*
You might think million times that the sentence is not truth, but
that is as it is. I'll give you several examples:
* Gems cannot express dependencies on system libraries such
sqlite3, libxml2, etc.
Doesn't matter for the system administrator who has already installed
the gem.
* Gems does not undergoing legal review, i.e. you have to trust to
the author of the gem, that the license is correct, which sadly
may not true.
Doesn't matter for the system administrator who has already installed
the gem.
* Bundling, quite common phenomenon.
Doesn't matter for the system administrator who has already installed
the gem.
Just this short list of issues should be enough. It might work for
you, when you decide to neglect all the issue mentioned above and
probably several else, but it does not work for distribution. Sorry.
I don't think this is necessarily targeted at changing how _Fedora_
distributes things; just giving system administrators a single command
that works on an installed system might be an useful improvement.
Mirek
Ok, so speaking for gem2rpm, it might be made more dumber to include
automatically * and if no license is found, then put there "foo" for
example. Actually, everybody is free to do such a template for himself,
or even submit such patch upstream, but then the RPM kind of loosing its
purpose.
Vít
3:55 a.m.
Dne Čt 23. května 2013 16:41:04, Vít Ondruch napsal(a):
Dne 23.5.2013 16:29, Miloslav Trmač napsal(a):
> On Thu, May 23, 2013 at 4:23 PM, Vít Ondruch <vondruch(a)redhat.com
>
> <mailto:vondruch@redhat.com>> wrote:
> *It is not possible to convert the packages technically nor
> philosophically*
>
> You might think million times that the sentence is not truth, but
> that is as it is. I'll give you several examples:
>
> * Gems cannot express dependencies on system libraries such
> sqlite3, libxml2, etc.
>
> Doesn't matter for the system administrator who has already installed
> the gem.
>
> * Gems does not undergoing legal review, i.e. you have to trust to
> the author of the gem, that the license is correct, which sadly
> may not true.
>
> Doesn't matter for the system administrator who has already installed
> the gem.
>
> * Bundling, quite common phenomenon.
>
> Doesn't matter for the system administrator who has already installed
> the gem.
>
> Just this short list of issues should be enough. It might work for
> you, when you decide to neglect all the issue mentioned above and
> probably several else, but it does not work for distribution. Sorry.
>
> I don't think this is necessarily targeted at changing how _Fedora_
> distributes things; just giving system administrators a single command
> that works on an installed system might be an useful improvement.
>
> Mirek
Ok, so speaking for gem2rpm, it might be made more dumber to include
automatically * and if no license is found, then put there "foo" for
example. Actually, everybody is free to do such a template for himself,
or even submit such patch upstream, but then the RPM kind of loosing its
purpose.
Agreed, at least to a certain degree.
If we are talking just about a single point of access for users, perhaps we
should rather consider putting this in one of the layers above, sort of gnome
software center approach.
Thanks
Jan
9:34 a.m.
On 05/23/2013 03:30 PM, Stijn Hoop wrote:
On Thu, 23 May 2013 14:02:34 +0200
Jan Zelený <jzeleny(a)redhat.com> wrote:
> The problem is that some of these languages have fundamentally
> different philosophy than Fedora and unfortunatelly it's not a
> mix-and-match situation. That being said, there already are different
> tools to create spec files from those upstream representations
> (gem2spec, cpan2spec, ...)
Yes, it is true that there sometimes is a different philosophy, but
fundamentally is "in the eye of the beholder". If there are domain2spec
tools available NOW, why would it not be possible technically?
The main difference is that most other packaging systems consider it
perfectly acceptable to depend on specific version (or even a specific
build of a specific version) of package. Fedora generally only ships a
single version of every package. Reverse dependencies are expected to
be rebuilt against the most recent version. If that is not possible,
those reverse dependencies have to be ported over.
This is a fundamental cultural difference. The Fedora way requires more
work upfront, but in the long term, it scales much better, and it is the
only approach that can guarantee you can fix critical bugs for all users
who are potentially affected by them.
--
Florian Weimer / Red Hat Product Security Team
9:12 a.m.
On Thu, May 23, 2013 at 4:53 AM, Stijn Hoop <stijn(a)sandcat.nl> wrote:
Hi,
I would like better integration with domain-specific package managers.
By which I mean npm (for node.js), gem (for ruby), pip (for python),
cpan (for perl), pecl/pear (for PHP), CRAN (for R), CTAN (for TeX), and
many more I'm sure.
You left out "maven". And first off, those are not "yum" behaviors,
they're "rpm" or "rpmbuild" behaviors. .
Second, the second you start letting yum and RPM grab modules
automatically from any of those third party repositories, you are in
dependency and compatibility hell. To take an example, look at the
perl-HTML-Mason package. It's been evolving reasonably quickly and
bopping back and forth between distinct authors with distinct styles,
so the dependency lists keep changing and updating to very recent
versions of other modules. As you bring in more Perl dependencies,
automatically, via local cpan builds, you lose track of which RPM you
built and installed in what fashion and can wind up with dependencies
based on the latest builds from CPAN that *break* your existing
codebase, or even wind up upgrading your version of perl. (Been there
with "cpan build", done that, had one hell of a time cleaning up the
mess!)
I've done a lot of work with CPAN -> RPM building, and it's painful
(That's why I keep updated versions of "cpan2rpm" and "cpanspec"
up at
https://github.com/nkadel/)
By integrating RPM with these package managers, I feel it would be
possible to provide a consistent view of the system, as well as a
consistent management interface for sysadmins as opposed to application
developers. The latter I might expect to continue to use the domain
specific package managers, simply because they add value to domain
experts -- but for the common usecase "install this app on the server"
it would be nice to use RPM only.
It's possible to have some tools to help with such building, but
that's a package creation issue, not a package management issue. The
same issues exist
Another advantage that I see is that it saves Fedora packager
manpower
-- if the "translation" is good enough, it should be possible to work
with upstream packages and simply automate the fedora rpm process as
much as possible. Current examples are R2spec and the TeXLive package
scripts.
TexLive is well defined. CPAN, rubygems, Maven, php/pear, etc.? Not so
much. Rubygems is particularly scary because the gems *do not publish
compilable source* for any well defined build environment, they rely
extensively on pre-built modules. Maven has the same "I publish
binaries, not buildable source code" problem. CPAN is a nightmare:
the stylistic differences among the developers make consistent RPm
building a very, very manual process: TexLive is so well organized as
a monolithic, buildable source environment that it works well, I'll
admit, but but not the others.
6:14 a.m.
On 05/22/2013 03:43 PM, Jan Zelený wrote:
Please send your requests as replies to this email so they can be
properly
discussed.
Have equivalent of apt-get autoremove.
I.e. If you run
yum install foo
and it will install packages "bar" and "bra" for dependencies.
Then when I remove package "foo", those two packages will be left on my
system.
dnf autoremove
should tell me that packages "bar" and "bra" were installed as
dependencies for package, which is no more present on disk (and no other
package requires them) and can be removed.
--
Miroslav Suchy
Red Hat Systems Management Engineering
6:21 a.m.
Am 28.05.2013 13:14, schrieb Miroslav Suchý:
On 05/22/2013 03:43 PM, Jan Zelený wrote:
> Please send your requests as replies to this email so they can be properly
> discussed.
Have equivalent of apt-get autoremove.
I.e. If you run
yum install foo
and it will install packages "bar" and "bra" for dependencies.
Then when I remove package "foo", those two packages will be left on my system
not on mine, the only big question is why it is not default
cat /etc/yum.conf | grep clean
clean_requirements_on_remove=1
6:25 a.m.
On Tue, 2013-05-28 at 13:14 +0200, Miroslav Suchý wrote:
On 05/22/2013 03:43 PM, Jan Zelený wrote:
> Please send your requests as replies to this email so they can be properly
> discussed.
Have equivalent of apt-get autoremove.
That's what yum-plugin-remove-with-leaves does.
--
Mathieu
7:08 a.m.
On 05/28/2013 06:25 AM, Mathieu Bridon wrote:
On Tue, 2013-05-28 at 13:14 +0200, Miroslav Suchý wrote:
> On 05/22/2013 03:43 PM, Jan Zelený wrote:
>> Please send your requests as replies to this email so they can be properly
>> discussed.
>
> Have equivalent of apt-get autoremove.
That's what yum-plugin-remove-with-leaves does.
Almost, but not quite (it must be done when you remove the package,
can't be done after).
The clean_requirements_on_remove=1 setting is also helpful, like
'autoremove' after each 'remove'. Still can't be done after-the-fact,
but is nice (and more accurate, AFAIK, since it uses the yumdb 'reason'
tag).
8:27 a.m.
On 05/28/2013 01:14 PM, Miroslav Suchý wrote:
dnf autoremove
should tell me that packages "bar" and "bra" were installed as
dependencies for package, which is no more present on disk (and no other
package requires them) and can be removed.
There's an RFE for this already:
https://bugzilla.redhat.com/show_bug.cgi?id=963345
Ales
8:33 a.m.
This is basically the major impediment to the "uninstall" of a product that
consists of several packages. Some of these packages may be, at time of uninstall, also
required by other packages, so the "and no other package requires them" part is
essential.
--Fernando
----- Original Message -----
From: "Ales Kozumplik" <akozumpl(a)redhat.com>
To: "Development discussions related to Fedora"
<devel(a)lists.fedoraproject.org>
Sent: Tuesday, May 28, 2013 9:27:23 AM
Subject: Re: Software Management call for RFEs
On 05/28/2013 01:14 PM, Miroslav Suchý wrote:
> dnf autoremove
> should tell me that packages "bar" and "bra" were installed as
> dependencies for package, which is no more present on disk (and no
> other
> package requires them) and can be removed.
There's an RFE for this already:
https://bugzilla.redhat.com/show_bug.cgi?id=963345
Ales
--
devel mailing list
devel(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
1:32 p.m.
Dne Út 28. května 2013 09:33:11, Fernando Nasser napsal(a):
This is basically the major impediment to the "uninstall"
of a product that
consists of several packages. Some of these packages may be, at time of
uninstall, also required by other packages, so the "and no other package
requires them" part is essential.
yum repo-pkgs should handle the other situation (other packages require them)
as well.
Thanks
Jan
4:52 a.m.
On Wed, May 22, 2013 at 3:43 PM, Jan Zelený <jzeleny(a)redhat.com> wrote:
Dear Fedora community,
several months ago, at the Developer conference in Brno, Software Management
team received a whole bunch of proposals for new functionality in RPM and
related software stack.
We acknowledge the need for some changes in Software Management stack in
Fedora but we don't want to make changes just by guessing what our
users want. Therefore I call to you, consumers of our products (dnf, yum and
rpm): what are the changes that you would like to see in the foreseeable
future (say 2-3 years) and why would you like to see them (what would they
help you with)?
There is already a list of some RFEs on rpm.org wiki, you can use it as an
inspiration, to see what RFEs we have already received:
http://rpm.org/wiki/FeaturePlanning
The only limitation for your requests is our manifest which defines the scope
of SW management stack for the future. It is attached to this email (note that
it's quite extensive but the first part should give you a good image of what is
the planned scope of SW management stack).
Please send your requests as replies to this email so they can be properly
discussed.
How about improving delta rpm performance? Currently we save download
time but require a lot of time to rebuild the rpms.
Can we just sign the deltas and then don't compress the generated
rpms? We waste time and cycles building xz
compressed rpms (from the deltas) just to decompress them a few
minutes later. Skipping this (just create uncompressed rpms when
building from deltas) should improve performance a lot.
9:13 a.m.
drago01 (drago01(a)gmail.com) said:
How about improving delta rpm performance? Currently we save
download
time but require a lot of time to rebuild the rpms.
Can we just sign the deltas and then don't compress the generated
rpms? We waste time and cycles building xz
compressed rpms (from the deltas) just to decompress them a few
minutes later. Skipping this (just create uncompressed rpms when
building from deltas) should improve performance a lot.
The entire delta framework is built around generating the original
RPM and handing that off wholesale to yum/rpm - it's how it can be
done as a yum plugin without changing anything in RPM itself. Changing
it to create something *different* than the full RPM (in terms of
checksum, size, signature, and so on) would be a non-trivial change
to how they work.
I think if you want to go down this road, you probably want to start
by integrating delta technology into RPM core itself, probably
redefining the delta format into something much more efficient.
Bill
9:29 a.m.
Am 30.05.2013 16:13, schrieb Bill Nottingham:
drago01 (drago01(a)gmail.com) said:
> How about improving delta rpm performance? Currently we save download
> time but require a lot of time to rebuild the rpms.
> Can we just sign the deltas and then don't compress the generated
> rpms? We waste time and cycles building xz
> compressed rpms (from the deltas) just to decompress them a few
> minutes later. Skipping this (just create uncompressed rpms when
> building from deltas) should improve performance a lot.
The entire delta framework is built around generating the original
RPM and handing that off wholesale to yum/rpm - it's how it can be
done as a yum plugin without changing anything in RPM itself. Changing
it to create something *different* than the full RPM (in terms of
checksum, size, signature, and so on) would be a non-trivial change
to how they work.
and it would break infrastructure like below where the one admin-machine
get any update, benefits from delta-rpms and deploy the updates later
to all other internal machines - you can calculate how much traffic
is saved on both sides (mine and fedora infrastructure) with this fro
more than 20 machines
[root@buildserver:~]$ cat /buildserver/repo-cache.sh
#!/usr/bin/bash
basearch=`uname -i`
releasever=`rpm -q --qf "%{version}\n" fedora-release`
# Alle Subfolder unter "/var/cache/yum" durchlaufen und RPM-Pakete
# in das eigene Repo verschieben
for g in `ls -1b /var/cache/yum`
do
if [ -d /var/cache/yum/$g/packages ]
then
echo "/var/cache/yum/$g/packages/ > /repo/cache/fc$releasever/"
sudo mv --verbose /var/cache/yum/$g/packages/*.rpm /repo/cache/fc$releasever/ 2>
/dev/null
fi
done
/buildserver/repo-create.sh
10:06 a.m.
On Thu, May 30, 2013 at 4:13 PM, Bill Nottingham <notting(a)redhat.com> wrote:
drago01 (drago01(a)gmail.com) said:
> How about improving delta rpm performance? Currently we save download
> time but require a lot of time to rebuild the rpms.
> Can we just sign the deltas and then don't compress the generated
> rpms? We waste time and cycles building xz
> compressed rpms (from the deltas) just to decompress them a few
> minutes later. Skipping this (just create uncompressed rpms when
> building from deltas) should improve performance a lot.
The entire delta framework is built around generating the original
RPM and handing that off wholesale to yum/rpm - it's how it can be
done as a yum plugin without changing anything in RPM itself. Changing
it to create something *different* than the full RPM (in terms of
checksum, size, signature, and so on) would be a non-trivial change
to how they work.
Yeah but the goal is to have the same files installed on disk, not really
a 1:1 copy of the rpm file (doing this just causes additional work).
I think if you want to go down this road, you probably want to start
by integrating delta technology into RPM core itself, probably
redefining the delta format into something much more efficient.
Yeah that makes sense.
7:43 a.m.
I'd like a tool similar to portaudit in FreeBSD or debscan in Debian. This
tool should list all packages which have a security issue. Currently there
is yum-security-plugin but it lists packages only if an update is
available. The new tool would list vulnerable packages even if no update is
available yet, so that the user can take precaution.
2:38 a.m.
On 2. 6. 2013 at 14:43:15, enclair wrote:
I'd like a tool similar to portaudit in FreeBSD or debscan in
Debian. This
tool should list all packages which have a security issue. Currently there
is yum-security-plugin but it lists packages only if an update is
available. The new tool would list vulnerable packages even if no update is
available yet, so that the user can take precaution.
Are you sure this is within the scope of Software management? I'd say that the
line is exactly between the existing and requested solution. The existing
solution is within our scope (there is a security update) while the requested
is more of an independent tool taking information from any place there is for
this sort of information (there has been a security risk reported at XXX, take
a look).
Thanks
Jan
2:58 a.m.
On 06/02/2013 02:43 PM, enclair wrote:
I'd like a tool similar to portaudit in FreeBSD or debscan in
Debian.
This tool should list all packages which have a security issue.
I don't know about portaudit, but debsecan works completely out of the
usual software management stack. Part of the reason for that is that
you even get reports if you haven't configured the security archive
properly (so that the package manager won't notice that there are
updates available). The real work is in the backend and the data
collection; debsecan is a short Python script which just runs a few
version comparisons. In Debian's case, this isn't fully integrated with
the repository management, either, which is mostly due to historical
accident and not deliberate design.
But the key point is that this is not a question of software. It's all
about the data that describes vulnerabilities and fixed packages, and
this is currently not available for Fedora in consistent,
machine-readable form.
--
Florian Weimer / Red Hat Product Security Team
3:46 a.m.
On Sun, Jun 02, 2013 at 02:43:15PM +0200, enclair wrote:
I'd like a tool similar to portaudit in FreeBSD or debscan in
Debian. This
tool should list all packages which have a security issue. Currently there
is yum-security-plugin but it lists packages only if an update is
available. The new tool would list vulnerable packages even if no update is
available yet, so that the user can take precaution.
You probably want SCAP (specifically OpenSCAP). It's the proper
solution for all of this.
However the real issue is going to be that the OpenSCAP vulnerability
databases for Fedora are not maintained.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
11:44 a.m.
On Wed, May 22, 2013 at 03:43:32PM +0200, Jan Zelený wrote:
Please send your requests as replies to this email so they can be
properly
discussed.
I know I'm a little slow on this, and I hope I didn't miss a duplicate in
the big thread, but here's mine:
Better handling of documentation and languages which aren't installed
initially. We often get requests to remove these things from the cloud
image (see for example http://groveronline.com/2013/06/fedora-for-servers/,
or look what oVirt node does).
This would save a considerable amount of space -- in the cloud images, docs
and man pages are 5% of the installed footprint and locale is 10%. That's a
pretty big win for one thing. We have been reluctant to use the big RPM flag
hammers to prevent these from being installed, though, because we want the
core to be a minimal _building block_, and there's no graceful way to add
those things back if they're initially not selected. So, handling this
better would really help.
One approach for docs might be to automatically separate %doc files into a
-doc subpackage, and have a yum option which automatically installs those
subpackages when the main package is installed. That's a bit ugly but
wouldn't take a lot of changes. Or something like deltarpm could be used to
add in missing doc and lang files if requested. But I'm not really wedded to
any solution; just offering up a problem we need solved.
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm(a)fedoraproject.org>
3975
days inactive
3990
days old
177 comments
54 participants
participants (54)
-
Adam Williamson -
Alek Paunov -
Ales Kozumplik -
Bill Nottingham -
Björn Persson -
Christophe Fergeau -
Christopher Meng -
Colin Walters -
Dan Fruehauf -
David Malcolm -
David Tardon -
drago01 -
enclair -
Fernando Nasser -
Florian Weimer -
James Antill -
Jan Zelený -
John Reiser -
John.Florian＠dart.biz -
Jonathan Dieter -
Kalev Lember -
Kevin Fenzi -
Lars Seipel -
Mathieu Bridon -
Matthew Miller -
Michael Ekstrand -
Michael Scherer -
Michael Schroeder -
Michal Schmidt -
Mike Pinkerton -
Miloslav Trmač -
Miroslav Suchý -
Nico Kadel-Garcia -
Nicolas Mailhot -
Orion Poplawski -
Oron Peled -
Panu Matilainen -
Paul Flo Williams -
Paulo César Pereira de Andrade -
Petr Pisar -
Phil Knirsch -
Przemek Klosowski -
Rahul Sundaram -
Ralf Corsepius -
Ravindra Kumar -
Reindl Harald -
Richard W.M. Jones -
Sandro Mani -
Seth Vidal -
Simone Caronni -
Stijn Hoop -
Till Maas -
Vít Ondruch -
Zdenek Pavlas