Proposal to limit file deps

[Toshio Kuratomi]

Hey all, I've just added a draft proposal for limiting file dependencies
in packages.  It's a "should" recommendation at this point.

http://www.fedoraproject.org/wiki/PackagingDrafts/FileDeps

= File Deps Guideline =

== Problem ==

Resolving file dependencies can be a bottleneck when running yum.  With
our current repodata format, information on which package owns files
in /etc and the bin directories (/bin /sbin /usr/bin /usr/sbin) are kept
in primary.xml.  All other file deps are kept in filelist.xml.  This
means that anytime yum has to resolve a file dep in /etc or a bin
directory it can just download primary.xml to resolve it.  If the file
dep is outside of those directories, yum has to download and parse a
second xml file, filelist.xml, which is much larger than primary.xml.

Fedora 6 repodata compressed and uncompressed:
||       ||primary.xml||filelist.xml||
||Core   ||1M/7.6M    ||3.1M/39.2M||
||Extras ||4.7M/11.7M ||4M/50.8M||
||Updates||0.4M/4.2M  ||2.1M/27.5M||

The compressed size has direct bearing on the additional time it will
take to download the additional filelist.xml.  The uncompressed size is
one indicator of how much additional time is needed to parse for the
file that satisfies the dependency.  If yum does not need to refresh its
primary.xml and filelist.xml (because it downloaded a copy previously
and the repository has not been updated since), yum won't have to parse
a new file and the filelist info will be quicker to find (as the file
will have been parsed into an sqlite database), however this is the
exception rather than the rule.

If we can avoid having to download and parse filelist.xml, we can
greatly speed up yum's dependency resolution times.

== Proposed Guideline ==

'''SHOULD''': If the package has file dependencies outside
of /etc, /bin, /sbin, /usr/bin, or /usr/sbin consider requiring the
package which provides the file instead of the file itself.

Using file dependencies outside of /etc, /bin, /sbin, /usr/bin,
or /usr/sbin requires yum (and other depsolvers using the repomd format)
to download and parse a large xml file looking for the dependency.
Helping the depsolvers avoid this processing by depending on the package
instead of the file saves our end users a lot of time.

-Toshio

[Panu Matilainen]

On Wed, 20 Dec 2006, Toshio Kuratomi wrote:
>  '''SHOULD''': If the package has file
> dependencies outside
>  of /etc, /bin, /sbin, /usr/bin, or /usr/sbin consider requiring the
>  package which provides the file instead of the file itself.
>
>  Using file dependencies outside of /etc, /bin, /sbin, /usr/bin,
>  or /usr/sbin requires yum (and other depsolvers using the repomd format)
>  to download and parse a large xml file looking for the dependency.
>  Helping the depsolvers avoid this processing by depending on the package
>  instead of the file saves our end users a lot of time. 
Just FWIW, this doesn't actually help smart and apt at all, unless file 
dependencies outside what's recorded in primary.xml are outright banned in 
the repodata specification itself by refusing to add dependencies to 
other paths than what's in primary.xml. Yum is the only one of the three 
doing dependency resolution "on demand", both apt and smart precalculate 
the full dependency tree for all operations so they need all the
information at all times.

Note that I've absolutely nothing against this proposal, this helps yum 
(meaning most Fedora users) without hurting the others so nothing wrong 
with it. File dependencies are a mixed blessing...

 	- Panu -

[Axel Thimm]

On Thu, Dec 21, 2006 at 12:59:37PM +0200, Panu Matilainen wrote:
>  On Wed, 20 Dec 2006, Toshio Kuratomi wrote:
>  >'''SHOULD''': If the package has file dependencies outside
>  >of /etc, /bin, /sbin, /usr/bin, or /usr/sbin consider requiring the
>  >package which provides the file instead of the file itself.
>  >
>  >Using file dependencies outside of /etc, /bin, /sbin, /usr/bin,
>  >or /usr/sbin requires yum (and other depsolvers using the repomd format)
>  >to download and parse a large xml file looking for the dependency.
>  >Helping the depsolvers avoid this processing by depending on the package
>  >instead of the file saves our end users a lot of time.
>  
>  Just FWIW, this doesn't actually help smart and apt at all, 
Why? If unneccessary file dependencies are removed the files sizes get
smaller. Of course, yum has the larger benefits, but apt/smart/etc
would benefit from say cutting off half of the file dependencies, too.

The only problem I see is to identify when some file dependencies were
used "unneccessarily". Using file dependencies was never really
preferred over proper package depedencies, so most likely the packager
really wanted file dependencies when he explcitely used them.

That means w/o a proper analysis of what we currently have, we
shouldn't simply ban them. Instead an analysis would show why
packagers used this method and whether we can offer better workarounds
for what they wanted to achieve.
-- 
Axel.Thimm at ATrpms.net

[Axel Thimm]

On Wed, Dec 20, 2006 at 08:55:25PM -0800, Toshio Kuratomi wrote:
>  Hey all, I've just added a draft proposal for limiting file
> dependencies
>  in packages.  It's a "should" recommendation at this point.
>  
>  http://www.fedoraproject.org/wiki/PackagingDrafts/FileDeps
>  
>  = File Deps Guideline =
>  
>  == Problem == 
How often does this problem really occur? I just checked core's x86_64
packages and there are exactly 12 files in such dependencies
affecting 18 packages out of 2422 (that's 0.7%):

  cman-kernel dlm-kernel gamin-python gdm GFS-kernel gnbd-kernel httpd
  ImageMagick-perl inn libvirt-python libxml2-python libxslt-python
  openldap-servers openmotif openssh-server python-lcms redhat-lsb
  vsftpd

That's 0.7%. I thought that the problem was much larger. I either
miscalculated the number of file dependencies or there is no problem :)

(At least not one that has to be solved by creating new guidelines,
most of these packages like lsb or kernel modules seem to need special
handling anyway)
-- 
Axel.Thimm at ATrpms.net

[Axel Thimm]

On Thu, Dec 21, 2006 at 02:10:18PM +0100, Axel Thimm wrote:
>  On Wed, Dec 20, 2006 at 08:55:25PM -0800, Toshio Kuratomi wrote:
>  > Hey all, I've just added a draft proposal for limiting file dependencies
>  > in packages.  It's a "should" recommendation at this point.
>  > 
>  > http://www.fedoraproject.org/wiki/PackagingDrafts/FileDeps
>  > 
>  > = File Deps Guideline =
>  > 
>  > == Problem ==
>  
>  How often does this problem really occur? I just checked core's x86_64
>  packages and there are exactly 12 files in such dependencies
>  affecting 18 packages out of 2422 (that's 0.7%):
>  
>    cman-kernel dlm-kernel gamin-python gdm GFS-kernel gnbd-kernel httpd
>    ImageMagick-perl inn libvirt-python libxml2-python libxslt-python
>    openldap-servers openmotif openssh-server python-lcms redhat-lsb
>    vsftpd
>  
>  That's 0.7%. I thought that the problem was much larger. I either
>  miscalculated the number of file dependencies or there is no problem :) 
And the same metrics for extras: 26 files required by 27 packages out
of 3660 packages:

   chess clamav-devel ctapi-cyberjack cvs2cl em8300-devel freefont
   ftnchek-emacs gnash-plugin libibverbs-devel mimetex
   mozilla-opensc-signer openbox openct pcsc-perl perl-Image-Info pyxdg
   SoQt-devel tuxpaint util-vserver util-vserver-legacy util-vserver-sysv
   uuid-devel uuid-pgsql uuid-php w3c-markup-validator-libs
   x11-ssh-askpass xmms

That's again 0.7%, so the combined Fedora is at 0.7%, too.

Still hoping my calculations aren't flawed ...

>  (At least not one that has to be solved by creating new guidelines,
>  most of these packages like lsb or kernel modules seem to need special
>  handling anyway) 
-- 
Axel.Thimm at ATrpms.net

[Thorsten Leemhuis]

On 21.12.2006 14:24, Axel Thimm wrote:
>  On Thu, Dec 21, 2006 at 02:10:18PM +0100, Axel Thimm wrote:
> > On Wed, Dec 20, 2006 at 08:55:25PM -0800, Toshio Kuratomi wrote:
> >> Hey all, I've just added a draft proposal for limiting file dependencies
> >> in packages.  It's a "should" recommendation at this point.
> >>
> >> http://www.fedoraproject.org/wiki/PackagingDrafts/FileDeps
> >>
> >> = File Deps Guideline =
> >>
> >> == Problem ==
> > How often does this problem really occur? I just checked core's x86_64
> > packages and there are exactly 12 files in such dependencies
> > affecting 18 packages out of 2422 (that's 0.7%):
> >
> >   cman-kernel dlm-kernel gamin-python gdm GFS-kernel gnbd-kernel httpd
> >   ImageMagick-perl inn libvirt-python libxml2-python libxslt-python
> >   openldap-servers openmotif openssh-server python-lcms redhat-lsb
> >   vsftpd
> >
> > That's 0.7%. I thought that the problem was much larger. I either
> > miscalculated the number of file dependencies or there is no problem :)
>  
>  And the same metrics for extras: 26 files required by 27 packages out
>  of 3660 packages:
>  
>     chess clamav-devel ctapi-cyberjack cvs2cl em8300-devel freefont
>     ftnchek-emacs gnash-plugin libibverbs-devel mimetex
>     mozilla-opensc-signer openbox openct pcsc-perl perl-Image-Info pyxdg
>     SoQt-devel tuxpaint util-vserver util-vserver-legacy util-vserver-sysv
>     uuid-devel uuid-pgsql uuid-php w3c-markup-validator-libs
>     x11-ssh-askpass xmms
>  
>  That's again 0.7%, so the combined Fedora is at 0.7%, too. 
Well, my initial calculations found 32 file based deps outside of "/etc 
{/usr,}/{s,}bin/" in Extras 6 x64 ;-) Just to note, I mentioned that 
when I started the whole mess after skvidal poked me about it. See 
https://www.redhat.com/archives/fedora-packaging/2006-December/msg00077.html

CU
thl

[Panu Matilainen]

On Thu, 21 Dec 2006, Axel Thimm wrote:

>  On Thu, Dec 21, 2006 at 12:59:37PM +0200, Panu Matilainen wrote:
> > On Wed, 20 Dec 2006, Toshio Kuratomi wrote:
> >> '''SHOULD''': If the package has file dependencies
> outside
> >> of /etc, /bin, /sbin, /usr/bin, or /usr/sbin consider requiring the
> >> package which provides the file instead of the file itself.
> >>
> >> Using file dependencies outside of /etc, /bin, /sbin, /usr/bin,
> >> or /usr/sbin requires yum (and other depsolvers using the repomd format)
> >> to download and parse a large xml file looking for the dependency.
> >> Helping the depsolvers avoid this processing by depending on the package
> >> instead of the file saves our end users a lot of time.
> >
> > Just FWIW, this doesn't actually help smart and apt at all,
>
>  Why? If unneccessary file dependencies are removed the files sizes get
>  smaller. Of course, yum has the larger benefits, but apt/smart/etc
>  would benefit from say cutting off half of the file dependencies, too. 
Looking up a file dependency is just as cheap as looking up a package 
dependency in apt and smart (AFAIK, haven't studien smart in too much 
detail), once the initial crunching through the available files is done.

Throwing out *unnecessary* dependencies (file or package) is obviously 
good, even if it's hardly noticeable in the filesizes in reality. This 
simply doesn't change the fact that apt and smart need to download the 
filelists.xml along with primary.xml at all times to guarantee working 
with any old repomd repository.

 	- Panu -

[Andrew Overholt]

On Thu, 2006-21-12 at 13:46 +0100, Axel Thimm wrote:
>  Instead an analysis would show why
>  packagers used this method and whether we can offer better workarounds
>  for what they wanted to achieve. 
In the Eclipse SDK package set, we recently went through the
multilib-ification process.  We had to add a few file-level dependencies
to get this to work.  This was due to the inability to specify %{arch}
in a Requires.  How would this have been better accomplished?

Ben can probably explain this better than I can.

Andrew

[Tom Callaway]

On Thu, 2006-12-21 at 10:10 -0500, Andrew Overholt wrote:
>  On Thu, 2006-21-12 at 13:46 +0100, Axel Thimm wrote:
>  > Instead an analysis would show why
>  > packagers used this method and whether we can offer better workarounds
>  > for what they wanted to achieve.
>  
>  In the Eclipse SDK package set, we recently went through the
>  multilib-ification process.  We had to add a few file-level dependencies
>  to get this to work.  This was due to the inability to specify %{arch}
>  in a Requires.  How would this have been better accomplished?
>  
>  Ben can probably explain this better than I can. 
Honestly, this is the only valid case for file-level dependencies I can
think of, and unfortunately, the only way to work around it is to fix
rpm to permit %{arch} specific Requires.

~spot

[Axel Thimm]

On Thu, Dec 21, 2006 at 04:25:10PM +0200, Panu Matilainen wrote:
>  This simply doesn't change the fact that apt and smart need to
>  download the filelists.xml along with primary.xml at all times to
>  guarantee working with any old repomd repository. 
What is an "old" and "new" repomd?
-- 
Axel.Thimm at ATrpms.net

[Fernando Nasser]

I have two questions:

1) If this is a problem that only affects yum and all the other 
depsolvers do it efficiently, would it be better to fix yum instead or 
working around it by spec file changes?

2) If we still use file deps for things in /bin /sbin /usr/bin 
/usr/sbin, which happens for all GCJ compiled packages, wouldn't it 
force the reading of the second file anyways?

regards,
Fernando

Panu Matilainen wrote:
>  On Thu, 21 Dec 2006, Axel Thimm wrote:
>  
> > On Thu, Dec 21, 2006 at 12:59:37PM +0200, Panu Matilainen wrote:
> >> On Wed, 20 Dec 2006, Toshio Kuratomi wrote:
> >>> '''SHOULD''': If the package has file dependencies
> outside
> >>> of /etc, /bin, /sbin, /usr/bin, or /usr/sbin consider requiring the
> >>> package which provides the file instead of the file itself.
> >>>
> >>> Using file dependencies outside of /etc, /bin, /sbin, /usr/bin,
> >>> or /usr/sbin requires yum (and other depsolvers using the repomd 
> >>> format)
> >>> to download and parse a large xml file looking for the dependency.
> >>> Helping the depsolvers avoid this processing by depending on the 
> >>> package
> >>> instead of the file saves our end users a lot of time.
> >>
> >> Just FWIW, this doesn't actually help smart and apt at all,
> >
> > Why? If unneccessary file dependencies are removed the files sizes get
> > smaller. Of course, yum has the larger benefits, but apt/smart/etc
> > would benefit from say cutting off half of the file dependencies, too.
>  
>  Looking up a file dependency is just as cheap as looking up a package 
>  dependency in apt and smart (AFAIK, haven't studien smart in too much 
>  detail), once the initial crunching through the available files is done.
>  
>  Throwing out *unnecessary* dependencies (file or package) is obviously 
>  good, even if it's hardly noticeable in the filesizes in reality. This 
>  simply doesn't change the fact that apt and smart need to download the 
>  filelists.xml along with primary.xml at all times to guarantee working 
>  with any old repomd repository.
>  
>      - Panu -
>  
>  -- 
>  Fedora-packaging mailing list
>  Fedora-packaging(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-packaging
>

[Rex Dieter]

Fernando Nasser wrote:
>  I have two questions:
>  
>  1) If this is a problem that only affects yum and all the other 
>  depsolvers do it efficiently, would it be better to fix yum instead or 
>  working around it by spec file changes? 
It's still generally a good idea to minimize file deps.

>  2) If we still use file deps for things in /bin /sbin /usr/bin 
>  /usr/sbin, which happens for all GCJ compiled packages, wouldn't it 
>  force the reading of the second file anyways? 
No, just for items *outside* of those locations does yum need extra 
processing.  That's the whole point of this exercise.

-- Rex

[Axel Thimm]

On Thu, Dec 21, 2006 at 11:39:38AM -0500, Fernando Nasser wrote:
>  1) If this is a problem that only affects yum and all the other 
>  depsolvers do it efficiently, would it be better to fix yum instead or 
>  working around it by spec file changes? 
No, it's the other way round: yum is more efficient in downloading
because it most often does not need to download everything, while the
other depsolvers always download the bist in advance. It's a bit
oversimplified, but that servers for a first picture.

The discussion is about whether some packages force yum to go download
and drop its bandwidth performance to that of the other depsolvers,
but that's just 0.7% of all packages, so not really worth attacking at
all IMHO.
-- 
Axel.Thimm at ATrpms.net

[Axel Thimm]

On Thu, Dec 21, 2006 at 02:43:31PM +0100, Thorsten Leemhuis wrote:
>  On 21.12.2006 14:24, Axel Thimm wrote:
>  >On Thu, Dec 21, 2006 at 02:10:18PM +0100, Axel Thimm wrote:
>  >>On Wed, Dec 20, 2006 at 08:55:25PM -0800, Toshio Kuratomi wrote:
>  >>>Hey all, I've just added a draft proposal for limiting file dependencies
>  >>>in packages.  It's a "should" recommendation at this point.
>  >>>
>  >>>http://www.fedoraproject.org/wiki/PackagingDrafts/FileDeps
>  >>>
>  >>>= File Deps Guideline =
>  >>>
>  >>>== Problem ==
>  >>How often does this problem really occur? I just checked core's x86_64
>  >>packages and there are exactly 12 files in such dependencies
>  >>affecting 18 packages out of 2422 (that's 0.7%): 
>  >And the same metrics for extras: 26 files required by 27 packages
> out
>  >of 3660 packages: 
>  >That's again 0.7%, so the combined Fedora is at 0.7%, too.
>  
>  Well, my initial calculations found 32 file based deps outside of "/etc 
>  {/usr,}/{s,}bin/" in Extras 6 x64 ;-) Just to note, I mentioned that 
>  when I started the whole mess after skvidal poked me about it. See 
>  https://www.redhat.com/archives/fedora-packaging/2006-December/msg00077.html 
But then it was known that this is negligible, why does it escalate
to become even a proposal for guidelines? Do we care that yum fetches
the filelists every 140 package updates?
-- 
Axel.Thimm at ATrpms.net

[Fernando Nasser]

Rex Dieter wrote:
>  Fernando Nasser wrote:
> > I have two questions:
> >
> > 1) If this is a problem that only affects yum and all the other 
> > depsolvers do it efficiently, would it be better to fix yum instead or 
> > working around it by spec file changes?
>  
>  It's still generally a good idea to minimize file deps.
>   
Sure, but there are cases where it is desirable.  I am afraid a dull 
guideline will put a straight jacket on developers.

For instance, I know a couple of folks working on a software that in the 
current release needs two independent packages.  Another developer needs 
one (a single one) file from them, but the file may change places.  They 
thought of adding a virtual provides, but it seems overkill in this case 
and it is probably only temporary (which would mean we would use a name 
unnecessarily and regret later). So the best way around it was to 
require the file, at least until the set of packages stabilize on a 
future version of the dependency.

Perhaps a "recommendation" guideline?  (if there is such a thing)

> > 2) If we still use file deps for things in /bin /sbin /usr/bin 
> > /usr/sbin, which happens for all GCJ compiled packages, wouldn't it 
> > force the reading of the second file anyways?
>  
>  No, just for items *outside* of those locations does yum need extra 
>  processing.  That's the whole point of this exercise.
>   
Ah, thanks for the clarification.  The use of file deps prevents an 
optimization to kick in.

Is this list complete: /bin /sbin /usr/bin /usr/sbin ?
Or there are others that do not cause the loading of the second file?


Regards,
Fernando

[Fernando Nasser]

Axel Thimm wrote:
>  On Thu, Dec 21, 2006 at 11:39:38AM -0500, Fernando Nasser wrote:
> > 1) If this is a problem that only affects yum and all the other 
> > depsolvers do it efficiently, would it be better to fix yum instead or 
> > working around it by spec file changes?
>  
>  No, it's the other way round: yum is more efficient in downloading
>  because it most often does not need to download everything, while the
>  other depsolvers always download the bist in advance. It's a bit
>  oversimplified, but that servers for a first picture.
>   
Thanks for the clarification.  The use of file deps prevents an 
optimization to kick in.

[Toshio Kuratomi]

On Thu, 2006-12-21 at 12:13 -0500, Fernando Nasser wrote:
>  Rex Dieter wrote:
>  > Fernando Nasser wrote:
>  >> I have two questions:
>  >>
>  >> 1) If this is a problem that only affects yum and all the other 
>  >> depsolvers do it efficiently, would it be better to fix yum instead or 
>  >> working around it by spec file changes?
>  > 
>  > It's still generally a good idea to minimize file deps.
>  > 
>   
Actually, what's happening is that all depsolvers have the ability to
optimize two time consuming steps if the file deps follow these
guidelines.  The two steps are 1) downloading filelist.xml.gz and 2)
parsing filelist.xml to find the package which provides the file deps.
[And this problem is multiplied the more repositories you have defined
as the depsolver will have to grab the filelist.xml file from multiple
repositories].

Panu points out that only yum currently optimizes this but there's
nothing stopping the others doing it as well.

>  Sure, but there are cases where it is desirable.  I am afraid a dull
>
>  guideline will put a straight jacket on developers.
>  
>  For instance, I know a couple of folks working on a software that in the 
>  current release needs two independent packages.  Another developer needs 
>  one (a single one) file from them, but the file may change places.  They 
>  thought of adding a virtual provides, but it seems overkill in this case 
>  and it is probably only temporary (which would mean we would use a name 
>  unnecessarily and regret later). So the best way around it was to 
>  require the file, at least until the set of packages stabilize on a 
>  future version of the dependency.
>  
>  Perhaps a "recommendation" guideline?  (if there is such a thing)
>   
Yes.  This proposal is for a "Should" rather than a "Must"
guideline.
In the case you outline, I'd like to see the package depend on the file
until it was settled which package would provide it.  Then the
dependency could be converted to a package dep instead.  We don't
currently have an ongoing QA requirement, though, so remembering to make
the change would largely be the responsibility of the packager.

>  >> 2) If we still use file deps for things in /bin /sbin
> /usr/bin 
>  >> /usr/sbin, which happens for all GCJ compiled packages, wouldn't it 
>  >> force the reading of the second file anyways?
>  > 
>  > No, just for items *outside* of those locations does yum need extra 
>  > processing.  That's the whole point of this exercise.
>  > 
>  
>  Ah, thanks for the clarification.  The use of file deps prevents an 
>  optimization to kick in.
>   
Correct.

>  Is this list complete: /bin /sbin /usr/bin /usr/sbin ?
>  Or there are others that do not cause the loading of the second file? 
/etc is the other one.

That's why the guideline applies to file deps outside
of /etc /bin /sbin /usr/bin and /usr/sbin.

-Toshio

[Thorsten Leemhuis]

Axel Thimm schrieb:
>  On Thu, Dec 21, 2006 at 02:43:31PM +0100, Thorsten Leemhuis wrote:
> > On 21.12.2006 14:24, Axel Thimm wrote:
> >> On Thu, Dec 21, 2006 at 02:10:18PM +0100, Axel Thimm wrote:
> >>> On Wed, Dec 20, 2006 at 08:55:25PM -0800, Toshio Kuratomi wrote:
> >>>> Hey all, I've just added a draft proposal for limiting file
> dependencies
> >>>> in packages.  It's a "should" recommendation at this
> point.
> >>>> http://www.fedoraproject.org/wiki/PackagingDrafts/FileDeps
> >>>> = File Deps Guideline =
> >>>> == Problem ==
> >>> How often does this problem really occur? I just checked core's x86_64
> >>> packages and there are exactly 12 files in such dependencies
> >>> affecting 18 packages out of 2422 (that's 0.7%):
> >> And the same metrics for extras: 26 files required by 27 packages out
> >> of 3660 packages:
> >> That's again 0.7%, so the combined Fedora is at 0.7%, too.
> > Well, my initial calculations found 32 file based deps outside of "/etc 
> > {/usr,}/{s,}bin/" in Extras 6 x64 ;-) Just to note, I mentioned that 
> > when I started the whole mess after skvidal poked me about it. See 
> > https://www.redhat.com/archives/fedora-packaging/2006-December/msg00077.html
>  But then it was known that this is negligible, 
"negligible" IMHO is debatable, but yes, it's not a big problem
(currently).

And just to clarify: I just started to look into it because skvidal
poked me about it. I had no interest in it myself and after the initial
research I did te thing rolled on its own.

>  why does it escalate
>  to become even a proposal for guidelines? 
To avoid that there get more of those added in the future?

>  Do we care that yum fetches
>  the filelists every 140 package updates? 
If it can be avoided easiy: yes.

There is also the OLPC hardware that seems to be quite slow and will
probably take ages to resolve such a dep. We should keep that in mind, too.

CU
thl

[Fernando Nasser]

Toshio Kuratomi wrote:
>  On Thu, 2006-12-21 at 12:13 -0500, Fernando Nasser wrote:
> > Rex Dieter wrote:
> >> Fernando Nasser wrote:
> >>> I have two questions:
> >>>
> >>> 1) If this is a problem that only affects yum and all the other 
> >>> depsolvers do it efficiently, would it be better to fix yum instead or 
> >>> working around it by spec file changes?
> >> It's still generally a good idea to minimize file deps.
> >>
>  Actually, what's happening is that all depsolvers have the ability to
>  optimize two time consuming steps if the file deps follow these
>  guidelines.  The two steps are 1) downloading filelist.xml.gz and 2)
>  parsing filelist.xml to find the package which provides the file deps.
>  [And this problem is multiplied the more repositories you have defined
>  as the depsolver will have to grab the filelist.xml file from multiple
>  repositories].
>  
>  Panu points out that only yum currently optimizes this but there's
>  nothing stopping the others doing it as well.
>   
OK

> > Sure, but there are cases where it is desirable.  I am afraid a
> dull 
> > guideline will put a straight jacket on developers.
> >
> > For instance, I know a couple of folks working on a software that in the 
> > current release needs two independent packages.  Another developer needs 
> > one (a single one) file from them, but the file may change places.  They 
> > thought of adding a virtual provides, but it seems overkill in this case 
> > and it is probably only temporary (which would mean we would use a name 
> > unnecessarily and regret later). So the best way around it was to 
> > require the file, at least until the set of packages stabilize on a 
> > future version of the dependency.
> >
> > Perhaps a "recommendation" guideline?  (if there is such a thing)
> >
>  Yes.  This proposal is for a "Should" rather than a "Must"
> guideline.
>  In the case you outline, I'd like to see the package depend on the file
>  until it was settled which package would provide it.  Then the
>  dependency could be converted to a package dep instead.  We don't
>  currently have an ongoing QA requirement, though, so remembering to make
>  the change would largely be the responsibility of the packager.
>   
That will work for them.  I think it is a sensible approach.
Perhaps adding a FIXME to the specfile above that dependency would be a 
good idea.

> >>> 2) If we still use file deps for things in /bin /sbin
> /usr/bin 
> >>> /usr/sbin, which happens for all GCJ compiled packages, wouldn't it 
> >>> force the reading of the second file anyways?
> >> No, just for items *outside* of those locations does yum need extra 
> >> processing.  That's the whole point of this exercise.
> >>
> > Ah, thanks for the clarification.  The use of file deps prevents an 
> > optimization to kick in.
> >
>  Correct.
>  
> > Is this list complete: /bin /sbin /usr/bin /usr/sbin ?
> > Or there are others that do not cause the loading of the second file?
>  
>  /etc is the other one.
>   
Thanks.  I guess the list of exceptions will be in the guidelines.

>  That's why the guideline applies to file deps outside
>  of /etc /bin /sbin /usr/bin and /usr/sbin.
>  
>  -Toshio
>  
>   
Thanks for the clarifications

Fernando

[Nicolas Mailhot]

Le jeudi 21 décembre 2006 à 19:00 +0100, Thorsten Leemhuis a écrit :
>  Axel Thimm schrieb: 
>  > why does it escalate
>  > to become even a proposal for guidelines?
>  
>  To avoid that there get more of those added in the future? 
As the numbers show no one is particularly inclined to add a large
number of such deps today. They won't become more common unless they
start providing some benefit to users or packagers. (for example proper
rpm directory handling at last). I don't see how we can decide today a
yum-specific optimisation will take precedence over this yet-to-be-known
benefit.

-- 
Nicolas Mailhot

[Panu Matilainen]

On Thu, 21 Dec 2006, Toshio Kuratomi wrote:
>  On Thu, 2006-12-21 at 12:13 -0500, Fernando Nasser wrote:
> > Rex Dieter wrote:
> >> Fernando Nasser wrote:
> >>> I have two questions:
> >>>
> >>> 1) If this is a problem that only affects yum and all the other
> >>> depsolvers do it efficiently, would it be better to fix yum instead or
> >>> working around it by spec file changes?
> >>
> >> It's still generally a good idea to minimize file deps.
> >
>  Actually, what's happening is that all depsolvers have the ability to
>  optimize two time consuming steps if the file deps follow these
>  guidelines.  The two steps are 1) downloading filelist.xml.gz and 2)
>  parsing filelist.xml to find the package which provides the file deps.
>  [And this problem is multiplied the more repositories you have defined
>  as the depsolver will have to grab the filelist.xml file from multiple
>  repositories].
>
>  Panu points out that only yum currently optimizes this but there's
>  nothing stopping the others doing it as well. 
Because of their very design, smart and apt CAN NOT optimize it as long as 
the repodata format permits arbitrary file dependencies. That's what I've 
been trying to say here.

Assume for a moment that FC + FE are cleaned up from all file dependencies 
outside of those present in primary.xml. Sure, it'd be possible to make a 
fedora-specific one-liner patch to avoid downloading the filelists.xml and 
that would "work." But as long as it's just a Fedora policy and not forced 
by the tools, things would break as soon as any 3rd party repository not 
enforcing that particular Fedora policy is used (Axel, this is what I 
meant with "any old repository", no such thing as "new repodata" at
the 
moment)

What does it mean? Heck, nothing really. Apt and smart have a radically 
different internal design from that of yum, each has their own benefits. 
The partial filelist info within primary.xml is really what I'd call a 
RH/Fedora specific performance hack, based on rough "these directories 
cover 99% of the filedeps in practice" heuristic. Which is all and well, 
except it's annoyingly close-but-no-cigar to working for apt and smart as 
well, they need to download the huge filelists.xml only because of that 
remaining 1% :)

Again, cleaning up obscure file dependencies is good, nothing against this 
thing at all. I just wanted to correct the idea that it would help "the 
other" depsolvers as well - it doesn't, but that doesn't make the idea 
bad.

 	- Panu -

[Ville Skyttä]

On Thu, 2006-12-21 at 09:19 -0600, Tom 'spot' Callaway wrote:
>  On Thu, 2006-12-21 at 10:10 -0500, Andrew Overholt wrote:
>  > 
>  > In the Eclipse SDK package set, we recently went through the
>  > multilib-ification process.  We had to add a few file-level dependencies
>  > to get this to work.  This was due to the inability to specify %{arch}
>  > in a Requires.  How would this have been better accomplished?
>  > 
>  > Ben can probably explain this better than I can.
>  
>  Honestly, this is the only valid case for file-level dependencies I can
>  think of, and unfortunately, the only way to work around it is to fix
>  rpm to permit %{arch} specific Requires. 
One thing for which a dir based dependency is the best we have at the
moment are plugins for Mozilla compatible browsers (Requires:
%{_libdir}/mozilla/plugins).  That could be changed if those browser
packages had a common Provides indicating that they load plugins from
that dir.

By the way, would it be possible/feasible to modify yum so that before
downloading filelists metadata, it would check the headers of packages
already inserted into a transaction to see if they already satisfy the
needed file based dependencies (even if those deps are not available in
primary metadata)?  One example case where this would be useful is what
I currently do in w3c-markup-validator-libs:

   Requires: xhtml1-dtds /usr/share/sgml/xhtml1/xhtml1-20020801/DTD

The DTDs are really expected to be found in that exact path.  Even
though I know it's currently that way in xhtml1-dtds, I'd like to have
the path based dependency there for additional safeguard in case the
path to the DTDs ever changes (there's a date embedded in the path so
it's inherently at least theoretically prone to that).

So when yum sees the above two dependencies without having full
filelists available, it'd first insert xhtml1-dtds and try it out if it
happened to satisfy /usr/share/sgml/xhtml1/xhtml1-20020801/DTD too,
without downloading filelists yet.  If yes, no need to download
filelists.