Yum .hdr files massive

[Phil Anderson]

Hi,

Is there any reason why the yum header files (.hdr) are about the same 
size as the RPM files?

e.g.
[pza@hallucination development]$ ll headers/thunderbird-0.8.0-4.i386.hdr 
packages/thunderbird-0.8.0-4.i386.rpm
-rw-r--r--  1 root root 11108767 Oct  9 03:27 
headers/thunderbird-0.8.0-4.i386.hdr
-rw-r--r--  1 root root 11109047 Oct  9 03:27 
packages/thunderbird-0.8.0-4.i386.rpm
[pza@hallucination development]$ du -s *
412296  headers
412320  packages
940     primary.xml.gz
6844    primary.xml.gz.0de5a02528ed14facfc0eaf9c7060209.pickle
8       repomd.xml

I thought the idea of the header files was to get all the important info 
from without downloading the whole rpm?

Anyone know what is going on here?  Keeping up with the development 
repositlry is draining my monthly download quota, as i'm downloading 
twice what is really required.

Phil

[Jef Spaleta]

On Sun, 10 Oct 2004 02:20:54 +1000, Phil Anderson <pza(a)pza.net.au&gt; wrote:
>  Anyone know what is going on here?  Keeping up with the development
>  repositlry is draining my monthly download quota, as i'm downloading
>  twice what is really required. 
can you clean out your headers and your packages
directory... and try
again with the latest yum from the development tree and doublecheck to
make sure this is still an issue. I dont see the issue here
yum-2.1.6-1:

ls -lah packages/totem-0.99.17-1.i386.rpm
-rw-r--r--  1 root root 684K Oct  8 17:24 packages/totem-0.99.17-1.i386.rpm
ls -lah headers/totem-0.99.17-1.i386.hdr
-rw-r--r--  1 root root 20K Oct  8 17:23 headers/totem-0.99.17-1.i386.hdr

ls -lah packages/openoffice.org-1.1.2-9.i386.rpm
-rw-r--r--  1 root root 44M Oct  7 11:22 packages/openoffice.org-1.1.2-9.i386.rp
ls -lah headers/openoffice.org-1.1.2-9.i386.hdr
-rw-r--r--  1 root root 373K Oct  7 23:15
headers/openoffice.org-1.1.2-9.i386.hdr

ls -lah packages/thunderbird-0.8.0-4.i386.rpm
-rw-r--r--  1 root root 11M Oct  9 12:30 packages/thunderbird-0.8.0-4.i386.rpm
 ls -lah headers/thunderbird-0.8.0-4.i386.hdr
-rw-r--r--  1 root root 62K Oct  9 12:29 headers/thunderbird-0.8.0-4.i386.hdr

du -s *
7792    headers
2536120 packages
940     primary.xml.gz
6844    primary.xml.gz.0de5a02528ed14facfc0eaf9c7060209.pickle
8       repomd.xml


-jef

[seth vidal]

On Sun, 2004-10-10 at 02:20 +1000, Phil Anderson wrote:
>  Hi,
>  
>  Is there any reason why the yum header files (.hdr) are about the same 
>  size as the RPM files?
>  
>  e.g.
>  [pza@hallucination development]$ ll headers/thunderbird-0.8.0-4.i386.hdr 
>  packages/thunderbird-0.8.0-4.i386.rpm
>  -rw-r--r--  1 root root 11108767 Oct  9 03:27 
>  headers/thunderbird-0.8.0-4.i386.hdr
>  -rw-r--r--  1 root root 11109047 Oct  9 03:27 
>  packages/thunderbird-0.8.0-4.i386.rpm
>  [pza@hallucination development]$ du -s *
>  412296  headers
>  412320  packages
>  940     primary.xml.gz
>  6844    primary.xml.gz.0de5a02528ed14facfc0eaf9c7060209.pickle
>  8       repomd.xml
>  
>  I thought the idea of the header files was to get all the important info 
>  from without downloading the whole rpm? 
What mirror are you using?

If you're not using a site that supports http 1.1 and byte-ranges then
yum will have to download the whole package.


-sv

[Ralf Ertzinger]

Hi.

seth vidal <skvidal(a)phy.duke.edu&gt; wrote:

>  If you're not using a site that supports http 1.1 and byte-ranges
> then
>  yum will have to download the whole package. 
You're grabbing the headers out of the RPM files via byte ranges?
Sweet idea.

-- 
Man who drive like hell, bound to get there.

[Nils Philippsen]

On Sun, 2004-10-10 at 12:46 +0200, Ralf Ertzinger wrote:
>  Hi.
>  
>  seth vidal <skvidal(a)phy.duke.edu&gt; wrote:
>  
>  > If you're not using a site that supports http 1.1 and byte-ranges then
>  > yum will have to download the whole package.
>  
>  You're grabbing the headers out of the RPM files via byte ranges?
>  Sweet idea. 
Though it would be even sweeter if byte-ranges fail you would start to
download the whole thing (or at least tell the server that) and break
the connection when you've read the entire header. Still better than
downloading the whole thing.

Nils
-- 
     Nils Philippsen    /    Red Hat    /    nphilipp(a)redhat.com
"They that can give up essential liberty to obtain a little temporary
 safety deserve neither liberty nor safety."     -- B. Franklin, 1759
 PGP fingerprint:  C4A8 9474 5C4C ADE3 2B8F  656D 47D8 9B65 6951 3011

[Ralf Ertzinger]

Hi.

Nils Philippsen <nphilipp(a)redhat.com&gt; wrote:

>  the connection when you've read the entire header. Still better
> than
>  downloading the whole thing. 
On the other hand, this is 2004. Everything claiming to be a HTTP server and
not supporting this ought to be slugged though the processor core :)

-- 
Never try to teach a pig to sing. It wastes your time, and it annoys
the pig.

[Nils Philippsen]

On Sun, 2004-10-10 at 13:50 +0200, Ralf Ertzinger wrote:
>  Hi.
>  
>  Nils Philippsen <nphilipp(a)redhat.com&gt; wrote:
>  
>  > the connection when you've read the entire header. Still better than
>  > downloading the whole thing.
>  
>  On the other hand, this is 2004. Everything claiming to be a HTTP server and
>  not supporting this ought to be slugged though the processor core :) 
I'd say let's change what we are able to change. The software running
the mirrors is probably beyond our area of control ;-).

Nils
-- 
     Nils Philippsen    /    Red Hat    /    nphilipp(a)redhat.com
"They that can give up essential liberty to obtain a little temporary
 safety deserve neither liberty nor safety."     -- B. Franklin, 1759
 PGP fingerprint:  C4A8 9474 5C4C ADE3 2B8F  656D 47D8 9B65 6951 3011

[seth vidal]

On Sun, 2004-10-10 at 14:17 +0200, Nils Philippsen wrote:
>  On Sun, 2004-10-10 at 13:50 +0200, Ralf Ertzinger wrote:
>  > Hi.
>  > 
>  > Nils Philippsen <nphilipp(a)redhat.com&gt; wrote:
>  > 
>  > > the connection when you've read the entire header. Still better than
>  > > downloading the whole thing.
>  > 
>  > On the other hand, this is 2004. Everything claiming to be a HTTP server and
>  > not supporting this ought to be slugged though the processor core :)
>  
>  I'd say let's change what we are able to change. The software running
>  the mirrors is probably beyond our area of control ;-).
>   
But it's not beyond the control of the users.

-sv

[seth vidal]

On Sun, 2004-10-10 at 13:45 +0200, Nils Philippsen wrote:
>  On Sun, 2004-10-10 at 12:46 +0200, Ralf Ertzinger wrote:
>  > Hi.
>  > 
>  > seth vidal <skvidal(a)phy.duke.edu&gt; wrote:
>  > 
>  > > If you're not using a site that supports http 1.1 and byte-ranges then
>  > > yum will have to download the whole package.
>  > 
>  > You're grabbing the headers out of the RPM files via byte ranges?
>  > Sweet idea.
>  
>  Though it would be even sweeter if byte-ranges fail you would start to
>  download the whole thing (or at least tell the server that) and break
>  the connection when you've read the entire header. Still better than
>  downloading the whole thing.
>   
I welcome patches to do just that but you should coordinate with the
urlgrabber authors.

Thanks for volunteering!
-sv

[Nils Philippsen]

On Sun, 2004-10-10 at 12:26 -0400, seth vidal wrote:
>  On Sun, 2004-10-10 at 13:45 +0200, Nils Philippsen wrote:
>  > On Sun, 2004-10-10 at 12:46 +0200, Ralf Ertzinger wrote:
>  > > Hi.
>  > > 
>  > > seth vidal <skvidal(a)phy.duke.edu&gt; wrote:
>  > > 
>  > > > If you're not using a site that supports http 1.1 and byte-ranges
> then
>  > > > yum will have to download the whole package.
>  > > 
>  > > You're grabbing the headers out of the RPM files via byte ranges?
>  > > Sweet idea.
>  > 
>  > Though it would be even sweeter if byte-ranges fail you would start to
>  > download the whole thing (or at least tell the server that) and break
>  > the connection when you've read the entire header. Still better than
>  > downloading the whole thing.
>  > 
>  
>  I welcome patches to do just that but you should coordinate with the
>  urlgrabber authors. 
And that are ...?

[...]
nils@gibraltar:~> rpm -qf /usr/lib/python2.3/site-packages/urlgrabber
yum-2.1.6-1
[...]

>  Thanks for volunteering! 
I'll add that to the list of things I want to do when I have the time
;-P.

Nils
-- 
     Nils Philippsen    /    Red Hat    /    nphilipp(a)redhat.com
"They that can give up essential liberty to obtain a little temporary
 safety deserve neither liberty nor safety."     -- B. Franklin, 1759
 PGP fingerprint:  C4A8 9474 5C4C ADE3 2B8F  656D 47D8 9B65 6951 3011

[seth vidal]

>  And that are ...? 
Go to the yum-devel list and you can meet them:

https://lists.dulug.duke.edu/mailman/listinfo/yum-devel/

Michael Stenner and Ryan Tomayko.



>  I'll add that to the list of things I want to do when I have the
> time
>  ;-P. 
Join the club, then.

-sv

[Stephen Pollei]

On Sun, 2004-10-10 at 09:26, seth vidal wrote:
>  On Sun, 2004-10-10 at 13:45 +0200, Nils Philippsen wrote:
>  > Though it would be even sweeter if byte-ranges fail you would start to
>  > download the whole thing (or at least tell the server that) and break
>  > the connection when you've read the entire header. Still better than
>  > downloading the whole thing. 
Well if you wanted to be super careful not to
waste bandwidth you might
choose to use setsockopt( ..., SO_RCVBUF, ...) at least. A value from
2800 to 4200 might be about right. The reason is that between the time
you do the final read(2) and the time you do the close(2) or shutdown(2)
you will probably still have other packets in flight. Of course to
really cut down on all wastage you'd have to micromanage the advertised
tcp window to a greater extent.

My initial guess at the 2800 to 4200 is based on 1500 sized packets,
guess of how much data you will need to read from a tcp stream, and the
slow start doubling .
I estimate that a tcp connection needs to download between 4-8k to
extract the rpm header information via http.
http headers(300 bytes) + lead(100 bytes) + signature(200 bytes) +
header(2,500 to 6,500 bytes) = 3,000 to 7,000 bytes.

  2800 | 4200  window size advertised
==============
  1460 | 1460 first trip total
  4260 | 4380 second trip total
  7060 | 8580 last trip total

Note that the first packet should have given you the total amount you
needed to download so in theory if you could micromanage the advertised
window you could set it to exactly how much more you needed. You'd need
extra kernel hooks if you wanted to always have it stop at exactly the
right place every time AFAIK.

-- 
http://dmoz.org/profiles/pollei.html
http://sourceforge.net/users/stephen_pollei/
http://www.orkut.com/Profile.aspx?uid=2455954990164098214
http://stephen_pollei.home.comcast.net/
GPG Key fingerprint = EF6F 1486 EC27 B5E7 E6E1  3C01 910F 6BB5 4A7D 9677

[seth vidal]

>  Well if you wanted to be super careful not to waste bandwidth you
> might
>  choose to use setsockopt( ..., SO_RCVBUF, ...) at least. A value from
>  2800 to 4200 might be about right. The reason is that between the time
>  you do the final read(2) and the time you do the close(2) or shutdown(2)
>  you will probably still have other packets in flight. Of course to
>  really cut down on all wastage you'd have to micromanage the advertised
>  tcp window to a greater extent.
>   
I don't want to be supercareful. In fact, getting this involved in the
way the packages get downloaded is not worth the time.

-sv

[Nils Philippsen]

On Mon, 2004-10-11 at 04:14, seth vidal wrote:
>  > Well if you wanted to be super careful not to waste bandwidth
> you might
>  > choose to use setsockopt( ..., SO_RCVBUF, ...) at least. A value from
>  > 2800 to 4200 might be about right. The reason is that between the time
>  > you do the final read(2) and the time you do the close(2) or shutdown(2)
>  > you will probably still have other packets in flight. Of course to
>  > really cut down on all wastage you'd have to micromanage the advertised
>  > tcp window to a greater extent.
>  > 
>  
>  I don't want to be supercareful. In fact, getting this involved in the
>  way the packages get downloaded is not worth the time. 
I agree totally. It would be cool that if the header is about 50KB and
the RPM is 5MB that we download about 50KB or maybe little more (well
that little more depends on how quick your connection is, but I
digress), but not the entire 5MB. Or in terms of the actual HTTP, that
we emulate "HTTP range from byte x to byte y" so that we stop shortly
after downloading y bytes (because you have to read from the start,
which is about what we have to do when extracting the header from an
RPM). Anything else is superfluous and possibly even harmful (changing
the kernel due to external web server not supporting HTTP range is just
a bit over the top ;-).

Nils
-- 
     Nils Philippsen    /    Red Hat    /    nphilipp(a)redhat.com
"They that can give up essential liberty to obtain a little temporary
 safety deserve neither liberty nor safety."     -- B. Franklin, 1759
 PGP fingerprint:  C4A8 9474 5C4C ADE3 2B8F  656D 47D8 9B65 6951 3011

[Jef Spaleta]

On Sun, 10 Oct 2004 22:14:15 -0400, seth vidal <skvidal(a)phy.duke.edu&gt; wrote:
>  I don't want to be supercareful. In fact, getting this involved
> in the
>  way the packages get downloaded is not worth the time. 
Instead of being fancy and trying to work around servers that dont
understand byteranges with connection breaking or whatnot...

How much work would it be to implement a check client side to prevent
a redownload if the full package was downloaded in the previous
byterange step into the headers directory? When the server doesn't
handle the byte range information correctly is there a problem just
copying over the full download from the header directory into the
packages directory?

And is there any textual information that yum can give to a user to
inform them their chosen mirror doesn't handle byteranges correctly? 
The best way to prevent this problem from being widespread, and thus a
way to prevent people from refiling this as a bug over and over again,
is to find a way to inform them to choose a different mirror.

-jef