package upload/import speed

[Naveen Gavini]

Hello all,

When importing or building RPM's from source is there an option that is 
throttling the upload speed of the package?
We are seeing speeds of 10kbps which makes importing/building large 
packages very slow.

Thanks.

-- 
Naveen Gavini
Student Systems Programmer
OSS/CSS - OIT Rutgers
ngavini(a)jla.rutgers.edu

[Bryce]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Naveen Gavini wrote:
>  Hello all,
>  
>  When importing or building RPM's from source is there an option that is
>  throttling the upload speed of the package?
>  We are seeing speeds of 10kbps which makes importing/building large
>  packages very slow.
>  
>  Thanks.
>   
On a local machine (where client and server, hub etc all reside) I get
140K/s (about x14 what you're seeing) but that seems to be an artificial
cap given that it's invariant no matter what the loading on the machine
is. Ie it's not CPU bound. Unfortunately it's a royal PITA for scratch
building since that means large src rpms like the kernel can take up to
5 minutes just to be 'uploaded' on the same damned box.

To be honest this is something I've been scratching my head over as
well. Various slim picked information I've gathered tries to point the
finger of slowness at SSL itself for the encryption speed and suggest
getting hold of SSL HW accelerators but I don't buy that argument due to
the constant transfer speed.

I haven't been able to decide if it's a client side issue or a server
side issue though 8/

Phil
=--=
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.7 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
>
> iD8DBQFIh5SgPcECCTpcVR0RAjZwAJ9s0giafZQmRHtxdyOefvq+QktiCgCfeFe+
> hoOV7CUIucOTbZI1PTtFgDs=
> =gMPu
> -----END PGP SIGNATURE-----

[Mike Bonnet]

On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote:
>  Hello all,
>  
>  When importing or building RPM's from source is there an option that is 
>  throttling the upload speed of the package?
>  We are seeing speeds of 10kbps which makes importing/building large 
>  packages very slow. 
Package upload happens via xml-rpc, and to avoid holding the entire
contents of the file in memory, it's done in chunks.  Unfortunately
python xml-rpc libs don't support keep-alive, so if you're using SSL
authentication you incur the non-trivial SSL connection setup/teardown
overhead for every chunk, in addition to the TCP connection
setup/teardown.  The chunk size in your version is probably very small
(64k?).  That has been increased in the Koji repo, but not released yet.
The next version will improve this situation somewhat.

[Toshio Kuratomi]

Mike Bonnet wrote:
>  On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote:
> > Hello all,
> >
> > When importing or building RPM's from source is there an option that is 
> > throttling the upload speed of the package?
> > We are seeing speeds of 10kbps which makes importing/building large 
> > packages very slow.
>  
>  Package upload happens via xml-rpc, and to avoid holding the entire
>  contents of the file in memory, it's done in chunks.  Unfortunately
>  python xml-rpc libs don't support keep-alive, so if you're using SSL
>  authentication you incur the non-trivial SSL connection setup/teardown
>  overhead for every chunk, in addition to the TCP connection
>  setup/teardown.  The chunk size in your version is probably very small
>  (64k?).  That has been increased in the Koji repo, but not released yet.
>  The next version will improve this situation somewhat.
>   
Any chance we could port to pycurl which does support keep-alive?  Or is 
urllib pretty deeply embedded in the code ATM?

-Toshio

[Mike Bonnet]

On Wed, 2008-07-23 at 14:03 -0700, Toshio Kuratomi wrote:
>  Mike Bonnet wrote:
>  > On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote:
>  >> Hello all,
>  >>
>  >> When importing or building RPM's from source is there an option that is 
>  >> throttling the upload speed of the package?
>  >> We are seeing speeds of 10kbps which makes importing/building large 
>  >> packages very slow.
>  > 
>  > Package upload happens via xml-rpc, and to avoid holding the entire
>  > contents of the file in memory, it's done in chunks.  Unfortunately
>  > python xml-rpc libs don't support keep-alive, so if you're using SSL
>  > authentication you incur the non-trivial SSL connection setup/teardown
>  > overhead for every chunk, in addition to the TCP connection
>  > setup/teardown.  The chunk size in your version is probably very small
>  > (64k?).  That has been increased in the Koji repo, but not released yet.
>  > The next version will improve this situation somewhat.
>  > 
>  Any chance we could port to pycurl which does support keep-alive?  Or is 
>  urllib pretty deeply embedded in the code ATM? 
I don't know anything about pycurl, but the transport layer is decently
separated from the XML-RPC layer in xmlrpclib, so it should be possible
to replace that it with a transport that supports SSL and keep-alive.  I
haven't looked at how much work this might be though.

>  -Toshio
>  
>  --
>  Fedora-buildsys-list mailing list
>  Fedora-buildsys-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

[Oliver Falk]

Mike Bonnet wrote:
>  On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote:
> > Hello all,
> >
> > When importing or building RPM's from source is there an option that is 
> > throttling the upload speed of the package?
> > We are seeing speeds of 10kbps which makes importing/building large 
> > packages very slow.
>  
>  Package upload happens via xml-rpc, and to avoid holding the entire
>  contents of the file in memory, it's done in chunks.  Unfortunately
>  python xml-rpc libs don't support keep-alive, so if you're using SSL
>  authentication you incur the non-trivial SSL connection setup/teardown
>  overhead for every chunk, in addition to the TCP connection
>  setup/teardown.  The chunk size in your version is probably very small
>  (64k?).  That has been increased in the Koji repo, but not released yet.
>  The next version will improve this situation somewhat. 
Just wanted to write the same...

Naveen, you could increase your blocksiize as a workaround. Just search 
for blocksize in /usr/bin/koji (function do_import)... I've done that 
here and it helps. BUT do not set to too large, as for every chunk some 
checksum (it seems) is being build (SSL does that I think) and the 
larger your chunk the more CPU resources it will use!!!

-of