Hello all,
When importing or building RPM's from source is there an option that is throttling the upload speed of the package? We are seeing speeds of 10kbps which makes importing/building large packages very slow.
Thanks.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Naveen Gavini wrote:
Hello all,
When importing or building RPM's from source is there an option that is throttling the upload speed of the package? We are seeing speeds of 10kbps which makes importing/building large packages very slow.
Thanks.
On a local machine (where client and server, hub etc all reside) I get 140K/s (about x14 what you're seeing) but that seems to be an artificial cap given that it's invariant no matter what the loading on the machine is. Ie it's not CPU bound. Unfortunately it's a royal PITA for scratch building since that means large src rpms like the kernel can take up to 5 minutes just to be 'uploaded' on the same damned box.
To be honest this is something I've been scratching my head over as well. Various slim picked information I've gathered tries to point the finger of slowness at SSL itself for the encryption speed and suggest getting hold of SSL HW accelerators but I don't buy that argument due to the constant transfer speed.
I haven't been able to decide if it's a client side issue or a server side issue though 8/
Phil =--=
On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote:
Hello all,
When importing or building RPM's from source is there an option that is throttling the upload speed of the package? We are seeing speeds of 10kbps which makes importing/building large packages very slow.
Package upload happens via xml-rpc, and to avoid holding the entire contents of the file in memory, it's done in chunks. Unfortunately python xml-rpc libs don't support keep-alive, so if you're using SSL authentication you incur the non-trivial SSL connection setup/teardown overhead for every chunk, in addition to the TCP connection setup/teardown. The chunk size in your version is probably very small (64k?). That has been increased in the Koji repo, but not released yet. The next version will improve this situation somewhat.
Mike Bonnet wrote:
On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote:
Hello all,
When importing or building RPM's from source is there an option that is throttling the upload speed of the package? We are seeing speeds of 10kbps which makes importing/building large packages very slow.
Package upload happens via xml-rpc, and to avoid holding the entire contents of the file in memory, it's done in chunks. Unfortunately python xml-rpc libs don't support keep-alive, so if you're using SSL authentication you incur the non-trivial SSL connection setup/teardown overhead for every chunk, in addition to the TCP connection setup/teardown. The chunk size in your version is probably very small (64k?). That has been increased in the Koji repo, but not released yet. The next version will improve this situation somewhat.
Any chance we could port to pycurl which does support keep-alive? Or is urllib pretty deeply embedded in the code ATM?
-Toshio
On Wed, 2008-07-23 at 14:03 -0700, Toshio Kuratomi wrote:
Mike Bonnet wrote:
On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote:
Hello all,
When importing or building RPM's from source is there an option that is throttling the upload speed of the package? We are seeing speeds of 10kbps which makes importing/building large packages very slow.
Package upload happens via xml-rpc, and to avoid holding the entire contents of the file in memory, it's done in chunks. Unfortunately python xml-rpc libs don't support keep-alive, so if you're using SSL authentication you incur the non-trivial SSL connection setup/teardown overhead for every chunk, in addition to the TCP connection setup/teardown. The chunk size in your version is probably very small (64k?). That has been increased in the Koji repo, but not released yet. The next version will improve this situation somewhat.
Any chance we could port to pycurl which does support keep-alive? Or is urllib pretty deeply embedded in the code ATM?
I don't know anything about pycurl, but the transport layer is decently separated from the XML-RPC layer in xmlrpclib, so it should be possible to replace that it with a transport that supports SSL and keep-alive. I haven't looked at how much work this might be though.
-Toshio
-- Fedora-buildsys-list mailing list Fedora-buildsys-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
Mike Bonnet wrote:
On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote:
Hello all,
When importing or building RPM's from source is there an option that is throttling the upload speed of the package? We are seeing speeds of 10kbps which makes importing/building large packages very slow.
Package upload happens via xml-rpc, and to avoid holding the entire contents of the file in memory, it's done in chunks. Unfortunately python xml-rpc libs don't support keep-alive, so if you're using SSL authentication you incur the non-trivial SSL connection setup/teardown overhead for every chunk, in addition to the TCP connection setup/teardown. The chunk size in your version is probably very small (64k?). That has been increased in the Koji repo, but not released yet. The next version will improve this situation somewhat.
Just wanted to write the same...
Naveen, you could increase your blocksiize as a workaround. Just search for blocksize in /usr/bin/koji (function do_import)... I've done that here and it helps. BUT do not set to too large, as for every chunk some checksum (it seems) is being build (SSL does that I think) and the larger your chunk the more CPU resources it will use!!!
-of
buildsys@lists.fedoraproject.org
-
Bryce -
Mike Bonnet -
Naveen Gavini -
Oliver Falk -
Toshio Kuratomi