upload blocksize

[Dennis Gregorovic]

Hi,

In the koji/__init__.py:fastUpload method there is a blocksize parameter that is used to determine how many bytes are uploaded to the koji hub per request.  By default it is set to 1MB and as far as I can tell that is the value actually used.  Is there a way to change this value via koji configuration?  A larger blocksize value could significantly improve the upload throughput.

-- Dennis

[Mike McLean]

On 03/01/2016 11:09 AM, Dennis Gregorovic wrote:
> Hi,
>
> In the koji/__init__.py:fastUpload method there is a blocksize
> parameter that is used to determine how many bytes are uploaded to
> the koji hub per request.  By default it is set to 1MB and as far as
> I can tell that is the value actually used.  Is there a way to change
> this value via koji configuration?  A larger blocksize value could
> significantly improve the upload throughput.

There is no way to change it via configuration. It used to be smaller
and we raised it for performance. At the time at least, 1M seemed to be
roughly the point of diminishing returns. It really depends on the
connection overhead.

Speaking of connection overhead, another factor to consider is whether
keepalive is enabled on the httpd server that runs the hub.

Worth noting that there is also a hard coded block size on the other end.

Does anyone have time to experiment with different permutations?
> -- Dennis _______________________________________________ koji-devel
> mailing list [e-mail] 
> https://lists.fedorahosted.org/admin/lists/[e-mail]

[Peter Robinson]

On Tue, Mar 1, 2016 at 4:48 PM, Mike McLean  wrote:
> On 03/01/2016 11:09 AM, Dennis Gregorovic wrote:
> > Hi,
> >
> > In the koji/__init__.py:fastUpload method there is a blocksize
> > parameter that is used to determine how many bytes are uploaded to
> > the koji hub per request.  By default it is set to 1MB and as far as
> > I can tell that is the value actually used.  Is there a way to change
> > this value via koji configuration?  A larger blocksize value could
> > significantly improve the upload throughput.

For people on the local LAN sure, but the further away you are the
more likely you'll have issues, I suspect people in India, Australia
and other places much more will cause issues.
> There is no way to change it via configuration. It used to be smaller
> and we raised it for performance. At the time at least, 1M seemed to be
> roughly the point of diminishing returns. It really depends on the
> connection overhead.

Is it hard to make it a config option?
> Worth noting that there is also a hard coded block size on the other end.

Why?
> Does anyone have time to experiment with different permutations?

Ah, time, I wish.

[Dennis Gregorovic]

On 03/01/2016 12:05 PM, Peter Robinson wrote:
> On Tue, Mar 1, 2016 at 4:48 PM, Mike McLean  wrote:
> > On 03/01/2016 11:09 AM, Dennis Gregorovic wrote:
> > > Hi,
> > >
> > > In the koji/__init__.py:fastUpload method there is a blocksize
> > > parameter that is used to determine how many bytes are uploaded to
> > > the koji hub per request.  By default it is set to 1MB and as far as
> > > I can tell that is the value actually used.  Is there a way to change
> > > this value via koji configuration?  A larger blocksize value could
> > > significantly improve the upload throughput.
>
> For people on the local LAN sure, but the further away you are the
> more likely you'll have issues, I suspect people in India, Australia
> and other places much more will cause issues.
> > There is no way to change it via configuration. It used to be smaller
> > and we raised it for performance. At the time at least, 1M seemed to be
> > roughly the point of diminishing returns. It really depends on the
> > connection overhead.
>
> Is it hard to make it a config option?
> > Worth noting that there is also a hard coded block size on the other end.
>
> Why?
> > Does anyone have time to experiment with different permutations?
>
> Ah, time, I wish.

I haven't tested with brew, but we did some brief testing with uploads to Pulp, which uses a similar upload model.  Over a wan, we saw an 8x increase in upload throughput when using a 10MB block size vs 1MB.

I don't know if upload speed is generally an issue or not for koji deployments, but I wanted to throw this out there. I've created an RFE for the backlog.

https://pagure.io/koji/issue/30

Cheers
-- Dennis

[Dennis Gilmore]

On Tuesday, March 01, 2016 01:44:12 PM Dennis Gregorovic wrote:
> On 03/01/2016 12:05 PM, Peter Robinson wrote:
> > On Tue, Mar 1, 2016 at 4:48 PM, Mike McLean  wrote:
> > > On 03/01/2016 11:09 AM, Dennis Gregorovic wrote:
> > > > Hi,
> > > >
> > > > In the koji/__init__.py:fastUpload method there is a blocksize
> > > > parameter that is used to determine how many bytes are uploaded to
> > > > the koji hub per request.  By default it is set to 1MB and as far as
> > > > I can tell that is the value actually used.  Is there a way to change
> > > > this value via koji configuration?  A larger blocksize value could
> > > > significantly improve the upload throughput.
> >
> > For people on the local LAN sure, but the further away you are the
> > more likely you'll have issues, I suspect people in India, Australia
> > and other places much more will cause issues.
> > > There is no way to change it via configuration. It used to be smaller
> > > and we raised it for performance. At the time at least, 1M seemed to be
> > > roughly the point of diminishing returns. It really depends on the
> > > connection overhead.
> >
> > Is it hard to make it a config option?
> > > Worth noting that there is also a hard coded block size on the other end.
> >
> > Why?
> > > Does anyone have time to experiment with different permutations?
> >
> > Ah, time, I wish.
>
> I haven't tested with brew, but we did some brief testing with uploads to
> Pulp, which uses a similar upload model.  Over a wan, we saw an 8x increase
> in upload throughput when using a 10MB block size vs 1MB.
>
> I don't know if upload speed is generally an issue or not for koji
> deployments, but I wanted to throw this out there. I've created an RFE for
> the backlog.
>
> https://pagure.io/koji/issue/30
>
> Cheers
> -- Dennis

Thanks Dennis,

Uploads of big items like live iso's, cloud and arm images are extremely slow 
in fedora. It takes somewhere around 10 to 20 Minutes to upload 1Gb

Dennis

[Dennis Gregorovic]

On 03/02/2016 08:57 AM, Dennis Gilmore wrote:
> On Tuesday, March 01, 2016 01:44:12 PM Dennis Gregorovic wrote:
> > On 03/01/2016 12:05 PM, Peter Robinson wrote:
> > > On Tue, Mar 1, 2016 at 4:48 PM, Mike McLean  wrote:
> > > > On 03/01/2016 11:09 AM, Dennis Gregorovic wrote:
> > > > > Hi,
> > > > >
> > > > > In the koji/__init__.py:fastUpload method there is a blocksize
> > > > > parameter that is used to determine how many bytes are uploaded to
> > > > > the koji hub per request.  By default it is set to 1MB and as far as
> > > > > I can tell that is the value actually used.  Is there a way to change
> > > > > this value via koji configuration?  A larger blocksize value could
> > > > > significantly improve the upload throughput.
> > >
> > > For people on the local LAN sure, but the further away you are the
> > > more likely you'll have issues, I suspect people in India, Australia
> > > and other places much more will cause issues.
> > > > There is no way to change it via configuration. It used to be smaller
> > > > and we raised it for performance. At the time at least, 1M seemed to be
> > > > roughly the point of diminishing returns. It really depends on the
> > > > connection overhead.
> > >
> > > Is it hard to make it a config option?
> > > > Worth noting that there is also a hard coded block size on the other end.
> > >
> > > Why?
> > > > Does anyone have time to experiment with different permutations?
> > >
> > > Ah, time, I wish.
> >
> > I haven't tested with brew, but we did some brief testing with uploads to
> > Pulp, which uses a similar upload model.  Over a wan, we saw an 8x increase
> > in upload throughput when using a 10MB block size vs 1MB.
> >
> > I don't know if upload speed is generally an issue or not for koji
> > deployments, but I wanted to throw this out there. I've created an RFE for
> > the backlog.
> >
> > https://pagure.io/koji/issue/30
> >
> > Cheers
> > -- Dennis
>
> Thanks Dennis,
>
> Uploads of big items like live iso's, cloud and arm images are extremely slow 
> in fedora. It takes somewhere around 10 to 20 Minutes to upload 1Gb
>
> Dennis

That very well could be related to the chunk size.  I'll follow up on the Fedora rel-eng list to see about testing that.