Emergency FBR: Lower concurrency on postgres to help load

[Stephen John Smoogen]

Currently the load on db-koji is 140+ and it is slow and
non-responsive to users. I believe it is the below change which has
affected this. I would like to lower down io to 32 to see if that
affects things and tune further if needed. I am not doing this until
Kevin or Patrick are around as it will affect koji and there is an
order needed

diff --git a/roles/postgresql_server/templates/postgresql.conf
b/roles/postgresql_server/templates/postgresql.conf
index cbaaaae..341340c 100644
--- a/roles/postgresql_server/templates/postgresql.conf
+++ b/roles/postgresql_server/templates/postgresql.conf
@@ -504,4 +504,4 @@ default_text_search_config = 'pg_catalog.english'
 #custom_variable_classes = ''           # list of custom variable class names
 #
 # Number of concurrent i/o operations at the same time. The default is 1.
-effective_io_concurrency = 100
+effective_io_concurrency = 32


-- 
Stephen J Smoogen.

[Pierre-Yves Chibon]

On Fri, Oct 11, 2019 at 09:52:20AM -0400, Stephen John Smoogen wrote:
>  Currently the load on db-koji is 140+ and it is slow and
>  non-responsive to users. I believe it is the below change which has
>  affected this. I would like to lower down io to 32 to see if that
>  affects things and tune further if needed. I am not doing this until
>  Kevin or Patrick are around as it will affect koji and there is an
>  order needed
>  
>  diff --git a/roles/postgresql_server/templates/postgresql.conf
>  b/roles/postgresql_server/templates/postgresql.conf
>  index cbaaaae..341340c 100644
>  --- a/roles/postgresql_server/templates/postgresql.conf
>  +++ b/roles/postgresql_server/templates/postgresql.conf
>  @@ -504,4 +504,4 @@ default_text_search_config = 'pg_catalog.english'
>   #custom_variable_classes = ''           # list of custom variable class names
>   #
>   # Number of concurrent i/o operations at the same time. The default is 1.
>  -effective_io_concurrency = 100
>  +effective_io_concurrency = 32 
+1 for me


Pierre

[Clement Verna]

On Fri, 11 Oct 2019 at 16:07, Pierre-Yves Chibon <pingou(a)pingoured.fr&gt;
wrote:

>  On Fri, Oct 11, 2019 at 09:52:20AM -0400, Stephen John Smoogen
> wrote:
>  > Currently the load on db-koji is 140+ and it is slow and
>  > non-responsive to users. I believe it is the below change which has
>  > affected this. I would like to lower down io to 32 to see if that
>  > affects things and tune further if needed. I am not doing this until
>  > Kevin or Patrick are around as it will affect koji and there is an
>  > order needed
>  >
>  > diff --git a/roles/postgresql_server/templates/postgresql.conf
>  > b/roles/postgresql_server/templates/postgresql.conf
>  > index cbaaaae..341340c 100644
>  > --- a/roles/postgresql_server/templates/postgresql.conf
>  > +++ b/roles/postgresql_server/templates/postgresql.conf
>  > @@ -504,4 +504,4 @@ default_text_search_config = 'pg_catalog.english'
>  >  #custom_variable_classes = ''           # list of custom variable class
>  names
>  >  #
>  >  # Number of concurrent i/o operations at the same time. The default is
>  1.
>  > -effective_io_concurrency = 100
>  > +effective_io_concurrency = 32
>
>  +1 for me
>
>  
+1 too

>  Pierre
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to
>  infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>

[Kevin Fenzi]

On Fri, Oct 11, 2019 at 09:52:20AM -0400, Stephen John Smoogen wrote:
>  Currently the load on db-koji is 140+ and it is slow and
>  non-responsive to users. I believe it is the below change which has
>  affected this. I would like to lower down io to 32 to see if that
>  affects things and tune further if needed. I am not doing this until
>  Kevin or Patrick are around as it will affect koji and there is an
>  order needed
>  
>  diff --git a/roles/postgresql_server/templates/postgresql.conf
>  b/roles/postgresql_server/templates/postgresql.conf
>  index cbaaaae..341340c 100644
>  --- a/roles/postgresql_server/templates/postgresql.conf
>  +++ b/roles/postgresql_server/templates/postgresql.conf
>  @@ -504,4 +504,4 @@ default_text_search_config = 'pg_catalog.english'
>   #custom_variable_classes = ''           # list of custom variable class names
>   #
>   # Number of concurrent i/o operations at the same time. The default is 1.
>  -effective_io_concurrency = 100
>  +effective_io_concurrency = 32 
We could try it, but I am not sure that would do the trick.

It was doing this before I had that value set... 

The issue started around the time we upgraded to 1.18.0 koji, and also
added memory, so it's hard to know what change really caused it. 

I guess we can try it, but I'd like to investgate some more today and
see if I can figure out a reason for this. 

I did kill the existing dump. 
I'm trying a different dump setting to see if it cause anything. 

kevin

[Stephen John Smoogen]

On Fri, 11 Oct 2019 at 12:10, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  On Fri, Oct 11, 2019 at 09:52:20AM -0400, Stephen John Smoogen wrote:
>  > Currently the load on db-koji is 140+ and it is slow and
>  > non-responsive to users. I believe it is the below change which has
>  > affected this. I would like to lower down io to 32 to see if that
>  > affects things and tune further if needed. I am not doing this until
>  > Kevin or Patrick are around as it will affect koji and there is an
>  > order needed
>  >
>  > diff --git a/roles/postgresql_server/templates/postgresql.conf
>  > b/roles/postgresql_server/templates/postgresql.conf
>  > index cbaaaae..341340c 100644
>  > --- a/roles/postgresql_server/templates/postgresql.conf
>  > +++ b/roles/postgresql_server/templates/postgresql.conf
>  > @@ -504,4 +504,4 @@ default_text_search_config = 'pg_catalog.english'
>  >  #custom_variable_classes = ''           # list of custom variable class
> names
>  >  #
>  >  # Number of concurrent i/o operations at the same time. The default is 1.
>  > -effective_io_concurrency = 100
>  > +effective_io_concurrency = 32
>
>  We could try it, but I am not sure that would do the trick.
>
>  It was doing this before I had that value set...
>
>  The issue started around the time we upgraded to 1.18.0 koji, and also
>  added memory, so it's hard to know what change really caused it.
>
>  I guess we can try it, but I'd like to investgate some more today and
>  see if I can figure out a reason for this.
>
>  I did kill the existing dump.
>  I'm trying a different dump setting to see if it cause anything. 
I am going to say cancel this FBR until other methods have been exhausted.



-- 
Stephen J Smoogen.