Hi all,
Tahrir is slow, and the endpoint that mediawiki-FedoraBadges uses to get its badge count and badge assertion data from takes about 35 seconds per request.
As a result, if a user enables both badge count and badge images on a user page, the two requests combined take well over a minute. This results in mediawiki often bombing the request and rendering an ugly error in place of badges.
We get tickets/support requests on this occasionally as well: - https://pagure.io/fedora-infrastructure/issue/7650
Some quick testing shows that the queries I care about for the mediawiki-FedoraBadges plugin take about 70ms to run (in prod). Incurring a 35 second penalty for that seems *crazy* to me.
As an alternative, I'd like to propose making a tahrir-readonly user, and modifying the mediawiki-FedoraBadges plugin to query the badges database directly.
I played around and got it working in staging, the process was something like:
- Create tahrir-readonly postgres user - Grant it readonly perms on the tahrir tables, and connect and usage perms. - Modify the mw plugin - Install php-pgsql on wiki01.stg so it can actually connect
After that it seems to work fine. I triggered a cache purge of all wiki pages on stg as well (but on prod, I'd probably just let the cache naturally expire). Pages with badges load almost instantly now (granted the stg data is way less than prod, but still).
I'm looking for comments because I don't know if there's anything security-wise I'm overlooking in doing this. At a quick glance, it seems fine -- the new pg user is readonly and very few people have access to LocalSettings.php where the connect info would be stored. But I might well be missing something and want to be safe.
Thoughts?
Rick
On Wed, 20 Mar 2019 at 14:27, Rick Elrod codeblock@elrod.me wrote:
Hi all,
Tahrir is slow, and the endpoint that mediawiki-FedoraBadges uses to get its badge count and badge assertion data from takes about 35 seconds per request.
Do we have an idea what makes Tahrir so slow ?
As a result, if a user enables both badge count and badge images on a user page, the two requests combined take well over a minute. This results in mediawiki often bombing the request and rendering an ugly error in place of badges.
We get tickets/support requests on this occasionally as well:
Some quick testing shows that the queries I care about for the mediawiki-FedoraBadges plugin take about 70ms to run (in prod). Incurring a 35 second penalty for that seems *crazy* to me.
As an alternative, I'd like to propose making a tahrir-readonly user, and modifying the mediawiki-FedoraBadges plugin to query the badges database directly.
I played around and got it working in staging, the process was something like:
- Create tahrir-readonly postgres user
- Grant it readonly perms on the tahrir tables, and connect and usage perms.
- Modify the mw plugin
- Install php-pgsql on wiki01.stg so it can actually connect
After that it seems to work fine. I triggered a cache purge of all wiki pages on stg as well (but on prod, I'd probably just let the cache naturally expire). Pages with badges load almost instantly now (granted the stg data is way less than prod, but still).
I'm looking for comments because I don't know if there's anything security-wise I'm overlooking in doing this. At a quick glance, it seems fine -- the new pg user is readonly and very few people have access to LocalSettings.php where the connect info would be stored. But I might well be missing something and want to be safe.
Thoughts?
Rick _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On 3/20/19 9:42 AM, Clement Verna wrote:
On Wed, 20 Mar 2019 at 14:27, Rick Elrod codeblock@elrod.me wrote:
Hi all,
Tahrir is slow, and the endpoint that mediawiki-FedoraBadges uses to get its badge count and badge assertion data from takes about 35 seconds per request.
Do we have an idea what makes Tahrir so slow ?
Way back when Tahrir was more actively developed, I tried to look into this some. From what I remember, it's just generating super inefficient queries, and lots of them. I recall the ORM it was using loves to generate queries that exhibit the N+1 query problem. When the N queries are inefficient ones (that might also generate more queries each), the result is slowness.
But that might have changed, and I haven't touched the Tahrir codebase in years. As smooge said, there are a number of Tahrir-upstream bugs that mention it being slow all around, so I don't think this is specific to the endpoint I'm using. I think this is just where we end up seeing it fairly often because of wiki userpages.
Rick
On Wed, Mar 20, 2019 at 09:17:45AM -0400, Rick Elrod wrote:
Hi all,
Tahrir is slow, and the endpoint that mediawiki-FedoraBadges uses to get its badge count and badge assertion data from takes about 35 seconds per request.
As a result, if a user enables both badge count and badge images on a user page, the two requests combined take well over a minute. This results in mediawiki often bombing the request and rendering an ugly error in place of badges.
We get tickets/support requests on this occasionally as well:
Some quick testing shows that the queries I care about for the mediawiki-FedoraBadges plugin take about 70ms to run (in prod). Incurring a 35 second penalty for that seems *crazy* to me.
As an alternative, I'd like to propose making a tahrir-readonly user, and modifying the mediawiki-FedoraBadges plugin to query the badges database directly.
I played around and got it working in staging, the process was something like:
- Create tahrir-readonly postgres user
- Grant it readonly perms on the tahrir tables, and connect and usage perms.
- Modify the mw plugin
- Install php-pgsql on wiki01.stg so it can actually connect
After that it seems to work fine. I triggered a cache purge of all wiki pages on stg as well (but on prod, I'd probably just let the cache naturally expire). Pages with badges load almost instantly now (granted the stg data is way less than prod, but still).
I'm looking for comments because I don't know if there's anything security-wise I'm overlooking in doing this. At a quick glance, it seems fine -- the new pg user is readonly and very few people have access to LocalSettings.php where the connect info would be stored. But I might well be missing something and want to be safe.
Thoughts?
Sounds fine to me :) As a ticket been opened upstream so it's not lost that this API needs optimization?
Pierre
On Wed, 20 Mar 2019 at 09:43, Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Wed, Mar 20, 2019 at 09:17:45AM -0400, Rick Elrod wrote:
Hi all,
Tahrir is slow, and the endpoint that mediawiki-FedoraBadges uses to get its badge count and badge assertion data from takes about 35 seconds per request.
As a result, if a user enables both badge count and badge images on a user page, the two requests combined take well over a minute. This results in mediawiki often bombing the request and rendering an ugly error in place of badges.
We get tickets/support requests on this occasionally as well:
Some quick testing shows that the queries I care about for the mediawiki-FedoraBadges plugin take about 70ms to run (in prod). Incurring a 35 second penalty for that seems *crazy* to me.
As an alternative, I'd like to propose making a tahrir-readonly user, and modifying the mediawiki-FedoraBadges plugin to query the badges database directly.
I played around and got it working in staging, the process was something like:
- Create tahrir-readonly postgres user
- Grant it readonly perms on the tahrir tables, and connect and usage perms.
- Modify the mw plugin
- Install php-pgsql on wiki01.stg so it can actually connect
After that it seems to work fine. I triggered a cache purge of all wiki pages on stg as well (but on prod, I'd probably just let the cache naturally expire). Pages with badges load almost instantly now (granted the stg data is way less than prod, but still).
I'm looking for comments because I don't know if there's anything security-wise I'm overlooking in doing this. At a quick glance, it seems fine -- the new pg user is readonly and very few people have access to LocalSettings.php where the connect info would be stored. But I might well be missing something and want to be safe.
Thoughts?
Sounds fine to me :) As a ticket been opened upstream so it's not lost that this API needs optimization?
To this particular one, no. I went and looked at the open issues and saw several 'tahrir is slow' so figured it might be caught up already in those.
Pierre _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On 3/20/19 9:48 AM, Stephen John Smoogen wrote:
On Wed, 20 Mar 2019 at 09:43, Pierre-Yves Chibon pingou@pingoured.fr wrote:
Sounds fine to me :) As a ticket been opened upstream so it's not lost that this API needs optimization?
To this particular one, no. I went and looked at the open issues and saw several 'tahrir is slow' so figured it might be caught up already in those.
Yeah, exactly this. I think the problems aren't specific to this particular endpoint. We've known (or at least I've known) about the slowness on this endpoint since the start of Tahrir's development, but I think it affects some other endpoints as well (as suggested by the tickets smooge referred to). I tried for a while to track it down a few years ago, but never did.
R.
infrastructure@lists.fedoraproject.org
-
Clement Verna -
Pierre-Yves Chibon -
Rick Elrod -
Stephen John Smoogen