Problems with CI - hitting github rate limits

[Rich Megginson]

We're having some problems with CI - getting errors like this:

ERROR: GitHub: You have triggered an abuse detection mechanism. Please 
wait a few minutes before you try again.

and

WARNING: Rate limiting hit, waiting for 1957 seconds

I'm going to scale down the number of pods used in production - perhaps 
there are too many concurrent operations.

[Till Maas]

Hi,

Am Do., 3. Sept. 2020 um 23:18 Uhr schrieb Rich Megginson <
[e-mail]>:
> We're having some problems with CI - getting errors like this:
>
> ERROR: GitHub: You have triggered an abuse detection mechanism. Please
> wait a few minutes before you try again.
>
> and
>
> WARNING: Rate limiting hit, waiting for 1957 seconds
>
> I'm going to scale down the number of pods used in production - perhaps
> there are too many concurrent operations.

to me it seems that the actual solution would be to have only one
system/pod that uses the GitHub API and then spawns/distributes the work to
workers. Since the number of roles and therefore PRs is increasing and the
number of different test environments, too, the number of pods should
actually be increased.

Thanks
Till

[Rich Megginson]

On 9/11/20 3:28 AM, Till Maas wrote:
> Hi,
>
> Am Do., 3. Sept. 2020 um 23:18 Uhr schrieb Rich Megginson 
> <[e-mail] >:
> We're having some problems with CI - getting errors like this:
>
> ERROR: GitHub: You have triggered an abuse detection mechanism.
> Please
> wait a few minutes before you try again.
>
> and
>
> WARNING: Rate limiting hit, waiting for 1957 seconds
>
> I'm going to scale down the number of pods used in production -
> perhaps
> there are too many concurrent operations.
>
>
> to me it seems that the actual solution would be to have only one 
> system/pod that uses the GitHub API and then spawns/distributes the 
> work to workers. Since the number of roles and therefore PRs is 
> increasing and the number of different test environments, too, the 
> number of pods should actually be increased.

I'd rather spend any development time/resources on moving to BaseOS CI 
than improving the current CI.
> Thanks
> Till
> -- 
> Till Maas
> He/His/Him
> Associate Manager, Software Engineering
> NetworkManager, Nmstate, Ansible RHEL Networking System Role
>
> Red Hat GmbH, https://de.redhat.com/, Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, 
> Michael O'Neill