On Mon, Sep 25, 2017 at 01:54:49PM -0700, Kevin Fenzi wrote:
Greetings.
This morning pkgs02 stopped answering to git:// clone urls from koji, breaking builds.
We investigated, but the machine was in a very weird state. Systemd was unable to talk to itself (systemctl/reboot didn't work, and journald wasn't logging new entries) and it was under very high load.
So, I applied updates to it and power cycled it.
systemd was happy after that, but load was still very very high. Looking I found a number of git clones from external ip's. Since there's no reason for this (external people should use https:// clone urls or ssh://) I blocked those except from 10.0.0.0/8.
Since this was outage causing for builds I went ahead and did all this, but would like to get retroactive +1s or any adjustments I might have missed.
Thanks for taking care of this +1
I wonder if this weird systemd state isn't something we hit earlier when we were deploying pagure there.
Pierre