Proposal: Ansible Spring Cleaning: Update/upgrade scripts

[Stephen John Smoogen]

Clement brought up that spring cleaning of our ansible playbooks would
be a good idea. This is painfully obvious during our previous
update/reboot cycles where we have had services not updated or
restarted correctly so that systems did not come up well when we
rebooted.

I have opened https://pagure.io/fedora-infrastructure/issue/7695 which
is the tracking ticket for this problem.  I am proposing that we do
major updates something like the following in the future. We can tweak
as we find better ways to do them in clusters later.


If you maintain a service, take that playbook and add comments for the
following:
a. Who is the current maintainer
b. Date when that was last updated
c. Who tested the upgrade and when
d. General comments to explain what things are doing.

If the playbook should be retired, removed, killed, etc please do so.
My goal will be to make our update schedules something like this:

Day 1:
a. Run update playbooks on staging instances.
b. Fix any problems shown by those.
c. Run general update vhost_update on staging instances
d. Reboot staging instances.
e. Fix problems found from this.

Day 2:
a. Access if day 1 was a complete failure and stop upgrade cycle
b. Run update playbooks on low priority systems
c. Fix any problems shown by those.
d. Run general update vhost_update on staging instances
e. Reboot staging instances.
f. Fix problems found from this.

Day 3:
a. Access if day 2 was a complete failure and stop upgrade cycle
b. Run update playbooks on high priority systems
c. Fix any problems shown by those.
d. Run general update vhost_update on staging instances
e. Reboot staging instances.
f. Fix problems found from this.

This should cut down the extra long hours and extended outages we have
needed to do in the last couple of reboot cycles.

-- 
Stephen J Smoogen.

[Kevin Fenzi]

On 4/5/19 8:59 AM, Stephen John Smoogen wrote:
>  Clement brought up that spring cleaning of our ansible playbooks
> would
>  be a good idea. This is painfully obvious during our previous
>  update/reboot cycles where we have had services not updated or
>  restarted correctly so that systems did not come up well when we
>  rebooted.
>  
>  I have opened https://pagure.io/fedora-infrastructure/issue/7695 which
>  is the tracking ticket for this problem.  I am proposing that we do
>  major updates something like the following in the future. We can tweak
>  as we find better ways to do them in clusters later. 
Thanks for bringing this up smooge!

I completely agree we should clean up/make sure the manual/upgrade/*
playbooks are good and up to date.

>  If you maintain a service, take that playbook and add comments for
> the
>  following:
>  a. Who is the current maintainer
>  b. Date when that was last updated
>  c. Who tested the upgrade and when
>  d. General comments to explain what things are doing.
>  
>  If the playbook should be retired, removed, killed, etc please do so. 
I think we need things like maintainer for all our apps. ;)

>  My goal will be to make our update schedules something like this:
>
>  Day 1:
>  a. Run update playbooks on staging instances.
>  b. Fix any problems shown by those.
>  c. Run general update vhost_update on staging instances
>  d. Reboot staging instances.
>  e. Fix problems found from this.
>  
>  Day 2:
>  a. Access if day 1 was a complete failure and stop upgrade cycle
>  b. Run update playbooks on low priority systems
>  c. Fix any problems shown by those.
>  d. Run general update vhost_update on staging instances
>  e. Reboot staging instances.
>  f. Fix problems found from this.
>  
>  Day 3:
>  a. Access if day 2 was a complete failure and stop upgrade cycle
>  b. Run update playbooks on high priority systems
>  c. Fix any problems shown by those.
>  d. Run general update vhost_update on staging instances
>  e. Reboot staging instances.
>  f. Fix problems found from this. 
Should all of those have staging? Or should it be staging then build
then the rest? or staging, low pri, then high pri?
>  
>  This should cut down the extra long hours and extended outages we have
>  needed to do in the last couple of reboot cycles. 
Well, so some background (not for smooge as he knows all this, but
others reading):

In the past the way we did mass/update reboots has changed a few times.
The most recent incarnation has been doing staging on a friday or
monday, then doing the 'build' machines on one day (basically anything
on bvirthost) and then doing 'the rest' on the next day. Sometimes due
to time we have compressed the two things into one (long) day. During
these we list out all the virthosts/hardware machines, and the sysadmins
take them, update then, reboot them and confirm they come up. Then at
the end we look at nagios and clear up any alerts before calling it
done. The reason for this seperation was because they each had different
'users': We could notify the build outage to just devel-announce, the
'everything else' to announce. Of course now we have to announce the
staging at least to centos folks due to keeping pkgs01.stg in sync via
repospanner.

One thing I have done a few times that I think helped a LOT as far as
time is to actually just apply updates to everything before the reboot
cycle. This saved us all the time waiting for updates to apply (which
can sometimes on some machines take a really long time). Of course this
means that machines run for a time with the updates, but with no
restarted processes.

I am not sure we will easily be able to seperate out what 'manual'
playbooks to run for what servers, and additionally in most cases
updates on our apps are done outside our updates cycles (ie, pagure
would update to 5.4 manually when it's out/desired, we wouldn't expect a
pending update when we do our normal OS update cycles)

So some radical ideas:

* What if we just daily auto-apply all updates. (We already do daily
apply security updates on fedora instances). This would break things
from time to time, but I suspect only particular things, not everything
all at once. We would also still need reboot cycles.

* What if we just daily auto-apply security updates? (This would reduce
the breakage from all updates some). Reboots still needed.

* I thought of the idea someday of doing reboots and having no outage.
Unfortunately, that requires database clustering. At the time the
clustering was all horrible, but it might be better these days. If we
did have that however, we could just do reboots when we liked.

I'm not sure there's a great answer here... I think when we decide to do
a update/reboot cycle we should apply updates up front to save time/pain
and suppose if we can easily determine what manual playbooks to run we
could run those too.

kevin

[Stephen John Smoogen]

On Fri, 5 Apr 2019 at 13:15, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  On 4/5/19 8:59 AM, Stephen John Smoogen wrote:
>  > Clement brought up that spring cleaning of our ansible playbooks would
>  > be a good idea. This is painfully obvious during our previous
>  > update/reboot cycles where we have had services not updated or
>  > restarted correctly so that systems did not come up well when we
>  > rebooted.
>  >
>  > I have opened https://pagure.io/fedora-infrastructure/issue/7695 which
>  > is the tracking ticket for this problem.  I am proposing that we do
>  > major updates something like the following in the future. We can tweak
>  > as we find better ways to do them in clusters later.
>
>  Thanks for bringing this up smooge!
>
>  I completely agree we should clean up/make sure the manual/upgrade/*
>  playbooks are good and up to date.
>
>  > If you maintain a service, take that playbook and add comments for the
>  > following:
>  > a. Who is the current maintainer
>  > b. Date when that was last updated
>  > c. Who tested the upgrade and when
>  > d. General comments to explain what things are doing.
>  >
>  > If the playbook should be retired, removed, killed, etc please do so.
>
>  I think we need things like maintainer for all our apps. ;)
>
>  > My goal will be to make our update schedules something like this:
>  >
>  > Day 1:
>  > a. Run update playbooks on staging instances.
>  > b. Fix any problems shown by those.
>  > c. Run general update vhost_update on staging instances
>  > d. Reboot staging instances.
>  > e. Fix problems found from this.
>  >
>  > Day 2:
>  > a. Access if day 1 was a complete failure and stop upgrade cycle
>  > b. Run update playbooks on low priority systems
>  > c. Fix any problems shown by those.
>  > d. Run general update vhost_update on staging instances
>  > e. Reboot staging instances.
>  > f. Fix problems found from this.
>  >
>  > Day 3:
>  > a. Access if day 2 was a complete failure and stop upgrade cycle
>  > b. Run update playbooks on high priority systems
>  > c. Fix any problems shown by those.
>  > d. Run general update vhost_update on staging instances
>  > e. Reboot staging instances.
>  > f. Fix problems found from this.
>
>  Should all of those have staging? Or should it be staging then build
>  then the rest? or staging, low pri, then high pri? 
So I was leaving the definitions vague as some build boxes are high
priority (no redundancy or major outage) and some are low priority
because they have a large amount of redundancy. I was figuring

low priority:
- proxies not shared on high priority external virthosts
- builders and other high redundancy build systems
- openshift systems IF updated and drained properly
- other external services which have low SLE

high priority:
services with no redundancy:
- databases
- pagure
- src
- etc etc
services with high outage effects
- koji?
- etc etc



>  >
>  > This should cut down the extra long hours and extended outages we have
>  > needed to do in the last couple of reboot cycles.
>
>  Well, so some background (not for smooge as he knows all this, but
>  others reading):
>
>  In the past the way we did mass/update reboots has changed a few times.
>  The most recent incarnation has been doing staging on a friday or
>  monday, then doing the 'build' machines on one day (basically anything
>  on bvirthost) and then doing 'the rest' on the next day. Sometimes due
>  to time we have compressed the two things into one (long) day. During
>  these we list out all the virthosts/hardware machines, and the sysadmins
>  take them, update then, reboot them and confirm they come up. Then at
>  the end we look at nagios and clear up any alerts before calling it
>  done. The reason for this seperation was because they each had different
>  'users': We could notify the build outage to just devel-announce, the
>  'everything else' to announce. Of course now we have to announce the
>  staging at least to centos folks due to keeping pkgs01.stg in sync via
>  repospanner.
>
>  One thing I have done a few times that I think helped a LOT as far as
>  time is to actually just apply updates to everything before the reboot
>  cycle. This saved us all the time waiting for updates to apply (which
>  can sometimes on some machines take a really long time). Of course this
>  means that machines run for a time with the updates, but with no
>  restarted processes.
>
>  I am not sure we will easily be able to seperate out what 'manual'
>  playbooks to run for what servers, and additionally in most cases
>  updates on our apps are done outside our updates cycles (ie, pagure
>  would update to 5.4 manually when it's out/desired, we wouldn't expect a
>  pending update when we do our normal OS update cycles)
>
>  So some radical ideas:
>
>  * What if we just daily auto-apply all updates. (We already do daily
>  apply security updates on fedora instances). This would break things
>  from time to time, but I suspect only particular things, not everything
>  all at once. We would also still need reboot cycles.
>
>  * What if we just daily auto-apply security updates? (This would reduce
>  the breakage from all updates some). Reboots still needed.
>
>  * I thought of the idea someday of doing reboots and having no outage.
>  Unfortunately, that requires database clustering. At the time the
>  clustering was all horrible, but it might be better these days. If we
>  did have that however, we could just do reboots when we liked.
>
>  I'm not sure there's a great answer here... I think when we decide to do
>  a update/reboot cycle we should apply updates up front to save time/pain
>  and suppose if we can easily determine what manual playbooks to run we
>  could run those too.
>
>  kevin
>
>
>
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...


-- 
Stephen J Smoogen.

[Kevin Fenzi]

On 4/5/19 10:34 AM, Stephen John Smoogen wrote:
>  On Fri, 5 Apr 2019 at 13:15, Kevin Fenzi <kevin(a)scrye.com&gt;
> wrote: 
>  So I was leaving the definitions vague as some build boxes are high
>  priority (no redundancy or major outage) and some are low priority
>  because they have a large amount of redundancy. I was figuring 
Yeah, but you still need to notify people about those, and the audience
is different, so you will need to notify everyone and mention the
specific services. ;(

>  
>  low priority:
>  - proxies not shared on high priority external virthosts
>  - builders and other high redundancy build systems
>  - openshift systems IF updated and drained properly
>  - other external services which have low SLE 
Yeah, many or all of these could be done anytime, users shouldn't notice
an outage (prodixes can be taken out of dns, builders disabled,
openshift can migrate things around), etc.

>  high priority:
>  services with no redundancy:
>  - databases
>  - pagure
>  - src
>  - etc etc
>  services with high outage effects
>  - koji?
>  - etc etc 
Yeah, although even here... for example, we can switch to koji02 and
reboot koji01 without people really noticing. Of course kojidb is
another matter.

kevin

[Stephen John Smoogen]

On Sun, 7 Apr 2019 at 13:44, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  On 4/5/19 10:34 AM, Stephen John Smoogen wrote:
>  > On Fri, 5 Apr 2019 at 13:15, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>
>  > So I was leaving the definitions vague as some build boxes are high
>  > priority (no redundancy or major outage) and some are low priority
>  > because they have a large amount of redundancy. I was figuring
>
>  Yeah, but you still need to notify people about those, and the audience
>  is different, so you will need to notify everyone and mention the
>  specific services. ;(
>  
Yes. That should still be part of the process. I should have had that.

>  >
>  > low priority:
>  > - proxies not shared on high priority external virthosts
>  > - builders and other high redundancy build systems
>  > - openshift systems IF updated and drained properly
>  > - other external services which have low SLE
>
>  Yeah, many or all of these could be done anytime, users shouldn't notice
>  an outage (prodixes can be taken out of dns, builders disabled,
>  openshift can migrate things around), etc.
>  
If there is one thing I have noticed in the last 10 years.. is that
someone always notices... and probably had something Important(TM)
that this broke and want to make sure we notify.  Even weekend reboots
of builders seem to get some module build to fail and an angry
developer comment in #fedora-devel (at least the last 3 reboot
cycles). The proxies get similar issues because even if we pull it out
of DNS.. we don't control everyone else who have DNS servers expecting
24 hours before stuff goes away.


>  > high priority:
>  > services with no redundancy:
>  > - databases
>  > - pagure
>  > - src
>  > - etc etc
>  > services with high outage effects
>  > - koji?
>  > - etc etc
>
>  Yeah, although even here... for example, we can switch to koji02 and
>  reboot koji01 without people really noticing. Of course kojidb is
>  another matter.
>  
That was the reason for the ?. Some things I know we can do and others
I do and find out that 02 is actually not a redundant system but used
for something else.. 'oh yeah I would rename that but was waiting
until we redesigned after the next major overhaul.' So I figure we
will need to go through each service and categorize this but we need
to do that anyway for other reasons.

>  kevin
>
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...


-- 
Stephen J Smoogen.