Re: Freeze Break Request: fix iscsi/mpath

[Stephen John Smoogen]

This looks sensible. If the outage causes further issues we can do
'drastic' action then.

On 21 March 2017 at 16:52, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  Greetings.
>
>  This thursday (2017-03-23) there's going to be some upgrades done on our
>  netapp storage. These upgrades should not cause any outages if
>  everything was working as expected, but there's an issue: A number of
>  our machines are logged into iscsi luns, but not properly multipathing
>  that connection, which means that those machines could see an outage to
>  the iscsi storage since they don't have the redundant paths.
>
>  However, we are not using iscsi as widely as we used to either, so a
>  number of these machines are logged in, but don't otherwise use the
>  storage at all.
>
>  So, I'd like to get a freeze break to do the following:
>
>  1. Wait until our f26rc2 is composed (which might mean waiting until
>  tomorrow).
>
>  2. Apply the following ansible patch:
>
>  diff --git a/playbooks/groups/virthost.yml b/playbooks/groups/virthost.yml
>  index e3d673f..e53d540 100644
>  --- a/playbooks/groups/virthost.yml
>  +++ b/playbooks/groups/virthost.yml
>  @@ -22,7 +22,7 @@
>     - hosts
>     - fas_client
>     - collectd/base
>  -  - { role: iscsi_client, when: datacenter == "phx2" }
>  +  - { role: iscsi_client, when:
>  inventory_hostname.startswith(('bvirthost','buildvmhost')) }
>     - sudo
>     - { role: openvpn/client, when: datacenter != "phx2" }
>     - virthost
>
>  This will restrict the iscsi client setup to only bvirthost and
>  buildvmhost machines. Those are the only ones that _should_ be using
>  this storage (but see below):
>
>  3. Run ansible commands over the non bvirthost/buildvmhost machines to
>  do a 'iscsi logout', disable iscsi and multipathd services and remove
>  the /var/lib/iscsi/nodes/* files.
>
>  4. Re-provision docker-registry02.stg.phx2.fedoraproject.org and
>  loopabull01.stg.phx2.fedoraproject.org that were provisioned on the
>  iscsi storage and make them use local storage.
>
>  5. Sadly, that leaves bvirthost01/02/03 still not working right.
>  However, they use 0 things from the iscsi storage, so we can fix them
>  after freeze.
>
>  That should hopefully do it. ;)
>
>  +1s?
>
>  kevin
>
>
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
>  


-- 
Stephen J Smoogen.