bastion03 outage this afternoon

[Kevin Fenzi]

We had an outage of bastion03 this afternoon.

Because it's the vpn hub, lots of services were affected.

It looks like it's running into
https://bugzilla.redhat.com/show_bug.cgi?id=725332

So, if this re-occurs, we should either:

a) switch it to use e1000 instead of virtio network devices.

or

b) switch back to bastion02 (which is still there and rhel5).

So, keep an eye out for outages of services where you see a lot of ping
loss or not answer at all from bastion.fedoraproject.org.

For the record, the outage started at 22:45 and ran until 22:55 or so.

kevin

[Tristan Santore]

On 14/09/11 00:32, Kevin Fenzi wrote:
> We had an outage of bastion03 this afternoon.
>
> Because it's the vpn hub, lots of services were affected.
>
> It looks like it's running into
> https://bugzilla.redhat.com/show_bug.cgi?id=725332
>
> So, if this re-occurs, we should either:
>
> a) switch it to use e1000 instead of virtio network devices.
>
> or
>
> b) switch back to bastion02 (which is still there and rhel5).
>
> So, keep an eye out for outages of services where you see a lot of ping
> loss or not answer at all from bastion.fedoraproject.org.
>
> For the record, the outage started at 22:45 and ran until 22:55 or so.
>
> kevin
>
> infrastructure mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/infrastructure

How about creating a failover VPN, which is doable in OpenVPN.

Regards,

Tristan

[Seth Vidal]

On Wed, 2011-09-14 at 06:01 +0100, Tristan Santore wrote:
> How about creating a failover VPN, which is doable in OpenVPN.

After being woken up this morning  by pages I'm looking into just that.

I'm trying to figure out why we've been doing cold-failover all this
time when it seems to handle hot failover just fine.

-sv

[Kevin Fenzi]

On Wed, 14 Sep 2011 10:18:28 -0400
seth vidal  wrote:
> On Wed, 2011-09-14 at 06:01 +0100, Tristan Santore wrote:
> > How about creating a failover VPN, which is doable in OpenVPN.
>
> After being woken up this morning  by pages I'm looking into just
> that.
>
> I'm trying to figure out why we've been doing cold-failover all this
> time when it seems to handle hot failover just fine.

It may be possible, but currently only the active node runs the openvpn
server. This is to avoid the case of some machines on one and some on
the other both using the same network. ;(

I think we can improve the setup here, but it's not as simple as
running the server on the secondary node. ;(

kevin