Mailing lists http and smtp

[Kevin Fenzi]

Greetings. 

So, we are getting closer to rolling out some hyperkitty goodness for
production. Now we need to determine how we want to manage the rollout
from a techincal angle. 

First, some history/description of how our current lists work:

HTTP/HTTPS:

Requests to admin.fedoraproject.org/mailman are proxied from any proxy
to collab03 (located at osuosl).

lists.fedoraproject.org is just a A record directly pointing to
collab03, so requests go direct to it. 

SMTP:

incoming: 

lists.fedoraproject.org has MX for our 3 smtp-mm machines. They are
located in 3 places around the globe. They accept emails for lists and
relay them to collab03. If collab03 is down or they can't reach it,
they simply queue and wait. 

outgoing: 

collab03 relays everything to bastion01. bastion01 sends the emails
out. This I think is to make sure all *.fedoraproject.org emails come
from the same ip address and don't shift around. 

(Note that the path for non lists emails is a bit different). 

So, my hyperkitty migration plan: 

1 Figure out how to get http and smtp to it with NO disruption of
  existing urls and emails. 
2 Setup test list and do some testing.
3 Once that looks good, move on to migrating a list over (this list
  perhaps :) 
4 Then, migrate a bunch more opt-in. 
5 Then, set a sunset date on the old setup and migrate the rest. 

Options for item 1: 

1) Setup forwards on collab03 for any lists going to mm3 and setup
httpd redirects for admin.fedoraproject.org/mailman/*listname* and
lists.fedoraproject.org/mailman/*listname*.

This means a lot of monkeying with the config and apache redirects are
sometimes very fragile. 

2. Make a new 'mailman3.fedoraproject.org' hostname, set smtp-mm's to
accept for it and send on to mailman01/02. Setup proxies to proxy
mailman3 into mailman01/02. 

This makes it nice and seperate, but means list names change? 

3) Hybrid: Setup mailman3.fedoraproject.org for the web interface only,
use forwards for the smtp side. 

This means list names could stay the same, but web interface is
different. Might confuse people where a list web interface is. 

4) Your brilliant plan here. ;) 

Thoughts?

kevin

[Ankur Sinha]

On Mon, 2014-02-03 at 11:42 -0700, Kevin Fenzi wrote:
>  3 Once that looks good, move on to migrating a list over (this list
>    perhaps :)  
Maybe move a list that receives less activity and is less critical to
the working of the community, like the fedora-join list? 

The infra list is far too important to pick for testing :P
-- 
Thanks,
Warm regards,
Ankur (FranciscoD)

http://fedoraproject.org/wiki/User:Ankursinha

Join Fedora! Come talk to us!
http://fedoraproject.org/wiki/Fedora_Join_SIG

[Jon Stanley]

On Mon, Feb 3, 2014 at 7:34 PM, Ankur Sinha <sanjay.ankur(a)gmail.com&gt; wrote:

>  Maybe move a list that receives less activity and is less critical
> to
>  the working of the community, like the fedora-join list?
>
>  The infra list is far too important to pick for testing :P 
I respectfully disagree - the infra team has another well-known way to
communicate (IRC) if something goes incredibly south. Not so sure that
the traffic on other lists could easily be redirected to other
channels like that.

$0.02

[Jon Stanley]

On Mon, Feb 3, 2014 at 1:42 PM, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:

>  1) Setup forwards on collab03 for any lists going to mm3 and setup
>  httpd redirects for admin.fedoraproject.org/mailman/*listname* and
>  lists.fedoraproject.org/mailman/*listname*. 
This seems the most sane (and the most work) to me of the options. I
think that the goal should be to minimize user impact as much as is
humanly possible. The other two options just seem somewhat unappealing
to me.

[Aurelien Bompard]

> > 1) Setup forwards on collab03 for any lists going to mm3 and
> setup
> >  httpd redirects for admin.fedoraproject.org/mailman/*listname* and
> >  lists.fedoraproject.org/mailman/*listname*.
>
>  This seems the most sane (and the most work) to me of the options. I
>   think that the goal should be to minimize user impact as much as is
>   humanly possible. The other two options just seem somewhat
>  unappealing to me. 
I agree with this. If we have to get a bit clever with redirects to
provide this continuity, well, so be it.
We (I) can even write a small basic WSGI script if things become too
unmaintainable with Apache redirects only.

Aurélien

http://aurelien.bompard.org    ~~~~~~    xmpp:[e-mail]
"The only function of economic forecasting is to make astrology
look respectable." -- John Kenneth Galbraith, economist

[Aurelien Bompard]

> > 1) Setup forwards on collab03 for any lists going to mm3 and
> setup
> >  httpd redirects for admin.fedoraproject.org/mailman/*listname* and
> >  lists.fedoraproject.org/mailman/*listname*.
>
>  This seems the most sane (and the most work) to me of the options. I
>   think that the goal should be to minimize user impact as much as is
>   humanly possible. The other two options just seem somewhat
>  unappealing to me. 
I agree with this. If we have to get a bit clever with redirects to
provide this continuity, well, so be it.
We (I) can even write a small basic WSGI script if things become too
unmaintainable with Apache redirects only.

Aurélien
-- 
http://aurelien.bompard.org    ~~~~~~    xmpp:[e-mail]
"The only function of economic forecasting is to make astrology
look respectable." -- John Kenneth Galbraith, economist

[Kevin Fenzi]

On Wed, 05 Feb 2014 09:50:02 +0100
Aurélien Bompard <gauret(a)free.fr&gt; wrote:

>  
>  >> 1) Setup forwards on collab03 for any lists going to mm3 and setup
>  >>  httpd redirects for admin.fedoraproject.org/mailman/*listname* and
>  >>  lists.fedoraproject.org/mailman/*listname*.
>  >
>  > This seems the most sane (and the most work) to me of the options. I
>  >  think that the goal should be to minimize user impact as much as is
>  >  humanly possible. The other two options just seem somewhat
>  > unappealing to me.
>  
>  I agree with this. If we have to get a bit clever with redirects to
>  provide this continuity, well, so be it.
>  We (I) can even write a small basic WSGI script if things become too
>  unmaintainable with Apache redirects only. 
Alright. 

Sounds like this is the way to go then... 

So, next steps: 

* Create some kind of test list on the prod instances. 

* Put in place aliases on collab03 -> mailman01/02 for smtp.
I guess we need to point at one? Or do MXes?

* Setup apache redirect for
  lists.fedoraproject.org/mailman/listinfo/testlist -> mailman01/02

* Test a bunch. ;) 

Let me know what of this I can help out with... 

kevin

[Aurelien Bompard]

>  So, next steps:
>  * Create some kind of test list on the prod instances.
>  * Put in place aliases on collab03 -> mailman01/02 for smtp.
>  I guess we need to point at one? Or do MXes?
>  * Setup apache redirect for
>    lists.fedoraproject.org/mailman/listinfo/testlist -> mailman01/02
>  * Test a bunch. ;) 
Alright, the two prod servers are setup, I've created a test list
which answers to test(a)lists.fedoraproject.org on them, so we can now
put redirects in place from collab03 to mailman01, in SMTP and HTTP.
We can start with mailman01 only for HTTP, I just realized that the
full-text index is file-based, so it needs to be on a shared
filesystem for mailman02 to access it. But that does not prevent us
from testing the rest.

Can I just have access to collab03 to setup the redirects, or does
someone with a better knowledge of the whole picture want to setup
them? (it's prod, after all)

Aurélien

[Kevin Fenzi]

On Mon, 24 Feb 2014 10:45:09 +0100
Aurélien Bompard <gauret(a)free.fr&gt; wrote:

>  > So, next steps:
>  > * Create some kind of test list on the prod instances.
>  > * Put in place aliases on collab03 -> mailman01/02 for smtp.
>  > I guess we need to point at one? Or do MXes?
>  > * Setup apache redirect for
>  >   lists.fedoraproject.org/mailman/listinfo/testlist -> mailman01/02
>  > * Test a bunch. ;)
>  
>  Alright, the two prod servers are setup, I've created a test list
>  which answers to test(a)lists.fedoraproject.org on them,  
Amusingly, thats already the name of our qa list. ;) 

Could you rename something like 'mailman3' or another random name thats
not taken? 

>  so we can now
>  put redirects in place from collab03 to mailman01, in SMTP and HTTP. 
Cool.

>  We can start with mailman01 only for HTTP, I just realized that the
>  full-text index is file-based, so it needs to be on a shared
>  filesystem for mailman02 to access it. But that does not prevent us
>  from testing the rest. 
ok. We do now have a gluster role (thanks Ralph!) So we could spin up
gluster between them. 

What all data needs to be shared again? Perhaps this is better a netapp
volume... 
 
>  Can I just have access to collab03 to setup the redirects, or does
>  someone with a better knowledge of the whole picture want to setup
>  them? (it's prod, after all) 
I'd be happy to help, but I also added you to sysadmin-tools a while
back so you should have sudo on collab03. 

I guess we need to add aliases for the testing list that send it all on
to mailman01.vpn and then tweak apache config to redirect the web side?

kevin

[Aurelien Bompard]

>  Amusingly, thats already the name of our qa list. ;)
>  Could you rename something like 'mailman3' or another random name thats
>  not taken? 
Haha, I knew I should have checked ;-)
OK, I created a new test-mailman3 list on this host, will remove the
"test" one later (before we migrate the qa list obviously)

>  ok. We do now have a gluster role (thanks Ralph!) So we could spin
> up
>  gluster between them. 
Cool.

>  What all data needs to be shared again? Perhaps this is better a
> netapp
>  volume... 
It's around 4GB right now.

>  I'd be happy to help, but I also added you to sysadmin-tools a
> while
>  back so you should have sudo on collab03. 
Nice, thanks.

>  I guess we need to add aliases for the testing list that send it all
> on
>  to mailman01.vpn and then tweak apache config to redirect the web side? 
Yes, that's the plan. What's mailman01.vpn ? Is it an alias for
mailman01.phx2 ? A local DNS resolution does not find it.

Aurélien

[Aurelien Bompard]

>  Yes, that's the plan. What's mailman01.vpn ? Is it an alias
> for
>  mailman01.phx2 ? A local DNS resolution does not find it. 
Hmm, I can't find a way for collab03 to reach mailman01.phx2, any
special hostname or IP address I should use?
I admit our network architecture is still a bit unclear to me (I did
read http://fedoraproject.org/wiki/Infrastructure/Architecture).

Aurélien

[Kevin Fenzi]

On Tue, 25 Feb 2014 18:09:02 +0100
Aurélien Bompard <gauret(a)free.fr&gt; wrote:

>  > Yes, that's the plan. What's mailman01.vpn ? Is it an
> alias for
>  > mailman01.phx2 ? A local DNS resolution does not find it.
>  
>  Hmm, I can't find a way for collab03 to reach mailman01.phx2, any
>  special hostname or IP address I should use?
>  I admit our network architecture is still a bit unclear to me (I did
>  read http://fedoraproject.org/wiki/Infrastructure/Architecture). 
They need to be on the vpn. ;) 

I have added them there and they should be reachable soon as
'mailman01.vpn.fedoraproject.org' and 'mailman02.vpn.fedoraproject.org'.

kevin

[Matthew Miller]

On Mon, Feb 03, 2014 at 11:42:54AM -0700, Kevin Fenzi wrote:
>  3 Once that looks good, move on to migrating a list over (this list
>    perhaps :)  
As mentioned before, I'm eager to offer up the cloud SIG mailing list,
although now that it's more active I suppose I should check with the other
regulars. Let me know when it's a good time to ask. :)


-- 
Matthew Miller    --   Fedora Project    --    <mattdm(a)fedoraproject.org&gt;

[Matthew Miller]

On Mon, Feb 03, 2014 at 11:42:54AM -0700, Kevin Fenzi wrote:
>  3) Hybrid: Setup mailman3.fedoraproject.org for the web interface
> only,
>  use forwards for the smtp side.  
I think it's a good idea to avoid particular technology names in hostnames.
Maybe it's okay as a temporary measure, but temporary becomes permanent very
quickly. And, as I have high hopes for this being a new, friendly face which
will be most users' and developers' _primarly_ interface to the project...
it seems sad to put it off on technicalname####.fedoraproject.org.


-- 
Matthew Miller    --   Fedora Project    --    <mattdm(a)fedoraproject.org&gt;