proposal: webserver/site configs in puppet

[seth vidal]

On Wed, 2011-06-08 at 12:44 -0400, seth vidal wrote:
> On Wed, 2011-06-08 at 08:29 -0400, Adam M. Dutko wrote:
> > It seems like the Apache "Include" directive is being overlooked.
> > Can't we have a base http/https file with essential "global
> > directives" like ServerTokens, some (not all) modules and etc. then
> > maintain a seperate set of include files for each site that has any
> > needed changes?
> > 
> > We could then use puppet to change which files are included on a per
> > host basis, no?
> 
> 
> > 
> > On Wed, 2011-06-08 at 19:24 +0530, ranjib dey wrote:
> > hi seth,
> > > I prefer to maintain the whole apache configs in debian style. Where
> > I have base apache module for minimal setup + sites-available +
> > mods-available, and individual use cases (passenger /mod_wsgi etc or
> > virtualhosts/proxypasses) are modelled as dependant modules of apache
> > where they selectively enable/disable sites or mods. Individual mods
> > are loaded by separate .load file and configured by separate .conf
> > files. Also we have separate puppet sub modules for individual mods.
> > This way i can readily assemble my node specific apache configs
> > without tampering the base puppet modules even if they need fin
> > grained customizations.
> > > 
> > 
> 
> Wanted to reply to both of these. I think this is what I'm aiming for.
> 
> I suspect our base apache configuration on all of our hosts of a single
> distro ver are shockingly similar. However, I do not believe the base
> apache config file that ships in rhel is limited enough.
> 
> So we'll probably need a minimal apache config and then a bunch of .conf
> files included in. Which is, ostensibly, what we do now, but we've
> abstracted that 2 or 3 layers more which is where the complexity is
> coming from.
> 
> so I think I'm in favor of not using the classes (or subclasses as that
> just means we end up going deeper into puppetisms when, realistically,
> we should be doing more apache-isms since that is what we're actually
> running)
> 
> If we have:
> 
>  httpd.conf
>   and a simple site-include directive in puppet to pull in a dir/.conf
> file.  
> 
> this makes it similar to how we lay things out on the systems now - but
> makes our puppet configs look more like what things do look like on the
> system. :)
> 
> thinking on this some more - seems like working on this for the rhel6
> change over for our various servers is a good idea. Instead of trying to
> force this back onto rhel5.
> 

Talking on irc and looking at infrastructure.fedoraproject.org as a
prototype helped congeal some ideas, these are a  melange from myself
and other people - I'm not taking credit for them, just posting them.


1. we modify the default httpd.conf to modify this line:

   Include conf.d/*.conf
  to
   Include fi-conf.d/*.conf

which is where we will store all of our site configs - this gets us
around the random welcome.conf issue and anyone else stepping on our
configs later.

It also means we need to make sure when we gather up things we may want
from pkgs we install that include a conf file into conf.d to actually be
included - but that's not much of a deal breaker I don't think.

2. we keep our current habit of:
    $website.fedoraproject.org.conf file and $website.fedoraproject.org
dir full of files.

3. we stop generating the items in #2 by templates.

4. under puppet/configs/httpd/websites/ we start housing the .conf files
and the dirs for items in #2.

5. those dirs and files are entirely (and recursively) deployed using a
simple module in puppet which just does that.
  
This means if we need to make an adjustment to a site we can make it to
JUST that site and not have to guess what else it might impact.

It also means if we want to make the same adjustment to several sites
that we may have to make it several times. But I believe in the power of
git-grep. :)

6. For ssl sites we will need to have a similar structure in private for
the keys/certs/etc - that will need some working out of details but it
is not terrible.



I'm going to work on a version of this for
infrastructure.fedoraproject.org and we'll go from there.

Thanks to StylusEater and mmcgrath for helping with their thoughts here.

-sv