[PATCH 0/2] templatize rsyncd.conf for master mirrors

[Matt Domsch]

Wanting some extra eyes on this, as I haven't mucked with puppet
templates that much.  This should cause the download* servers
/etc/rsyncd.conf file to get populated from a template, so it isn't a
cut-and-paste plus file copy x7 operation just to update the rsync
ACLs.

Thanks,
Matt

-- 
Matt Domsch
Technology Strategist
Dell | Office of the CTO

[PATCH 1/2] templatize rsyncd.conf for master mirrors

[Matt Domsch]

> From 77af1729ab406762cc2c66c8ca7b6043578f0ae9 Mon Sep 17 00:00:00 2001
From: Matt Domsch <mdomsch(a)fedoraproject.org&gt;
Date: Wed, 6 Jul 2011 15:52:27 +0000
Subject: [PATCH 1/2] templatize master mirror rsyncd.conf files

---
 manifests/services/download-server.pp              |    2 +-
 modules/rsync/manifests/mastermirror.pp            |   10 +++
 .../rsync/templates/rsyncd.conf.mastermirror.erb   |   83 ++++++++++++++++++++
 3 files changed, 94 insertions(+), 1 deletions(-)
 create mode 100644 modules/rsync/manifests/mastermirror.pp
 create mode 100644 modules/rsync/templates/rsyncd.conf.mastermirror.erb

diff --git a/manifests/services/download-server.pp
b/manifests/services/download-server.pp
index 9364398..1d994f7 100644
--- a/manifests/services/download-server.pp
+++ b/manifests/services/download-server.pp
@@ -9,7 +9,7 @@ class download-server::base {
   include http_log
   include httpd::proxy
   include mod_limitipconn::mod_limitipconn
-  include rsync::server
+  include rsync::mastermirror
   include nfs-utils::package
 
   collectd::collectd { 'log02': }
diff --git a/modules/rsync/manifests/mastermirror.pp
b/modules/rsync/manifests/mastermirror.pp
new file mode 100644
index 0000000..1f1e0da
--- /dev/null
+++ b/modules/rsync/manifests/mastermirror.pp
@@ -0,0 +1,10 @@
+class rsync::mastermirror inherits rsync::server {
+    $hosts_allow = "152.46.7.122 200.17.202.1/28 zeus1.kernel.org zeus2.kernel.org
zeus3.kernel.org zeus4.kernel.org 149.20.20.132 204.152.191.36 199.6.1.170 130.239.17.3
sinclair.wpi.edu bonaparte.hrz.tu-chemnitz.de josephine.hrz.tu-che mirror.speedpartner.de
rsyncer.ftp.heanet.ie archive.linux.duke.edu lists.us.dell.com auslistsprd01.us.dell.com
auslistsdr01.us.dell.com 65.18.175.19 198.129.224.34 mirror.hiwaay.net sagres.c3sl.ufpr.br
mail.fedoraunity.org scrye.com odysseus.fi.muni.cz rhlx01.hs-esslingen.de"
+  
+    file { "/etc/rsyncd.conf":
+        content =>
template("puppet:///rsync/rsyncd.conf.mastermirror.erb"),
+        mode    => 0644,
+        require => Package["rsync"],
+    }
+}
+
diff --git a/modules/rsync/templates/rsyncd.conf.mastermirror.erb
b/modules/rsync/templates/rsyncd.conf.mastermirror.erb
new file mode 100644
index 0000000..d55256d
--- /dev/null
+++ b/modules/rsync/templates/rsyncd.conf.mastermirror.erb
@@ -0,0 +1,83 @@
+pid file = /var/run/rsyncd.pid
+syslog facility = daemon
+max connections = 25
+timeout = 600
+use chroot = yes
+uid = nobody
+gid = nobody
+#transfer logging = false
+motd file = /etc/motd_fedora
+log file = /var/log/rsyncd-fedora.log
+
+dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.bz2 *.iso *.ogg *.ogv *.tbz
+exclude = .snapshot/ .~tmp~/ /.private/ /.private/** **/.nfs*
+ignore nonreadable = yes
+list = true
+read only = yes
+refuse options = checksum
+
+[ fedora-alt ]
+        comment = non-Fedora Alternative Content
+        path = /srv/pub/alt
+
+[fedora-archive]
+        comment = Fedora Release Archives
+        path = /srv/pub/archive
+     
+[ fedora-enchilada ]
+        comment = Fedora - The whole enchilada
+        path = /srv/pub/fedora
+
+[ fedora-epel ]
+        comment = Extra Packages for Enterprise Linux
+        path = /srv/pub/epel
+
+[ fedora-linux-releases ]
+        comment = Fedora Linux Releases
+        path = /srv/pub/fedora/linux/releases
+
+[ fedora-linux-development ]
+        comment = Fedora Linux Development
+        path = /srv/pub/fedora/linux/development
+
+[ fedora-linux-updates ]
+        comment = Fedora Linux Updates
+        path = /srv/pub/fedora/linux/updates
+
+[fedora-secondary]
+        comment = Fedora Secondary Archs
+        path = /srv/pub/fedora-secondary
+
+[ fedora-stage ]
+        comment = Staging directory
+	path = /srv/pub/alt/stage/
+
+##
+## The following are not seen and are limited by IP.
+##
+
+[fedora-enchilada0]
+       comment = Fedora Enchilada for Tier0|1 Mirrors
+       path = /srv/pub/fedora/
+       list = no
+       uid = 263
+       gid = 263
+       hosts allow = <%= hosts_allow %>
+
+[fedora-epel0]
+       comment = Fedora EPEL for Tier0|1 Mirrors
+       path = /srv/pub/epel/
+       list = no
+       uid = 263
+       gid = 263
+       hosts allow = <%= hosts_allow %>
+
+# For distributing applications
+[log]
+       comment = Server Logs
+       path = /var/log
+       uid = root
+       gid = root
+       read only = yes
+       hosts allow = 10.5.126.29
+       list = no
-- 
1.7.2.1


-- 
Matt Domsch
Technology Strategist
Dell | Office of the CTO

[PATCH 2/2] templatize rsyncd.conf for master mirrors

[Matt Domsch]

> From 4a42171734609da4c1e57012e64f9004ccd6f704 Mon Sep 17 00:00:00 2001
From: Matt Domsch <mdomsch(a)fedoraproject.org&gt;
Date: Wed, 6 Jul 2011 15:54:07 +0000
Subject: [PATCH 2/2] fix rsync mastermirror template syntax

---
 modules/rsync/manifests/mastermirror.pp |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/modules/rsync/manifests/mastermirror.pp
b/modules/rsync/manifests/mastermirror.pp
index 1f1e0da..a33d990 100644
--- a/modules/rsync/manifests/mastermirror.pp
+++ b/modules/rsync/manifests/mastermirror.pp
@@ -2,7 +2,7 @@ class rsync::mastermirror inherits rsync::server {
     $hosts_allow = "152.46.7.122 200.17.202.1/28 zeus1.kernel.org zeus2.kernel.org
zeus3.kernel.org zeus4.kernel.org 149.20.20.132 204.152.191.36 199.6.1.170 130.239.17.3
sinclair.wpi.edu bonaparte.hrz.tu-chemnitz.de josephine.hrz.tu-che mirror.speedpartner.de
rsyncer.ftp.heanet.ie archive.linux.duke.edu lists.us.dell.com auslistsprd01.us.dell.com
auslistsdr01.us.dell.com 65.18.175.19 198.129.224.34 mirror.hiwaay.net sagres.c3sl.ufpr.br
mail.fedoraunity.org scrye.com odysseus.fi.muni.cz rhlx01.hs-esslingen.de"
   
     file { "/etc/rsyncd.conf":
-        content =>
template("puppet:///rsync/rsyncd.conf.mastermirror.erb"),
+        content => template("rsync/rsyncd.conf.mastermirror.erb"),
         mode    => 0644,
         require => Package["rsync"],
     }
-- 
1.7.2.1


-- 
Matt Domsch
Technology Strategist
Dell | Office of the CTO

[PATCH 2/2] templatize rsyncd.conf for master mirrors

[Seth Vidal]

On Wed, 2011-07-06 at 11:02 -0500, Matt Domsch wrote:
>  >From 4a42171734609da4c1e57012e64f9004ccd6f704 Mon Sep 17 00:00:00
> 2001
>  From: Matt Domsch <mdomsch(a)fedoraproject.org&gt;
>  Date: Wed, 6 Jul 2011 15:54:07 +0000
>  Subject: [PATCH 2/2] fix rsync mastermirror template syntax
>  
>  ---
>   modules/rsync/manifests/mastermirror.pp |    2 +-
>   1 files changed, 1 insertions(+), 1 deletions(-)
>  
>  diff --git a/modules/rsync/manifests/mastermirror.pp
> b/modules/rsync/manifests/mastermirror.pp
>  index 1f1e0da..a33d990 100644
>  --- a/modules/rsync/manifests/mastermirror.pp
>  +++ b/modules/rsync/manifests/mastermirror.pp
>  @@ -2,7 +2,7 @@ class rsync::mastermirror inherits rsync::server {
>       $hosts_allow = "152.46.7.122 200.17.202.1/28 zeus1.kernel.org zeus2.kernel.org
> zeus3.kernel.org zeus4.kernel.org 149.20.20.132 204.152.191.36 199.6.1.170 130.239.17.3
> sinclair.wpi.edu bonaparte.hrz.tu-chemnitz.de josephine.hrz.tu-che mirror.speedpartner.de
> rsyncer.ftp.heanet.ie archive.linux.duke.edu lists.us.dell.com auslistsprd01.us.dell.com
> auslistsdr01.us.dell.com 65.18.175.19 198.129.224.34 mirror.hiwaay.net sagres.c3sl.ufpr.br
> mail.fedoraunity.org scrye.com odysseus.fi.muni.cz rhlx01.hs-esslingen.de"
>     
>       file { "/etc/rsyncd.conf":
>  -        content =>
> template("puppet:///rsync/rsyncd.conf.mastermirror.erb"),
>  +        content => template("rsync/rsyncd.conf.mastermirror.erb"),
>           mode    => 0644,
>           require => Package["rsync"],
>       } 
Argh.
 why are we using templates for this? Why not just use
rsyncd.conf.$hostname or rsyncd.conf.$group_of_some_kind?


it's WAY easier to read and we don't have to do this
templating-necromancy to figure out what we're going to break later.

-sv

[PATCH 2/2] templatize rsyncd.conf for master mirrors

[Matt Domsch]

On Wed, Jul 06, 2011 at 11:08:14AM -0500, seth vidal wrote:
>  On Wed, 2011-07-06 at 11:02 -0500, Matt Domsch wrote:
>  > >From 4a42171734609da4c1e57012e64f9004ccd6f704 Mon Sep 17 00:00:00 2001
>  > From: Matt Domsch <mdomsch(a)fedoraproject.org&gt;
>  > Date: Wed, 6 Jul 2011 15:54:07 +0000
>  > Subject: [PATCH 2/2] fix rsync mastermirror template syntax
>  > 
>  > ---
>  >  modules/rsync/manifests/mastermirror.pp |    2 +-
>  >  1 files changed, 1 insertions(+), 1 deletions(-)
>  > 
>  > diff --git a/modules/rsync/manifests/mastermirror.pp
> b/modules/rsync/manifests/mastermirror.pp
>  > index 1f1e0da..a33d990 100644
>  > --- a/modules/rsync/manifests/mastermirror.pp
>  > +++ b/modules/rsync/manifests/mastermirror.pp
>  > @@ -2,7 +2,7 @@ class rsync::mastermirror inherits rsync::server {
>  >      $hosts_allow = "152.46.7.122 200.17.202.1/28 zeus1.kernel.org
> zeus2.kernel.org zeus3.kernel.org zeus4.kernel.org 149.20.20.132 204.152.191.36
> 199.6.1.170 130.239.17.3 sinclair.wpi.edu bonaparte.hrz.tu-chemnitz.de
> josephine.hrz.tu-che mirror.speedpartner.de rsyncer.ftp.heanet.ie archive.linux.duke.edu
> lists.us.dell.com auslistsprd01.us.dell.com auslistsdr01.us.dell.com 65.18.175.19
> 198.129.224.34 mirror.hiwaay.net sagres.c3sl.ufpr.br mail.fedoraunity.org scrye.com
> odysseus.fi.muni.cz rhlx01.hs-esslingen.de"
>  >    
>  >      file { "/etc/rsyncd.conf":
>  > -        content =>
> template("puppet:///rsync/rsyncd.conf.mastermirror.erb"),
>  > +        content => template("rsync/rsyncd.conf.mastermirror.erb"),
>  >          mode    => 0644,
>  >          require => Package["rsync"],
>  >      }
>  
>  Argh.
>   why are we using templates for this? Why not just use
>  rsyncd.conf.$hostname or rsyncd.conf.$group_of_some_kind?
>  
>  
>  it's WAY easier to read and we don't have to do this
>  templating-necromancy to figure out what we're going to break later. 

What we have now is awful.  edit rsyncd.conf.download0 in 2 places.
Copy that file to rsyncd.conf.download0[2345] and
rsyncd.conf.download-rdu0[123].  Commit same change to all files.


Could do $group_of_some_kind w/o templates I suppose.  Still leaves
the editing in 2 places (which I've forgotten to do the second one as
recently as yesterday) though.  The template makes that one edit.


-- 
Matt Domsch
Technology Strategist
Dell | Office of the CTO

[PATCH 2/2] templatize rsyncd.conf for master mirrors

[Seth Vidal]

On Wed, 2011-07-06 at 11:12 -0500, Matt Domsch wrote:

>  What we have now is awful.  edit rsyncd.conf.download0 in 2 places.
>  Copy that file to rsyncd.conf.download0[2345] and
>  rsyncd.conf.download-rdu0[123].  Commit same change to all files.
>  
>  
>  Could do $group_of_some_kind w/o templates I suppose.  Still leaves
>  the editing in 2 places (which I've forgotten to do the second one as
>  recently as yesterday) though.  The template makes that one edit. 
one edit which can break places you didn't intend.

It's the fundamental problem with our templates we end up not knowing
what else is going to be impacted by the changes.

Our current rsyncd.* layout is not good, I agree. but this is why we
have

$datacenter
$hostname
$arbitrary_variable_we_can_use

and give those in any order we want for the source list for the files.

I'm strongly against using the templates.

I'd rather have a set of files I can more easily grep for the changes
than have to know what the template definition might include from
elsewhere.

-sv