puppet update broken?

[Dennis Jacobfeuerborn]

Hi,
I just installed a new machine in our infrastructure but I can't get the 
puppet package to work. I see it has recently been updated to 0.25.4 while 
the previous version i EPEL was 0.24.8. The other machines that are still 
running 0.24.8 are all doing fine but on the one with the new puppet 
version I just get a "Reopening log files" in the syslog. On the other 
machines this is followed by "Starting Puppet client version 0.24.8" but on 
the new one nothing happens.
Starting puppet manually with "puppetd --no-damonize -d" looks good but 
ends with the following output:
...
debug: /File[/var/lib/puppet/ssl/public_keys]/ensure: created
debug: /File[/var/lib/puppet/ssl/private]: Changing ensure
debug: /File[/var/lib/puppet/ssl/private]: 1 change(s)
debug: /File[/var/lib/puppet/ssl/private]/ensure: created
debug: /File[/var/lib/puppet/ssl/certs]: Changing ensure
debug: /File[/var/lib/puppet/ssl/certs]: 1 change(s)
debug: /File[/var/lib/puppet/ssl/certs]/ensure: created
debug: Finishing transaction 23882872773280 with 12 changes
info: Creating a new SSL key for wsuptf01.tmm.cvsn.de
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
info: Creating a new SSL certificate request for wsuptf01.tmm.cvsn.de
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session

After that nothing happens.

I also noticed that the previous version has been deleted on the EPEL 
servers. Was this intended? That makes any rollback to a previous version 
impossible in case someone updates his machines and runs into similar problems.

Regards,
   Dennis

[Dennis Jacobfeuerborn]

I found a copy of the previous version on the net. After installing that 
everything is fine again. That still leaves the question why the EPEL 
version was updated to an apparently incompatible revision.

Regards,
   Dennis

On 03/12/2010 10:47 PM, Dennis J. wrote:
>  Hi,
>  I just installed a new machine in our infrastructure but I can't get the
>  puppet package to work. I see it has recently been updated to 0.25.4
>  while the previous version i EPEL was 0.24.8. The other machines that
>  are still running 0.24.8 are all doing fine but on the one with the new
>  puppet version I just get a "Reopening log files" in the syslog. On the
>  other machines this is followed by "Starting Puppet client version
>  0.24.8" but on the new one nothing happens.
>  Starting puppet manually with "puppetd --no-damonize -d" looks good but
>  ends with the following output:
>  ...
>  debug: /File[/var/lib/puppet/ssl/public_keys]/ensure: created
>  debug: /File[/var/lib/puppet/ssl/private]: Changing ensure
>  debug: /File[/var/lib/puppet/ssl/private]: 1 change(s)
>  debug: /File[/var/lib/puppet/ssl/private]/ensure: created
>  debug: /File[/var/lib/puppet/ssl/certs]: Changing ensure
>  debug: /File[/var/lib/puppet/ssl/certs]: 1 change(s)
>  debug: /File[/var/lib/puppet/ssl/certs]/ensure: created
>  debug: Finishing transaction 23882872773280 with 12 changes
>  info: Creating a new SSL key for wsuptf01.tmm.cvsn.de
>  warning: peer certificate won't be verified in this SSL session
>  warning: peer certificate won't be verified in this SSL session
>  info: Creating a new SSL certificate request for wsuptf01.tmm.cvsn.de
>  warning: peer certificate won't be verified in this SSL session
>  warning: peer certificate won't be verified in this SSL session
>  warning: peer certificate won't be verified in this SSL session
>
>  After that nothing happens.
>
>  I also noticed that the previous version has been deleted on the EPEL
>  servers. Was this intended? That makes any rollback to a previous
>  version impossible in case someone updates his machines and runs into
>  similar problems.
>
>  Regards,
>  Dennis
>
>  _______________________________________________
>  epel-devel-list mailing list
>  epel-devel-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/epel-devel-list

[Ricky Zhou]

On 2010-03-12 11:13:19 PM, Dennis J. wrote:
>  I found a copy of the previous version on the net. After installing
>  that everything is fine again. That still leaves the question why
>  the EPEL version was updated to an apparently incompatible revision. 
The server is
usually backwards-compatible, but not the client.  The 
advice from the puppet folks is to always update the server first.  
Still, it would have been a good idea to send an announcement to 
epel-announce about this update earlier.

Thanks,
Ricky

[Stephen John Smoogen]

On Fri, Mar 12, 2010 at 2:47 PM, Dennis J. <dennisml(a)conversis.de&gt; wrote:
>  Hi,
>  I just installed a new machine in our infrastructure but I can't get the
>  puppet package to work. I see it has recently been updated to 0.25.4 while
>  the previous version i EPEL was 0.24.8. The other machines that are still
>  running 0.24.8 are all doing fine but on the one with the new puppet version
>  I just get a "Reopening log files" in the syslog. On the other machines this
>  is followed by "Starting Puppet client version 0.24.8" but on the new one
>  nothing happens. 
A bad part of this our fault. We need to send out FLAG days on certain
packages (puppet being one of them) when an update goes into testing
and when it goes into production.

A second bad part is we do not keep many old versions of our packages
around due to limits on resources on our part.

Part of this is due to how puppet upstream works. Puppet only gives a
partial guarantee for backwards compatibility between 'major' version
upgrades (and only on the server), and the puppet recommended way of
upgrades is always:

Upgrade the server first, Upgrade the clients next.



-- 
Stephen J Smoogen.

Ah, but a man's reach should exceed his grasp. Or what's a heaven for?
-- Robert Browning

[Dennis Jacobfeuerborn]

On 03/12/2010 11:37 PM, Stephen John Smoogen wrote:
>  On Fri, Mar 12, 2010 at 2:47 PM, Dennis
> J.<dennisml(a)conversis.de>  wrote:
> > Hi,
> > I just installed a new machine in our infrastructure but I can't get the
> > puppet package to work. I see it has recently been updated to 0.25.4 while
> > the previous version i EPEL was 0.24.8. The other machines that are still
> > running 0.24.8 are all doing fine but on the one with the new puppet version
> > I just get a "Reopening log files" in the syslog. On the other machines
> this
> > is followed by "Starting Puppet client version 0.24.8" but on the new one
> > nothing happens.
>
>  A bad part of this our fault. We need to send out FLAG days on certain
>  packages (puppet being one of them) when an update goes into testing
>  and when it goes into production.
>
>  A second bad part is we do not keep many old versions of our packages
>  around due to limits on resources on our part. 
I don't know about many versions but it would be good to have at least the 
previous version around for a potential rollback.

>  Part of this is due to how puppet upstream works. Puppet only gives
> a
>  partial guarantee for backwards compatibility between 'major' version
>  upgrades (and only on the server), and the puppet recommended way of
>  upgrades is always:
>
>  Upgrade the server first, Upgrade the clients next. 
Should this 'major' update have been put into EPEL at all if there are 
known backwards compatibility issues? That seems to be quite a risky policy 
for a distribution which most people expect to be mostly free from any 
surprises (as opposed to the faster moving desktop distros like Fedora).

Would a backup of /var/lib/puppet and /etc/puppet be sufficient when I 
attempt to update the server? I manage 50+ machines with puppet and it 
would be extremely painful to not be able to go back to a known working 
setup if 0.25.4 shows problems.

Regards,
   Dennis

[Stephen John Smoogen]

On Fri, Mar 12, 2010 at 4:16 PM, Dennis J. <dennisml(a)conversis.de&gt; wrote:
>  On 03/12/2010 11:37 PM, Stephen John Smoogen wrote:
> >
> > On Fri, Mar 12, 2010 at 2:47 PM, Dennis J.&lt;dennisml(a)conversis.de&gt;  wrote:
> >>
> >> Hi,
> >> I just installed a new machine in our infrastructure but I can't get the
> >> puppet package to work. I see it has recently been updated to 0.25.4
> >> while
> >> the previous version i EPEL was 0.24.8. The other machines that are still
> >> running 0.24.8 are all doing fine but on the one with the new puppet
> >> version
> >> I just get a "Reopening log files" in the syslog. On the other
> machines
> >> this
> >> is followed by "Starting Puppet client version 0.24.8" but on the new
> one
> >> nothing happens.
> >
> > A bad part of this our fault. We need to send out FLAG days on certain
> > packages (puppet being one of them) when an update goes into testing
> > and when it goes into production.
> >
> > A second bad part is we do not keep many old versions of our packages
> > around due to limits on resources on our part.
>
>  I don't know about many versions but it would be good to have at least the
>  previous version around for a potential rollback. 
We would need to keep several versions around because the rollback
would have been 0.25.2 versus the 0.24.8 one you had on the system.

> > Part of this is due to how puppet upstream works. Puppet only
> gives a
> > partial guarantee for backwards compatibility between 'major' version
> > upgrades (and only on the server), and the puppet recommended way of
> > upgrades is always:
> >
> > Upgrade the server first, Upgrade the clients next.
>
>  Should this 'major' update have been put into EPEL at all if there are known
>  backwards compatibility issues? That seems to be quite a risky policy for a
>  distribution which most people expect to be mostly free from any surprises
>  (as opposed to the faster moving desktop distros like Fedora). 
Here is where EPEL breaks down... certain software that people want
very much does not stabilize and is moving forward all the time. If we
were strictly going to do things, puppet and most of the things people
ask for in EPEL would never be allowed in because their up-streams do
not back-port and change their code so much between versions that
back-porting is not possible.

However, that would mean that we would only allow for software that
hasn't changed since Red Hat Linux 7.1 which I think would be maybe
bsd-games and fortune (maybe). People want newer stuff, and that comes
with certain risks that really can't be dealt without serious
dedicated resources. We should have announced this, but that would
require people to 'know' they needed to read epel-announce or some rss
feed before doing yum update.

>  Would a backup of /var/lib/puppet and /etc/puppet be sufficient when
> I
>  attempt to update the server? I manage 50+ machines with puppet and it would
>  be extremely painful to not be able to go back to a known working setup if
>  0.25.4 shows problems. 
We have been using 0.25.x in Fedora infrastructure (100+ machines) for
about a month before it got pushed to epel-stable. Backing up those
directories should be enough.

If you want to lower your risk lower, mirror what of the the
repositories locally and use that to do updates to your boxes from. It
is the method that Dag has pointed out for years and makes issues like
this much easier to deal with no matter what the repository is.

>  Regards,
>   Dennis
>
>  _______________________________________________
>  epel-devel-list mailing list
>  epel-devel-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/epel-devel-list
>  


-- 
Stephen J Smoogen.

Ah, but a man's reach should exceed his grasp. Or what's a heaven for?
-- Robert Browning