hitch is broken in epel7, fix in fedora may or may not break production systems (bz #1731420)

[Ingvar Hagelund]

Hello

hitch is a TLS terminating network proxy, made to be lean and mean and do nothing else than terminating TLS. It fits hand-in-glove with varnish cache. I maintain hitch in Fedora and EPEL.

There is a bug in the current epel7 config that is fixed in the latest rawhide update. In short, the bug is that with the default config, hitch forks a daemon, while the systemd hitch service says Type=simple. See Bugzilla bug #1731420.

The fedora update fixes the problem by changing the systemd service to Type=forking.

There were two ways to get around the bug:

- Set daemon=off in hitch.conf. That file is marked with noreplace, so the update will not overwrite this fix. As this does not match the updated Type=forking in hitch.service, hitch will not start after the update.

- Set Type=forking in hitch.service. This is the same fix as in the update, so this should be safe.

Also, the Fedora update adds a systemd limits.conf including LimitNOFILE=10240 that is important, as the default value (1024) would trig network problems on a medium busy site (true story).

Is it safe to push this update to epel7?

Ingvar

[Stephen John Smoogen]

On Tue, 12 Nov 2019 at 17:06, Ingvar Hagelund  wrote:
> Hello
>
> hitch is a TLS terminating network proxy, made to be lean and mean and do nothing else than terminating TLS. It fits hand-in-glove with varnish cache. I maintain hitch in Fedora and EPEL.
>
> There is a bug in the current epel7 config that is fixed in the latest rawhide update. In short, the bug is that with the default config, hitch forks a daemon, while the systemd hitch service says Type=simple. See Bugzilla bug #1731420.
>
> The fedora update fixes the problem by changing the systemd service to Type=forking.
>
> There were two ways to get around the bug:
>
> Set daemon=off in hitch.conf. That file is marked with noreplace, so the update will not overwrite this fix. As this does not match the updated Type=forking in hitch.service, hitch will not start after the update.
>
> Set Type=forking in hitch.service. This is the same fix as in the update, so this should be safe.
>
> Also, the Fedora update adds a systemd limits.conf including LimitNOFILE=10240 that is important, as the default value (1024) would trig network problems on a medium busy site (true story).
>
> Is it safe to push this update to epel7?

This was discussed at today's EPEL meeting and approved. Please push
this to epel-testing and let users know in any tickets that it can be
used there. After that, push to stable after regular feedback time.

[Leon Fauster]

Am 13.11.19 um 20:44 schrieb Stephen John Smoogen:
> On Tue, 12 Nov 2019 at 17:06, Ingvar Hagelund  wrote:
> > Hello
> >
> > hitch is a TLS terminating network proxy, made to be lean and mean and do nothing else than terminating TLS. It fits hand-in-glove with varnish cache. I maintain hitch in Fedora and EPEL.
> >
> > There is a bug in the current epel7 config that is fixed in the latest rawhide update. In short, the bug is that with the default config, hitch forks a daemon, while the systemd hitch service says Type=simple. See Bugzilla bug #1731420.
> >
> > The fedora update fixes the problem by changing the systemd service to Type=forking.
> >
> > There were two ways to get around the bug:
> >
> > Set daemon=off in hitch.conf. That file is marked with noreplace, so the update will not overwrite this fix. As this does not match the updated Type=forking in hitch.service, hitch will not start after the update.
> >
> > Set Type=forking in hitch.service. This is the same fix as in the update, so this should be safe.
> >
> > Also, the Fedora update adds a systemd limits.conf including LimitNOFILE=10240 that is important, as the default value (1024) would trig network problems on a medium busy site (true story).
> >
> > Is it safe to push this update to epel7?
>
> This was discussed at today's EPEL meeting and approved. Please push
> this to epel-testing and let users know in any tickets that it can be
> used there. After that, push to stable after regular feedback time.

I'd really appreciate if it could also be branched into EPEL8?

[Leon Fauster]

Am 14.11.19 um 14:01 schrieb Leon Fauster:
> Am 13.11.19 um 20:44 schrieb Stephen John Smoogen:
> > On Tue, 12 Nov 2019 at 17:06, Ingvar Hagelund 
> >  wrote:
> > > Hello
> > >
> > > hitch is a TLS terminating network proxy, made to be lean and mean 
> > > and do nothing else than terminating TLS. It fits hand-in-glove with 
> > > varnish cache. I maintain hitch in Fedora and EPEL.
> > >
> > > There is a bug in the current epel7 config that is fixed in the 
> > > latest rawhide update. In short, the bug is that with the default 
> > > config, hitch forks a daemon, while the systemd hitch service says 
> > > Type=simple. See Bugzilla bug #1731420.
> > >
> > > The fedora update fixes the problem by changing the systemd service 
> > > to Type=forking.
> > >
> > > There were two ways to get around the bug:
> > > Set daemon=off in hitch.conf. That file is marked with noreplace,
> > >
> > > so the update will not overwrite this fix. As this does not match the 
> > > updated Type=forking in hitch.service, hitch will not start after the 
> > > update.
> > > Set Type=forking in hitch.service. This is the same fix as in the
> > >
> > > update, so this should be safe.
> > >
> > > Also, the Fedora update adds a systemd limits.conf including 
> > > LimitNOFILE=10240 that is important, as the default value (1024) 
> > > would trig network problems on a medium busy site (true story).
> > >
> > > Is it safe to push this update to epel7?
> >
> > This was discussed at today's EPEL meeting and approved. Please push
> > this to epel-testing and let users know in any tickets that it can be
> > used there. After that, push to stable after regular feedback time.
>
> I'd really appreciate if it could also be branched into EPEL8?

Thanks for pushing it into fedora-epel/testing/8/ !

--
Leon

[Matthew Blissett]

On 13/11/2019 20:44, Stephen John Smoogen wrote:
> On Tue, 12 Nov 2019 at 17:06, Ingvar Hagelund <ingvar(a)redpill-linpro.com> wrote:
> > Hello
> >
> > hitch is a TLS terminating network proxy, made to be lean and mean and do nothing else than terminating TLS. It fits hand-in-glove with varnish cache. I maintain hitch in Fedora and EPEL.
> >
> > There is a bug in the current epel7 config that is fixed in the latest rawhide update. In short, the bug is that with the default config, hitch forks a daemon, while the systemd hitch service says Type=simple. See Bugzilla bug #1731420.
> >
> > The fedora update fixes the problem by changing the systemd service to Type=forking.
> >
> > There were two ways to get around the bug:
> >
> > Set daemon=off in hitch.conf. That file is marked with noreplace, so the update will not overwrite this fix. As this does not match the updated Type=forking in hitch.service, hitch will not start after the update.
> >
> > Set Type=forking in hitch.service. This is the same fix as in the update, so this should be safe.
> >
> > Also, the Fedora update adds a systemd limits.conf including LimitNOFILE=10240 that is important, as the default value (1024) would trig network problems on a medium busy site (true story).
> >
> > Is it safe to push this update to epel7?
>
> This was discussed at today's EPEL meeting and approved. Please push
> this to epel-testing and let users know in any tickets that it can be
> used there. After that, push to stable after regular feedback time.

As anticipated by Ingvar, this update broke my production systems.

The EPEL site [1] says
> it is possible that occasionally an incompatible update will be released
> such that administrator action is required. By policy these are announced
> in advance in order to give administrators time to test and provide 
> suggestions.
>
> It is strongly recommended that if you make use of EPEL, and especially
> if you rely upon it, that you subscribe to the epel-announce list.
> Traffic on this list is kept to a minimum needed to notify administrators
> of important updates.

This update wasn't announced there -- is that an oversight, or should I
change my approach to updates to account for possible incompatible
updates in the future?

Thanks,

Matthew Blissett

[1]https://fedoraproject.org/wiki/EPEL#Can_I_rely_on_these_packages.3F

[Stephen John Smoogen]

On Thu, 12 Dec 2019 at 05:28, Matthew Blissett  wrote:
> On 13/11/2019 20:44, Stephen John Smoogen wrote:
> > On Tue, 12 Nov 2019 at 17:06, Ingvar Hagelund <ingvar(a)redpill-linpro.com> wrote:
> > > Hello
> > >
> > > hitch is a TLS terminating network proxy, made to be lean and mean and do nothing else than terminating TLS. It fits hand-in-glove with varnish cache. I maintain hitch in Fedora and EPEL.
> > >
> > > There is a bug in the current epel7 config that is fixed in the latest rawhide update. In short, the bug is that with the default config, hitch forks a daemon, while the systemd hitch service says Type=simple. See Bugzilla bug #1731420.
> > >
> > > The fedora update fixes the problem by changing the systemd service to Type=forking.
> > >
> > > There were two ways to get around the bug:
> > >
> > > Set daemon=off in hitch.conf. That file is marked with noreplace, so the update will not overwrite this fix. As this does not match the updated Type=forking in hitch.service, hitch will not start after the update.
> > >
> > > Set Type=forking in hitch.service. This is the same fix as in the update, so this should be safe.
> > >
> > > Also, the Fedora update adds a systemd limits.conf including LimitNOFILE=10240 that is important, as the default value (1024) would trig network problems on a medium busy site (true story).
> > >
> > > Is it safe to push this update to epel7?
> >
> > This was discussed at today's EPEL meeting and approved. Please push
> > this to epel-testing and let users know in any tickets that it can be
> > used there. After that, push to stable after regular feedback time.
>
> As anticipated by Ingvar, this update broke my production systems.
>
> The EPEL site [1] says
> > it is possible that occasionally an incompatible update will be released
> > such that administrator action is required. By policy these are announced
> > in advance in order to give administrators time to test and provide
> > suggestions.
> >
> > It is strongly recommended that if you make use of EPEL, and especially
> > if you rely upon it, that you subscribe to the epel-announce list.
> > Traffic on this list is kept to a minimum needed to notify administrators
> > of important updates.
>
> This update wasn't announced there -- is that an oversight, or should I
> change my approach to updates to account for possible incompatible
> updates in the future?

So by policy an update should have been announced to epel-announce. It
wasn't, it was an oversight but there isn't anything we can do about
that :(. This is very much a volunteer effort and things like this
will happen. I apologize for the problems it caused you, and will
reword that paragraph however it should be.
> Thanks,
>
> Matthew Blissett
>
> [1]https://fedoraproject.org/wiki/EPEL#Can_I_rely_on_these_packages.3F
>
> epel-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]...