[atomic-wg] Issue #405: blog post/email about ostree HTTP2 issues and work arounds for F27 AH/AW
miabbott reported a new issue against the project: `atomic-wg` that you are following: `` I was the latest victim of issues pulling updates via HTTP2 when using `rpm-ostree upgrade` on my F27AW system. The problem is covered here:
https://github.com/ostreedev/ostree/issues/1362
Thankfully, the latest release of `ostree` (2017.15) contains the fix (workaround?) for the issue, but users are going to need to install that fixed version before they'll be able to successfully update their Atomic Host/Workstations. The ol' chicken vs egg problem.
We should publicize this issue via a blog post or email to `atomic-devel` (or both) because it has the potential to impact a lot of users. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
walters added a new comment to an issue you are following: `` FAH should mostly hit deltas which don't seem likely to trigger this right? Though of course that isn't the same as "won't".
I think my vote is to ask infra to turn off http2 server side for a bit, let updates trickle out that disable it client side. Though I saw some discussion about a libcurl fix - if someone has more info can they add it to the libostree issue? ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
miabbott added a new comment to an issue you are following: `` @puiterwijk might have been working on a libcurl fix...any comments Patrick? ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
dustymabe added a new comment to an issue you are following: ``
@puiterwijk might have been working on a libcurl fix...any comments Patrick?
talked with patrick in IRC, follow discussion in https://github.com/ostreedev/ostree/issues/1362 ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
dustymabe added a new comment to an issue you are following: ``
talked with patrick in IRC, follow discussion in https://github.com/ostreedev/ostree/issues/1362
and here: https://github.com/ostreedev/ostree/issues/878#issuecomment-354803583 ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
walters added a new comment to an issue you are following: `` Here's how you can hotfix this issue:
``` ostree admin unlock rpm -Uvh https://kojipkgs.fedoraproject.org//packages/ostree/2017.15/1.fc27/x86_64/os... systemctl restart rpm-ostreed ```
Then appy updates as normal, e.g.:
``` rpm-ostree upgrade ```
``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
dustymabe added a new comment to an issue you are following: `` one thing to consider here is that for FAH we produce new media every two weeks. For FAW we just have the ISO that was generated before F27 major release. So basically when doing an upgrade from FAW ISO to latest users will always hit the http2 bug (because the connection resets after a certain number of files have been downloaded). This explains why FAW users are seeing this more than FAH users.
One potentially ugly workaround (to prevent us from having to build new media) is to redirect `https://dl.fedoraproject.org/ostree/27/%60 to a non https location (Since h2 negotiation is done during the TLS handshake) and make that non https location only serve content that is gpg signed. We have signature checking enabled in the image so this should be ok. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
smilner added a new comment to an issue you are following: `` @miabbott / @walters / @dustymabe:
Any of you folks want to take this blog post? If not, who do you think would be a good person to do this? If no one else comes to mind I will be the fall back and write one. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
walters added a new comment to an issue you are following: `` AIUI the current status is that Fedora Infrastructure has disabled HTTP2 at entrypoints:
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=076286e... https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=5f4ea31...
I think the agreement was to do that for a short period of time - 2 weeks? ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
dustymabe added a new comment to an issue you are following: ``
I think the agreement was to do that for a short period of time - 2 weeks?
Right, I wonder if this is even worth a blog post at this point. I think as long as we can fix the [FAW ISO install+upgrade path](https://pagure.io/atomic-wg/issue/405#comment-486841) then we don't need to. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
miabbott added a new comment to an issue you are following: ``
Right, I wonder if this is even worth a blog post at this point. I think as long as we can fix the FAW ISO install+upgrade path then we don't need to.
Agreed; with HTTP2 disabled for a while, most users should avoid this. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
smilner added a new comment to an issue you are following: ``
Right, I wonder if this is even worth a blog post at this point. I think as long as we can fix the FAW ISO install+upgrade path then we don't need to.
Agreed; with HTTP2 disabled for a while, most users should avoid the error/problem.
In that case is it OK to close this issue? ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
dustymabe added a new comment to an issue you are following: `` probably need to open a new ticket to address https://pagure.io/atomic-wg/issue/405#comment-486841 and then close this ticket. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
dustymabe added a new comment to an issue you are following: `` FYI request to have ostree pass version information which will help us workaround issues like this server side in the future: https://github.com/ostreedev/ostree/issues/1405 ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
dustymabe added a new comment to an issue you are following: ``
FYI request to have ostree pass version information which will help us workaround issues like this server side in the future: https://github.com/ostreedev/ostree/issues/1405
That issue is now fixed and ostree will start passing version information in our next two week release (in a few weeks). ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
dustymabe added a new comment to an issue you are following: ``
That issue is now fixed and ostree will start passing version information in our next two week release (in a few weeks).
ostree [made it to stable](https://bodhi.fedoraproject.org/updates/FEDORA-2018-325f84e2bb) so we should get the **version** information in our http requests and can use this data to redirect clients that we need to redirect ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
sinnykumari added a new comment to an issue you are following: ``
probably need to open a new ticket to address https://pagure.io/atomic-wg/issue/405#comment-486841 and then close this ticket.
Are we looking for creating ticket about this comment or fixes made in ostree is sufficent for future workaround for OSTree HTTP2 issue? ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
The status of the issue: `blog post/email about ostree HTTP2 issues and work arounds for F27 AH/AW` of project: `atomic-wg` has been updated to: Closed as Fixed by dustymabe.
dustymabe added a new comment to an issue you are following: `` I think we are good now. HTTP2 has been disabled in ostree and no one should have issues with media from F28. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/405
atomic@lists.fedoraproject.org
-
Colin Walters -
Dusty Mabe
-
Micah Abbott
-
Sinny Kumari
-
Steve Milner