[releng] Issue #8330: no f30 silverblue updates: ostree composes started by bodhi are failing

[Dusty Mabe]

dustymabe reported a new issue against the project: `releng` that you are following:
``
* Describe the issue

The ostree composes seem to be failing with an error:


```
2019-03-12 16:00:50 [ERROR   ] [FAIL] Ostree (variant Everything, arch x86_64) failed, but
going on anyway.
2019-03-12 16:00:50 [ERROR   ] 'ascii' codec can't decode byte 0xc2 in
position 6271: ordinal not in range(128)
```

An example failed composes issue for this:
https://pagure.io/dusty/failed-composes/issue/1642


* When do you need this? (YYYY/MM/DD)

ASAP so people on Fedora SB 30 can get updates

* When is this no longer needed or useful? (YYYY/MM/DD)

F30 EOL

* If we cannot complete your request, what is the impact?

Users of Fedora SB move to something else (possibly not Fedora).  
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Dusty Mabe]

dustymabe added a new comment to an issue you are following:
``
The ostree composes weren't failing in the normal pungi runs before release (i.e. not
run through bodhi). Mohan pointed out one difference between those setups:

```
09:55:46*       lsedlar | mboddu, is there the same python version as well?
09:56:29        +mboddu | lsedlar: Nope
09:57:06*       +mboddu | bodhi-backend01.phx2.fedoraproject.org has
python3-3.7.3-1.fc29.x86_64
09:57:19*       +mboddu | where as the compose-x86-01.phx2.fedoraproject.org has
                        | python3-3.7.2-5.fc29.x86_64
09:58:01        +mboddu | lsedlar: bodhi-bakend01 has the latest version compared to both
of them

```
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Colin Walters]

walters added a new comment to an issue you are following:
``
One thing I'm a bit confused on is why Pungi isn't doing UTF-8 for subprocesses by
default.  Something like this?

```
diff --git a/pungi/wrappers/kojiwrapper.py b/pungi/wrappers/kojiwrapper.py
index 3cd9ac9..02855bb 100644
--- a/pungi/wrappers/kojiwrapper.py
+++ b/pungi/wrappers/kojiwrapper.py
@@ -147,7 +147,8 @@ class KojiWrapper(object):
         task_id = None
         with self.get_koji_cmd_env() as env:
             retcode, output = run(command, can_fail=True, logfile=log_file,
-                                  show_cmd=True, env=env, universal_newlines=True)
+                                  show_cmd=True, env=env, universal_newlines=True,
+                                  encoding='UTF-8')
         if "--task-id" in command:
             first_line = output.splitlines()[0]
             if re.match(r'^\d+$', first_line):
```
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Jonathan Lebon]

jlebon added a new comment to an issue you are following:
``
So reading the backstory here in https://pagure.io/dusty/failed-composes/issue/1642, it
seems like it's getting thrown off by some specific Unicode characters rpm-ostree
emits. And some changes in Unicode handling in 3.7.3 is making this reliably fail. Seems
like we should be able to just tweak pungi to decode rpm-ostree output using utf-8?
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Jonathan Lebon]

jlebon added a new comment to an issue you are following:
``
In the worst case we *could* tweak rpm-ostree for this, but... it's utf-8. I'm
sure rpm-ostree isn't the only app we'd want to run in a runroot that outputs
utf-8. :)
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Colin Walters]

walters added a new comment to an issue you are following:
``
This part also looks suspicious:

```
bin/pungi-koji
9:import locale
29:# force C locales
30:locale.setlocale(locale.LC_ALL, "C")
```

I think that should probably be `C.UTF-8` nowadays.


``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Colin Walters]

walters added a new comment to an issue you are following:
``
This kobo function (used in a few places in pungi) also looks suspicious:

```
def save_to_file(filename, text, append=False, mode=0o644):
    """Save text to a file."""
    if append and os.path.exists(filename):
        fd = os.open(filename, os.O_RDWR | os.O_APPEND, mode)
    else:
        fd = os.open(filename, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, mode)
    if not isinstance(text, bytes):
        text = bytes(text, encoding=locale.getpreferredencoding())
    os.write(fd, text)
    os.close(fd)
```

Having one function to create both binary data and text seems odd...given the use cases
I'd just assume any `bytes()` instances are `UTF-8`?

``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Colin Walters]

walters added a new comment to an issue you are following:
``
This bug is pretty urgent IMO because we shipped people an operating system and aren't
giving them security updates - anything I can do to help please let me know.  If truly
necessary we can try to strip out the UTF-8 usage in rpm-ostree but...can a Pungi
maintainer comment on https://pagure.io/releng/issue/8330#comment-569167 ?
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Mohan Boddu]

mohanboddu added a new comment to an issue you are following:
``
I downgraded bodhi-backend01.phx2.fp.o to python3-3.7.2.-5, and started f30-updates and
f30-updates-testing push. Lets see what happens.
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Dusty Mabe]

dustymabe added a new comment to an issue you are following:
``
>  I downgraded bodhi-backend01.phx2.fp.o to python3-3.7.2.-5, and
> started f30-updates and f30-updates-testing push. Lets see what happens.
>  
Thanks @mohanboddu.

He did this so we can see if we can get a new update out today. The pungi maintainers
would only be able to get back to us at the earliest tomorrow morning and we'd like to
get an update out sooner than later. Since the attempt at a workaround here is a downgrade
this is only a temporary bandaid so we'll have to dig in and possibly implement the
suggestions from @walters and @jlebon soon. 
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Mohan Boddu]

mohanboddu added a new comment to an issue you are following:
``
Okay, I am still seeing this after downgrading python3 in the [latest bodhi
compose](https://kojipkgs.fedoraproject.org/compose/updates/Fedora-30-upd...
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Jonathan Lebon]

jlebon added a new comment to an issue you are following:
``
OK, testing a temporary hack in
https://src.fedoraproject.org/rpms/rpm-ostree/pull-request/32.
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Dusty Mabe]

dustymabe added a new comment to an issue you are following:
``
>  OK, testing a temporary hack in
> https://src.fedoraproject.org/rpms/rpm-ostree/pull-request/32.
>  
cool - once we get a build would we need to use a buildroot override? 
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Jonathan Lebon]

jlebon added a new comment to an issue you are following:
``
OK, new rpm-ostree in buildroot with the patch above. Can someone retry the updates &
updates-testing push?

/cc @kevin 
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Kevin Fenzi]

kevin added a new comment to an issue you are following:
``
f30 updates-testing away. (if that works we can push updates too, but best to just test
with updates-testing)

``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Kevin Fenzi]

kevin added a new comment to an issue you are following:
``
It seems to have worked. If we get a f30-updates I think we can be happy... and look for
longer term fix in pungi. 
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Sinny Kumari]

sinnykumari added a new comment to an issue you are following:
``
[f30-updates](https://kojipkgs.fedoraproject.org/compose/updates/Fedora-30...
 succeeded as well. Thanks all!

Both updates and testing ref has not yet synced to production ostree/repo/refs/
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Dusty Mabe]

dustymabe added a new comment to an issue you are following:
``
>  @walters 
>  One thing I'm a bit confused on is why Pungi isn't doing UTF-8 for subprocesses
> by default.
>  
open PR in https://pagure.io/pungi/pull-request/1185

>  @sinnykumari 
>  Both updates and testing ref has not yet synced to production ostree/repo/refs/
>  
right - we'll need to look into getting those synced out and also alias the stable ref
to the updates ref so that they are one in the same
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Dusty Mabe]

dustymabe added a new comment to an issue you are following:
``
ok - we have an updates ref now - i'm running a quick update on my laptop to make sure
it's not obviously broken before I alias the stable ref to it. 
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Dusty Mabe]

dustymabe added a new comment to an issue you are following:
``
it looked good. I got the stable ref aliased over and everyone should be able to update
now 🌈 
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Dusty Mabe]

The status of the issue: `no f30 silverblue updates: ostree composes started by bodhi are
failing ` of project: `releng` has been updated to: Closed as Fixed by dustymabe.

https://pagure.io/releng/issue/8330

[Dusty Mabe]

dustymabe added a new comment to an issue you are following:
``
actually i'll re-open this as we applied a bandaid (patch to rpm-ostree) but the real
fix is in pungi. I'll re-open until we confirm it's fixed (open PR is
[here](https://pagure.io/pungi/pull-request/1185))
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330

[Dusty Mabe]

The status of the issue: `no f30 silverblue updates: ostree composes started by bodhi are
failing ` of project: `releng` has been updated to: Open by dustymabe.

https://pagure.io/releng/issue/8330

[Dusty Mabe]

The status of the issue: `no f30 silverblue updates: ostree composes started by bodhi are
failing ` of project: `releng` has been updated to: Closed as Fixed by dustymabe.

https://pagure.io/releng/issue/8330

[Dusty Mabe]

dustymabe added a new comment to an issue you are following:
``
we can close this now since it's been a while 
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8330