Logging in to f32 image on pi

[Tim Coote]

Hullo

It's been a while since I've spun up fedoraiot, so I thought I'd see where
we'd got to, as I've now got a concrete usecase monitoring some solar panels. 
Unfortunately, when I put Fedora-IoT-32-20200603.0.aarch64.raw.xz onto an SDCard, the pi
(3b+) boots and gets to a login prompt, but I cannot login; neither as `root` with no
password on the console, nor ssh'ing as `root` after using the Zezere portal. I'm
pretty sure that I'm using the right key (although the documentation is a bit light on
what a good key content looks like).  The `/portal/devices/` page shows that there's
an action to upload a key, with a `cancel runrequest` button, but no indication that more
is happening.

Any hints or tips / troubleshooting approaches?

Tim

[Tim Coote]

I managed to fix the access by mounting /dev/sdb3 and copying a known good
`authorized_keys` to `ostree/deploy/fedora-iot/var/roothome/.ssh/authorized_keys`. The
version on the image looked like a version of the public key that I thought I'd
deleted on zerere before trying to claim this device. It was just the key, and lacked the
key type at the start of the line and userid at the end.

[Peter Robinson]

Hi Tim,

>  It's been a while since I've spun up fedoraiot, so I thought
> I'd see where we'd got to, as I've now got a concrete usecase monitoring some
> solar panels.  Unfortunately, when I put Fedora-IoT-32-20200603.0.aarch64.raw.xz onto an
> SDCard, the pi (3b+) boots and gets to a login prompt, but I cannot login; neither as
> `root` with no password on the console, nor ssh'ing as `root` after using the Zezere
> portal. I'm pretty sure that I'm using the right key (although the documentation
> is a bit light on what a good key content looks like).  The `/portal/devices/` page shows
> that there's an action to upload a key, with a `cancel runrequest` button, but no
> indication that more is happening.
>
>  Any hints or tips / troubleshooting approaches? 
There's details in the docs for key provisioning:
https://docs.fedoraproject.org/en-US/iot/ignition/

[TimC]

Hi Peter

Thanks. I had spotted that, but couldn’t find the detail that I needed (eg definition of
the key - does it include the key type? is it just a copy of an entry in
~/.ssh/authorized_keys); should all keys be dropped into /root/.ssh/authorized_keys? What
triggers the upload? etc. As a user, with a pi that you cannot log into, there’s not a lot
to go on.

Tim
>  On 21 Aug 2020, at 07:34, Peter Robinson <pbrobinson(a)gmail.com&gt;
> wrote:
>  
>  Hi Tim,
>  
> > It's been a while since I've spun up fedoraiot, so I thought I'd see
> where we'd got to, as I've now got a concrete usecase monitoring some solar
> panels.  Unfortunately, when I put Fedora-IoT-32-20200603.0.aarch64.raw.xz onto an SDCard,
> the pi (3b+) boots and gets to a login prompt, but I cannot login; neither as `root` with
> no password on the console, nor ssh'ing as `root` after using the Zezere portal.
> I'm pretty sure that I'm using the right key (although the documentation is a bit
> light on what a good key content looks like).  The `/portal/devices/` page shows that
> there's an action to upload a key, with a `cancel runrequest` button, but no
> indication that more is happening.
> > 
> > Any hints or tips / troubleshooting approaches?
>  
>  There's details in the docs for key provisioning:
>  https://docs.fedoraproject.org/en-US/iot/ignition/
>  _______________________________________________
>  IoT mailing list -- iot(a)lists.fedoraproject.org
>  To unsubscribe send an email to iot-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

layered packages

[TimC]

Hullo

I’m sure that I’m being a bit dim here and that I’ve seen this working. However.

Starting with a downloaded image for a Pi (3b+), after a `rpm-ostree upgrade`, I seem
unable to deploy layered packages. They seem to install, but after the reboot, they’re not
there. As an example, I tried to drop `toolbox` onto the image so that I could build some
of my own rpms to run under systemd, using the approach here: https://bit.ly/3b8Ivri. The
initial status before installation looks like this:
```
rpm-ostree status
State: idle
Deployments:
* ostree://fedora-iot:fedora/stable/aarch64/iot
                  Version: 32.20200817.0 (2020-08-17T15:28:18Z)
                   Commit:
ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
             GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

 ostree://fedora-iot:fedora/stable/aarch64/iot
                  Version: 32.20200603.0 (2020-06-03T10:45:43Z)
                   Commit:
c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b
             GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
```

`sudo rpm-ostree install toolbox` progresses as I’d expect, creating a new bootable
deployment:
```
[tim@localhost ~]$ sudo rpm-ostree install toolbox
Checking out tree ace3685... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora fedora-modular
updates-modular
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-03-17T20:10:43Z
rpm-md repo 'updates' (cached); generated: 2020-08-27T14:18:53Z
rpm-md repo 'fedora' (cached); generated: 2020-04-22T22:22:16Z
rpm-md repo 'fedora-modular' (cached); generated: 2020-04-22T20:58:40Z
rpm-md repo 'updates-modular' (cached); generated: 2020-08-27T14:52:01Z
Importing rpm-md... done
Resolving dependencies... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
⠉ Staging deployment... 
Staging deployment... done

Freed: 354.4?MB (pkgcache branches: 217)
Added:
 flatpak-session-helper-1.6.5-1.fc32.aarch64
 toolbox-0.0.93-1.fc32.aarch64
Run "systemctl reboot" to start a reboot

```

and the status looks ok (ignoring the niggling error):
```
[tim@localhost ~]$ rpm-ostree status
error: Timeout was reached
[tim@localhost ~]$ rpm-ostree status
State: idle
Deployments:
 ostree://fedora-iot:fedora/stable/aarch64/iot
                  Version: 32.20200817.0 (2020-08-17T15:28:18Z)
               BaseCommit:
ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
             GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
                     Diff: 2 added
          LayeredPackages: toolbox

* ostree://fedora-iot:fedora/stable/aarch64/iot
                  Version: 32.20200817.0 (2020-08-17T15:28:18Z)
                   Commit:
ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
             GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

 ostree://fedora-iot:fedora/stable/aarch64/iot
                  Version: 32.20200603.0 (2020-06-03T10:45:43Z)
                   Commit:
c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b
             GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
```

But when I reboot, that deployment isn’t the image used and it’s disappeared from
rpm-ostree’s world:
```
[tim@localhost ~]$ sudo rpm-ostree status
State: idle
Deployments:
* ostree://fedora-iot:fedora/stable/aarch64/iot
                  Version: 32.20200817.0 (2020-08-17T15:28:18Z)
                   Commit:
ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
             GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

 ostree://fedora-iot:fedora/stable/aarch64/iot
                  Version: 32.20200603.0 (2020-06-03T10:45:43Z)
                   Commit:
c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b
             GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
```

Did I forget how this type of layered installation should work?

Tim

layered packages

[Dusty Mabe]

On 8/28/20 5:43 PM, TimC wrote:
>  Hullo
>  
>  I’m sure that I’m being a bit dim here and that I’ve seen this working. However.
>  
>  Starting with a downloaded image for a Pi (3b+), after a `rpm-ostree upgrade`, I seem
> unable to deploy layered packages. They seem to install, but after the reboot, they’re not
> there. As an example, I tried to drop `toolbox` onto the image so that I could build some
> of my own rpms to run under systemd, using the approach here: https://bit.ly/3b8Ivri. The
> initial status before installation looks like this:
>  ```
>  rpm-ostree status
>  State: idle
>  Deployments:
>  * ostree://fedora-iot:fedora/stable/aarch64/iot
>                    Version: 32.20200817.0 (2020-08-17T15:28:18Z)
>                     Commit:
> ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
>               GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
>  
>   ostree://fedora-iot:fedora/stable/aarch64/iot
>                    Version: 32.20200603.0 (2020-06-03T10:45:43Z)
>                     Commit:
> c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b
>               GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
>  ```
>  
>  `sudo rpm-ostree install toolbox` progresses as I’d expect, creating a new bootable
> deployment:
>  ```
>  [tim@localhost ~]$ sudo rpm-ostree install toolbox
>  Checking out tree ace3685... done
>  Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora fedora-modular
> updates-modular
>  rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-03-17T20:10:43Z
>  rpm-md repo 'updates' (cached); generated: 2020-08-27T14:18:53Z
>  rpm-md repo 'fedora' (cached); generated: 2020-04-22T22:22:16Z
>  rpm-md repo 'fedora-modular' (cached); generated: 2020-04-22T20:58:40Z
>  rpm-md repo 'updates-modular' (cached); generated: 2020-08-27T14:52:01Z
>  Importing rpm-md... done
>  Resolving dependencies... done
>  Checking out packages... done
>  Running pre scripts... done
>  Running post scripts... done
>  Running posttrans scripts... done
>  Writing rpmdb... done
>  Writing OSTree commit... done
>  ⠉ Staging deployment... 
>  Staging deployment... done
>  
>  Freed: 354.4?MB (pkgcache branches: 217)
>  Added:
>   flatpak-session-helper-1.6.5-1.fc32.aarch64
>   toolbox-0.0.93-1.fc32.aarch64
>  Run "systemctl reboot" to start a reboot
>  
>  ```
>  
>  and the status looks ok (ignoring the niggling error):
>  ```
>  [tim@localhost ~]$ rpm-ostree status
>  error: Timeout was reached
>  [tim@localhost ~]$ rpm-ostree status
>  State: idle
>  Deployments:
>   ostree://fedora-iot:fedora/stable/aarch64/iot
>                    Version: 32.20200817.0 (2020-08-17T15:28:18Z)
>                 BaseCommit:
> ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
>               GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
>                       Diff: 2 added
>            LayeredPackages: toolbox
>  
>  * ostree://fedora-iot:fedora/stable/aarch64/iot
>                    Version: 32.20200817.0 (2020-08-17T15:28:18Z)
>                     Commit:
> ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
>               GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
>  
>   ostree://fedora-iot:fedora/stable/aarch64/iot
>                    Version: 32.20200603.0 (2020-06-03T10:45:43Z)
>                     Commit:
> c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b
>               GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
>  ```
>  
>  But when I reboot, that deployment isn’t the image used and it’s disappeared from
> rpm-ostree’s world:
>  ```
>  [tim@localhost ~]$ sudo rpm-ostree status
>  State: idle
>  Deployments:
>  * ostree://fedora-iot:fedora/stable/aarch64/iot
>                    Version: 32.20200817.0 (2020-08-17T15:28:18Z)
>                     Commit:
> ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
>               GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
>  
>   ostree://fedora-iot:fedora/stable/aarch64/iot
>                    Version: 32.20200603.0 (2020-06-03T10:45:43Z)
>                     Commit:
> c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b
>               GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
>  ```
>  
>  Did I forget how this type of layered installation should work? 

Seems like it should be working. This vaguely sounds like the deployment not getting
finalized.

Does `sudo journalctl -u ostree-finalize-staged.service` show any errors? If it is timing
out for
some reason you could try to manually run `sudo ostree admin finalize-staged` *before* you
type
reboot to reboot into the new deployment so that you know it finishes.

Dusty

layered packages

[TimC]

Hi Dusty

Is this a likely culprit?
```
-- Reboot --
Aug 28 20:56:24 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged
Deployment.
Aug 28 21:07:23 localhost.localdomain systemd[1]: Stopping OSTree Finalize Staged
Deployment...
Aug 28 21:07:23 localhost.localdomain ostree[14561]: Finalizing staged deployment
Aug 28 21:07:32 localhost.localdomain ostree[14561]: Copying /etc changes: 10 modified, 0
removed, 34 added
Aug 28 21:07:32 localhost.localdomain ostree[14561]: Copying /etc changes: 10 modified, 0
removed, 34 added
Aug 28 21:08:30 localhost.localdomain ostree[14561]: **
Aug 28 21:08:30 localhost.localdomain ostree[14561]:
OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel:
assertion failed (kernel_layout->bootcsum == bootcsum):
("02ec07fa832b4485887f6dbac19d0311e3>
Aug 28 21:08:30 localhost.localdomain ostree[14561]: Bail out!
OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel:
assertion failed (kernel_layout->bootcsum == bootcsum):
("02ec07fa832b4485887f6dba>
Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Control
process exited, code=dumped, status=6/ABRT
Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Failed
with result 'core-dump'.
Aug 28 21:08:30 localhost.localdomain systemd[1]: Stopped OSTree Finalize Staged
Deployment.
Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Consumed
8.211s CPU time.
```

Note that this is from yesterday’s fail. When I tried again today, I only get this (before
reboot):
```
-- Reboot --
Aug 29 11:12:12 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged
Deployment.
```


When I run `sudo ostree admin finalize-staged`, it fails, thus:
```
 sudo ostree admin finalize-staged
Copying /etc changes: 10 modified, 0 removed, 34 added
**
OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel:
assertion failed (kernel_layout->bootcsum == bootcsum):
("02ec07fa832b4485887f6dbac19d0311e3039855cc6ebd57aefaafa4a3997d1e" ==
"fc8629fdb6f5355dd5359350296b619216cbc32e19887511caf75ee9d6aafbad")
Bail out!
OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel:
assertion failed (kernel_layout->bootcsum == bootcsum):
("02ec07fa832b4485887f6dbac19d0311e3039855cc6ebd57aefaafa4a3997d1e" ==
"fc8629fdb6f5355dd5359350296b619216cbc32e19887511caf75ee9d6aafbad")
Aborted

```

I’d guess that’s a fail originating somehow in my trying to increase the image and
filesystem in the download as my first attempts to layer packages just ran out of space.
The approach worked with Fedora-IoT 29 and was based on decompressing, appending 0’s to
the end of the file and then resizing the filesystem.

Would it make sense to confirm the cause by layering a single package onto the original
download, if that’s useful? I think that I will need some mechanism to increase the image
size, tho’ or I won’t be able to get many tools onto the pi.

Tim

>  On 29 Aug 2020, at 06:02, Dusty Mabe <dusty(a)dustymabe.com&gt;
> wrote:
>  
>  
>  
>  On 8/28/20 5:43 PM, TimC wrote:
> > Hullo
> > 
> > I’m sure that I’m being a bit dim here and that I’ve seen this working. However.
> > 
> > Starting with a downloaded image for a Pi (3b+), after a `rpm-ostree upgrade`, I seem
> unable to deploy layered packages. They seem to install, but after the reboot, they’re not
> there. As an example, I tried to drop `toolbox` onto the image so that I could build some
> of my own rpms to run under systemd, using the approach here: https://bit.ly/3b8Ivri. The
> initial status before installation looks like this:
> > ```
> > rpm-ostree status
> > State: idle
> > Deployments:
> > * ostree://fedora-iot:fedora/stable/aarch64/iot
> >                  Version: 32.20200817.0 (2020-08-17T15:28:18Z)
> >                   Commit:
> ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
> >             GPGSignature: Valid signature by
> 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
> > 
> > ostree://fedora-iot:fedora/stable/aarch64/iot
> >                  Version: 32.20200603.0 (2020-06-03T10:45:43Z)
> >                   Commit:
> c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b
> >             GPGSignature: Valid signature by
> 7D22D5867F2A4236474BF7B850CB390B3C3359C4
> > ```
> > 
> > `sudo rpm-ostree install toolbox` progresses as I’d expect, creating a new bootable
> deployment:
> > ```
> > [tim@localhost ~]$ sudo rpm-ostree install toolbox
> > Checking out tree ace3685... done
> > Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora fedora-modular
> updates-modular
> > rpm-md repo 'fedora-cisco-openh264' (cached); generated:
> 2020-03-17T20:10:43Z
> > rpm-md repo 'updates' (cached); generated: 2020-08-27T14:18:53Z
> > rpm-md repo 'fedora' (cached); generated: 2020-04-22T22:22:16Z
> > rpm-md repo 'fedora-modular' (cached); generated: 2020-04-22T20:58:40Z
> > rpm-md repo 'updates-modular' (cached); generated: 2020-08-27T14:52:01Z
> > Importing rpm-md... done
> > Resolving dependencies... done
> > Checking out packages... done
> > Running pre scripts... done
> > Running post scripts... done
> > Running posttrans scripts... done
> > Writing rpmdb... done
> > Writing OSTree commit... done
> > ⠉ Staging deployment... 
> > Staging deployment... done
> > 
> > Freed: 354.4?MB (pkgcache branches: 217)
> > Added:
> > flatpak-session-helper-1.6.5-1.fc32.aarch64
> > toolbox-0.0.93-1.fc32.aarch64
> > Run "systemctl reboot" to start a reboot
> > 
> > ```
> > 
> > and the status looks ok (ignoring the niggling error):
> > ```
> > [tim@localhost ~]$ rpm-ostree status
> > error: Timeout was reached
> > [tim@localhost ~]$ rpm-ostree status
> > State: idle
> > Deployments:
> > ostree://fedora-iot:fedora/stable/aarch64/iot
> >                  Version: 32.20200817.0 (2020-08-17T15:28:18Z)
> >               BaseCommit:
> ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
> >             GPGSignature: Valid signature by
> 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
> >                     Diff: 2 added
> >          LayeredPackages: toolbox
> > 
> > * ostree://fedora-iot:fedora/stable/aarch64/iot
> >                  Version: 32.20200817.0 (2020-08-17T15:28:18Z)
> >                   Commit:
> ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
> >             GPGSignature: Valid signature by
> 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
> > 
> > ostree://fedora-iot:fedora/stable/aarch64/iot
> >                  Version: 32.20200603.0 (2020-06-03T10:45:43Z)
> >                   Commit:
> c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b
> >             GPGSignature: Valid signature by
> 7D22D5867F2A4236474BF7B850CB390B3C3359C4
> > ```
> > 
> > But when I reboot, that deployment isn’t the image used and it’s disappeared from
> rpm-ostree’s world:
> > ```
> > [tim@localhost ~]$ sudo rpm-ostree status
> > State: idle
> > Deployments:
> > * ostree://fedora-iot:fedora/stable/aarch64/iot
> >                  Version: 32.20200817.0 (2020-08-17T15:28:18Z)
> >                   Commit:
> ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
> >             GPGSignature: Valid signature by
> 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
> > 
> > ostree://fedora-iot:fedora/stable/aarch64/iot
> >                  Version: 32.20200603.0 (2020-06-03T10:45:43Z)
> >                   Commit:
> c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b
> >             GPGSignature: Valid signature by
> 7D22D5867F2A4236474BF7B850CB390B3C3359C4
> > ```
> > 
> > Did I forget how this type of layered installation should work?
>  
>  
>  Seems like it should be working. This vaguely sounds like the deployment not getting
> finalized.
>  
>  Does `sudo journalctl -u ostree-finalize-staged.service` show any errors? If it is timing
> out for
>  some reason you could try to manually run `sudo ostree admin finalize-staged` *before*
> you type
>  reboot to reboot into the new deployment so that you know it finishes.
>  
>  Dusty

layered packages

[TimC]

fwiw, there may also be an issue with the log entries being visible to journalctl: after a
reboot, there are more log entries, that, I think, originate from before the reboot:

…[last entry after yesterday’s reboot, followed by the logs post `sudo rpm-ostree install
toolbox`, then `sudo ostree admin finalize-staged`]
```
Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Consumed
8.211s CPU time.
-- Reboot --
Aug 29 11:12:12 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged
Deployment.
Aug 29 11:32:23 localhost.localdomain systemd[1]: Stopping OSTree Finalize Staged
Deployment...
Aug 29 11:32:23 localhost.localdomain ostree[26417]: No deployment staged for
finalization
Aug 29 11:32:25 localhost.localdomain systemd[1]: ostree-finalize-staged.service:
Succeeded.
Aug 29 11:32:25 localhost.localdomain systemd[1]: Stopped OSTree Finalize Staged
Deployment.
```
>  On 29 Aug 2020, at 12:34, TimC
> <tim+fedoraproject.org(a)coote.org&gt; wrote:
>  
>  Hi Dusty
>  
>  Is this a likely culprit?
>  ```
>  -- Reboot --
>  Aug 28 20:56:24 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged
> Deployment.
>  Aug 28 21:07:23 localhost.localdomain systemd[1]: Stopping OSTree Finalize Staged
> Deployment...
>  Aug 28 21:07:23 localhost.localdomain ostree[14561]: Finalizing staged deployment
>  Aug 28 21:07:32 localhost.localdomain ostree[14561]: Copying /etc changes: 10 modified, 0
> removed, 34 added
>  Aug 28 21:07:32 localhost.localdomain ostree[14561]: Copying /etc changes: 10 modified, 0
> removed, 34 added
>  Aug 28 21:08:30 localhost.localdomain ostree[14561]: **
>  Aug 28 21:08:30 localhost.localdomain ostree[14561]:
> OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel:
> assertion failed (kernel_layout->bootcsum == bootcsum):
> ("02ec07fa832b4485887f6dbac19d0311e3>
>  Aug 28 21:08:30 localhost.localdomain ostree[14561]: Bail out!
> OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel:
> assertion failed (kernel_layout->bootcsum == bootcsum):
> ("02ec07fa832b4485887f6dba>
>  Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Control
> process exited, code=dumped, status=6/ABRT
>  Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Failed
> with result 'core-dump'.
>  Aug 28 21:08:30 localhost.localdomain systemd[1]: Stopped OSTree Finalize Staged
> Deployment.
>  Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service:
> Consumed 8.211s CPU time.
>  ```
>  
>  Note that this is from yesterday’s fail. When I tried again today, I only get this
> (before reboot):
>  ```
>  -- Reboot --
>  Aug 29 11:12:12 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged
> Deployment.
>  ```
>  
>  
>  When I run `sudo ostree admin finalize-staged`, it fails, thus:
>  ```
>  sudo ostree admin finalize-staged
>  Copying /etc changes: 10 modified, 0 removed, 34 added
>  **
>  OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel:
> assertion failed (kernel_layout->bootcsum == bootcsum):
> ("02ec07fa832b4485887f6dbac19d0311e3039855cc6ebd57aefaafa4a3997d1e" ==
> "fc8629fdb6f5355dd5359350296b619216cbc32e19887511caf75ee9d6aafbad")
>  Bail out!
> OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel:
> assertion failed (kernel_layout->bootcsum == bootcsum):
> ("02ec07fa832b4485887f6dbac19d0311e3039855cc6ebd57aefaafa4a3997d1e" ==
> "fc8629fdb6f5355dd5359350296b619216cbc32e19887511caf75ee9d6aafbad")
>  Aborted
>  
>  ```
>  
>  I’d guess that’s a fail originating somehow in my trying to increase the image and
> filesystem in the download as my first attempts to layer packages just ran out of space.
>  The approach worked with Fedora-IoT 29 and was based on decompressing, appending 0’s to
> the end of the file and then resizing the filesystem.
>  
>  Would it make sense to confirm the cause by layering a single package onto the original
> download, if that’s useful? I think that I will need some mechanism to increase the image
> size, tho’ or I won’t be able to get many tools onto the pi.
>  
>  Tim
>  
> > On 29 Aug 2020, at 06:02, Dusty Mabe <dusty(a)dustymabe.com&gt; wrote:
> > 
> > 
> > 
> > On 8/28/20 5:43 PM, TimC wrote:
> >> Hullo
> >> 
> >> I’m sure that I’m being a bit dim here and that I’ve seen this working. However.
> >> 
> >> Starting with a downloaded image for a Pi (3b+), after a `rpm-ostree upgrade`, I
> seem unable to deploy layered packages. They seem to install, but after the reboot,
> they’re not there. As an example, I tried to drop `toolbox` onto the image so that I could
> build some of my own rpms to run under systemd, using the approach here:
> https://bit.ly/3b8Ivri. The initial status before installation looks like this:
> >> ```
> >> rpm-ostree status
> >> State: idle
> >> Deployments:
> >> * ostree://fedora-iot:fedora/stable/aarch64/iot
> >>                 Version: 32.20200817.0 (2020-08-17T15:28:18Z)
> >>                  Commit:
> ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
> >>            GPGSignature: Valid signature by
> 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
> >> 
> >> ostree://fedora-iot:fedora/stable/aarch64/iot
> >>                 Version: 32.20200603.0 (2020-06-03T10:45:43Z)
> >>                  Commit:
> c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b
> >>            GPGSignature: Valid signature by
> 7D22D5867F2A4236474BF7B850CB390B3C3359C4
> >> ```
> >> 
> >> `sudo rpm-ostree install toolbox` progresses as I’d expect, creating a new
> bootable deployment:
> >> ```
> >> [tim@localhost ~]$ sudo rpm-ostree install toolbox
> >> Checking out tree ace3685... done
> >> Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora fedora-modular
> updates-modular
> >> rpm-md repo 'fedora-cisco-openh264' (cached); generated:
> 2020-03-17T20:10:43Z
> >> rpm-md repo 'updates' (cached); generated: 2020-08-27T14:18:53Z
> >> rpm-md repo 'fedora' (cached); generated: 2020-04-22T22:22:16Z
> >> rpm-md repo 'fedora-modular' (cached); generated: 2020-04-22T20:58:40Z
> >> rpm-md repo 'updates-modular' (cached); generated: 2020-08-27T14:52:01Z
> >> Importing rpm-md... done
> >> Resolving dependencies... done
> >> Checking out packages... done
> >> Running pre scripts... done
> >> Running post scripts... done
> >> Running posttrans scripts... done
> >> Writing rpmdb... done
> >> Writing OSTree commit... done
> >> ⠉ Staging deployment... 
> >> Staging deployment... done
> >> 
> >> Freed: 354.4?MB (pkgcache branches: 217)
> >> Added:
> >> flatpak-session-helper-1.6.5-1.fc32.aarch64
> >> toolbox-0.0.93-1.fc32.aarch64
> >> Run "systemctl reboot" to start a reboot
> >> 
> >> ```
> >> 
> >> and the status looks ok (ignoring the niggling error):
> >> ```
> >> [tim@localhost ~]$ rpm-ostree status
> >> error: Timeout was reached
> >> [tim@localhost ~]$ rpm-ostree status
> >> State: idle
> >> Deployments:
> >> ostree://fedora-iot:fedora/stable/aarch64/iot
> >>                 Version: 32.20200817.0 (2020-08-17T15:28:18Z)
> >>              BaseCommit:
> ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
> >>            GPGSignature: Valid signature by
> 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
> >>                    Diff: 2 added
> >>         LayeredPackages: toolbox
> >> 
> >> * ostree://fedora-iot:fedora/stable/aarch64/iot
> >>                 Version: 32.20200817.0 (2020-08-17T15:28:18Z)
> >>                  Commit:
> ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
> >>            GPGSignature: Valid signature by
> 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
> >> 
> >> ostree://fedora-iot:fedora/stable/aarch64/iot
> >>                 Version: 32.20200603.0 (2020-06-03T10:45:43Z)
> >>                  Commit:
> c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b
> >>            GPGSignature: Valid signature by
> 7D22D5867F2A4236474BF7B850CB390B3C3359C4
> >> ```
> >> 
> >> But when I reboot, that deployment isn’t the image used and it’s disappeared from
> rpm-ostree’s world:
> >> ```
> >> [tim@localhost ~]$ sudo rpm-ostree status
> >> State: idle
> >> Deployments:
> >> * ostree://fedora-iot:fedora/stable/aarch64/iot
> >>                 Version: 32.20200817.0 (2020-08-17T15:28:18Z)
> >>                  Commit:
> ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea
> >>            GPGSignature: Valid signature by
> 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
> >> 
> >> ostree://fedora-iot:fedora/stable/aarch64/iot
> >>                 Version: 32.20200603.0 (2020-06-03T10:45:43Z)
> >>                  Commit:
> c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b
> >>            GPGSignature: Valid signature by
> 7D22D5867F2A4236474BF7B850CB390B3C3359C4
> >> ```
> >> 
> >> Did I forget how this type of layered installation should work?
> > 
> > 
> > Seems like it should be working. This vaguely sounds like the deployment not getting
> finalized.
> > 
> > Does `sudo journalctl -u ostree-finalize-staged.service` show any errors? If it is
> timing out for
> > some reason you could try to manually run `sudo ostree admin finalize-staged`
> *before* you type
> > reboot to reboot into the new deployment so that you know it finishes.
> > 
> > Dusty
>  _______________________________________________
>  IoT mailing list -- iot(a)lists.fedoraproject.org
>  To unsubscribe send an email to iot-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

layered packages

[Dusty Mabe]

On 8/29/20 7:34 AM, TimC wrote:
>  Hi Dusty
>  
>  Is this a likely culprit?
>  ```
>  -- Reboot --
>  Aug 28 20:56:24 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged
> Deployment.
>  Aug 28 21:07:23 localhost.localdomain systemd[1]: Stopping OSTree Finalize Staged
> Deployment...
>  Aug 28 21:07:23 localhost.localdomain ostree[14561]: Finalizing staged deployment
>  Aug 28 21:07:32 localhost.localdomain ostree[14561]: Copying /etc changes: 10 modified, 0
> removed, 34 added
>  Aug 28 21:07:32 localhost.localdomain ostree[14561]: Copying /etc changes: 10 modified, 0
> removed, 34 added
>  Aug 28 21:08:30 localhost.localdomain ostree[14561]: **
>  Aug 28 21:08:30 localhost.localdomain ostree[14561]:
> OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel:
> assertion failed (kernel_layout->bootcsum == bootcsum):
> ("02ec07fa832b4485887f6dbac19d0311e3>
>  Aug 28 21:08:30 localhost.localdomain ostree[14561]: Bail out!
> OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel:
> assertion failed (kernel_layout->bootcsum == bootcsum):
> ("02ec07fa832b4485887f6dba>
>  Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Control
> process exited, code=dumped, status=6/ABRT
>  Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Failed
> with result 'core-dump'.
>  Aug 28 21:08:30 localhost.localdomain systemd[1]: Stopped OSTree Finalize Staged
> Deployment.
>  Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service:
> Consumed 8.211s CPU time.
>  ``` 
I would say so. It's either a bug or the things you did to modify the system
(mentioned below) caused some
corruption maybe.

>  
>  Note that this is from yesterday’s fail. When I tried again today, I only get this
> (before reboot):
>  ```
>  -- Reboot --
>  Aug 29 11:12:12 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged
> Deployment.
>  ```
>  
>  
>  When I run `sudo ostree admin finalize-staged`, it fails, thus:
>  ```
>   sudo ostree admin finalize-staged
>  Copying /etc changes: 10 modified, 0 removed, 34 added
>  **
>  OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel:
> assertion failed (kernel_layout->bootcsum == bootcsum):
> ("02ec07fa832b4485887f6dbac19d0311e3039855cc6ebd57aefaafa4a3997d1e" ==
> "fc8629fdb6f5355dd5359350296b619216cbc32e19887511caf75ee9d6aafbad")
>  Bail out!
> OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel:
> assertion failed (kernel_layout->bootcsum == bootcsum):
> ("02ec07fa832b4485887f6dbac19d0311e3039855cc6ebd57aefaafa4a3997d1e" ==
> "fc8629fdb6f5355dd5359350296b619216cbc32e19887511caf75ee9d6aafbad")
>  Aborted
>  
>  ```
>  
>  I’d guess that’s a fail originating somehow in my trying to increase the image and
> filesystem in the download as my first attempts to layer packages just ran out of space.
>  The approach worked with Fedora-IoT 29 and was based on decompressing, appending 0’s to
> the end of the file and then resizing the filesystem. 
I'm not sure if that is OK or not. I haven't used IoT much, but I do know the
rpm-ostree ecosystem a bit. Somebody else might
be able to answer better than I.

>  
>  Would it make sense to confirm the cause by layering a single package onto the original
> download, if that’s useful? I think that I will need some mechanism to increase the image
> size, tho’ or I won’t be able to get many tools onto the pi. 
That would be good information. 
> 
> Tim

layered packages

[Peter Robinson]

>  I’d guess that’s a fail originating somehow in my trying to increase
> the image and filesystem in the download as my first attempts to layer packages just ran
> out of space.
>  The approach worked with Fedora-IoT 29 and was based on decompressing, appending 0’s to
> the end of the file and then resizing the filesystem. 
How are you running this image? Is it a VM image or on a physical
piece of hardware?

If a VM you should be able to do this:
https://nullr0ute.com/2018/08/increasing-a-libvirt-kvm-virtual-machine-di...