I'd consider focusing on using checkisomd5 as the main way to
test if FMW
works ok. First the offline one and then online one running from the flash
drive before bootup.
That's actually a very good advice for the test case. As Martin showed me recently,
running simply "checkisomd5 --verbose /dev/sdX" after you've written the
image to /dev/sdX is actually a fully satisfactory verification that the image was written
properly (or you could also do "cmp /dev/sdX image.iso"). Of course booting the
thumb stick and letting it verify consistency from the syslinux menu is a bonus point.
There's a potential issue here. Sometimes it can happen that the written partitions
are automounted and some filesystem metadata are changed before unplugging (at least
that's my guess). This happened to me several times during the last cycle, and the
checksum verification then fails (but the installation proceeds completely OK). This
happens also on Windows, according to Martin, and it's quite difficult to prevent it.
We should mention it in the test case, but I'm not sure what advice we can give. We
don't want to of course handwaive such issues completely, because they can also mean
some data writing corruption (I found one such bug a few months back, and it was FMW's
fault).
If the integrity check passes, all other errors are either in the compose
process or the packages. Anything above this I would consider testing the
actual written image, not FMW.
Yes. In our particular case, in the "Default boot and install" section, we also
test default installation (anaconda), so I guess we can't optimize there. But for that
particular test case, the offline/online verification should be enough, I believe.