Hello,
On Wednesday, September 22, 2021 7:21:42 AM EDT Miro Hrončok wrote:
for many releases, Fedora has the brp-mangle-sehbangs BuildRoot
Policy
Script that does the following:
1) Gets all executable files in the buildroot
2) Gets all "text" files from those
3a) Mangles shebangs that are "wrong"
(e.g. #!/usr/bin/env node -> #!/usr/bin/node)
3b) Removes executable bits from "text" files without shebangs
This is interesting. I didn't know Fedora had such a policy. I have been
doing studies of this myself because fapolicyd wants correctly identified
scripts.
The idea behind this is that all "text" files that are
executable need a
shebang and if they don't have it, something is wrong. OTOH files that are
"binary" don't need it.
I intentionally put the terms "text" and "binary" in quotation marks,
as
the definition is somewhat fuzzy. Up until now, the script did the
detection by utilizing the file tool to get the MIME type. If the MIME
type starts with text/, it considered the executable to be a text file.
I find the file utility to be almost reliable. It changes how it identifies ELF
files every couple releases. So, to stabilize this, fapolicyd-cli uses it's
own logic to determine what kind of ELF file it finds. I also regularly find
text/plain files where it cannot identify the language and files that are
application/octet-stream which are also misidentified.
However, a bug [1] has been discovered. Some obvious text files, such
as
executable JavaScript scripts, are detected as application/ (e.g.
application/javascript), and hence are not considered "text".
This is another inconsistency with libmagic that we do battle with. It can
change on th next release. Another example of this is python
misidentification. In order to have any stability and correctness, fapolicyd
ships with it's own libmagic override file. You might find
fapolicyd-cli --ftype a bit more stable. I also put new languages we discover
in the override while we are waiting for the patch to be accepted upstream.
And I think upstream has not accepted a couple patches for languages libmagic
can't detect right.
If a JavaScript executable script has the #!/usr/bin/env node
shebang, the
brp-mangle-sehbangs script does not mangle it.
One possible solution [2] to this problem is to limit the number of bytes
the MIME detection reads. My experiments showed that limiting the number
of bytes to 8 always recognizes JavaScript (and other scripting languages)
files as text/plain and binary files as application/octet-stream. As a
side effect, it might make the BRP script faster. However, I am not sure
if this approach is deterministic enough.
Another solution, suggested by Florian Weimer [3], is to not detect MIME
type at all, but use eu-elfclassify instead. The idea is quite simple:
If (and only if) the executable file is ELF [4], it does not require a
shebang. Instead of some fragile idea about what files are text and what
files are binary, this is quite deterministic. It allows mangling shebangs
of executable ZIP files etc.
I've drafted the eu-elfclassify solution in a pull request [5]. However, we
have discovered that several non-elf binary formats in Fedora are
possibly legitimately executable. E.g. .exe files (for mono or wine) or
other formats registered with the kernel [6].
We are presented with 3 possible actions:
1) Keep the script as it is, say the text/ MIME type limitation is how this
BRP script was scoped. Affected packages would need to correct shebangs
manually.
2) Limit the MIME type detection to 8 bytes and hope it will not yield
incorrect results.
3) Use eu-elfclassify. Consider non-ELF executables without shebangs bogus
and document this. Packages that are affected would need to opt-out.
What do you think?
4) maybe fapolicyd-cli has better detection? Or at least, its more closely
maintained. It also has it's own ELF detection so that it's stable from
release to release.
-Steve