A number of Fedora packages have adopted a GOT audit test that I wrote after the xz-utils attack. It looks for signs of ELF namespace tampering in running processes. I'd like to propose that Fedora deploy it as a generic test.
I talked to Cristian about that in the CI channel in June, and he said "it's fine from Fedora-CI technical perspective but needs more discussion on the layer deeper down". Cristian asked Maxwell G to review the test, and Maxwell suggested that we discuss it here when I followed up with him.
https://codeberg.org/gordonmessmer/got-audit-pipeline https://codeberg.org/gordonmessmer/got-audit
The version of "got-audit" above is a machine-generated port of the original:
https://github.com/hugsy/gef-extras/blob/main/scripts/got_audit.py https://github.com/hugsy/gef/blob/main/gef.py#L10700
As is often the case, the Python GDB extension was great for prototyping the tool, but it has required a lot of effort to keep it building from release to release.
The generic pipeline should check packages for systemd service units, start the units if they can be started automatically, and then audit each PID started by the service. Auditing non-service binaries is out of scope for the CI test.
On Mon, Aug 31, 2026, at 3:34 PM, Gordon Messmer wrote:
A number of Fedora packages have adopted a GOT audit test that I wrote after the xz-utils attack. It looks for signs of ELF namespace tampering in running processes. I'd like to propose that Fedora deploy it as a generic test.
I talked to Cristian about that in the CI channel in June, and he said "it's fine from Fedora-CI technical perspective but needs more discussion on the layer deeper down". Cristian asked Maxwell G to review the test, and Maxwell suggested that we discuss it here when I followed up with him.
My instinct is that while it does seem good to say "we have a thing here" in response to the xz attack, in practice that particular vector is probably considered "burned" (i.e. unlikely to be reused in the same way) by the attackers. The times this tool flags a problem will probably be 99% false positives (tool chain bugs, etc). There's quite a whole lot of possible attacks here of which mutating the GOT is just one.
Also a core problem in the general case is it's running from the system it's trying to scan, which has all the same problems that antivirus systems have.
But on the other hand, it will probably pass 99.9% of the time and so it will come down to measuring its cost over time, if it's pretty cheap and quick, that's probably fine - especially if it does happen to find toolchain bugs.
On 2026-08-31 2:04 PM, Colin Walters wrote:
My instinct is that while it does seem good to say "we have a thing here" in response to the xz attack, in practice that particular vector is probably considered "burned" (i.e. unlikely to be reused in the same way) by the attackers.
I think it's important to not confuse the instance for the class. Function hijacking is a class of attacks. XZ-utils was an instance of function hijacking.
When an adversary actively uses an instance of an attack, they risk burning that instance. If the attack is detected, and if it is traced to an instance of a vulnerability, then that instance can be mitigated, and the attack can no longer be used against anyone who deploys the mitigation.
It's not enough that the vulnerability is revealed, deploying the mitigation is the action that burns the instance of the vulnerability. That is also true of attack classes. If no one deploys a mitigation against a class of attacks, then that class of attacks remains viable.
I can't actually think of an example of a class of attacks that has been burned. We know about SQL Injection, but we continue to see SQL Injection attacks. We know about about JS Prototype Poisoning, but we continue to see Prototype Poisoning attacks.
Function hijacking was a well known technique before 2024, and IFUNC was already considered risky. There are more people aware of that vector now, sure. But I don't think there's any reason to believe that it won't be used again unless it's something we actively look for. Especially so since I'm not aware of any changes in the linker that would prevent future recurrence.
The times this tool flags a problem will probably be 99% false positives (tool chain bugs, etc).
Yes, I'm sure that's true. In particular, I think the handling of symbols that appear in multiple libraries needs more thought.
There's quite a whole lot of possible attacks here of which mutating the GOT is just one.
Also a core problem in the general case is it's running from the system it's trying to scan, which has all the same problems that antivirus systems have.
Antivirus systems are imperfect, and they wouldn't solve all possible problems even if they were. Yet anti-virus software is still widely used. An A-V system is not a guarantee against malware, but they do reduce the number of successful attacks on endpoints.
I don't think any security solution ever eliminates the need for other security tools, and I don't think it makes sense to run nothing while waiting for one comprehensive solution to all problems.
On Mon, 2026-08-31 at 12:34 -0700, Gordon Messmer wrote:
A number of Fedora packages have adopted a GOT audit test that I wrote after the xz-utils attack. It looks for signs of ELF namespace tampering in running processes. I'd like to propose that Fedora deploy it as a generic test.
I talked to Cristian about that in the CI channel in June, and he said "it's fine from Fedora-CI technical perspective but needs more discussion on the layer deeper down". Cristian asked Maxwell G to review the test, and Maxwell suggested that we discuss it here when I followed up with him.
https://codeberg.org/gordonmessmer/got-audit-pipeline https://codeberg.org/gordonmessmer/got-audit
The version of "got-audit" above is a machine-generated port of the original:
In case you are not aware - this seems to be against Codeberg's new terms of service
https://blog.codeberg.org/protecting-our-floss-commons-from-llms.html
Projects written and maintained with heavy use of LLMs
You might want to move it somewhere else before Codeberg admins take action.
HTH,
Not adding anything, I just realize that I tend to write subjects that aren't meaningful without context. Perhaps more context in the subject will invite more comments.
On 2026-08-31 12:34 PM, Gordon Messmer wrote:
A number of Fedora packages have adopted a GOT audit test that I wrote after the xz-utils attack. It looks for signs of ELF namespace tampering in running processes. I'd like to propose that Fedora deploy it as a generic test.
I talked to Cristian about that in the CI channel in June, and he said "it's fine from Fedora-CI technical perspective but needs more discussion on the layer deeper down". Cristian asked Maxwell G to review the test, and Maxwell suggested that we discuss it here when I followed up with him.
https://codeberg.org/gordonmessmer/got-audit-pipeline https://codeberg.org/gordonmessmer/got-audit
The version of "got-audit" above is a machine-generated port of the original:
https://github.com/hugsy/gef-extras/blob/main/scripts/got_audit.py https://github.com/hugsy/gef/blob/main/gef.py#L10700
As is often the case, the Python GDB extension was great for prototyping the tool, but it has required a lot of effort to keep it building from release to release.
The generic pipeline should check packages for systemd service units, start the units if they can be started automatically, and then audit each PID started by the service. Auditing non-service binaries is out of scope for the CI test.
On 2026-08-31 12:34 PM, Gordon Messmer wrote:
A number of Fedora packages have adopted a GOT audit test that I wrote after the xz-utils attack. It looks for signs of ELF namespace tampering in running processes. I'd like to propose that Fedora deploy it as a generic test.
I think this relates to https://forge.fedoraproject.org/fesco/tickets/issues/3683 and that should probibly be figured out first?
But aside that, do you mean that you want this to be a global test that is not gating? Or what exactly?
I definitely think we should make new global tests no gating at first to make sure there's not a bunch of false positives or infra problems before making them gating.
kevin
On 2026-09-04 10:05 AM, Kevin Fenzi wrote:
I think this relates to https://forge.fedoraproject.org/fesco/tickets/issues/3683 and that should probibly be figured out first?
Yeah, I would guess that Maxwell opened the ticket after I followed up with him (after Cristian asked him to review the test).
But aside that, do you mean that you want this to be a global test that is not gating? Or what exactly?
Yes, a global test that is not gating.
I definitely think we should make new global tests no gating at first to make sure there's not a bunch of false positives or infra problems before making them gating.
Definitely.
The things that I am concerned are mostly
On 2026-08-31 12:34 PM, Gordon Messmer wrote:
A number of Fedora packages have adopted a GOT audit test that I wrote after the xz-utils attack. It looks for signs of ELF namespace tampering in running processes. I'd like to propose that Fedora deploy it as a generic test.
I talked to Cristian about that in the CI channel in June, and he said "it's fine from Fedora-CI technical perspective but needs more discussion on the layer deeper down". Cristian asked Maxwell G to review the test, and Maxwell suggested that we discuss it here when I followed up with him.
https://codeberg.org/gordonmessmer/got-audit-pipeline https://codeberg.org/gordonmessmer/got-audit
The version of "got-audit" above is a machine-generated port of the original:
Apart from the codeberg policy violation, I am concerned about the age of the tool. Would it not have the same issues that python gdb extension is having or other? Mostly I would like someone with some domain knowledge to review the implementation, for example from the CMake perspective I would have multiple complaints.
https://github.com/hugsy/gef-extras/blob/main/scripts/got_audit.py https://github.com/hugsy/gef/blob/main/gef.py#L10700
As is often the case, the Python GDB extension was great for prototyping the tool, but it has required a lot of effort to keep it building from release to release.
Is the trade-off worth it? Can we not work on the "a lot of effort to keep it building" part? It seems to me that that would have a greater benefit to the overall ecosystem. Some more details on this would be good.
The generic pipeline should check packages for systemd service units, start the units if they can be started automatically, and then audit each PID started by the service. Auditing non-service binaries is out of scope for the CI test.
Why is it out of scope? To me the implementation against service units is fragile e.g. on a fresh installation of `mariadb` you would not be able to just systemctl start it since it needs to be configured first. My proposal here would be to hook it more generically, and of course I am biased towards having it as a tmt check plugin. The primary distinction would be that something/somebody else is in charge of defining how to start the service or other simple functional tests. I can work on an example repo to get you started on this.
On 2026-09-07 4:08 AM, Cristian Le via devel wrote:
On 2026-08-31 12:34 PM, Gordon Messmer wrote:
https://codeberg.org/gordonmessmer/got-audit-pipeline https://codeberg.org/gordonmessmer/got-audit
Apart from the codeberg policy violation, I am concerned about the age of the tool. Would it not have the same issues that python gdb extension is having or other? Mostly I would like someone with some domain knowledge to review the implementation, for example from the CMake perspective I would have multiple complaints.
I can easily move the repos elsewhere. For the most part I've been using codeberg because Fedora is adopting Forgejo and Matthew wanted me to make sure all of my work was functional with Codeberg actions.
"gef" has been a lot of work to keep building because it's a general purpose tool with a very broad test suite. Changes in gdb and in glibc have both cause test failures that require updates in "gef" to adapt. I don't think that will be true of got-audit because its purpose is much narrower. Less functionality means it's less sensitive to changes in the platform.
The generic pipeline should check packages for systemd service units, start the units if they can be started automatically, and then audit each PID started by the service. Auditing non-service binaries is out of scope for the CI test.
Why is it out of scope?
The intention is to detect back-doors in services that listen on network sockets. It's possible that someone might insert malware in software that doesn't run as a service, but I am not personally as concerned with that possibility.
To me the implementation against service units is fragile e.g. on a fresh installation of `mariadb` you would not be able to just systemctl start it since it needs to be configured first. My proposal here would be to hook it more generically, and of course I am biased towards having it as a tmt check plugin. The primary distinction would be that something/somebody else is in charge of defining how to start the service or other simple functional tests. I can work on an example repo to get you started on this.
There are services that may require extra steps to audit, and I'd be happy to work on tmt tests for cases where the generic test doesn't work. But it would be nice to get as much coverage as possible, and address the exceptions as needed, rather than doing every service one by one.
* Gordon Messmer:
A number of Fedora packages have adopted a GOT audit test that I wrote after the xz-utils attack. It looks for signs of ELF namespace tampering in running processes. I'd like to propose that Fedora deploy it as a generic test.
I talked to Cristian about that in the CI channel in June, and he said "it's fine from Fedora-CI technical perspective but needs more discussion on the layer deeper down". Cristian asked Maxwell G to review the test, and Maxwell suggested that we discuss it here when I followed up with him.
https://codeberg.org/gordonmessmer/got-audit-pipeline https://codeberg.org/gordonmessmer/got-audit
Does this light up on Red Hat's build of Fedora? It has heavy run-time patching of libc.so.6, apparently.
Thanks, Florian
On 2026-09-04 2:17 AM, Florian Weimer wrote:
https://codeberg.org/gordonmessmer/got-audit-pipeline https://codeberg.org/gordonmessmer/got-audit
Does this light up on Red Hat's build of Fedora? It has heavy run-time patching of libc.so.6, apparently.
I'm not sure what you mean by run-time patching... There are libraries in Fedora like libattr.so and libtirpc.so that provide identically named functions as functions in libc.so (although, with different symbol versions). The scanner does look for libraries that export identical function names, which might be a sign of function hijacking, but it has an allow-list for functions in a few libraries. I think that needs a better and more generalized approach.
Are you referring to some other run-time patching?
* Gordon Messmer:
On 2026-09-04 2:17 AM, Florian Weimer wrote:
https://codeberg.org/gordonmessmer/got-audit-pipeline https://codeberg.org/gordonmessmer/got-audit
Does this light up on Red Hat's build of Fedora? It has heavy run-time patching of libc.so.6, apparently.
I'm not sure what you mean by run-time patching... There are libraries in Fedora like libattr.so and libtirpc.so that provide identically named functions as functions in libc.so (although, with different symbol versions). The scanner does look for libraries that export identical function names, which might be a sign of function hijacking, but it has an allow-list for functions in a few libraries. I think that needs a better and more generalized approach.
Are you referring to some other run-time patching?
It's very narrow if this only detects on very specific form of run-time patching. Fedora CSB has this patch of libc.so.6. (I believe it's not malicious.)
000000000001e740 <putenv>: - 1e740: f3 0f 1e fa endbr64 + 1e740: cc int3 + 1e741: 0f 1e fa nop %edx 1e744: 55 push %rbp 1e745: be 3d 00 00 00 mov $0x3d,%esi 1e74a: 48 89 e5 mov %rsp,%rbp @@ -33543,7 +33544,8 @@ 2027b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
0000000000020280 <setenv>: - 20280: f3 0f 1e fa endbr64 + 20280: cc int3 + 20281: 0f 1e fa nop %edx 20284: 55 push %rbp 20285: 48 89 e5 mov %rsp,%rbp 20288: 48 83 ec 20 sub $0x20,%rsp @@ -33675,7 +33677,8 @@ 2047c: 00 00 00 00
0000000000020480 <clearenv>: - 20480: f3 0f 1e fa endbr64 + 20480: cc int3 + 20481: 0f 1e fa nop %edx 20484: 55 push %rbp 20485: 31 c0 xor %eax,%eax 20487: ba 01 00 00 00 mov $0x1,%edx
Thanks, Florian
On Mon, Sep 7, 2026, at 2:20 PM, Florian Weimer via devel wrote:
000000000001e740 <putenv>:
- 1e740: f3 0f 1e fa endbr64
- 1e740: cc int3
- 1e741: 0f 1e fa nop %edx 1e744: 55 push %rbp 1e745: be 3d 00 00 00 mov $0x3d,%esi 1e74a: 48 89 e5 mov %rsp,%rbp
@@ -33543,7 +33544,8 @@ 2027b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
0000000000020280 <setenv>:
- 20280: f3 0f 1e fa endbr64
- 20280: cc int3
- 20281: 0f 1e fa nop %edx 20284: 55 push %rbp 20285: 48 89 e5 mov %rsp,%rbp 20288: 48 83 ec 20 sub $0x20,%rsp
@@ -33675,7 +33677,8 @@ 2047c: 00 00 00 00
0000000000020480 <clearenv>:
- 20480: f3 0f 1e fa endbr64
- 20480: cc int3
- 20481: 0f 1e fa nop %edx 20484: 55 push %rbp 20485: 31 c0 xor %eax,%eax 20487: ba 01 00 00 00 mov $0x1,%edx
What does this do? It seems to SIGTRAP any call to environment modifying functions, is something listening to those as "runtime integrity" protection (or snooping) and if so what is that?
* Simon de Vlieger:
On Mon, Sep 7, 2026, at 2:20 PM, Florian Weimer via devel wrote:
000000000001e740 <putenv>:
- 1e740: f3 0f 1e fa endbr64
- 1e740: cc int3
- 1e741: 0f 1e fa nop %edx 1e744: 55 push %rbp 1e745: be 3d 00 00 00 mov $0x3d,%esi 1e74a: 48 89 e5 mov %rsp,%rbp
@@ -33543,7 +33544,8 @@ 2027b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
0000000000020280 <setenv>:
- 20280: f3 0f 1e fa endbr64
- 20280: cc int3
- 20281: 0f 1e fa nop %edx 20284: 55 push %rbp 20285: 48 89 e5 mov %rsp,%rbp 20288: 48 83 ec 20 sub $0x20,%rsp
@@ -33675,7 +33677,8 @@ 2047c: 00 00 00 00
0000000000020480 <clearenv>:
- 20480: f3 0f 1e fa endbr64
- 20480: cc int3
- 20481: 0f 1e fa nop %edx 20484: 55 push %rbp 20485: 31 c0 xor %eax,%eax 20487: ba 01 00 00 00 mov $0x1,%edx
What does this do? It seems to SIGTRAP any call to environment modifying functions, is something listening to those as "runtime integrity" protection (or snooping) and if so what is that?
Those are kernel uprobes. Instead of trapping, the kernel runs an eBPF script when these functions are called. But it breaks tools that see these int3 instructions and interpret them as traps.
Thanks, Florian
On 2026-09-07 5:20 AM, Florian Weimer wrote:
It's very narrow if this only detects on very specific form of run-time patching.
So, yes, got-audit only checks for function hijacking via GOT tampering. But we can add new tools to look for other types of run-time patching. It looks like this example would be easy to detect. We'd just need to find out if any of this exists in non-CSB environments (i.e. Fedora CI), and figure out how to allow-list anything we're aware of that's legitimate and expected in that environment.
This is interesting. I'll start work on it.
On 2026-09-07 5:20 AM, Florian Weimer wrote:
000000000001e740 <putenv>:
- 1e740: f3 0f 1e fa endbr64
- 1e740: cc int3
- 1e741: 0f 1e fa nop %edx 1e744: 55 push %rbp 1e745: be 3d 00 00 00 mov $0x3d,%esi 1e74a: 48 89 e5 mov %rsp,%rbp
Before I do any work on a new CI test: How did you generate this diff?
* Gordon Messmer:
On 2026-09-07 5:20 AM, Florian Weimer wrote:
000000000001e740 <putenv>:
- 1e740: f3 0f 1e fa endbr64
- 1e740: cc int3
- 1e741: 0f 1e fa nop %edx 1e744: 55 push %rbp 1e745: be 3d 00 00 00 mov $0x3d,%esi 1e74a: 48 89 e5 mov %rsp,%rbp
Before I do any work on a new CI test: How did you generate this diff?
I used the program below to dum /usr/lib64/libc.so.6, and then ran objdump -d --reloc.
Thanks, Florian
#include <err.h> #include <errno.h> #include <fcntl.h> #include <string.h> #include <sys/mman.h> #include <sys/stat.h> #include <unistd.h>
struct mapped { void *buffer; size_t length; };
static bool open_mmap (const char *file, int flags, struct mapped *result) { int fd = open (file, O_RDONLY); if (fd < 0) return false; struct statx st; if (statx (fd, "", AT_EMPTY_PATH, STATX_SIZE, &st) != 0) { close (fd); return false; } void *ptr; if (st.stx_size == 0) ptr = NULL; else ptr = mmap (NULL, st.stx_size, PROT_READ, MAP_FILE | flags, fd, 0); close (fd); if (ptr == MAP_FAILED) return false; result->buffer = ptr; result->length = st.stx_size; return true; }
static void open_mmap_unmap (struct mapped *result) { if (result->length > 0) { munmap (result->buffer, result->length); result->buffer = NULL; result->length = 0; } }
int main (int argc, char **argv) { for (int i = 1; i < argc; ++i) { struct mapped mprivate = {NULL, 0}; if (open_mmap (argv[i], MAP_PRIVATE, &mprivate)) { void *p = mprivate.buffer; void *end = p + mprivate.length; while (p != end) { ssize_t count = write (STDOUT_FILENO, p, end - p); if (count == 0) { errno = ENOSPC; count = -1; } if (count < 0) err (1, "write"); p += count; } } open_mmap_unmap (&mprivate); } }
On 2026-09-15 10:57 PM, Florian Weimer wrote:
I used the program below to dum /usr/lib64/libc.so.6, and then ran objdump -d --reloc.
Thanks!
I've added: https://codeberg.org/gordonmessmer/maps-audit
and updated: https://codeberg.org/gordonmessmer/got-audit-pipeline
"maps-audit" successfully detects CSB's run-time patching.
I've added a "scan" script that iterates over all of the packages in Fedora that have service units, and scans each of them. Just under 2/3 start without configuration and can be scanned.
With that, I found just one more set of libraries that had symbol duplication. And at that point, I think there was enough information available to try to put in place a general heuristic to minimize the number of symbols that have to be simply allow-listed.
I've checked the got-audit tool against the compromised library and confirmed that the back-door is still detected, and I've run the scan against Fedora 44 and found no alerts (as expected).
https://github.com/gordonmessmer/got-audit
https://github.com/gordonmessmer/maps-audit
https://github.com/gordonmessmer/got-audit-pipeline
https://github.com/gordonmessmer/got-audit-scan
https://copr.fedorainfracloud.org/coprs/gordonmessmer/got-audit/
-
Colin Walters -
Cristian Le -
Florian Weimer -
Gordon Messmer -
Kevin Fenzi -
Michel Lind -
Simon de Vlieger