memory testing

[Chris Murphy]

Hi,

While bad RAM is uncommon, it comes up with some regularity to cause
folks a lot of grief. I'm wondering if there's a way to make it easier
to get bad news :-\ In particular there are cases where RAM defects
just don't show up with a few hours of memtest86+, it can take days of
contiguous testing, which is so inconvenient the test itself seems
worse.

Here's what I've got so far:

1. Fedora includes /boot/memtest86+-5.01 on every installation. But
this is a legacy/BIOS program. The idea of recommending folks enable
CSM/legacy BIOS just to test their RAM is questionable because it
means disabling UEFI Secure Boot to do it. Lie in wait malware is
perhaps rare but plausible.  UEFI native memtest86+ is not free so it
can't be included. I kinda wonder if including this should be
deprecated?

2. The kernel has a built-in memory tester. Therefore it can run on
anything. But how good is it? Is it worth enabling? Should it be
enabled for all kernels or just debug kernels? The code is pretty
simple, so will it catch only the worst cases of bad RAM?
# CONFIG_MEMTEST is not set
https://elixir.bootlin.com/linux/v5.8-rc4/source/mm/memtest.c

3. "memory interface test" used at Google, Apache 2.0 license
https://github.com/stressapptest/stressapptest

4. "multiple concurrent kernel compiles" and "GCC seems to have memory
usage patterns that reliably trigger memory errors that
aren't caught by memtest"
https://lore.kernel.org/linux-btrfs/799cf552-4612-56c5-b44d-59458119e2b0@...

Example of btrfs catching a bit flip:
https://lore.kernel.org/linux-btrfs/f42fc0d6-5dc9-dd15-9d61-53efb04fad33@...
And also, this is not a good example of a memory tester. Some of the
time the corruption happens before the csum is computed so, it's not
going to catch everything.

Any other ideas how to make this better?

Thanks,
-- 
Chris Murphy

[Laura Abbott]

On 7/15/20 1:11 PM, Chris Murphy wrote:
>  Hi,
>  
>  While bad RAM is uncommon, it comes up with some regularity to cause
>  folks a lot of grief. I'm wondering if there's a way to make it easier
>  to get bad news :-\ In particular there are cases where RAM defects
>  just don't show up with a few hours of memtest86+, it can take days of
>  contiguous testing, which is so inconvenient the test itself seems
>  worse.
>  
>  Here's what I've got so far:
>  
>  1. Fedora includes /boot/memtest86+-5.01 on every installation. But
>  this is a legacy/BIOS program. The idea of recommending folks enable
>  CSM/legacy BIOS just to test their RAM is questionable because it
>  means disabling UEFI Secure Boot to do it. Lie in wait malware is
>  perhaps rare but plausible.  UEFI native memtest86+ is not free so it
>  can't be included. I kinda wonder if including this should be
>  deprecated?
>  
>  2. The kernel has a built-in memory tester. Therefore it can run on
>  anything. But how good is it? Is it worth enabling? Should it be
>  enabled for all kernels or just debug kernels? The code is pretty
>  simple, so will it catch only the worst cases of bad RAM?
>  # CONFIG_MEMTEST is not set
>  https://elixir.bootlin.com/linux/v5.8-rc4/source/mm/memtest.c
>   
I wouldn't bother with CONFIG_MEMTEST. It's designed to work
around the problem by calling memblock_reserve so the bad
memory doesn't get used. It's nice if you want to keep a machine
running but that doesn't sound like what we're going for.

>  3. "memory interface test" used at Google, Apache 2.0
> license
>  https://github.com/stressapptest/stressapptest
>  
>  4. "multiple concurrent kernel compiles" and "GCC seems to have memory
>  usage patterns that reliably trigger memory errors that
>  aren't caught by memtest"
>  https://lore.kernel.org/linux-btrfs/799cf552-4612-56c5-b44d-59458119e2b0@...
>  
>  Example of btrfs catching a bit flip:
>  https://lore.kernel.org/linux-btrfs/f42fc0d6-5dc9-dd15-9d61-53efb04fad33@...
>  And also, this is not a good example of a memory tester. Some of the
>  time the corruption happens before the csum is computed so, it's not
>  going to catch everything.
>  
>  Any other ideas how to make this better?
>   
Detecting hardware faults is a very hard problem unfortunately. I
brought up this question at a conference a few years ago in the
context of determining real bugs from hardware issues and nobody
had any great suggestions. Much of it ends up coming down to analyzing
the crash you are seeing and trying to figure out what doesn't make
sense.

Doing multiple kernel compiles is probably about as effective as
anything I know about.

Thanks,
Laura

[Chris Murphy]

On Wed, Jul 15, 2020 at 12:09 PM Laura Abbott <laura(a)labbott.name&gt; wrote:
>  Detecting hardware faults is a very hard problem unfortunately. I
>  brought up this question at a conference a few years ago in the
>  context of determining real bugs from hardware issues and nobody
>  had any great suggestions. Much of it ends up coming down to analyzing
>  the crash you are seeing and trying to figure out what doesn't make
>  sense.
>
>  Doing multiple kernel compiles is probably about as effective as
>  anything I know about. 
As tedious as this is, I think it's way more helpful to hear
confirmation that this is in fact a difficult problem, and what's
important is to stick with the testing and process of elimination,
rather than suffering through it.


-- 
Chris Murphy

[Brandon Nielsen]

On 7/15/20 12:11 PM, Chris Murphy wrote:
>  Hi,
>  
>  While bad RAM is uncommon, it comes up with some regularity to cause
>  folks a lot of grief. I'm wondering if there's a way to make it easier
>  to get bad news :-\ In particular there are cases where RAM defects
>  just don't show up with a few hours of memtest86+, it can take days of
>  contiguous testing, which is so inconvenient the test itself seems
>  worse.
>  
>  Here's what I've got so far:
>  
>  1. Fedora includes /boot/memtest86+-5.01 on every installation. But
>  this is a legacy/BIOS program. The idea of recommending folks enable
>  CSM/legacy BIOS just to test their RAM is questionable because it
>  means disabling UEFI Secure Boot to do it. Lie in wait malware is
>  perhaps rare but plausible.  UEFI native memtest86+ is not free so it
>  can't be included. I kinda wonder if including this should be
>  deprecated?
>   
<Snip>

While I consider the effort to keep Memtest86+ working in Fedora pretty 
heroic, and I've never had it fail to detect bad memory for me, I feel 
like shipping a tool that spuriously fails on modern hardware when 
attempting to detect spurious hardware failures[0][1] is more of a 
negative than a positive. While it seems fixed in Rawhide, with a dead 
upstream I'm sure it's just a matter of time until it happens again 
(likely when the necessary GCC compat package gets dropped).

0 - 
https://lists.fedoraproject.org/archives/list/[e-mail]...
1 - https://bugzilla.redhat.com/show_bug.cgi?id=1811353

[Michael Catanzaro]

Note that in my mail, I was referring to proprietary memtest86. I 
haven't tried memtest86+ on my current system since it's clearly 
obsolete.

On Wed, Jul 15, 2020 at 1:20 pm, Brandon Nielsen <nielsenb(a)jetfuse.net&gt; 
wrote:
>  While I consider the effort to keep Memtest86+ working in Fedora 
>  pretty
>  heroic, and I've never had it fail to detect bad memory for me, I feel
>  like shipping a tool that spuriously fails on modern hardware when
>  attempting to detect spurious hardware failures[0][1] is more of a
>  negative than a positive. While it seems fixed in Rawhide, with a dead
>  upstream I'm sure it's just a matter of time until it happens again
>  (likely when the necessary GCC compat package gets dropped). 
Note: memtest86+ actually had an upstream release recently after a 
*very* long hiatus, so I guess it's no longer dead. But I agree 
memtest86+ should be dropped, since it's incompatible with UEFI and 
surely not what we want people to try using.

[Solomon Peachy]

On Wed, Jul 15, 2020 at 01:41:27PM -0500, Michael Catanzaro wrote:
>  Note: memtest86+ actually had an upstream release recently after a
> *very*
>  long hiatus, so I guess it's no longer dead. But I agree memtest86+ should
>  be dropped, since it's incompatible with UEFI and surely not what we want
>  people to try using. 
Yes and no -- having it available on the fedora installer media is quite 
useful, and since that's going to have to support dual BIOS/UEFI 
booting, there's not really any downside.

(beyond keeping it building anyway..)

 - Solomon
-- 
Solomon Peachy			      pizza at shaftnet dot org (email&xmpp)
                                      @pizza:shaftnet dot org   (matrix)
High Springs, FL                      speachy (freenode)

[Chris Murphy]

On Wed, Jul 15, 2020 at 12:49 PM Solomon Peachy <pizza(a)shaftnet.org&gt; wrote:
>  On Wed, Jul 15, 2020 at 01:41:27PM -0500, Michael Catanzaro wrote:
>  > Note: memtest86+ actually had an upstream release recently after a *very*
>  > long hiatus, so I guess it's no longer dead. But I agree memtest86+ should
>  > be dropped, since it's incompatible with UEFI and surely not what we want
>  > people to try using.
>
>  Yes and no -- having it available on the fedora installer media is quite
>  useful, and since that's going to have to support dual BIOS/UEFI
>  booting, there's not really any downside. 
The same binary can't support both firmware types. As far as I know,
only the proprietary version offers a UEFI binary. And long ago the
plan for the free version was not to have dual support, but to
eventually deprecate the BIOS tester and only offer a UEFI binary.

But another difficulty even if there will be a free UEFI memtest86+ is
that it would need to be signed for UEFI Secure Boot, same as shim and
the kernel (and kernel modules). Seems like its own burden.

Does anyone know if Microsoft has a signed UEFI memory tester? I don't
exactly like the idea of recommending something outside our purview,
and also not being available within Fedora, but a signed memory tester
is better than one not signed.


-- 
Chris Murphy

[Vitaly Zaitsev]

On 15.07.2020 21:17, Chris Murphy wrote:
>  Does anyone know if Microsoft has a signed UEFI memory tester?
Yes, but it requires Windows Native API in order to work.

-- 
Sincerely,
  Vitaly Zaitsev (vitaly(a)easycoding.org)

[Peter Jones]

On Wed, Jul 15, 2020 at 01:17:50PM -0600, Chris Murphy wrote:
>  On Wed, Jul 15, 2020 at 12:49 PM Solomon Peachy
> <pizza(a)shaftnet.org&gt; wrote:
>  >
>  > On Wed, Jul 15, 2020 at 01:41:27PM -0500, Michael Catanzaro wrote:
>  > > Note: memtest86+ actually had an upstream release recently after a *very*
>  > > long hiatus, so I guess it's no longer dead. But I agree memtest86+ should
>  > > be dropped, since it's incompatible with UEFI and surely not what we want
>  > > people to try using.
>  >
>  > Yes and no -- having it available on the fedora installer media is quite
>  > useful, and since that's going to have to support dual BIOS/UEFI
>  > booting, there's not really any downside.
>  
>  The same binary can't support both firmware types. As far as I know,
>  only the proprietary version offers a UEFI binary. And long ago the
>  plan for the free version was not to have dual support, but to
>  eventually deprecate the BIOS tester and only offer a UEFI binary. 
If someone decided they wanted to spend some time making memtest86+ work
as an EFI binary, I'd be happy to mentor them.  I think getting it to
the point where it can test everything except the RAM UEFI is actually
/using/ is probably a medium difficulty but not huge project.  It's been
waaay down on my todo list forever, and I've got some ideas about how to
go about it, just not the time to do it myself.

>  But another difficulty even if there will be a free UEFI memtest86+
> is
>  that it would need to be signed for UEFI Secure Boot, same as shim and
>  the kernel (and kernel modules). Seems like its own burden. 
If we had a UEFI memory tester, you could reasonably add a boot entry
for shim with memtest.efi as the optional loader data (like we do for
fwupd) and enroll it in MoK with something like:

	pesign -i memtest.efi -P -h | while read name hash ; do
		mokutil --import-hash ${hash}
	done

There's not a super *huge* benefit to actually signing it, since you're
sure to be rebooting when you need to use it anyway.

-- 
        Peter

[Michael Catanzaro]

On Wed, Jul 15, 2020 at 11:11 am, Chris Murphy 
<lists(a)colorremedies.com&gt; wrote:
>  4. "multiple concurrent kernel compiles" and "GCC
> seems to have memory
>  usage patterns that reliably trigger memory errors that
>  aren't caught by memtest" 
In my experience, GCC is a really good test for RAM that is obviously 
bad. By "obviously bad," I mean RAM that will fail memtest86 every 
time. I've had two previous computers with obviously-bad RAM. I don't 
remember the first computer well, but on the second, GCC would often 
fail trying to compile assembler that was suspiciously one character 
away from being valid assembler. That's a pretty clear indication of 
trouble.

Sadly, because I am apparently cursed, my current system is exhibiting 
more subtle hardware errors. E.g. yesterday my desktop hung during a 
BlueJeans call [1]. I get occasional system lockups and crashes, 
sometimes many times in the same day; other times, the system might be 
stable for a week or more. I've finally been convinced this is really a 
hardware rather than software problem, so I'm ready to try swapping out 
components in hopes of finding what's wrong, but without a reliable 
test, it will be extremely hard to be sure if swapping out hardware has 
fixed the issue. GCC works flawlessly; I build large projects on a 
regular basis, and I've yet to see issues like I used to see regularly 
in the past on my system with obviously bad RAM. I've lost track of how 
much time I've spent running memtest86. I did manage to see a bad write 
in memtest86 once, but only once out of probably 20 different runs. 
Maybe if I were to run memtest for an entire week, that might be enough 
to reliably catch the problem, but I don't have that much patience, and 
even then, I honestly don't think it would be enough to be reliable. 
Just running it over long weekends is not enough; I've run memtest86 
for 30+ hours multiple times without a single error. It's very 
frustrating. I've tried mprime, stress, stress-ng, stressapptest, no 
luck with any of these. I might try replacing all my RAM with ECC RAM, 
just because I can't think of what else to try at this point, but my 
niggling worry is that the problem could just as likely be the CPU or 
the motherboard. I have no idea.

I've half a mind to take this system to Microcenter and see if they can 
figure it out, but my fear is that if I can't find a test that catches 
the issue, they won't be able to either.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1856846#c2

[Ken Gaillot]

On Wed, 2020-07-15 at 11:11 -0600, Chris Murphy wrote:
>  Hi,
>  
>  While bad RAM is uncommon, it comes up with some regularity to cause
>  folks a lot of grief. I'm wondering if there's a way to make it
>  easier
>  to get bad news :-\ In particular there are cases where RAM defects
>  just don't show up with a few hours of memtest86+, it can take days
>  of
>  contiguous testing, which is so inconvenient the test itself seems
>  worse. 
An interesting feature many people don't know about is EDAC for ECC
RAM. When a memory error occurs, the kernel will log a message like:

EDAC MC0: CE page 0x6ba7a, offset 0x800, grain 128, syndrome 0xf8, row
0, channel 0, label "": i3000 CE

and keep a running count (since boot) under
/sys/devices/system/edac/mc. You can track down errors to a specific
memory stick (if you have a secret decoder ring for your motherboard).

At a previous employer, we wrote a custom nagios plugin to monitor that
and alert us for errors on our servers.

For more info, see edac-util and edac-ctl from the edac-utils package
and:

https://buttersideup.com/mediawiki/index.php/Main_Page

https://www.kernel.org/doc/html/latest/driver-api/edac.html

Of course you need ECC RAM, but if you care about memory errors, you
should be using it anyway.

>  Here's what I've got so far:
>  
>  1. Fedora includes /boot/memtest86+-5.01 on every installation. But
>  this is a legacy/BIOS program. The idea of recommending folks enable
>  CSM/legacy BIOS just to test their RAM is questionable because it
>  means disabling UEFI Secure Boot to do it. Lie in wait malware is
>  perhaps rare but plausible.  UEFI native memtest86+ is not free so it
>  can't be included. I kinda wonder if including this should be
>  deprecated?
>  
>  2. The kernel has a built-in memory tester. Therefore it can run on
>  anything. But how good is it? Is it worth enabling? Should it be
>  enabled for all kernels or just debug kernels? The code is pretty
>  simple, so will it catch only the worst cases of bad RAM?
>  # CONFIG_MEMTEST is not set
>  https://elixir.bootlin.com/linux/v5.8-rc4/source/mm/memtest.c
>  
>  3. "memory interface test" used at Google, Apache 2.0 license
>  https://github.com/stressapptest/stressapptest
>  
>  4. "multiple concurrent kernel compiles" and "GCC seems to have
>  memory
>  usage patterns that reliably trigger memory errors that
>  aren't caught by memtest"
>  https://lore.kernel.org/linux-btrfs/799cf552-4612-56c5-b44d-59458119e2b0@...
>  
>  Example of btrfs catching a bit flip:
>  https://lore.kernel.org/linux-btrfs/f42fc0d6-5dc9-dd15-9d61-53efb04fad33@...
>  And also, this is not a good example of a memory tester. Some of the
>  time the corruption happens before the csum is computed so, it's not
>  going to catch everything.
>  
>  Any other ideas how to make this better?
>  
>  Thanks,
>  -- 
>  Chris Murphy 
-- 
Ken Gaillot <kgaillot(a)redhat.com&gt;

[Przemek Klosowski]

On 7/15/20 1:11 PM, Chris Murphy wrote:
>  Hi,
>
>  While bad RAM is uncommon, it comes up with some regularity to cause
>  folks a lot of grief. I'm wondering if there's a way to make it easier
>  to get bad news :-\ In particular there are cases where RAM defects
>  just don't show up with a few hours of memtest86+, it can take days of
>  contiguous testing, which is so inconvenient the test itself seems
>  worse.
>
>  Here's what I've got so far:
>
>  1. Fedora includes /boot/memtest86+-5.01 on every installation. But this is a legacy/BIOS
> program. [..]
>  2. The kernel has a built-in memory tester. [..]
>  3. "memory interface test" used at Google,[..]
>  4. "multiple concurrent kernel compiles" 
There's also 5. memtester, http://pyropus.ca/software/memtester/ ,  
packaged in Fedora. It runs in userland, so obvously isn't as thorough 
but I'd think it has got to better than gcc and other userland 
utilities, because it tries the 'known-tricky' access patterns rather 
than hoping to see them in compiler output.

I remember the beginnings of memtest86+: its predecessor was written at 
the SGI hardware division by folks who understood how e.g specific 
access paterns appear on the physical traces on the memory modules, and 
can lead to crosstalk, but only if you simulataneously and repeatedly 
toggle very specific bits. This type of knowledge was essential to 
writing good tests, and probably also to discovering vulnerabilites like 
rowhammer.

Have you looked at memtester? What do you think of it?

[Chris Murphy]

On Wed, Jul 15, 2020 at 2:55 PM Przemek Klosowski via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  On 7/15/20 1:11 PM, Chris Murphy wrote:
>  > Hi,
>  >
>  > While bad RAM is uncommon, it comes up with some regularity to cause
>  > folks a lot of grief. I'm wondering if there's a way to make it easier
>  > to get bad news :-\ In particular there are cases where RAM defects
>  > just don't show up with a few hours of memtest86+, it can take days of
>  > contiguous testing, which is so inconvenient the test itself seems
>  > worse.
>  >
>  > Here's what I've got so far:
>  >
>  > 1. Fedora includes /boot/memtest86+-5.01 on every installation. But this is a
> legacy/BIOS program. [..]
>  > 2. The kernel has a built-in memory tester. [..]
>  > 3. "memory interface test" used at Google,[..]
>  > 4. "multiple concurrent kernel compiles"
>
>  There's also 5. memtester, http://pyropus.ca/software/memtester/ ,
>  packaged in Fedora. It runs in userland, so obvously isn't as thorough
>  but I'd think it has got to better than gcc and other userland
>  utilities, because it tries the 'known-tricky' access patterns rather
>  than hoping to see them in compiler output.
>
>  I remember the beginnings of memtest86+: its predecessor was written at
>  the SGI hardware division by folks who understood how e.g specific
>  access paterns appear on the physical traces on the memory modules, and
>  can lead to crosstalk, but only if you simulataneously and repeatedly
>  toggle very specific bits. This type of knowledge was essential to
>  writing good tests, and probably also to discovering vulnerabilites like
>  rowhammer.
>
>  Have you looked at memtester? What do you think of it? 
I haven't. It might be useful to know what Michael Catanzaro thinks of
it, before he goes off to buy and install ECC RAM!


-- 
Chris Murphy

[Michael Catanzaro]

On Wed, Jul 15, 2020 at 4:05 pm, Chris Murphy <lists(a)colorremedies.com&gt; 
wrote:
>  I haven't. It might be useful to know what Michael Catanzaro
> thinks of
>  it, before he goes off to buy and install ECC RAM! 
I've been running it since last night and it hasn't turned up any 
problems. Very tricky. :/

[Kamil Paral]

On Wed, Jul 15, 2020 at 10:56 PM Przemek Klosowski via devel <
devel(a)lists.fedoraproject.org&gt; wrote:

>  Have you looked at memtester? What do you think of it?
>  
I've successfully used memtester in the past to detect suspend-resume
memory corruption on my desktop, just by periodically suspending and
resuming the PC while memtester was running. It helped me to confirm that
that particular desktop really has some memory corruption issues (while
other PCs I tested on didn't). So in this particular case I was satisfied
with the tool.

The downside is that it's a userspace utility, so it can't test the whole
memory. When I e.g. purchase a new RAM stick, I prefer to boot into
memtest86+ (I switch to BIOS mode to be able to boot it) and test the whole
stick with it. Anything running inside an OS can't provide you with the
same level of verification.