Hi,
I've been having some issues with LMDB on i686/s390x when building python-zarr and its dependants. I filed a bug report [1], but the maintainer suggested asking here to get more reach. I have replicated my original message from the report below:
The documentation for LMDB [2] states that the map size should be "chosen as large as possible". Consequently, the zarr developers have chosen a map size of 2**40 on 64-bit systems and 2**28 on 32-bit systems [3]. When the package is built on a 64-bit system [4], everything works fine. However, if the build is run on a 32-bit system [5], it fails with a MemoryError.
I have narrowed this down to the mmap call at [6] which attempts to map the backing file into memory. AFAICT, the mapped size is far below the RAM on the build machine as well as far below the normal 32-bit VM limit. So I don't know why the call is failing.
I can reproduce this in mock with the fedora-rawhide-i386 config, but what's weird is it also fails in the fedora-rawhide-s390x config. This machine has 16G of RAM, so it's enough for 2**28, but not 2**40. So it fails on 32-bit x86 with more than enough RAM, works on 64-bit x86 with nowhere near enough RAM, yet fails with 64-bit s390x.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1695525 [2] http://www.lmdb.tech/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5 [3] https://github.com/zarr-developers/zarr/blob/f6ced1e31b919065f8834d813ec081d... [4] https://koji.fedoraproject.org/koji/taskinfo?taskID=33910305 [5] https://koji.fedoraproject.org/koji/taskinfo?taskID=33881973 [6] http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=libraries/l...
On 04/06/2019 10:03, Elliott Sales de Andrade wrote:
I have narrowed this down to the mmap call at [6] which attempts to map the backing file into memory. AFAICT, the mapped size is far below the RAM on the build machine as well as far below the normal 32-bit VM limit. So I don't know why the call is failing.
It's not really about the physical memory because just doing the mmap won't actually read it all in and in any case the kernel will be quite happy to page bits in and out as needed.
The issue will be whether there is that much contiguous address space available - on a 32 bit machine 2**28 is, at best, one sixteenth of the potential address space.
I say at best because even with the best possible user:kernel split at least some address space is reserved to the kernel although that can be fairly small with some kernel configurations.
Then you have to take the memory map into account though, and how address randomisation may have placed other mappings in locations that mean such a large contiguous mapping is not in fact available.
Take a look an /proc/NNN/maps for the process when the mmap fails and see if it looks like there it a range of that size that is actually available...
Tom
Hi Tom,
On Tue, 4 Jun 2019 at 05:22, Tom Hughes tom@compton.nu wrote:
On 04/06/2019 10:03, Elliott Sales de Andrade wrote:
I have narrowed this down to the mmap call at [6] which attempts to map the backing file into memory. AFAICT, the mapped size is far below the RAM on the build machine as well as far below the normal 32-bit VM limit. So I don't know why the call is failing.
It's not really about the physical memory because just doing the mmap won't actually read it all in and in any case the kernel will be quite happy to page bits in and out as needed.
The issue will be whether there is that much contiguous address space available - on a 32 bit machine 2**28 is, at best, one sixteenth of the potential address space.
I say at best because even with the best possible user:kernel split at least some address space is reserved to the kernel although that can be fairly small with some kernel configurations.
Then you have to take the memory map into account though, and how address randomisation may have placed other mappings in locations that mean such a large contiguous mapping is not in fact available.
Take a look an /proc/NNN/maps for the process when the mmap fails and see if it looks like there it a range of that size that is actually available...
Thanks for the hint. I ran another test and patched it to print /proc/self/maps whenever loading the lmdb file failed [1] (ignore other arch failures; I just forgot to disable debuginfo.) Flattening down the ranges, the largest free space is only about 50% of what the test would need.
However, looking at the maps also hinted at a way out. There were a _lot_ of lmdb mappings and looking at the tests, they never explicitly close the files and rely on garbage collection to clean them up. So I wrote a somewhat hacky patch to do:
try: self.db = lmdb.open(...) except lmdb.MemoryError: gc.collect() self.db = lmdb.open(...)
and now it passes [2]. I will report something upstream so that they can modify the tests to be more explicit about closing things.
That being said, this does not explain why it fails in s390x mock (and continues to fail even with gc). But I guess getting it to work on koji is good enough for me.
[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=35282558 [2] https://koji.fedoraproject.org/koji/taskinfo?taskID=35291525
-
Elliott Sales de Andrade -
Tom Hughes