Bugs have been filed asking packagers to build with openblas instead of atlas. But there are multiple openblas libraries. I can link with any of the following:
-lopenblas (openblas-serial) -lopenblas64 (openblas-serial64) -lopenblas64_ (openblas-serial64_) -lopenblaso (openblas-openmp) -lopenblaso64 (openblas-openmp64) -lopenblaso64_ (openblas-openmp64_) -lopenblasp (openblas-threads) -lopenblasp64 (openblas-threads64) -lopenblasp64_ (openblas-threads64_)
If I understand correctly, the default is a library that cannot tolerate multiple threads, uses 32-bit integers in the interface, and does not use "a symbol name suffix", whatever that means. The various suffixes mean: o = use OpenMP (instead of serial) p = use pthreads (instead of serial) 64 = use 64-bit integers in the interface _ = use "a symbol name suffix"
The bugs that have been filed have not made any mention of how to select which of these libraries to link against. I'm looking at how to switch the sagemath stack over. Since suitesparse sits at the bottom of the stack, I just followed that package's example at first and linked with -lopenblas. But now sagemath says:
OpenBLAS : Program will terminate because you tried to start too many threads.
And kills it dead. I think this switch from atlas to openblas was not planned well. The entire sagemath stack, for example, will need to be linked with the *same* openblas library. The serial library with the 32-bit integer interface is not the right choice, and I'm going to need suitesparse to make the same choice as the rest of the stack. Some of the elements of that stack are OpenMP-enabled, so that may be the right choice, but I'm not sure. Suitesparse, for example, uses TBB, which suggests that the thread alternative may be the right choice for that package. The serial option is definitely not right for suitesparse, anyway.
How do we select which packages get linked against which openblas library? That question should have been answered *before* filing all of these bugs.
Also, I see this in /usr/include/openblas/openblas_config.h:
#define OPENBLAS_HAVE_SSE3 #define OPENBLAS_HAVE_SSSE3
That suggests that SSE3 and SSSE3 instructions have been built into one or more of the openblas libraries, which means that users with machines that do not support those instruction sets will get illegal instruction errors if they try to use anything linked with openblas.
Jerry James wrote:
If I understand correctly, the default is a library that cannot tolerate multiple threads, uses 32-bit integers in the interface, and does not use "a symbol name suffix", whatever that means.
Hmmm, normally, the serial version is not itself parallelized but should tolerate threaded applications. In fact, that is the main advantage over the threaded versions that may interfere with the application's own threading, and the reason why serial is still the default.
However, that does not appear to be entirely true, judging from your error message:
OpenBLAS : Program will terminate because you tried to start too many threads.
which leaves me slightly confused.
Also, I see this in /usr/include/openblas/openblas_config.h:
#define OPENBLAS_HAVE_SSE3 #define OPENBLAS_HAVE_SSSE3
That suggests that SSE3 and SSSE3 instructions have been built into one or more of the openblas libraries, which means that users with machines that do not support those instruction sets will get illegal instruction errors if they try to use anything linked with openblas.
OpenBLAS detects the available instruction set at runtime on x86/x86_64, so SSE3 and SSSE3 instructions should only be used when the hardware actually supports them (though I have not personally verified that this actually works as documented).
Kevin Kofler
Kevin Kofler kevin.kofler@chello.at writes:
Jerry James wrote:
If I understand correctly, the default is a library that cannot tolerate multiple threads, uses 32-bit integers in the interface, and does not use "a symbol name suffix", whatever that means.
Hmmm, normally, the serial version is not itself parallelized but should tolerate threaded applications. In fact, that is the main advantage over the threaded versions that may interfere with the application's own threading, and the reason why serial is still the default.
Yes, you normally want to parallelize only at the top level. Blithely switching on threading of some sort in libraries typically causes trouble with HPC applications in particular. Assuming only the public BLAS(/LAPACK) interface is used, you can change which implementation if used (including threaded or not) with LD_PRELOAD.
However, that does not appear to be entirely true, judging from your error message:
OpenBLAS : Program will terminate because you tried to start too many threads.
which leaves me slightly confused.
I don't know where that's coming from, but it's not from libopenblas as far as I can tell, so it needs debugging.
Also, I see this in /usr/include/openblas/openblas_config.h:
#define OPENBLAS_HAVE_SSE3 #define OPENBLAS_HAVE_SSSE3
That suggests that SSE3 and SSSE3 instructions have been built into one or more of the openblas libraries, which means that users with machines that do not support those instruction sets will get illegal instruction errors if they try to use anything linked with openblas.
OpenBLAS detects the available instruction set at runtime on x86/x86_64, so SSE3 and SSSE3 instructions should only be used when the hardware actually supports them (though I have not personally verified that this actually works as documented).
OpenBLAS's dynamic selection works fine (for x86 cpuids it recognizes), but is highly suboptimal on avx512, at least for dgemm. (You currently want BLIS for avx512, but the BLIS packaging is waiting for me to sort out the effect of hardening flags.) OB doesn't currently dispatch on micro-architecture for any other architectures as far as I know, but BLIS does to a limited extent, and I don't think you can optimally use just one of OB or BLIS everywhere. However, I don't think there's a need to use ATLAS anywhere in Fedora now.
Overall, the situation with linear algebra support in Fedora is a mess. There was a proposal to fix it, which was rejected in committee some time ago. (It looked to me as though the committee didn't understand the issues.) I wrote something about it and the fiddle I've used at https://loveshack.fedorapeople.org/blas-subversion.html, which probably needs some updating.
-- Frustrated of HPC-town
-
Dave Love -
Jerry James -
Kevin Kofler