OpenBLAS: link with which library?

[Jerry James]

Bugs have been filed asking packagers to build with openblas instead
of atlas.  But there are multiple openblas libraries. I can link with
any of the following:

  -lopenblas (openblas-serial)
  -lopenblas64 (openblas-serial64)
  -lopenblas64_ (openblas-serial64_)
  -lopenblaso (openblas-openmp)
  -lopenblaso64 (openblas-openmp64)
  -lopenblaso64_ (openblas-openmp64_)
  -lopenblasp (openblas-threads)
  -lopenblasp64 (openblas-threads64)
  -lopenblasp64_ (openblas-threads64_)

If I understand correctly, the default is a library that cannot
tolerate multiple threads, uses 32-bit integers in the interface, and
does not use "a symbol name suffix", whatever that means.  The various
suffixes mean:
  o = use OpenMP (instead of serial)
  p = use pthreads (instead of serial)
  64 = use 64-bit integers in the interface
  _ = use "a symbol name suffix"

The bugs that have been filed have not made any mention of how to
select which of these libraries to link against.  I'm looking at how
to switch the sagemath stack over.  Since suitesparse sits at the
bottom of the stack, I just followed that package's example at first
and linked with -lopenblas.  But now sagemath says:

OpenBLAS : Program will terminate because you tried to start too many threads.

And kills it dead.  I think this switch from atlas to openblas was not
planned well.  The entire sagemath stack, for example, will need to be
linked with the *same* openblas library.  The serial library with the
32-bit integer interface is not the right choice, and I'm going to
need suitesparse to make the same choice as the rest of the stack.
Some of the elements of that stack are OpenMP-enabled, so that may be
the right choice, but I'm not sure.  Suitesparse, for example, uses
TBB, which suggests that the thread alternative may be the right
choice for that package.  The serial option is definitely not right
for suitesparse, anyway.

How do we select which packages get linked against which openblas
library?  That question should have been answered *before* filing all
of these bugs.

Also, I see this in /usr/include/openblas/openblas_config.h:

#define OPENBLAS_HAVE_SSE3
#define OPENBLAS_HAVE_SSSE3

That suggests that SSE3 and SSSE3 instructions have been built into
one or more of the openblas libraries, which means that users with
machines that do not support those instruction sets will get illegal
instruction errors if they try to use anything linked with openblas.
-- 
Jerry James
http://www.jamezone.org/

[Kevin Kofler]

Jerry James wrote:
>  If I understand correctly, the default is a library that cannot
>  tolerate multiple threads, uses 32-bit integers in the interface, and
>  does not use "a symbol name suffix", whatever that means. 
Hmmm, normally, the serial version is not itself parallelized but should 
tolerate threaded applications. In fact, that is the main advantage over the 
threaded versions that may interfere with the application's own threading, 
and the reason why serial is still the default.

However, that does not appear to be entirely true, judging from your error 
message:
>  OpenBLAS : Program will terminate because you tried to start too
> many
>  threads. 
which leaves me slightly confused.

>  Also, I see this in /usr/include/openblas/openblas_config.h:
>  
>  #define OPENBLAS_HAVE_SSE3
>  #define OPENBLAS_HAVE_SSSE3
>  
>  That suggests that SSE3 and SSSE3 instructions have been built into
>  one or more of the openblas libraries, which means that users with
>  machines that do not support those instruction sets will get illegal
>  instruction errors if they try to use anything linked with openblas. 
OpenBLAS detects the available instruction set at runtime on x86/x86_64, so 
SSE3 and SSSE3 instructions should only be used when the hardware actually 
supports them (though I have not personally verified that this actually 
works as documented).

        Kevin Kofler

[Dave Love]

Kevin Kofler <kevin.kofler(a)chello.at&gt; writes:

>  Jerry James wrote:
> > If I understand correctly, the default is a library that cannot
> > tolerate multiple threads, uses 32-bit integers in the interface, and
> > does not use "a symbol name suffix", whatever that means.
>
>  Hmmm, normally, the serial version is not itself parallelized but should 
>  tolerate threaded applications. In fact, that is the main advantage over the 
>  threaded versions that may interfere with the application's own threading, 
>  and the reason why serial is still the default. 
Yes, you normally want to parallelize only at the top level.  Blithely
switching on threading of some sort in libraries typically causes
trouble with HPC applications in particular.  Assuming only the public
BLAS(/LAPACK) interface is used, you can change which implementation if
used (including threaded or not) with LD_PRELOAD.

>  However, that does not appear to be entirely true, judging from your
> error 
>  message:
> > OpenBLAS : Program will terminate because you tried to start too many
> > threads.
>  which leaves me slightly confused. 
I don't know where that's coming from, but it's not from libopenblas as
far as I can tell, so it needs debugging.

> > Also, I see this in /usr/include/openblas/openblas_config.h:
> > 
> > #define OPENBLAS_HAVE_SSE3
> > #define OPENBLAS_HAVE_SSSE3
> > 
> > That suggests that SSE3 and SSSE3 instructions have been built into
> > one or more of the openblas libraries, which means that users with
> > machines that do not support those instruction sets will get illegal
> > instruction errors if they try to use anything linked with openblas.
>
>  OpenBLAS detects the available instruction set at runtime on x86/x86_64, so 
>  SSE3 and SSSE3 instructions should only be used when the hardware actually 
>  supports them (though I have not personally verified that this actually 
>  works as documented). 
OpenBLAS's dynamic selection works fine (for x86 cpuids it recognizes),
but is highly suboptimal on avx512, at least for dgemm.  (You currently
want BLIS for avx512, but the BLIS packaging is waiting for me to sort
out the effect of hardening flags.)  OB doesn't currently dispatch on
micro-architecture for any other architectures as far as I know, but
BLIS does to a limited extent, and I don't think you can optimally use
just one of OB or BLIS everywhere.  However, I don't think there's a
need to use ATLAS anywhere in Fedora now.

Overall, the situation with linear algebra support in Fedora is a mess.
There was a proposal to fix it, which was rejected in committee some
time ago.  (It looked to me as though the committee didn't understand
the issues.)  I wrote something about it and the fiddle I've used at
<https://loveshack.fedorapeople.org/blas-subversion.html>;, which
probably needs some updating.

-- Frustrated of HPC-town