FlexiBLAS as BLAS/LAPACK manager - Fedora 33 System-Wide Change proposal

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/FlexiBLAS_as_BLAS/LAPACK_manager

== Summary ==
BLAS/LAPACK packages will be compiled against the FlexiBLAS wrapper
library, which will set OpenBLAS as system-wide default backend, and
at the same time will provide a proper switching mechanism that
currently Fedora lacks.

== Owner ==
* Name: [[User:iucar| Iñaki Úcar]]
* Email: i.ucar86 at gmail


== Detailed Description ==

BLAS and LAPACK are API standards for basic linear algebra operations
(such as vector and matrix multiplication). Fedora packages the
reference implementation from Netlib as well as several optimized
backends, such as ATLAS, BLIS and OpenBLAS. Historically, there have
been two unresolved issues regarding this API:

# '''Fedora lacks a system-wide default'''. Due to implementation
differences, it is important that all components of a particular
software stack link to the same BLAS/LAPACK implementation. Although
there have been [[Changes/OpenBLAS_as_default_BLAS|efforts]] to
standardize OpenBLAS as a system-wide default, they are incomplete.
# '''Fedora lacks a proper switching mechanism'''. Different
implementations work differently depending on the workload, but also
depending on whether the consumer of this API implements some kind of
parallelization, for instance. Therefore, users may want to choose a
particular implementation that works best for them at run time.
Mechanisms such as update-alternatives and modules have been discussed
in the past, but were considered improper (the former) or faced
technical issues (the former).

This change proposal aims to solve both issues at the same time with
minimal changes and minimal impact by using FlexiBLAS.

FlexiBLAS is a framework that wraps the BLAS/LAPACK API with
interfaces for both 32- and 64-bit integers. It provides runtime
exchangeable backends without recompilation, with a transparent
fallback mechanism to Netlib's reference implementation if a certain
symbol is not present in the selected backend. It also supports
flexible per-system/user/host configuration files and basic profiling.

Therefore, this change proposal requires:

# Recompilation of all BLAS/LAPACK-dependent packages linking against
FlexiBLAS instead of the current implementation they are using (just
changing a BuildRequires line should be sufficient in most cases,
unless a SPEC has something hardcoded somewhere else).
# Changing the packaging guidelines to reflect this new requirement
for BLAS/LAPACK consumers (there is [[PackagingDrafts/BLAS_LAPACK| a
draft]] already.

In this way, all the BLAS/LAPACK-dependent packages will automatically
use the system-wide default backend, set and centralized in the
FlexiBLAS SPEC, and will benefit from the straightforward switching
mechanism that FlexiBLAS provides.


== Benefit to Fedora ==

* '''Packaging of BLAS/LAPACK-dependent packages will be easier and
safer'''. FlexiBLAS is 100% compatible with the reference API. It uses
optimized symbols that are present in the selected backend and
transparently falls back to Netlib for the missing ones, so
compatibility is always guaranteed.
* '''Provides a centralized way of managing the system-wide
default''', just by changing a global variable in the FlexiBLAS SPEC.
* '''Provides sysadmins with tools to manage the system-wide
configuration'''. FlexiBLAS ships system-wide configuration files to,
e.g., change the default or point to new backends. A CLI tool is also
provided to easily manage such configurations.
* '''Provides users with tools to switch the backend'''. This can
be
managed by the same CLI tool too in user mode, and it generates
per-user configuration files.
* '''Provides developers with an API to hook into the BLAS/LAPACK API
calls'''. E.g., FlexiBLAS comes with a plugin that enables basic
profiling support.

== Scope ==
* Proposal owners: Modify the SPECs of the BLAS/LAPACK-dependent
packages to build against FlexiBLAS instead of the current backend
they are using.
* Other developers: Maintainers of the affected packages need to merge
the changes and rebuild. Alternatively, a provenpackager could help
with this process (it's 30-40 packages).

* Release engineering: N/A
* Policies and guidelines:
[https://pagure.io/packaging-committee/issue/995 #995] Package
guidelines should be updated to require BLAS/LAPACK-dependent packages
to link against FlexiBLAS.
* Trademark approval: N/A (not needed for this Change)

== Upgrade/compatibility impact ==
There is no upgrade/compatibility impact.

== How To Test ==
BLAS/LAPACK-dependent packages should work normally, without any (new)
issue. To test the switching capabilities, testers may want to run a
BLAS/LAPACK benchmark. There are many out there for different
software, but we will describe the workflow for R as an example:

* Install R, which requires BLAS/LAPACK for operations with matrices.
* Run any benchmark involving calculations with matrices (e.g.,
R-benchmark-25.R provided by an R-core member
[https://mac.r-project.org/benchmarks here]).
* Switch to the reference implementation:
 flexiblas -u default netlib
* Run the benchmark again. Timings should be much worse.

== User Experience ==
Users will have a new CLI tool, called flexiblas, which will allow
them to properly switch the BLAS/LAPACK backend without administrative
privileges and any compatibility issues.

== Dependencies ==
Packages depedent on the BLAS/LAPACK API will depend on FlexiBLAS
after this change. Maintainers of those packages are required to
collaborate to change their BuildRequires or, alternatively, this can
be done with the help of a provenpackager.

== Contingency Plan ==

* Contingency mechanism: Revert BLAS/LAPACK-dependent packages to
their prior BuildRequires and rebuild them.
* Contingency deadline: Beta freeze.
* Blocks release? No.
* Blocks product? No.

== Documentation ==
See the
[https://gitlab.mpi-magdeburg.mpg.de/software/flexiblas-release/-/blob/mas...
README] of the upstream project and their
[https://www.mpi-magdeburg.mpg.de/projects/flexiblas homepage] for
further information about FlexiBLAS.

== Release Notes ==
TBD


-- 
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis

[Jerry James]

On Wed, Jul 1, 2020 at 8:25 AM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
> https://fedoraproject.org/wiki/Changes/FlexiBLAS_as_BLAS/LAPACK_manager
>
>  == Summary ==
>  BLAS/LAPACK packages will be compiled against the FlexiBLAS wrapper
>  library, which will set OpenBLAS as system-wide default backend, and
>  at the same time will provide a proper switching mechanism that
>  currently Fedora lacks. 
I like this idea.  Comments below.

>  # '''Fedora lacks a system-wide default'''.
> Due to implementation
>  differences, it is important that all components of a particular
>  software stack link to the same BLAS/LAPACK implementation. Although
>  there have been [[Changes/OpenBLAS_as_default_BLAS|efforts]] to
>  standardize OpenBLAS as a system-wide default, they are incomplete.
>  # '''Fedora lacks a proper switching mechanism'''. Different
>  implementations work differently depending on the workload, but also
>  depending on whether the consumer of this API implements some kind of
>  parallelization, for instance. Therefore, users may want to choose a
>  particular implementation that works best for them at run time.
>  Mechanisms such as update-alternatives and modules have been discussed
>  in the past, but were considered improper (the former) or faced
>  technical issues (the former). 
I think the latter instance of "the former" should be "the latter".
:-)

>  # Recompilation of all BLAS/LAPACK-dependent packages linking
> against
>  FlexiBLAS instead of the current implementation they are using (just
>  changing a BuildRequires line should be sufficient in most cases,
>  unless a SPEC has something hardcoded somewhere else). 
Speaking as maintainer of several packages that use BLAS/LAPACK, I
think the effort required may be a bit more than this paragraph
suggests, but it is worth doing.  Thanks for working on this, Iñaki!
-- 
Jerry James
http://www.jamezone.org/

[Iñaki Ucar]

On Wed, 1 Jul 2020 at 17:53, Jerry James <loganjerry(a)gmail.com&gt; wrote:
>  On Wed, Jul 1, 2020 at 8:25 AM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  >
>  > Mechanisms such as update-alternatives and modules have been discussed
>  > in the past, but were considered improper (the former) or faced
>  > technical issues (the former).
>
>  I think the latter instance of "the former" should be "the latter".
> :-) 
Yes, already fixed in the change page, thanks. :)

>  > # Recompilation of all BLAS/LAPACK-dependent packages linking
> against
>  > FlexiBLAS instead of the current implementation they are using (just
>  > changing a BuildRequires line should be sufficient in most cases,
>  > unless a SPEC has something hardcoded somewhere else).
>
>  Speaking as maintainer of several packages that use BLAS/LAPACK, I
>  think the effort required may be a bit more than this paragraph
>  suggests, but it is worth doing.  Thanks for working on this, Iñaki! 
I just did the easy part. :) Thanks to the upstream maintainer,
Martin, who has listened to all the feedback I had, fixed a bunch of
things and prepared a new release ready for this proposal in no time.

BTW, I would also like to discuss here, as part of this proposal,
which backend should be the system-wide default. I believe we all
would agree that OpenBLAS nowadays is the best choice. But then, the
serial or the openmp version?

-- 
Iñaki Úcar

[Jerry James]

On Wed, Jul 1, 2020 at 10:26 AM Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote:
>  BTW, I would also like to discuss here, as part of this proposal,
>  which backend should be the system-wide default. I believe we all
>  would agree that OpenBLAS nowadays is the best choice. But then, the
>  serial or the openmp version? 
First, I want to make sure I understand the current openblas
packaging.  Is this correct?

openblas-openmp: use if the application uses OpenMP
openblas-serial: use if the application is multithreaded
openblas-threads: use if the application is single-threaded

I've always found the naming of the openblas packages to be confusing,
so that may not be right.  Somebody set me straight if so.

The question of the default is a hard one.  What happens if a
multithreaded application that does not use OpenMP is linked with the
OpenMP build of OpenBLAS?
-- 
Jerry James
http://www.jamezone.org/

[Iñaki Ucar]

On Wed, 1 Jul 2020 at 19:03, Jerry James <loganjerry(a)gmail.com&gt; wrote:
>  On Wed, Jul 1, 2020 at 10:26 AM Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote:
>  > BTW, I would also like to discuss here, as part of this proposal,
>  > which backend should be the system-wide default. I believe we all
>  > would agree that OpenBLAS nowadays is the best choice. But then, the
>  > serial or the openmp version?
>
>  First, I want to make sure I understand the current openblas
>  packaging.  Is this correct?
>
>  openblas-openmp: use if the application uses OpenMP
>  openblas-serial: use if the application is multithreaded
>  openblas-threads: use if the application is single-threaded
>
>  I've always found the naming of the openblas packages to be confusing,
>  so that may not be right.  Somebody set me straight if so. 
The naming means (the same for BLIS):

-serial: runs single-threaded.
-openmp: some parts use OpenMP.
-threads: runs multi-threaded.

So it's a property of the OpenBLAS build, not a property of the
application. In fact, the serial version is not thread-safe unless you
activate a particular option at build time. We found that issue
recently and fixed it, and BTW the workaround was to switch to
openblas-openmp.

Potentially, you could run every version regardless of the
application, but some combinations are clearly a no-go, like a
threaded OpenBLAS in a threaded application.

>  The question of the default is a hard one.  What happens if a
>  multithreaded application that does not use OpenMP is linked with the
>  OpenMP build of OpenBLAS? 
That combination is fine AFAIK. But I don't know whether the serial
version would perform better in that situation or not at all.

I'll bother the FlexiBLAS maintainer about this. :) They use FlexiBLAS
as a research tool in high-performance computing, so maybe he knows
better about all these possible combinations.

-- 
Iñaki Úcar

[Jerry James]

On Wed, Jul 1, 2020 at 11:20 AM Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote:
>  On Wed, 1 Jul 2020 at 19:03, Jerry James <loganjerry(a)gmail.com&gt;
> wrote:
>  > openblas-openmp: use if the application uses OpenMP
>  > openblas-serial: use if the application is multithreaded
>  > openblas-threads: use if the application is single-threaded
>  >
>  > I've always found the naming of the openblas packages to be confusing,
>  > so that may not be right.  Somebody set me straight if so.
>
>  The naming means (the same for BLIS):
>
>  -serial: runs single-threaded.
>  -openmp: some parts use OpenMP.
>  -threads: runs multi-threaded.
>
>  So it's a property of the OpenBLAS build, not a property of the
>  application. In fact, the serial version is not thread-safe unless you
>  activate a particular option at build time. We found that issue
>  recently and fixed it, and BTW the workaround was to switch to
>  openblas-openmp. 
I think the way the OpenBLAS wiki is written confuses me:

https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded

It starts with "If your application is already multi-threaded",
implying that the default case is suitable for a single-threaded
application.  The default would be USE_THREAD=1, which corresponds to
the openblas-threads package.  The wiki's remedy for an application
that is multi-threaded is to build OpenBLAS with USE_THREAD=0
USE_LOCKING=1, which corresponds to the openblas-serial package.  If
the list I wrote above is wrong, I blame the wiki.

>  > The question of the default is a hard one.  What happens if a
>  > multithreaded application that does not use OpenMP is linked with the
>  > OpenMP build of OpenBLAS?
>
>  That combination is fine AFAIK. But I don't know whether the serial
>  version would perform better in that situation or not at all.
>
>  I'll bother the FlexiBLAS maintainer about this. :) They use FlexiBLAS
>  as a research tool in high-performance computing, so maybe he knows
>  better about all these possible combinations. 
As long as the default can be overridden, it doesn't matter all that
much, I think.  It's probably best to pick the "safest" option as the
default, although I confess that I'm not sure which one that is.
-- 
Jerry James
http://www.jamezone.org/

[Susi Lehtola]

On Wed, 1 Jul 2020 10:54:16 -0600
Jerry James <loganjerry(a)gmail.com&gt; wrote:

>  On Wed, Jul 1, 2020 at 10:26 AM Iñaki Ucar
> <iucar(a)fedoraproject.org&gt;
>  wrote:
>  > BTW, I would also like to discuss here, as part of this proposal,
>  > which backend should be the system-wide default. I believe we all
>  > would agree that OpenBLAS nowadays is the best choice. But then, the
>  > serial or the openmp version?  
>  
>  First, I want to make sure I understand the current openblas
>  packaging.  Is this correct?
>  
>  openblas-openmp: use if the application uses OpenMP 
Yes; then OpenBLAS calls get run in parallel in single-threaded regions
of the code, and if sequential mode in regions that are already
running parallel.

>  openblas-serial: use if the application is multithreaded
>  openblas-threads: use if the application is single-threaded 
No, this is exactly the wrong way around. You should use the serial
library for code that you want to be running in serial (this way you
can get several instances of the program running efficiently), and the
pthreads version if you want to run the BLAS/LAPACK regions in parallel
(but are somehow opposed to OpenMP!)..

>  The question of the default is a hard one.  What happens if a
>  multithreaded application that does not use OpenMP is linked with the
>  OpenMP build of OpenBLAS? 
Then you get too much parallellism, i.e. every call to OpenBLAS will
launch several threads, and this will naturally ruin your scaling.
-- 
Susi Lehtola
Fedora Project Contributor
jussilehtola(a)fedoraproject.org

[Iñaki Ucar]

On Wed, 1 Jul 2020 at 20:13, Susi Lehtola
<jussilehtola(a)fedoraproject.org&gt; wrote:
>  On Wed, 1 Jul 2020 10:54:16 -0600
>  Jerry James <loganjerry(a)gmail.com&gt; wrote:
>
>  > On Wed, Jul 1, 2020 at 10:26 AM Iñaki Ucar <iucar(a)fedoraproject.org&gt;
>  > wrote:
>  > > BTW, I would also like to discuss here, as part of this proposal,
>  > > which backend should be the system-wide default. I believe we all
>  > > would agree that OpenBLAS nowadays is the best choice. But then, the
>  > > serial or the openmp version?
>  >
>  > First, I want to make sure I understand the current openblas
>  > packaging.  Is this correct?
>  >
>  > openblas-openmp: use if the application uses OpenMP
>
>  Yes; then OpenBLAS calls get run in parallel in single-threaded regions
>  of the code, and if sequential mode in regions that are already
>  running parallel.
>
>  > openblas-serial: use if the application is multithreaded
>  > openblas-threads: use if the application is single-threaded
>
>  No, this is exactly the wrong way around. You should use the serial
>  library for code that you want to be running in serial (this way you
>  can get several instances of the program running efficiently), and the
>  pthreads version if you want to run the BLAS/LAPACK regions in parallel
>  (but are somehow opposed to OpenMP!).. 
I think I'm more confused than before. :D

>  > The question of the default is a hard one.  What happens if a
>  > multithreaded application that does not use OpenMP is linked with the
>  > OpenMP build of OpenBLAS?
>
>  Then you get too much parallellism, i.e. every call to OpenBLAS will
>  launch several threads, and this will naturally ruin your scaling. 
So would you say that openblas-serial is the most sensible system-wide default?

-- 
Iñaki Úcar

[Susi Lehtola]

On Wed, 1 Jul 2020 20:16:36 +0200
Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote:
>  > No, this is exactly the wrong way around. You should use the
> serial
>  > library for code that you want to be running in serial (this way you
>  > can get several instances of the program running efficiently), and
>  > the pthreads version if you want to run the BLAS/LAPACK regions in
>  > parallel (but are somehow opposed to OpenMP!)..  
>  
>  I think I'm more confused than before. :D
>  
>  > > The question of the default is a hard one.  What happens if a
>  > > multithreaded application that does not use OpenMP is linked with
>  > > the OpenMP build of OpenBLAS?  
>  >
>  > Then you get too much parallellism, i.e. every call to OpenBLAS will
>  > launch several threads, and this will naturally ruin your scaling.  
>  
>  So would you say that openblas-serial is the most sensible
>  system-wide default? 
THERE SHOULD BE NO SYSTEM-WIDE DEFAULT.

The thing is that the maintainer needs to be able to pick the correct
flavor. If you use the wrong one, you may seriously handicap
performance.

But, if one *needs* a system-wide default, in my opinion it
should be the OpenMP version, since it plays well together with both
sequential and OpenMP parallel programs.
-- 
Susi Lehtola
Fedora Project Contributor
jussilehtola(a)fedoraproject.org

[Iñaki Ucar]

On Fri, 3 Jul 2020 at 13:02, Susi Lehtola
<jussilehtola(a)fedoraproject.org&gt; wrote:
>  On Wed, 1 Jul 2020 20:16:36 +0200
>  Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote:
>  > > No, this is exactly the wrong way around. You should use the serial
>  > > library for code that you want to be running in serial (this way you
>  > > can get several instances of the program running efficiently), and
>  > > the pthreads version if you want to run the BLAS/LAPACK regions in
>  > > parallel (but are somehow opposed to OpenMP!)..
>  >
>  > I think I'm more confused than before. :D
>  >
>  > > > The question of the default is a hard one.  What happens if a
>  > > > multithreaded application that does not use OpenMP is linked with
>  > > > the OpenMP build of OpenBLAS?
>  > >
>  > > Then you get too much parallellism, i.e. every call to OpenBLAS will
>  > > launch several threads, and this will naturally ruin your scaling.
>  >
>  > So would you say that openblas-serial is the most sensible
>  > system-wide default?
>
>  THERE SHOULD BE NO SYSTEM-WIDE DEFAULT.
>
>  The thing is that the maintainer needs to be able to pick the correct
>  flavor. If you use the wrong one, you may seriously handicap
>  performance. 
I don't agree, because in many (most?) programs you never know what
kind of workload is going to run. E.g., in R, octave... there may be
parallelization involved, there are packages using OpenMP... Then,
even if there really is a best option for a certain program, the
maintainer would need to know, which I don't think is a good
assumption (see Jerry's comments in this thread). Even if there is a
recommendation from upstream, many times it's not the best option
either.

>  But, if one *needs* a system-wide default, in my opinion it
>  should be the OpenMP version, since it plays well together with both
>  sequential and OpenMP parallel programs. 
Thanks. In my opinion, we need a system-wide default, plus we will be
offering the most safe and straightforward way to change the backend.
And then, if a particular package really needs a particular flavour
and really there are arguments not to allow users to switch it, then
an exception could be granted for that package.

-- 
Iñaki Úcar

[Jerry James]

On Wed, Jul 1, 2020 at 12:06 PM Susi Lehtola
<jussilehtola(a)fedoraproject.org&gt; wrote:
>  On Wed, 1 Jul 2020 10:54:16 -0600
>  Jerry James <loganjerry(a)gmail.com&gt; wrote:
>  > openblas-serial: use if the application is multithreaded
>  > openblas-threads: use if the application is single-threaded
>
>  No, this is exactly the wrong way around. You should use the serial
>  library for code that you want to be running in serial (this way you
>  can get several instances of the program running efficiently), and the
>  pthreads version if you want to run the BLAS/LAPACK regions in parallel
>  (but are somehow opposed to OpenMP!).. 
Okay, I think the wiki's wording is hard to understand
(https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded), at least
for me.  Let me see if I've got this straight.

1. openblas-opemp computes on subranges in parallel using OpenMP
2. openblas-threads computes on subranges in parallel using pthreads
3. openblas-serial uses a single thread for the entire computation

Right?

What I'm looking for is a set of rules I can apply when I've got a
package that uses BLAS, but upstream has given no guidance on what
kind of BLAS library is suitable.  I've got several of those, so I
want to check that they're linking with the best version of the
library.  How should I make that determination?  Also, how do I know
whether to use, say, openblas-openmp vs. openblas-openmp64 vs.
openblas-openmp64_?
-- 
Jerry James
http://www.jamezone.org/

[Susi Lehtola]

On Wed, 1 Jul 2020 21:03:09 -0600
Jerry James <loganjerry(a)gmail.com&gt; wrote:
>  On Wed, Jul 1, 2020 at 12:06 PM Susi Lehtola
>  <jussilehtola(a)fedoraproject.org&gt; wrote:
>  > On Wed, 1 Jul 2020 10:54:16 -0600
>  > Jerry James <loganjerry(a)gmail.com&gt; wrote:  
>  > > openblas-serial: use if the application is multithreaded
>  > > openblas-threads: use if the application is single-threaded  
>  >
>  > No, this is exactly the wrong way around. You should use the serial
>  > library for code that you want to be running in serial (this way you
>  > can get several instances of the program running efficiently), and
>  > the pthreads version if you want to run the BLAS/LAPACK regions in
>  > parallel (but are somehow opposed to OpenMP!)..  
>  
>  Okay, I think the wiki's wording is hard to understand
>  (https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded), at least
>  for me.  Let me see if I've got this straight.
>  
>  1. openblas-opemp computes on subranges in parallel using OpenMP
>  2. openblas-threads computes on subranges in parallel using pthreads
>  3. openblas-serial uses a single thread for the entire computation
>  
>  Right? 
Exactly. But the key bit here is that if openblas-openmp is called
within a code segment that is already OpenMP parallel, then OpenBLAS
does not parallellize further: if you have 8 cores, and the program is
running on 8 threads, further parallellism (going to 8*8=64 threads)
would just ruin the performance.

>  What I'm looking for is a set of rules I can apply when I've
> got a
>  package that uses BLAS, but upstream has given no guidance on what
>  kind of BLAS library is suitable.  I've got several of those, so I
>  want to check that they're linking with the best version of the
>  library.  How should I make that determination?  Also, how do I know
>  whether to use, say, openblas-openmp vs. openblas-openmp64 vs.
>  openblas-openmp64_? 
openblas-openmp is generally the safe choice. As for the latter
question, it depends on whether you compile your code with 4-bit or
8-bit integers.
-- 
Susi Lehtola
Fedora Project Contributor
jussilehtola(a)fedoraproject.org

[Miro Hrončok]

On 01. 07. 20 16:24, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/FlexiBLAS_as_BLAS/LAPACK_manager
>  
>  == Summary ==
>  BLAS/LAPACK packages will be compiled against the FlexiBLAS wrapper
>  library, which will set OpenBLAS as system-wide default backend, and
>  at the same time will provide a proper switching mechanism that
>  currently Fedora lacks.
>  
>  ...
>  
>  == Scope ==
>  * Proposal owners: Modify the SPECs of the BLAS/LAPACK-dependent
>  packages to build against FlexiBLAS instead of the current backend
>  they are using. 
I wonder, given FlexiBLAS is released under GPL (and not LGPL), whether this 
means we would need to change the licenses of all non-GPL packages that will be 
linked to FlexiBLAS to GPL.

CCing legal.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Iñaki Ucar]

On Wed, 1 Jul 2020 at 18:39, Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 01. 07. 20 16:24, Ben Cotton wrote:
>  > https://fedoraproject.org/wiki/Changes/FlexiBLAS_as_BLAS/LAPACK_manager
>  >
>  > == Summary ==
>  > BLAS/LAPACK packages will be compiled against the FlexiBLAS wrapper
>  > library, which will set OpenBLAS as system-wide default backend, and
>  > at the same time will provide a proper switching mechanism that
>  > currently Fedora lacks.
>  >
>  > ...
>  >
>  > == Scope ==
>  > * Proposal owners: Modify the SPECs of the BLAS/LAPACK-dependent
>  > packages to build against FlexiBLAS instead of the current backend
>  > they are using.
>
>  I wonder, given FlexiBLAS is released under GPL (and not LGPL), whether this
>  means we would need to change the licenses of all non-GPL packages that will be
>  linked to FlexiBLAS to GPL.
>
>  CCing legal. 
I'm no expert, but the FAQ says:

"You have a GPLed program that I'd like to link with my code to build
a proprietary program. Does the fact that I link with your program
mean I have to GPL my program? (#LinkingWithGPL)

Not exactly. It means you must release your program under a license
compatible with the GPL (more precisely, compatible with one or more
GPL versions accepted by all the rest of the code in the combination
that you link). The combination itself is then available under those
GPL versions."

So my understanding is that it's ok for a program to link to FlexiBLAS
if its license is GPL-compatible, not necessarily GPL. But of course
we would need confirmation from legal.

-- 
Iñaki Úcar

[Susi Lehtola]

On Wed, 1 Jul 2020 19:28:53 +0200
Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote:
>  I'm no expert, but the FAQ says:
>  
>  "You have a GPLed program that I'd like to link with my code to build
>  a proprietary program. Does the fact that I link with your program
>  mean I have to GPL my program? (#LinkingWithGPL)
>  
>  Not exactly. It means you must release your program under a license
>  compatible with the GPL (more precisely, compatible with one or more
>  GPL versions accepted by all the rest of the code in the combination
>  that you link). The combination itself is then available under those
>  GPL versions."
>  
>  So my understanding is that it's ok for a program to link to FlexiBLAS
>  if its license is GPL-compatible, not necessarily GPL. But of course
>  we would need confirmation from legal. 
The library is GPLv3 only

https://gitlab.mpi-magdeburg.mpg.de/software/flexiblas-release#license

which already makes it incompatible with many GNU Licenses

https://fedoraproject.org/wiki/Licensing:Main#GPL_Compatibility_Matrix

not to mention many other free licenses that are allowed in Fedora

https://fedoraproject.org/wiki/Licensing:Main?rd=Licensing#SoftwareLicenses
-- 
Susi Lehtola
Fedora Project Contributor
jussilehtola(a)fedoraproject.org

[Iñaki Ucar]

On Wed, 1 Jul 2020 at 20:24, Susi Lehtola
<jussilehtola(a)fedoraproject.org&gt; wrote:
>  On Wed, 1 Jul 2020 19:28:53 +0200
>  Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote:
>  > I'm no expert, but the FAQ says:
>  >
>  > "You have a GPLed program that I'd like to link with my code to build
>  > a proprietary program. Does the fact that I link with your program
>  > mean I have to GPL my program? (#LinkingWithGPL)
>  >
>  > Not exactly. It means you must release your program under a license
>  > compatible with the GPL (more precisely, compatible with one or more
>  > GPL versions accepted by all the rest of the code in the combination
>  > that you link). The combination itself is then available under those
>  > GPL versions."
>  >
>  > So my understanding is that it's ok for a program to link to FlexiBLAS
>  > if its license is GPL-compatible, not necessarily GPL. But of course
>  > we would need confirmation from legal.
>
>  The library is GPLv3 only
>
>  https://gitlab.mpi-magdeburg.mpg.de/software/flexiblas-release#license
>
>  which already makes it incompatible with many GNU Licenses
>
>  https://fedoraproject.org/wiki/Licensing:Main#GPL_Compatibility_Matrix 
With many? With GPLv2 only. But most interestingly, GPLv3 states:

"Corresponding Source includes interface definition files associated
with source files for the work, and the source code for shared
libraries and dynamically linked subprograms that the work is
specifically designed to require"

Technically, there's nothing in FlexiBLAS that a BLAS/LAPACK
application would require to work. These are designed to require
BLAS/LAPACK, and FlexiBLAS just duplicates that interface and glues
symbols together.

Maybe I'm completely misunderstanding the sentence above, but GPL
restrictions of course apply to any program that uses the FlexiBLAS
API for profiling, etc., but not to those that just use the
BLAS/LAPACK API.

-- 
Iñaki Úcar

[Iñaki Ucar]

On Wed, 1 Jul 2020 at 21:00, Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote:
>  On Wed, 1 Jul 2020 at 20:24, Susi Lehtola
>  <jussilehtola(a)fedoraproject.org&gt; wrote:
>  >
>  > On Wed, 1 Jul 2020 19:28:53 +0200
>  > Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote:
>  > > I'm no expert, but the FAQ says:
>  > >
>  > > "You have a GPLed program that I'd like to link with my code to build
>  > > a proprietary program. Does the fact that I link with your program
>  > > mean I have to GPL my program? (#LinkingWithGPL)
>  > >
>  > > Not exactly. It means you must release your program under a license
>  > > compatible with the GPL (more precisely, compatible with one or more
>  > > GPL versions accepted by all the rest of the code in the combination
>  > > that you link). The combination itself is then available under those
>  > > GPL versions."
>  > >
>  > > So my understanding is that it's ok for a program to link to FlexiBLAS
>  > > if its license is GPL-compatible, not necessarily GPL. But of course
>  > > we would need confirmation from legal.
>  >
>  > The library is GPLv3 only
>  >
>  > https://gitlab.mpi-magdeburg.mpg.de/software/flexiblas-release#license
>  >
>  > which already makes it incompatible with many GNU Licenses
>  >
>  > https://fedoraproject.org/wiki/Licensing:Main#GPL_Compatibility_Matrix
>
>  With many? With GPLv2 only. But most interestingly, GPLv3 states:
>
>  "Corresponding Source includes interface definition files associated
>  with source files for the work, and the source code for shared
>  libraries and dynamically linked subprograms that the work is
>  specifically designed to require"
>
>  Technically, there's nothing in FlexiBLAS that a BLAS/LAPACK
>  application would require to work. These are designed to require
>  BLAS/LAPACK, and FlexiBLAS just duplicates that interface and glues
>  symbols together.
>
>  Maybe I'm completely misunderstanding the sentence above, but GPL
>  restrictions of course apply to any program that uses the FlexiBLAS
>  API for profiling, etc., but not to those that just use the
>  BLAS/LAPACK API. 
In line with this, can FlexiBLAS be considered a "system library"? It
sounds reasonable to me, but I read the definition in section 1
several times and still don't know.

-- 
Iñaki Úcar

[Iñaki Ucar]

On Wed, 1 Jul 2020 at 18:39, Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 01. 07. 20 16:24, Ben Cotton wrote:
>  > https://fedoraproject.org/wiki/Changes/FlexiBLAS_as_BLAS/LAPACK_manager
>  >
>  > == Summary ==
>  > BLAS/LAPACK packages will be compiled against the FlexiBLAS wrapper
>  > library, which will set OpenBLAS as system-wide default backend, and
>  > at the same time will provide a proper switching mechanism that
>  > currently Fedora lacks.
>  >
>  > ...
>  >
>  > == Scope ==
>  > * Proposal owners: Modify the SPECs of the BLAS/LAPACK-dependent
>  > packages to build against FlexiBLAS instead of the current backend
>  > they are using.
>
>  I wonder, given FlexiBLAS is released under GPL (and not LGPL), whether this
>  means we would need to change the licenses of all non-GPL packages that will be
>  linked to FlexiBLAS to GPL.
>
>  CCing legal. 
Ok, let me recap here, because I wasn't subscribed to the legal list.
Basically, FlexiBLAS is a wrapper for an API (BLAS/LAPACK) that is
BSD. But to act as a wrapper, a program needs to link against the part
of FlexiBLAS that replicates that API, and they just connect symbols
to the appropriate backend (this is use 1). On the other hand,
FlexiBLAS provides another API that allows programs to hook into the
first API, profile, debug, switch the backend in real time... (this is
use 2).

Here we are talking about use 1, and the question is whether this can
be considered under the exception of a "system library", as defined in
the first section of GPLv3. If not, the program must be
GPL-compatible? Or the program needs to change the license altogether?

I contacted the upstream maintainer about this, and he really wants to
permit the use case in this change proposal (I've been working with
him 3 weeks to fix bugs and bring a suitable version to Fedora for
this), so if this can't be considered a "system library", he is
willing to add an exception to the license. This is what he told me:

>  About the linking problem I could think of an exception, similar to
> the linking exception in gcc/glibc, which coincides with our "Free for free use"
> idea. I could add that FlexiBLAS is allowed to be linked against free software with an OSI
> approved license as none of the flexiblas headers are used at compile-time. In this way
> all open-source software can use FlexiBLAS no matter if it is BSD/MIT/APACHE/GPL code but
> those who want to use the special features like switching BLAS within the program or
> developing debug hooks like the profile hook, have to be compatible to GPL. 
Is this possible? Would this be accepted?

Thanks.
-- 
Iñaki Úcar

[Zbigniew Jędrzejewski-Szmek]

On Thu, Jul 02, 2020 at 09:50:47AM +0200, Iñaki Ucar wrote:
>  On Wed, 1 Jul 2020 at 18:39, Miro Hrončok <mhroncok(a)redhat.com&gt;
> wrote:
>  >
>  > On 01. 07. 20 16:24, Ben Cotton wrote:
>  > > https://fedoraproject.org/wiki/Changes/FlexiBLAS_as_BLAS/LAPACK_manager
>  > >
>  > > == Summary ==
>  > > BLAS/LAPACK packages will be compiled against the FlexiBLAS wrapper
>  > > library, which will set OpenBLAS as system-wide default backend, and
>  > > at the same time will provide a proper switching mechanism that
>  > > currently Fedora lacks.
>  > >
>  > > ...
>  > >
>  > > == Scope ==
>  > > * Proposal owners: Modify the SPECs of the BLAS/LAPACK-dependent
>  > > packages to build against FlexiBLAS instead of the current backend
>  > > they are using.
>  >
>  > I wonder, given FlexiBLAS is released under GPL (and not LGPL), whether this
>  > means we would need to change the licenses of all non-GPL packages that will be
>  > linked to FlexiBLAS to GPL.
>  >
>  > CCing legal.
>  
>  Ok, let me recap here, because I wasn't subscribed to the legal list.
>  Basically, FlexiBLAS is a wrapper for an API (BLAS/LAPACK) that is
>  BSD. But to act as a wrapper, a program needs to link against the part
>  of FlexiBLAS that replicates that API, and they just connect symbols
>  to the appropriate backend (this is use 1). On the other hand,
>  FlexiBLAS provides another API that allows programs to hook into the
>  first API, profile, debug, switch the backend in real time... (this is
>  use 2).
>  
>  Here we are talking about use 1, and the question is whether this can
>  be considered under the exception of a "system library", as defined in
>  the first section of GPLv3. If not, the program must be
>  GPL-compatible? Or the program needs to change the license altogether? 
The "system library exception" is about allowing a GPL program to be
distributed without the source code for some of the libraries it links
to. But here we're talking about the opposite issue: whether we can
distribute *other* works linked to the GPLv3 library under their
original licenses. The "system library exception" is not relevant and
§5c says "You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This License
will therefore apply, along with any applicable section 7 additional
terms, to the whole of the work, and all its parts, regardless of how
they are packaged."

FSF insists that dynamically linking with a GPL-covered work is making
a combined work based on the GPL-covered part, and thus the final
product may only be distributed under the GPL, as quoted above [1].
I think this interpretation is dubious in some cases, but here, when
we're talking about compiling and linking programs against the GPLv3
headers, it seems reasonable.

If it would be possible to instead compile packages against some other
blas library, and only substitute flexiblas during dynamic linking at
users' systems, the GPLv3-covered work would only be created then and
we wouldn't need to care about licensing of other packages.

[1] https://www.gnu.org/licenses/gpl-faq.html#GPLStaticVsDynamic

>  I contacted the upstream maintainer about this, and he really wants
> to
>  permit the use case in this change proposal (I've been working with
>  him 3 weeks to fix bugs and bring a suitable version to Fedora for
>  this), so if this can't be considered a "system library", he is
>  willing to add an exception to the license. This is what he told me:
>  
>  > About the linking problem I could think of an exception, similar to the linking
> exception in gcc/glibc, which coincides with our "Free for free use" idea. I
> could add that FlexiBLAS is allowed to be linked against free software with an OSI
> approved license as none of the flexiblas headers are used at compile-time. In this way
> all open-source software can use FlexiBLAS no matter if it is BSD/MIT/APACHE/GPL code but
> those who want to use the special features like switching BLAS within the program or
> developing debug hooks like the profile hook, have to be compatible to GPL. 
Would the maintainer consider switching the whole thing to LGPLv3?
This would preserve the freeness of his code and be much less hassle
for everyone involved, with no interpretation of new legal texts required.

Zbyszek

[Fedora-legal-list] Re: FlexiBLAS as BLAS/LAPACK manager - Fedora 33 System-Wide Change proposal

[Iñaki Ucar]

On Fri, 3 Jul 2020 at 16:15, Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Thu, Jul 02, 2020 at 09:50:47AM +0200, Iñaki Ucar wrote:
>  > On Wed, 1 Jul 2020 at 18:39, Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  > >
>  > > On 01. 07. 20 16:24, Ben Cotton wrote:
>  > > > https://fedoraproject.org/wiki/Changes/FlexiBLAS_as_BLAS/LAPACK_manager
>  > > >
>  > > > == Summary ==
>  > > > BLAS/LAPACK packages will be compiled against the FlexiBLAS wrapper
>  > > > library, which will set OpenBLAS as system-wide default backend, and
>  > > > at the same time will provide a proper switching mechanism that
>  > > > currently Fedora lacks.
>  > > >
>  > > > ...
>  > > >
>  > > > == Scope ==
>  > > > * Proposal owners: Modify the SPECs of the BLAS/LAPACK-dependent
>  > > > packages to build against FlexiBLAS instead of the current backend
>  > > > they are using.
>  > >
>  > > I wonder, given FlexiBLAS is released under GPL (and not LGPL), whether this
>  > > means we would need to change the licenses of all non-GPL packages that will
> be
>  > > linked to FlexiBLAS to GPL.
>  > >
>  > > CCing legal.
>  >
>  > Ok, let me recap here, because I wasn't subscribed to the legal list.
>  > Basically, FlexiBLAS is a wrapper for an API (BLAS/LAPACK) that is
>  > BSD. But to act as a wrapper, a program needs to link against the part
>  > of FlexiBLAS that replicates that API, and they just connect symbols
>  > to the appropriate backend (this is use 1). On the other hand,
>  > FlexiBLAS provides another API that allows programs to hook into the
>  > first API, profile, debug, switch the backend in real time... (this is
>  > use 2).
>  >
>  > Here we are talking about use 1, and the question is whether this can
>  > be considered under the exception of a "system library", as defined in
>  > the first section of GPLv3. If not, the program must be
>  > GPL-compatible? Or the program needs to change the license altogether?
>
>  The "system library exception" is about allowing a GPL program to be
>  distributed without the source code for some of the libraries it links
>  to. But here we're talking about the opposite issue: whether we can
>  distribute *other* works linked to the GPLv3 library under their
>  original licenses. The "system library exception" is not relevant and
>  §5c says "You must license the entire work, as a whole, under this
>  License to anyone who comes into possession of a copy. This License
>  will therefore apply, along with any applicable section 7 additional
>  terms, to the whole of the work, and all its parts, regardless of how
>  they are packaged."
>
>  FSF insists that dynamically linking with a GPL-covered work is making
>  a combined work based on the GPL-covered part, and thus the final
>  product may only be distributed under the GPL, as quoted above [1].
>  I think this interpretation is dubious in some cases, but here, when
>  we're talking about compiling and linking programs against the GPLv3
>  headers, it seems reasonable.
>
>  If it would be possible to instead compile packages against some other
>  blas library, and only substitute flexiblas during dynamic linking at
>  users' systems, the GPLv3-covered work would only be created then and
>  we wouldn't need to care about licensing of other packages. 
That's the thing: this is certainly possible, but from the packaging
point of view, it's more difficult, because BLAS and LAPACK are
separate shared libraries. You could, for instance, duplicate yet
again the BLAS/LAPACK interface into a single library that could be
e.g. BSD, then build all the packages against that one, then provide
that with FlexiBLAS instead. You could use the separate libraries
libblas and liblapack and then have other stubs pointing to FlexiBLAS
instead.

All this is possible, but it's a ton of duplicated code, effort, etc.,
and it's ridiculous. So if the above it's perfectly possible and
complies with the license, it's also ridiculous that avoiding this
extra unnecessary stuff doesn't.

>  [1] https://www.gnu.org/licenses/gpl-faq.html#GPLStaticVsDynamic
>
>  > I contacted the upstream maintainer about this, and he really wants to
>  > permit the use case in this change proposal (I've been working with
>  > him 3 weeks to fix bugs and bring a suitable version to Fedora for
>  > this), so if this can't be considered a "system library", he is
>  > willing to add an exception to the license. This is what he told me:
>  >
>  > > About the linking problem I could think of an exception, similar to the linking
> exception in gcc/glibc, which coincides with our "Free for free use" idea. I
> could add that FlexiBLAS is allowed to be linked against free software with an OSI
> approved license as none of the flexiblas headers are used at compile-time. In this way
> all open-source software can use FlexiBLAS no matter if it is BSD/MIT/APACHE/GPL code but
> those who want to use the special features like switching BLAS within the program or
> developing debug hooks like the profile hook, have to be compatible to GPL.
>
>  Would the maintainer consider switching the whole thing to LGPLv3?
>  This would preserve the freeness of his code and be much less hassle
>  for everyone involved, with no interpretation of new legal texts required. 
LGPL has other implications towards proprietary software, and that's
what the authors specifically want to protect, so that's a hard line.

Wouldn't the Classpath Exception [1] be appropriate here? This
wouldn't require the interpretation of a new legal text.

[1] https://fedoraproject.org/wiki/Licensing/GPL_Classpath_Exception

-- 
Iñaki Úcar

[Fedora-legal-list] Re: FlexiBLAS as BLAS/LAPACK manager - Fedora 33 System-Wide Change proposal

[Zbigniew Jędrzejewski-Szmek]

On Fri, Jul 03, 2020 at 06:11:08PM +0200, Iñaki Ucar wrote:
>  On Fri, 3 Jul 2020 at 16:15, Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > On Thu, Jul 02, 2020 at 09:50:47AM +0200, Iñaki Ucar wrote:
>  > > On Wed, 1 Jul 2020 at 18:39, Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  > > >
>  > > > On 01. 07. 20 16:24, Ben Cotton wrote:
>  > > > >
> https://fedoraproject.org/wiki/Changes/FlexiBLAS_as_BLAS/LAPACK_manager
>  > > > >
>  > > > > == Summary ==
>  > > > > BLAS/LAPACK packages will be compiled against the FlexiBLAS wrapper
>  > > > > library, which will set OpenBLAS as system-wide default backend, and
>  > > > > at the same time will provide a proper switching mechanism that
>  > > > > currently Fedora lacks.
>  > > > >
>  > > > > ...
>  > > > >
>  > > > > == Scope ==
>  > > > > * Proposal owners: Modify the SPECs of the BLAS/LAPACK-dependent
>  > > > > packages to build against FlexiBLAS instead of the current backend
>  > > > > they are using.
>  > > >
>  > > > I wonder, given FlexiBLAS is released under GPL (and not LGPL), whether
> this
>  > > > means we would need to change the licenses of all non-GPL packages that
> will be
>  > > > linked to FlexiBLAS to GPL.
>  > > >
>  > > > CCing legal.
>  > >
>  > > Ok, let me recap here, because I wasn't subscribed to the legal list.
>  > > Basically, FlexiBLAS is a wrapper for an API (BLAS/LAPACK) that is
>  > > BSD. But to act as a wrapper, a program needs to link against the part
>  > > of FlexiBLAS that replicates that API, and they just connect symbols
>  > > to the appropriate backend (this is use 1). On the other hand,
>  > > FlexiBLAS provides another API that allows programs to hook into the
>  > > first API, profile, debug, switch the backend in real time... (this is
>  > > use 2).
>  > >
>  > > Here we are talking about use 1, and the question is whether this can
>  > > be considered under the exception of a "system library", as defined
> in
>  > > the first section of GPLv3. If not, the program must be
>  > > GPL-compatible? Or the program needs to change the license altogether?
>  >
>  > The "system library exception" is about allowing a GPL program to be
>  > distributed without the source code for some of the libraries it links
>  > to. But here we're talking about the opposite issue: whether we can
>  > distribute *other* works linked to the GPLv3 library under their
>  > original licenses. The "system library exception" is not relevant and
>  > §5c says "You must license the entire work, as a whole, under this
>  > License to anyone who comes into possession of a copy. This License
>  > will therefore apply, along with any applicable section 7 additional
>  > terms, to the whole of the work, and all its parts, regardless of how
>  > they are packaged."
>  >
>  > FSF insists that dynamically linking with a GPL-covered work is making
>  > a combined work based on the GPL-covered part, and thus the final
>  > product may only be distributed under the GPL, as quoted above [1].
>  > I think this interpretation is dubious in some cases, but here, when
>  > we're talking about compiling and linking programs against the GPLv3
>  > headers, it seems reasonable.
>  >
>  > If it would be possible to instead compile packages against some other
>  > blas library, and only substitute flexiblas during dynamic linking at
>  > users' systems, the GPLv3-covered work would only be created then and
>  > we wouldn't need to care about licensing of other packages.
>  
>  That's the thing: this is certainly possible, but from the packaging
>  point of view, it's more difficult, because BLAS and LAPACK are
>  separate shared libraries. You could, for instance, duplicate yet
>  again the BLAS/LAPACK interface into a single library that could be
>  e.g. BSD, then build all the packages against that one, then provide
>  that with FlexiBLAS instead. You could use the separate libraries
>  libblas and liblapack and then have other stubs pointing to FlexiBLAS
>  instead.
>  
>  All this is possible, but it's a ton of duplicated code, effort, etc.,
>  and it's ridiculous. So if the above it's perfectly possible and
>  complies with the license, it's also ridiculous that avoiding this
>  extra unnecessary stuff doesn't.
>  
>  >
>  > [1] https://www.gnu.org/licenses/gpl-faq.html#GPLStaticVsDynamic
>  >
>  > > I contacted the upstream maintainer about this, and he really wants to
>  > > permit the use case in this change proposal (I've been working with
>  > > him 3 weeks to fix bugs and bring a suitable version to Fedora for
>  > > this), so if this can't be considered a "system library", he is
>  > > willing to add an exception to the license. This is what he told me:
>  > >
>  > > > About the linking problem I could think of an exception, similar to the
> linking exception in gcc/glibc, which coincides with our "Free for free use"
> idea. I could add that FlexiBLAS is allowed to be linked against free software with an OSI
> approved license as none of the flexiblas headers are used at compile-time. In this way
> all open-source software can use FlexiBLAS no matter if it is BSD/MIT/APACHE/GPL code but
> those who want to use the special features like switching BLAS within the program or
> developing debug hooks like the profile hook, have to be compatible to GPL.
>  >
>  > Would the maintainer consider switching the whole thing to LGPLv3?
>  > This would preserve the freeness of his code and be much less hassle
>  > for everyone involved, with no interpretation of new legal texts required.
>  
>  LGPL has other implications towards proprietary software, and that's
>  what the authors specifically want to protect, so that's a hard line. 
I'm not sure what you mean. LGPL keeps the code free but allows it to be
freely combined with software under different licenses, which is what we
want in this case.

>  Wouldn't the Classpath Exception [1] be appropriate here? This
>  wouldn't require the interpretation of a new legal text. 
Classpath exception talks about "executable". This isn't very precise,
but at least in normal speech, a library is not an executable, so the
classpath exception would not cover other libraries which link to
flexiblass. So for example, numpy would not be covered by the exception.

Zbyszek

[Fedora-legal-list] Re: FlexiBLAS as BLAS/LAPACK manager - Fedora 33 System-Wide Change proposal

[Iñaki Ucar]

On Sat, 4 Jul 2020 at 16:20, Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  > > Would the maintainer consider switching the whole thing to LGPLv3?
>  > > This would preserve the freeness of his code and be much less hassle
>  > > for everyone involved, with no interpretation of new legal texts required.
>  >
>  > LGPL has other implications towards proprietary software, and that's
>  > what the authors specifically want to protect, so that's a hard line.
>
>  I'm not sure what you mean. LGPL keeps the code free but allows it to be
>  freely combined with software under different licenses, which is what we
>  want in this case. 
Yeap, but it's more permissive also with the FlexiBLAS interface (the
one that enables hooking into the duplicated BLAS/LAPACK interface,
the one that BLAS/LAPACK consumers are not using), and this is what
the authors do not want.

>  > Wouldn't the Classpath Exception [1] be appropriate here?
> This
>  > wouldn't require the interpretation of a new legal text.
>
>  Classpath exception talks about "executable". This isn't very precise,
>  but at least in normal speech, a library is not an executable, so the
>  classpath exception would not cover other libraries which link to
>  flexiblass. So for example, numpy would not be covered by the exception. 
True. But what about the "Linking over a controlled interface
exception"? That sounds like exactly this case:
https://www.gnu.org/licenses/gpl-faq.en.html#LinkingOverControlledInterface

-- 
Iñaki Úcar

[Fedora-legal-list] Re: FlexiBLAS as BLAS/LAPACK manager - Fedora 33 System-Wide Change proposal

[Zbigniew Jędrzejewski-Szmek]

On Sat, Jul 04, 2020 at 05:20:48PM +0200, Iñaki Ucar wrote:
>  On Sat, 4 Jul 2020 at 16:20, Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > > > Would the maintainer consider switching the whole thing to LGPLv3?
>  > > > This would preserve the freeness of his code and be much less hassle
>  > > > for everyone involved, with no interpretation of new legal texts
> required.
>  > >
>  > > LGPL has other implications towards proprietary software, and that's
>  > > what the authors specifically want to protect, so that's a hard line.
>  >
>  > I'm not sure what you mean. LGPL keeps the code free but allows it to be
>  > freely combined with software under different licenses, which is what we
>  > want in this case.
>  
>  Yeap, but it's more permissive also with the FlexiBLAS interface (the
>  one that enables hooking into the duplicated BLAS/LAPACK interface,
>  the one that BLAS/LAPACK consumers are not using), and this is what
>  the authors do not want. 
OK. Thanks for the clarification.

Maybe talk with the authors and tell them that a few functions to provide
this extra interface are not important enough to create all the hassle with
GPLv3 for a commonly used library and that LGPL would protect their code
almost as well? I assume that they want their library to be widely used,
and this strict licensing will be a constant source of problems because
many existing scientific packages are using more liberal licensing and will
not want to change their licensing to accommodate flexiblas.

>  > > Wouldn't the Classpath Exception [1] be appropriate
> here? This
>  > > wouldn't require the interpretation of a new legal text.
>  >
>  > Classpath exception talks about "executable". This isn't very
> precise,
>  > but at least in normal speech, a library is not an executable, so the
>  > classpath exception would not cover other libraries which link to
>  > flexiblass. So for example, numpy would not be covered by the exception.
>  
>  True. But what about the "Linking over a controlled interface
>  exception"? That sounds like exactly this case:
>  https://www.gnu.org/licenses/gpl-faq.en.html#LinkingOverControlledInterface 
Yep, that seems like it would work. The first para contains a legal
interpretation of GPLv3 and thus doesn't belong in the exception text.
But the rest is OK.

Zbyszek

[Fedora-legal-list] Re: FlexiBLAS as BLAS/LAPACK manager - Fedora 33 System-Wide Change proposal

[Kevin Kofler]

Iñaki Ucar wrote:
>  LGPL has other implications towards proprietary software, and
> that's
>  what the authors specifically want to protect, so that's a hard line.
>  
>  Wouldn't the Classpath Exception [1] be appropriate here? This
>  wouldn't require the interpretation of a new legal text.
>  
>  [1] https://fedoraproject.org/wiki/Licensing/GPL_Classpath_Exception 
The Classpath Exception is at least as permissive as the LGPL, so if upstram 
is not OK with the LGPL, they will not be OK with the Classpath Exception 
either (at least they should not, rationally).

        Kevin Kofler

[Fedora-legal-list] Re: FlexiBLAS as BLAS/LAPACK manager - Fedora 33 System-Wide Change proposal

[Iñaki Ucar]

On Wed, 15 Jul 2020 at 12:35, Kevin Kofler <kevin.kofler(a)chello.at&gt; wrote:
>  Iñaki Ucar wrote:
>  > LGPL has other implications towards proprietary software, and that's
>  > what the authors specifically want to protect, so that's a hard line.
>  >
>  > Wouldn't the Classpath Exception [1] be appropriate here? This
>  > wouldn't require the interpretation of a new legal text.
>  >
>  > [1] https://fedoraproject.org/wiki/Licensing/GPL_Classpath_Exception
>
>  The Classpath Exception is at least as permissive as the LGPL, so if upstram
>  is not OK with the LGPL, they will not be OK with the Classpath Exception
>  either (at least they should not, rationally). 
They are going to apply this one:
https://www.gnu.org/licenses/gpl-faq.en.html#LinkingOverControlledInterface

-- 
Iñaki Úcar

[Zbigniew Jędrzejewski-Szmek]

On Wed, Jul 01, 2020 at 10:24:27AM -0400, Ben Cotton wrote:
>  == User Experience ==
>  Users will have a new CLI tool, called flexiblas, which will allow
>  them to properly switch the BLAS/LAPACK backend without administrative
>  privileges and any compatibility issues. 
Based on this description, the switching is done by setting some flag
in local configuration. Is there a way to set the configuration through
an environment variable? Having only "global" configuration (in the sense
of a single setting for a user) is going to make testing much harder,
and also it doesn't work for the case of multiple programs being started
independently when those programs require a different backend.

Zbyszek

[Zbigniew Jędrzejewski-Szmek]

On Fri, Jul 03, 2020 at 01:15:29PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  On Wed, Jul 01, 2020 at 10:24:27AM -0400, Ben Cotton wrote:
>  > == User Experience ==
>  > Users will have a new CLI tool, called flexiblas, which will allow
>  > them to properly switch the BLAS/LAPACK backend without administrative
>  > privileges and any compatibility issues.
>  
>  Based on this description, the switching is done by setting some flag
>  in local configuration. Is there a way to set the configuration through
>  an environment variable? Having only "global" configuration (in the sense
>  of a single setting for a user) is going to make testing much harder,
>  and also it doesn't work for the case of multiple programs being started
>  independently when those programs require a different backend. 
To answer myself: $FLEXIBLAS envvar can be set. Maybe add to the description?

Zbyszek

[Iñaki Ucar]

On Fri, 3 Jul 2020 at 15:29, Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Fri, Jul 03, 2020 at 01:15:29PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > On Wed, Jul 01, 2020 at 10:24:27AM -0400, Ben Cotton wrote:
>  > > == User Experience ==
>  > > Users will have a new CLI tool, called flexiblas, which will allow
>  > > them to properly switch the BLAS/LAPACK backend without administrative
>  > > privileges and any compatibility issues.
>  >
>  > Based on this description, the switching is done by setting some flag
>  > in local configuration. Is there a way to set the configuration through
>  > an environment variable? Having only "global" configuration (in the sense
>  > of a single setting for a user) is going to make testing much harder,
>  > and also it doesn't work for the case of multiple programs being started
>  > independently when those programs require a different backend.
>
>  To answer myself: $FLEXIBLAS envvar can be set. Maybe add to the description? 
Indeed. I'm on it. Thanks!

-- 
Iñaki Úcar

[Iñaki Ucar]

On Fri, 3 Jul 2020 at 15:23, Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Wed, Jul 01, 2020 at 10:24:27AM -0400, Ben Cotton wrote:
>  > == User Experience ==
>  > Users will have a new CLI tool, called flexiblas, which will allow
>  > them to properly switch the BLAS/LAPACK backend without administrative
>  > privileges and any compatibility issues.
>
>  Based on this description, the switching is done by setting some flag
>  in local configuration. Is there a way to set the configuration through
>  an environment variable? Having only "global" configuration (in the sense
>  of a single setting for a user) is going to make testing much harder,
>  and also it doesn't work for the case of multiple programs being started
>  independently when those programs require a different backend. 
Yes, the FLEXIBLAS env variable overrides the system-wide and
user-defined backends. So, e.g.:

$ FLEXIBLAS=blis-openmp someprogram

changes the backend to Blis with OpenMP support (provided that the
flexiblas-blis-openmp subpackage is installed).

-- 
Iñaki Úcar

[Zbigniew Jędrzejewski-Szmek]

On Wed, Jul 01, 2020 at 10:24:27AM -0400, Ben Cotton wrote:
>  == Documentation ==
>  See the
> [https://gitlab.mpi-magdeburg.mpg.de/software/flexiblas-release/-/blob/mas...
>  README] of the upstream project and their
>  [https://www.mpi-magdeburg.mpg.de/projects/flexiblas homepage] for
>  further information about FlexiBLAS. 
Hmm, is there an actual repo with the code?
gitlab.mpi-magdeburg.mpg.de/software/flexiblas-release just contains
fake history with one commit per release.

Zbyszek

[Iñaki Ucar]

On Fri, 3 Jul 2020 at 15:43, Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Wed, Jul 01, 2020 at 10:24:27AM -0400, Ben Cotton wrote:
>  > == Documentation ==
>  > See the
> [https://gitlab.mpi-magdeburg.mpg.de/software/flexiblas-release/-/blob/mas...
>  > README] of the upstream project and their
>  > [https://www.mpi-magdeburg.mpg.de/projects/flexiblas homepage] for
>  > further information about FlexiBLAS.
>
>  Hmm, is there an actual repo with the code?
>  gitlab.mpi-magdeburg.mpg.de/software/flexiblas-release just contains
>  fake history with one commit per release. 
That is the actual repo, and there is a mirror on GitHub to collect
public issues (https://github.com/mpimd-csc/flexiblas).

This is because they develop and use this software for research in
high-performance computing, so the day-to-day development repo is
private (to have a competitive advantage for their publications), and
then they publish releases in those repos afterwards.

-- 
Iñaki Úcar

[Zbigniew Jędrzejewski-Szmek]

On Fri, Jul 03, 2020 at 03:57:04PM +0200, Iñaki Ucar wrote:
>  On Fri, 3 Jul 2020 at 15:43, Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > On Wed, Jul 01, 2020 at 10:24:27AM -0400, Ben Cotton wrote:
>  > > == Documentation ==
>  > > See the
> [https://gitlab.mpi-magdeburg.mpg.de/software/flexiblas-release/-/blob/mas...
>  > > README] of the upstream project and their
>  > > [https://www.mpi-magdeburg.mpg.de/projects/flexiblas homepage] for
>  > > further information about FlexiBLAS.
>  >
>  > Hmm, is there an actual repo with the code?
>  > gitlab.mpi-magdeburg.mpg.de/software/flexiblas-release just contains
>  > fake history with one commit per release.
>  
>  That is the actual repo, and there is a mirror on GitHub to collect
>  public issues (https://github.com/mpimd-csc/flexiblas).
>  
>  This is because they develop and use this software for research in
>  high-performance computing, so the day-to-day development repo is
>  private (to have a competitive advantage for their publications), and
>  then they publish releases in those repos afterwards. 
Just... wow. People still believe that crap about competitive advantage.

Zbyszek

[Dave Love]

[I found I hadn't sent this earlier, as I should have.]

> https://fedoraproject.org/wiki/Changes/FlexiBLAS_as_BLAS/LAPACK_manager
>
>  == Summary ==
>  BLAS/LAPACK packages will be compiled against the FlexiBLAS wrapper
>  library, which will set OpenBLAS as system-wide default backend, and
>  at the same time will provide a proper switching mechanism that
>  currently Fedora lacks.
>  
I oppose this (in favour of a different approach) from experience in
research computing system management, general support, and
implementation.  It doesn't solve any problem I (have) had, as far as I
can tell, and looks as if it produces more.  The licence seems to me to
rule it out a priori.

The proposal doesn't justify things, including its dismissal of the
simple, clean alternative in similar to Debian's, with which I have some
experience.  (I don't know what the environment modules alternative is,
since that's one way of specifying the late binding.)  BLAS isn't alone
in presenting a substitute interface like that.  It works well with a
heterogeneous HPC cluster where you want different BLAS implementations
on different nodes (think KNL, A64FX).

There will be hoops to jump through to get packages to configure when
they don't know about the library.  If I want to use a library that's
not included, I'm in the same position.  It's not clear to me a priori
what happens if you try to use just BLIS even, given that OpenBLAS'
LAPACK implementation isn't the vanilla one, and depends on OpenBLAS as
far as I know.  The choice of OpenBLAS isn't justified.  It's not even
obvious that you want the same implementation for serial and
multi-threaded as OpenBLAS seems to have had continual problems with
threading which BLIS hasn't as far as I know.  I expect OpenBLAS is the
best serial option on average, at least, but that needs data for
different architectures.

Profiling is listed as a feature, but I wouldn't want a LA-specific
profiler rather than a more general one.  (Scorep on el7 currently
doesn't do that for want of a suitable clang (or LLVM?) component to do
the library wrapping, but it could.)

I realize no-one is going to be running Fedora on HPC systems, where
this really matters, but presumably it filters down to RHEL, which is
looking less and less like a good HPC platform to me.

[Iñaki Ucar]

On Fri, 17 Jul 2020 at 10:34, Dave Love <loveshack(a)fedoraproject.org&gt; wrote:
>  [I found I hadn't sent this earlier, as I should have.]
>
>  > https://fedoraproject.org/wiki/Changes/FlexiBLAS_as_BLAS/LAPACK_manager
>  >
>  > == Summary ==
>  > BLAS/LAPACK packages will be compiled against the FlexiBLAS wrapper
>  > library, which will set OpenBLAS as system-wide default backend, and
>  > at the same time will provide a proper switching mechanism that
>  > currently Fedora lacks.
>  >
>  I oppose this (in favour of a different approach) from experience in
>  research computing system management, general support, and
>  implementation.  It doesn't solve any problem I (have) had, as far as I
>  can tell, and looks as if it produces more.  The licence seems to me to
>  rule it out a priori. 
The authors are going to add an exception, so the license won't be a
problem. What problems do you think it produces?

>  The proposal doesn't justify things, including its dismissal of
> the
>  simple, clean alternative in similar to Debian's, with which I have some 
I don't justify that because Debian's alternative has been repeatedly
dismissed as an option for Fedora for a simple reason: one could end
up with libblas pointing to one implementation and liblapack pointing
to another. This problem, that Debian has, is solved with this
library.

>  There will be hoops to jump through to get packages to configure
> when
>  they don't know about the library. 
I don't think so, but if there are serious issues, rolling back the
change is pretty straightforward.

>  If I want to use a library that's
>  not included, I'm in the same position. 
No, you are not. You just need to point to that library using the
FLEXIBLAS environment variable.

>  It's not clear to me a priori
>  what happens if you try to use just BLIS even, given that OpenBLAS'
>  LAPACK implementation isn't the vanilla one, and depends on OpenBLAS as
>  far as I know. 
I don't understand this. You can switch to BLIS easily. The default
one doesn't matter.

>  The choice of OpenBLAS isn't justified.  It's not even
>  obvious that you want the same implementation for serial and
>  multi-threaded as OpenBLAS seems to have had continual problems with
>  threading which BLIS hasn't as far as I know.  I expect OpenBLAS is the
>  best serial option on average, at least, but that needs data for
>  different architectures. 
This is about having a sane default one. That's it. You can switch
then to whatever implementation you want.

>  I realize no-one is going to be running Fedora on HPC systems, where
>  this really matters, but presumably it filters down to RHEL, which is
>  looking less and less like a good HPC platform to me. 
Actually, FlexiBLAS is developed by researchers in HPC.

-- 
Iñaki Úcar

[Dave Love]

Iñaki Ucar <iucar(a)fedoraproject.org&gt; writes:

> > The licence seems to me to
> > rule it out a priori.
>
>  The authors are going to add an exception, so the license won't be a
>  problem. What problems do you think it produces? 
Obviously GPLv3 is incompatible with GPLv2, for instance, and typically
with other strong copyleft.

> > The proposal doesn't justify things, including its dismissal
> of the
> > simple, clean alternative in similar to Debian's, with which I have some
>
>  I don't justify that because Debian's alternative has been repeatedly
>  dismissed as an option for Fedora for a simple reason: one could end
>  up with libblas pointing to one implementation and liblapack pointing
>  to another. This problem, that Debian has, is solved with this
>  library. 
You'll have to explain; I'd have thought the explanation should have
been in the proposal.  I don't know what it means to have "another
implementation" of LAPACK in relation to BLIS for example.  I explicitly
want to be able to substitute LAPACK and BLAS implementations, including
serial and OpenMP.

> > There will be hoops to jump through to get packages to configure
> when
> > they don't know about the library.
>
>  I don't think so, but if there are serious issues, rolling back the
>  change is pretty straightforward. 
That sounds like insufficient experience!

> > If I want to use a library that's
> > not included, I'm in the same position.
>
>  No, you are not. You just need to point to that library using the
>  FLEXIBLAS environment variable. 
I'm unclear on exactly how flexiblas currently works, but the above
seems to be saying that I must have a limited choice of BLAS and LAPACK
anyway.  I can flip an environment variable already to subvert other
BLASs with BLIS.  (I considered dynamically loading different
implementations when I tackled this initially.)  Setting the environment
other than explicitly in a batch script isn't necessarily easy or robust
anyway, as opposed to ldconfig on sets of nodes.

> > It's not clear to me a priori
> > what happens if you try to use just BLIS even, given that OpenBLAS'
> > LAPACK implementation isn't the vanilla one, and depends on OpenBLAS as
> > far as I know.
>
>  I don't understand this. You can switch to BLIS easily. The default
>  one doesn't matter. 
See above about an independent choice of BLAS and LAPACK.  What happens
if I want to use BLIS functions and link against a library that's using
OpenBLAS and don't know about whatever magic I need in the environment
to make it work?

> > The choice of OpenBLAS isn't justified.  It's not even
> > obvious that you want the same implementation for serial and
> > multi-threaded as OpenBLAS seems to have had continual problems with
> > threading which BLIS hasn't as far as I know.  I expect OpenBLAS is the
> > best serial option on average, at least, but that needs data for
> > different architectures.
>
>  This is about having a sane default one. That's it. You can switch
>  then to whatever implementation you want. 
So what's the basis for choosing the default?
https://github.com/flame/blis/blob/80e6c10b72d50863b4b64d79f784df7befedfc...
for instance?  An OpenMP default that performs serially if you set
OMP_NUM_THREADS and OMP_PROC_BIND doesn't seem a sane default.  Again
this seems to be saying I can switch things at will, despite what it
says above, and that the mechanism is robust.

> > I realize no-one is going to be running Fedora on HPC systems,
> where
> > this really matters, but presumably it filters down to RHEL, which is
> > looking less and less like a good HPC platform to me.
>
>  Actually, FlexiBLAS is developed by researchers in HPC. 
I know.  That doesn't necessarily mean they have substantial experience
of HPC research support, system management, and packaging; that helps
when evaluating any necessary engineering trades-off.  (If I listened to
the HPC mainstream on software provision, I wouldn't be a packager and
wouldn't have had a easily manageable cluster.)  I'm waiting to be
convinced what I've concluded from R&D is invalid.

[Iñaki Ucar]

On Tue, 21 Jul 2020 at 19:59, Dave Love <loveshack(a)fedoraproject.org&gt; wrote:
>  Iñaki Ucar <iucar(a)fedoraproject.org&gt; writes:
>
>  >> The licence seems to me to
>  >> rule it out a priori.
>  >
>  > The authors are going to add an exception, so the license won't be a
>  > problem. What problems do you think it produces?
>
>  Obviously GPLv3 is incompatible with GPLv2, for instance, and typically
>  with other strong copyleft. 
The "Linked over a controlled interface" exception has been added in
version 3.0.1. So GPLv3 incompatibilities don't apply to linking over
the BLAS/LAPACK API anymore.

>  >> The proposal doesn't justify things, including its
> dismissal of the
>  >> simple, clean alternative in similar to Debian's, with which I have some
>  >
>  > I don't justify that because Debian's alternative has been repeatedly
>  > dismissed as an option for Fedora for a simple reason: one could end
>  > up with libblas pointing to one implementation and liblapack pointing
>  > to another. This problem, that Debian has, is solved with this
>  > library.
>
>  You'll have to explain; I'd have thought the explanation should have
>  been in the proposal.  I don't know what it means to have "another
>  implementation" of LAPACK in relation to BLIS for example.  I explicitly
>  want to be able to substitute LAPACK and BLAS implementations, including
>  serial and OpenMP. 
So here you are exclusively talking about BLIS, because other
libraries implement BLAS + some parts of LAPACK. I may agree that
doing the latter is a bad idea given that the reference implementation
offers two distinct libraries. But that's the landscape we have
unfortunately.

Being able to switch LAPACK independently while avoiding symbol
collisions is an interesting feature that could be added in FlexiBLAS
(at least I'm gonna propose it upstream). That's certainly not in the
feature set right now, but it's not a possibility either without
FlexiBLAS. E.g., many applications in Fedora are linked against
OpenBLAS serial. How would you change BLAS and LAPACK in those?

>  >> There will be hoops to jump through to get packages to
> configure when
>  >> they don't know about the library.
>  >
>  > I don't think so, but if there are serious issues, rolling back the
>  > change is pretty straightforward.
>
>  That sounds like insufficient experience! 
It's very easy to say "there will be hoops to jump through". That is
insufficient experience. I don't have any evidence suggesting that.
But I acknowledge that software is very heterogeneous out there and
sometimes you find unexpected problems. Those problems cannot be
uncovered unless someone tries to make this happen. Following the same
logic, many (most?) Change Proposals have insufficient experience.

>  >> If I want to use a library that's
>  >> not included, I'm in the same position.
>  >
>  > No, you are not. You just need to point to that library using the
>  > FLEXIBLAS environment variable.
>
>  I'm unclear on exactly how flexiblas currently works, but the above
>  seems to be saying that I must have a limited choice of BLAS and LAPACK
>  anyway.  I can flip an environment variable already to subvert other
>  BLASs with BLIS.  (I considered dynamically loading different
>  implementations when I tackled this initially.)  Setting the environment
>  other than explicitly in a batch script isn't necessarily easy or robust
>  anyway, as opposed to ldconfig on sets of nodes. 
We have a more limited choice right now. See my comment above about
applications linked against OpenBLAS, or ATLAS. With FlexiBLAS, there
are less limitations and more flexibility (and I believe the specific
use case you have in mind for BLIS could be discussed upstream).

>  >> It's not clear to me a priori
>  >> what happens if you try to use just BLIS even, given that OpenBLAS'
>  >> LAPACK implementation isn't the vanilla one, and depends on OpenBLAS as
>  >> far as I know.
>  >
>  > I don't understand this. You can switch to BLIS easily. The default
>  > one doesn't matter.
>
>  See above about an independent choice of BLAS and LAPACK.  What happens
>  if I want to use BLIS functions and link against a library that's using
>  OpenBLAS and don't know about whatever magic I need in the environment
>  to make it work? 
I'm not sure I'm following you anymore. You may also want to use 3
functions from OpenBLAS serial, 4 functions from OpenBLAS OpenMP, 1
from ATLAS, 5 from BLIS and the rest from whatever other library. But
it doesn't scale.

That's obviously a dramatization. My point is that it's almost
impossible to cover all the very specialised HPC use cases out there.
But I would argue that, if there's a way to cover them all, that could
be achieved by adding features to FlexiBLAS, because it's the most
general way to solve the issue of the implementation disparity in the
BLAS/LAPACK landscape: just exposing a complete API and then
internally rewiring those calls to the appropriate libraries given
some configuration.

>  >> The choice of OpenBLAS isn't justified.  It's not
> even
>  >> obvious that you want the same implementation for serial and
>  >> multi-threaded as OpenBLAS seems to have had continual problems with
>  >> threading which BLIS hasn't as far as I know.  I expect OpenBLAS is the
>  >> best serial option on average, at least, but that needs data for
>  >> different architectures.
>  >
>  > This is about having a sane default one. That's it. You can switch
>  > then to whatever implementation you want.
>
>  So what's the basis for choosing the default?
>
> https://github.com/flame/blis/blob/80e6c10b72d50863b4b64d79f784df7befedfc...
>  for instance?  An OpenMP default that performs serially if you set
>  OMP_NUM_THREADS and OMP_PROC_BIND doesn't seem a sane default.  Again
>  this seems to be saying I can switch things at will, despite what it
>  says above, and that the mechanism is robust. 
We support a number of architectures in which BLIS doesn't perform
well, so I think we would agree that this rules out BLIS as a default.
Then, both @jussilehtola and the authors of FlexiBLAS independently
agree that the OpenMP version of OpenBLAS would be the best default.
For the record, I copy&paste below the author's assessment:

| * OpenBLAS / Serial performance comparable to BLIS and MKL, require
compilation with the USE_LOCKING=1 flag set to be safe in
multithreaded applications.
| * OpenBLAS-PThread, fast, flexible and good automatic threading, let
OpenMP applications crash if BLAS is called from the inside of a
OpenMP parallel region, preferred version on Debian/Ubuntu. Not usable
on ppc64le architectures.
| * OpenBLAS-OpenMP. around 2 % slower than the Pthread version but
less complicated with multithreaded applications, works on all
platforms, good automatic threading. Can get in trouble with StarPU
parallel applications. but I do not know of any application outside
research that uses this.
|
| So I prefer setting OpenBLAS-OpenMP as default on all systems I have
under my control and colleagues do the same with their machines.

>  >> I realize no-one is going to be running Fedora on HPC
> systems, where
>  >> this really matters, but presumably it filters down to RHEL, which is
>  >> looking less and less like a good HPC platform to me.
>  >
>  > Actually, FlexiBLAS is developed by researchers in HPC.
>
>  I know.  That doesn't necessarily mean they have substantial experience
>  of HPC research support, system management, and packaging; that helps
>  when evaluating any necessary engineering trades-off.  (If I listened to
>  the HPC mainstream on software provision, I wouldn't be a packager and
>  wouldn't have had a easily manageable cluster.)  I'm waiting to be
>  convinced what I've concluded from R&D is invalid. 
You are arguing from a narrow niche. And it may be possible that I'm
not following you correctly and it's my fault, but from your comments,
my impression is that you didn't take the time to read how FlexiBLAS
works (there are links in the change proposal; the documentation is
extensive, and there are links in their home page to their papers, to
a very informative presentation...).

--
Iñaki Úcar

[Dave Love]

Iñaki Ucar <iucar(a)fedoraproject.org&gt; writes:

>  So here you are exclusively talking about BLIS, because other
>  libraries implement BLAS + some parts of LAPACK. I may agree that
>  doing the latter is a bad idea given that the reference implementation
>  offers two distinct libraries. But that's the landscape we have
>  unfortunately. 
No, I'm talking more generally.  That's just an example.

>  Being able to switch LAPACK independently while avoiding symbol
>  collisions is an interesting feature that could be added in FlexiBLAS
>  (at least I'm gonna propose it upstream). That's certainly not in the
>  feature set right now, but it's not a possibility either without
>  FlexiBLAS. E.g., many applications in Fedora are linked against
>  OpenBLAS serial. How would you change BLAS and LAPACK in those? 
It's not just a possibility, it just works with dynamic libraries.  The
fact that things are linked against a specific library rather than
effectively a generic ABI is the problem, but I can define a shim to
subvert them, the same as I did for OpenBLAS versus ATLAS, Rblas, etc.,
and the same as I'd have to do for FlexiBLAS if I wanted something
different.

Now, there are cases where there's the same API provided by different
implementations which have a different ABI, MPI being the obvious
example in this area.  Some sort of multiplexing may be appropriate
there -- I'm not sure.  I've had trouble making the implementation work.

>  It's very easy to say "there will be hoops to jump
> through". That is
>  insufficient experience. I don't have any evidence suggesting that.
>  But I acknowledge that software is very heterogeneous out there and
>  sometimes you find unexpected problems. Those problems cannot be
>  uncovered unless someone tries to make this happen. Following the same
>  logic, many (most?) Change Proposals have insufficient experience. 
I'm offering the experience of doing all this work from various
different points of view.  I don't off-hand remember instances where
particular problems have occurred, because I've done quite a of this.
Isn't engineering experience valuable?  No-one seems to be offering
counter-experience to evaluate trades-off, just assertions about what
requirements.

> > I'm unclear on exactly how flexiblas currently works, but the
> above
> > seems to be saying that I must have a limited choice of BLAS and LAPACK
> > anyway.  I can flip an environment variable already to subvert other
> > BLASs with BLIS.  (I considered dynamically loading different
> > implementations when I tackled this initially.)  Setting the environment
> > other than explicitly in a batch script isn't necessarily easy or robust
> > anyway, as opposed to ldconfig on sets of nodes.
>
>  We have a more limited choice right now. See my comment above about
>  applications linked against OpenBLAS, or ATLAS. With FlexiBLAS, there
>  are less limitations and more flexibility (and I believe the specific
>  use case you have in mind for BLIS could be discussed upstream). 
I don't see how that's true.  It's clearly more flexible to be able to
have different implementations of an interface more-or-less trivially --
a one-liner -- than to have a fixed implementation with limited choice.
Please realize that there's more to research computing than what's
packaged in Fedora, and there's more than x86 or single-threaded stuff.
(More of it could be there if it wasn't such a dispiriting business.)

> > See above about an independent choice of BLAS and LAPACK.  What
> happens
> > if I want to use BLIS functions and link against a library that's using
> > OpenBLAS and don't know about whatever magic I need in the environment
> > to make it work?
>
>  I'm not sure I'm following you anymore. You may also want to use 3
>  functions from OpenBLAS serial, 4 functions from OpenBLAS OpenMP, 1
>  from ATLAS, 5 from BLIS and the rest from whatever other library. But
>  it doesn't scale. 
That wouldn't make sense, and I'm worried by the suggestion.

>  That's obviously a dramatization. My point is that it's
> almost
>  impossible to cover all the very specialised HPC use cases out there.
>  But I would argue that, if there's a way to cover them all, that could
>  be achieved by adding features to FlexiBLAS, because it's the most
>  general way to solve the issue of the implementation disparity in the
>  BLAS/LAPACK landscape: just exposing a complete API and then
>  internally rewiring those calls to the appropriate libraries given
>  some configuration. 
It isn't the most general way to replace things at runtime.  The most
general way is to substitute different implementations of the ABI with
dynamic linking.  Note there's a de facto ABI, not just an API.
Consider I want to speed up R by replacing the serial BLAS with a
parallel one; that's fine, as in the reference I posted.  You're saying
I shouldn't have that choice because you're going to define what's
serial and parallel in that case.  Also I apparently shouldn't be able
to substitute a shim to do tracing by this logic, or a malloc
implementation to do profiling/debugging.

You can't even distribute all the interesting implementations --
specifically CUDA, c.f. AMD's stuff (hooray!).  I'd really like to hear
experience of doing this stuff seriously for a user population on
heterogeneous hardware over a number of years.

> > So what's the basis for choosing the default?
> >
> https://github.com/flame/blis/blob/80e6c10b72d50863b4b64d79f784df7befedfc...
> > for instance?  An OpenMP default that performs serially if you set
> > OMP_NUM_THREADS and OMP_PROC_BIND doesn't seem a sane default.  Again
> > this seems to be saying I can switch things at will, despite what it
> > says above, and that the mechanism is robust.
>
>  We support a number of architectures in which BLIS doesn't perform
>  well, so I think we would agree that this rules out BLIS as a default.
>  Then, both @jussilehtola and the authors of FlexiBLAS independently
>  agree that the OpenMP version of OpenBLAS would be the best default. 
I haven't seen measurements to support these statements.  Where are
they?  It's not obvious from the ones I referenced that OpenBLAS is
generally better than BLIS in the reference I posted.

But I'm not arguing for a fixed choice, because it's been time- and
(micro-)architecture-dependent as well as threading-dependent in my
experience.  What users get on specific nodes of a system should be
determined by the system manager's installation, and not by fragile
environment variables.  ldconfig provides that, even in a single image
used over stateless nodes, assuming you have something like oneSIS to
manage it.  (I'm aware that's not something you'd be using Fedora for.)
I haven't heard any real argument why that's the wrong thing to do, when
it's proven robust in my use.

>  For the record, I copy&paste below the author's assessment:
>
>  | * OpenBLAS / Serial performance comparable to BLIS and MKL, require
>  compilation with the USE_LOCKING=1 flag set to be safe in
>  multithreaded applications.
>  | * OpenBLAS-PThread, fast, flexible and good automatic threading, let
>  OpenMP applications crash if BLAS is called from the inside of a
>  OpenMP parallel region, preferred version on Debian/Ubuntu. Not usable
>  on ppc64le architectures. 
[Debian's choice of pthreads looked like a mistake to cover a specific
case, the last remember.  I don't think it should be a default.]

>  | * OpenBLAS-OpenMP. around 2 % slower than the Pthread version but
>  less complicated with multithreaded applications, works on all
>  platforms, good automatic threading. Can get in trouble with StarPU
>  parallel applications. but I do not know of any application outside
>  research that uses this.
>  
OpenBLAS threading seems to have been a continual source of problems,
and if you do the obvious thing, you get serial performance, as I said.
It's not clear to me it performs better than BLIS with OpenMP in
measurements.  Again, I don't want to argue specifically for or against
any one implementation.

>  | So I prefer setting OpenBLAS-OpenMP as default on all systems I
> have
>  under my control and colleagues do the same with their machines.
>
>  You are arguing from a narrow niche. 
Yes, if multiple aspect of research computing support using rpm
packaging is narrow, but it's the home of this stuff.  (I realize linear
algebra gets everywhere.)  I'm specifically trying not to preclude
things, because applications differ, and there's typically a
counter-example from research support over the years to assertions I
see.

>  And it may be possible that I'm
>  not following you correctly and it's my fault, but from your comments,
>  my impression is that you didn't take the time to read how FlexiBLAS
>  works (there are links in the change proposal; the documentation is
>  extensive, and there are links in their home page to their papers, to
>  a very informative presentation...). 
I think I understand enough about how it works, though I only studied
the previous version, and I did read a paper.  Having no development
history doesn't help.  As I said, I considered implementing something
like that, though I was hoping ELF features would help more than they
do, but there was a better way.  I've also investigated a case where
multiplexing might be appropriate, and another like BLAS where you can
use the ABI, so this isn't entirely from ignorance.  I just haven't seen
anything to persuade me that a more general, simpler, more robust
approach is wrong.

[Dave Love]

I meant to say that wrapping symbols may cause some confusion with
debugging and profiling, though I don't think that's critical where its
necessary.

[Iñaki Ucar]

On Tue, 28 Jul 2020 at 18:02, Dave Love <loveshack(a)fedoraproject.org&gt; wrote:
>  I'm offering the experience of doing all this work from various
>  different points of view.  I don't off-hand remember instances where
>  particular problems have occurred, because I've done quite a of this.
>  Isn't engineering experience valuable?  No-one seems to be offering
>  counter-experience to evaluate trades-off, just assertions about what
>  requirements. 
It is very valuable, and I appreciate your comments. But note that the
author of FlexiBLAS has also engineering experience and has been
managing a HPC cluster for a number of years too, apart from his
research in academia.

I'm certainly not an expert here. But one thing I learnt talking to
the various stakeholders is that basically everyone seems to have a
strong opinion around this, when no option seems to be universally
right for every application.

I think we can agree that this has been stalled for many years in
Fedora. Now, I'm willing to get my hands dirty and I'm just doing my
best to bring something better than we had.

>  > We have a more limited choice right now. See my comment above
> about
>  > applications linked against OpenBLAS, or ATLAS. With FlexiBLAS, there
>  > are less limitations and more flexibility (and I believe the specific
>  > use case you have in mind for BLIS could be discussed upstream).
>
>  I don't see how that's true.  It's clearly more flexible to be able to
>  have different implementations of an interface more-or-less trivially --
>  a one-liner -- than to have a fixed implementation with limited choice.
>  Please realize that there's more to research computing than what's
>  packaged in Fedora, and there's more than x86 or single-threaded stuff.
>  (More of it could be there if it wasn't such a dispiriting business.) 
Yes, and if you want something different,

FLEXIBLAS=/some/other/library.so ./some_program

and you are good to go. And you still have the LD_PRELOAD trick if you
want to bypass FlexiBLAS entirely, as you can do for something that is
currently linked against e.g. OpenBLAS. In this regard, I don't see
how FlexiBLAS is any more problematic than the current status quo.

>  > I'm not sure I'm following you anymore. You may also
> want to use 3
>  > functions from OpenBLAS serial, 4 functions from OpenBLAS OpenMP, 1
>  > from ATLAS, 5 from BLIS and the rest from whatever other library. But
>  > it doesn't scale.
>
>  That wouldn't make sense, and I'm worried by the suggestion. 
That was the point. Glad we are aligned here.

>  > That's obviously a dramatization. My point is that it's
> almost
>  > impossible to cover all the very specialised HPC use cases out there.
>  > But I would argue that, if there's a way to cover them all, that could
>  > be achieved by adding features to FlexiBLAS, because it's the most
>  > general way to solve the issue of the implementation disparity in the
>  > BLAS/LAPACK landscape: just exposing a complete API and then
>  > internally rewiring those calls to the appropriate libraries given
>  > some configuration.
>
>  It isn't the most general way to replace things at runtime.  The most
>  general way is to substitute different implementations of the ABI with
>  dynamic linking.  Note there's a de facto ABI, not just an API.
>  Consider I want to speed up R by replacing the serial BLAS with a
>  parallel one; that's fine, as in the reference I posted.  You're saying
>  I shouldn't have that choice because you're going to define what's
>  serial and parallel in that case.  Also I apparently shouldn't be able
>  to substitute a shim to do tracing by this logic, or a malloc
>  implementation to do profiling/debugging. 
I didn't say that. You still are. You manage an HPC cluster. You know
how to do that in several ways, with and without FlexiBLAS. But what
if a regular R user wants to replace the serial BLAS? They don't know
how. I replied recently in the R-SIG-Fedora mailing list precisely
because an R user was having trouble with OpenBLAS serial. With
FlexiBLAS, that's very easy. You can replace the default, temporarily
or permanently, with another implementation shipped in Fedora, or with
anything else you got from another source (e.g., MKL).

>  > We support a number of architectures in which BLIS doesn't
> perform
>  > well, so I think we would agree that this rules out BLIS as a default.
>  > Then, both @jussilehtola and the authors of FlexiBLAS independently
>  > agree that the OpenMP version of OpenBLAS would be the best default.
>
>  I haven't seen measurements to support these statements.  Where are
>  they?  It's not obvious from the ones I referenced that OpenBLAS is
>  generally better than BLIS in the reference I posted. 
The ones you referenced are arguably too artificial. AFAIK, they have
a set of fine-tuned parameters that are hard to get right. There's no
way to know which are reasonable values for a specific application.
Also they use CPU affinity for BLIS and disabled it for OpenBLAS. Is
there any other third-party benchmark replicating these results?

--
Iñaki Úcar

[Dominik 'Rathann' Mierzejewski]

On Friday, 17 July 2020 at 10:25, Dave Love wrote:
>  [I found I hadn't sent this earlier, as I should have.]
>  
>  > https://fedoraproject.org/wiki/Changes/FlexiBLAS_as_BLAS/LAPACK_manager
>  >
>  > == Summary ==
>  > BLAS/LAPACK packages will be compiled against the FlexiBLAS wrapper
>  > library, which will set OpenBLAS as system-wide default backend, and
>  > at the same time will provide a proper switching mechanism that
>  > currently Fedora lacks.
>  >
>  I oppose this (in favour of a different approach) from experience in
>  research computing system management, general support, and
>  implementation.  It doesn't solve any problem I (have) had, as far as I
>  can tell, and looks as if it produces more.  The licence seems to me to
>  rule it out a priori. 
Licence issues seem to have been cleared according to a message further
down the thread.

>  The proposal doesn't justify things, including its dismissal of
> the
>  simple, clean alternative in similar to Debian's, with which I have some
>  experience. 
What exact alternative are you proposing, then? Carrying downstream
patches to all BLAS/LAPACK implementations like Debian? I don't think
that's sustainable.

>  (I don't know what the environment modules alternative is, since
>  that's one way of specifying the late binding.) 
Environment modules are user-selectable sets of environment variable
values and require affected packages to be built in a specific way.
https://docs.fedoraproject.org/en-US/packaging-guidelines/EnvironmentModu...

>  BLAS isn't alone in presenting a substitute interface like that. 
> It
>  works well with a heterogeneous HPC cluster where you want different
>  BLAS implementations on different nodes (think KNL, A64FX). 
I'm not sure what you mean here. What "substitute interface" are you
talking about?

>  There will be hoops to jump through to get packages to configure
> when
>  they don't know about the library. 
From what I understand from the proposal, FlexiBLAS looks like vanilla
BLAS and LAPACK to the consumers. No hoop jumping required.

>   If I want to use a library that's
>  not included, I'm in the same position.  It's not clear to me a priori
>  what happens if you try to use just BLIS even, given that OpenBLAS' 
From what I understand, if you select BLIS, then it'll be used for all
symbols it implements and netlib reference will be used for the rest.

Regards,
Dominik
-- 
Fedora   https://getfedora.org  |  RPM Fusion  http://rpmfusion.org
There should be a science of discontent. People need hard times and
oppression to develop psychic muscles.
        -- from "Collected Sayings of Muad'Dib" by the Princess Irulan

[Iñaki Ucar]

On Fri, 17 Jul 2020 at 11:19, Dominik 'Rathann' Mierzejewski
<dominik(a)greysector.net&gt; wrote:
>  On Friday, 17 July 2020 at 10:25, Dave Love wrote:
>  > There will be hoops to jump through to get packages to configure when
>  > they don't know about the library.
>
>  From what I understand from the proposal, FlexiBLAS looks like vanilla
>  BLAS and LAPACK to the consumers. No hoop jumping required. 
Correct.

>  >  If I want to use a library that's
>  > not included, I'm in the same position.  It's not clear to me a priori
>  > what happens if you try to use just BLIS even, given that OpenBLAS'
>
>  From what I understand, if you select BLIS, then it'll be used for all
>  symbols it implements and netlib reference will be used for the rest. 
And correct.

-- 
Iñaki Úcar

[Dave Love]

Dominik 'Rathann' Mierzejewski <dominik(a)greysector.net&gt; writes:

>  What exact alternative are you proposing, then? Carrying downstream
>  patches to all BLAS/LAPACK implementations like Debian? I don't think
>  that's sustainable. 
You don't need patches to be able to substitute an ABI-compatible
version (with the soname counting towards the ABI).  See the BLIS
packaging, though it should have more shims.
https://loveshack.fedorapeople.org/blas-subversion.html describes what I
used in production, but it probably should be updated.

> > (I don't know what the environment modules alternative is,
> since
> > that's one way of specifying the late binding.)
>
>  Environment modules are user-selectable sets of environment variable
>  values and require affected packages to be built in a specific way.
>  https://docs.fedoraproject.org/en-US/packaging-guidelines/EnvironmentModu... 
I meant I don't know what specific use of the environment is being
rejected, especially as I'm told that's what I need to use, but not
LD_LIBRARY_PATH.  (For what it's worth, I think typical use of modules
is mostly a sign of engineering failure, but it's a reasonable way for
users to flip a runtime path.)

> > BLAS isn't alone in presenting a substitute interface like
> that.  It
> > works well with a heterogeneous HPC cluster where you want different
> > BLAS implementations on different nodes (think KNL, A64FX).
>
>  I'm not sure what you mean here. What "substitute interface" are you
>  talking about? 
I guess that should have been "substitutable".  I can dynamically
substitute reference BLAS with another implementation of libblas.so.3 as
I can substitute libgomp.so.1 with a shim over libiomp (to get OMPT, for
instance) -- and I realize I should have submitted a patch for that.
That assumes you don't use symbols in the one implementation which
aren't in the substitute, of course.

> > There will be hoops to jump through to get packages to configure
> when
> > they don't know about the library.
>
>  From what I understand from the proposal, FlexiBLAS looks like vanilla
>  BLAS and LAPACK to the consumers. No hoop jumping required. 
?? libflexiblas.so(.3) doesn't look like libblas.so(.3).  I've had to
prat about in hostile build systems even to use openblas at times.

> >  If I want to use a library that's
> > not included, I'm in the same position.  It's not clear to me a priori
> > what happens if you try to use just BLIS even, given that OpenBLAS'
>
>  From what I understand, if you select BLIS, then it'll be used for all
>  symbols it implements and netlib reference will be used for the rest. 
But I might want to use libflame, for instance.