OpenBLAS Installation at Mathnet

BLAS means Basic Linear Algebra Subprograms. See this Wikipedia article about BLAS and its variants. It does basic operations such as matrix multiplication (DGEMM) which are building blocks for more complex operations like eigenvector extraction (DGEEV in LAPACK). At UCLA-Mathnet for a long time we have used the stock BLAS which is standard in the OpenSuSE distro: blas-devel-3.4.2 in OpenSuSE 13.1. However, this is the reference implementation which is locally optimized but does not take into account details of the processor architecture such as the hierarchical cache, and multiple cores.

It has been pointed out that we could get a considerable speed improvement by using one of the globally optimized BLAS implementations. These offer the same API (subroutine names and arguments) but the algorithms take cache characteristics into account.

Benchmark Program

My first step was to write a simplified version of the LAPACK benchmark. Following its lead, I devoted equal time to each of DGEMM (matrix multiplication), DGESV (solving linear equations), and DGEEV (eigenvectors and values). The time to solve these matrix problems scales as N³ where N is the dimension of the matrix, and so an appropriate speed measure is the time (seconds) to do each matrix divided by N³. For each one I do some preliminary tests to estimate the speed, and then I set N so one big matrix will take up the rest of the assigned time, assuming the speed truly scales with N³, which is not always the case. And of course checking can be turned on, to make sure that the subroutines are being called correctly.

Here are links to the program source and the Makefile.

Available BLAS Implementations

There are two optimzed BLAS packages that are credible for Intel-type processors and which are available in OpenSuSE:

• ATLAS: Automatically Tuned Linear Algebra Software. The main package is for x86_64 but Pentium variants are also available. You need to recompile it to turn on threading.

• OpenBLAS, based on GotoBLAS2 by Kazushige Goto. (In OpenSuSE 13.1 find it in the Science unstable sub-repo.) It is available in three variants: single core, pthreads (multi-core), and OpenMP parallelism. Available for both x86_64 and i586 (generic Intel 32bit processors).

For both of them, SuSE gives a version that performs well on the range of machines on which SuSE is likely to be used, but which is not optimized for any specific processor. You can, however, compile the source code on the target machine and get the best performance for that architecture. Documentation suggests 10% to 15% performance improvement if you do this. However, Mathnet has a variety of processors, and managing multiple architecture-specific versions would be a lot of work, so I decided, at least initially, to install the generic packages.

Another issue is multi-threading: SuSE offers a threaded version of OpenBLAS. Naively you would expect a 4x speedup on a quad core machine, but recruiting a thread in Linux (and Solaris and other architectures) is expensive, and you don't get full proportionality. I recommend that the programmer should do threading at the highest possible level and not use BLAS threading. In fact, many researchers at Mathnet make effective use of low-tech manual threading: submit N single-thread jobs that do not interact, keeping the code very simple. There is contention for the memory bus and possibly I/O (as there would be with an internally threaded algorithm) so you don't speed up by a factor of N, but you usually come fairly close. Nonetheless, I did install the pthreaded version of OpenBLAS.

Since ATLAS was the package requested, I tested it first. Unfortunately it got a segmentation fault in DGEMM, and since my benchmark runs error-free (including checking) on the stock BLAS on a range of Mathnet and home machines, I point the finger of blame at SuSE's ATLAS, some obscure compatibility issue. Debugging their code and/or configuration did not sound attractive.

Having read blog and forum postings (follow the link for one example) saying that ATLAS and OpenBLAS give generally similar improved performance, and OpenBLAS is easier to set up, I decided to give it a try first, before attempting to debug ATLAS. Indeed, OpenBLAS is spectacularly faster than the stock BLAS.

Installing and Linking

To install OpenBLAS do these steps:

zypper install libopenblas0 libopenblasp0
update-alternatives --config libblas.so.3 #And select libopenblas0.so.0

In the first step, I install both the single and multiple thread versions (libopenblasp0) even though not every problem benefits from threading. Mathnet has an enterprise mirror into which I have downloaded the RPM files. At other sites, since libopenblas0 is in an unstable sub-repo which you are unlikely to have turned on, you may need to download it to temporary storage and then give the full path name to Zypper. In OpenSuSE 13.2, OpenBLAS is in the main distro so the download step will not be necessary.

The second step causes a symbolic link to be made so libblas.so.3 really points to OpenBLAS and not to the stock BLAS. Thus when a user builds a program with -lblas on the linker command line, he will get the then-prevailing alternative, without needing to recompile if a different package is installed. I wrote a daily housekeeping script that reviews such non-default alternative settings (presently this is the only one) and complains if any machine is set contrary to policy.

To link with the threaded version, replace -lblas with either -lopenblasp0 or the full path to the library, currently /usr/lib64/libopenblasp.so.0 .

Performance Improvement

How much performance improvement is achieved? Here are some results on several representative machines; others were tested and give generally similar results according to the CPU speed.

• Achilles: Intel i7-4770S 3.10GHz x4, currently (2015-03-xx) Mathnet's top of the line quad core research workstation.
• Joshua02: Intel Xeon X5650 2.67GHz x6, our main cluster has 16 of these.
• Nemo01: Intel Xeon 3.40GHz x2, an older but still useful cluster with 10 of these.
• Diamond: Intel i7-3517UE 1.70GHz x2, the CFT compute server.

How much speedup was there, as a ratio?

So in summary, DGEMM (matrix multiply) speeds up by 7x to 14x, the newer and faster machines getting more benefit. DGESV (linear equations) speeds up by 6x to 10x; for some reason Joshua has the least benefit. DGEEV (eigenvectors) speeds up by 2.6x to 4x, the faster machines doing better.

As for threading, the most spectacular was 63x with matrix multiplication on Joshua02 (6 cores). OpenBLAS promises to avoid using Intel hyperthread, that is, it should limit the number of threads to the number of cores (not doubled), to avoid the overhead of spawning threads that cannot bring any benefit. However, hyper CPUs were fully utilized. For DGEMM (matrix multiply) and DGESV (linear equations) the speedup versus serial OpenBLAS was roughly proportional to the number of cores -- very roughly, and the overhead of recruiting threads is clear. In fact, Joshua02 speeds up more than the number of cores; likely OpenBLAS is not optimal for its cache configuration and when the problem is sliced up for threading it becomes more cache-friendly. But for DGEEV (eigenvectors) only Joshua02 ran a bit faster with threading, a disappointing result.

So my conclusion is, serial OpenBLAS is very successful as a drop-in replacement for the stock BLAS, and pthreaded OpenBLAS can give spectacular performance for particular kinds of problems. But the user needs to compare his or her particular algorithm with and without threading. Also, manual threading, i.e. multiple non-interacting single thread jobs, can be very effective with little effort.