openblas threads slow for small matrices

[guolas]

Hi,

I have just downloaded the latest version available on Github, and I have compiled it in two different systems:

• MacOS, Intel Core i5- @ 1.7GHz, with 4GB of memory.
• Linux, Intel Core i7-3960X CPU @ 3.30GHz, with 32GB of memory.

In both systems I have compiled the following version:

Version 0.2.0-prerelease+3005
Commit 0d39c66 2013-08-06 17:44:39 UTC

After executing the same Julia code, I have that the execution times are as follows:

• ~100s on the Mac.
• ~135s on the Linux.

Does it really make any sense?

I mean, the Linux computer is way more powerful, with 12 cores, each of which almost double the clock speed. And as it is executing, I can see, using top that all the CPUs are being used in the Linux machine, and still, it takes longer to finish the execution.

I have tried this several times, and it consistently yields the same execution times.

Thank you for any comment on this.

Regards,

Juanjo.

[Keno]

Without knowing the code you're running, we can't possibly make an assessment.

[guolas]

The code I am using can be found in:

julia_test

If that can help understanding what happens.

[pao]

Direct link to source: https://github.com/guolas/julia_test/blob/master/complete.jl

[staticfloat]

First, I would try using the profiler to figure out which lines of code are sucking up most of the CPU time. It's possible there's some operation that is just taking longer on Linux than OSX for some strange reason, and the profiler will help track that down.

If that doesn't solve the mystery, could you post the output of versioninfo() from both of your systems? It's possible we're comparing apples and oranges here, as your BLAS implementations could make a significant difference, as we've seen in some (admittedly linear algebra-heavy) tests, which may or may not be happening here. In any case, that should show up in the profiler as well, as it will show the BLAS/LAPACK functions taking up all the CPU time.

[guolas]

The versioninfo() I can provide it now. For the Mac (that runs faster):

julia> versioninfo()
Julia Version 0.2.0-prerelease+3005
Commit 0d39c66 2013-08-06 17:44:39 UTC
Platform Info:
  System: Darwin (x86_64-apple-darwin12.4.0)
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY)
  LAPACK: libopenblas
  LIBM: libopenlibm

For the Linux:

julia> versioninfo()
Julia Version 0.2.0-prerelease+3005
Commit 0d39c66 2013-08-06 17:44:39 UTC
Platform Info:
  System: Linux (x86_64-redhat-linux)
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY)
  LAPACK: libopenblas
  LIBM: libopenlibm

I guess that this does not help much, as the versions are the same in both systems. I will post information about the profiling when I got them.

Thank you.

[ViralBShah]

Yes, knowing what the profiler says will be useful. For example, is it a case of openblas tuning, or is it a julia compiler issue? We certainly need to chase this one down.

[JeffBezanson]

It would be interesting to try this with export OPENBLAS_NUM_THREADS=1. I get:

[jeff ~/src/julia]$ env|grep BLAS
OPENBLAS_NUM_THREADS=1
[jeff ~/src/julia]$ ./julia i3966.jl
elapsed time: 58.775533131 seconds
[jeff ~/src/julia]$ export OPENBLAS_NUM_THREADS=2
[jeff ~/src/julia]$ ./julia i3966.jl
elapsed time: 77.535673898 seconds

@xianyi

[xianyi]

Does it call GEMM or other BLAS functions? Could you give me a list?

[JeffBezanson]

I believe it is mostly dgemm, zgemm, and whatever zgesdd calls.

[xianyi]

I see. It is an old single thread / multi-threading adjustment issue.

[JeffBezanson]

More info: most of the time is doing many zgesdd calls of size 18x30. Can be reproduced by @time for i=1:10000; svd(rand(18,30),false); end

[staticfloat]

@guolas, can you tell how many threads are being started up in each case? Perhaps via htop or something....

[ViralBShah]

@xianyi Why not use multi threading only for problems bigger than something like 200x200? We can do some experiments and find these thresholds.

[guolas]

I have tried limiting the number of threads as @JeffBezanson suggested, and I get this:

For export OPENBLAS_NUM_THREADS=1:

• Mac: elapsed time: 64.820198286 seconds
• Linux: elapsed time: 59.656547279 seconds

For export OPENBLAS_NUM_THREADS=2:

• Mac: elapsed time: 101.788748919 seconds
• Linux: elapsed time: 83.597288193 seconds

And the time keeps increasing for the Linux machine, until I use the whole 12 threads that it can use.

When I don't set anything, according to top, it takes 1200% of the CPU, which probably means that it is using 12 threads.

[ViralBShah]

@blakejohnson In your experiments, how does MKL fare on small problems, compared to openblas?