symv broken on development version for ZEN

[chriselrod]

I've reproduced the issue in Julia and Fortran, both linked to a fresh source build of the latest master.

program symvtest
implicit none
integer, parameter :: n = 20, p = 12
real, dimension(n, p) :: Xb
real, dimension(p, p) :: A
real, dimension(p) :: x, y, ys

call random_number(Xb)
call random_number(x)

A = matmul( transpose(Xb), Xb )
y = matmul( A, x )

ys = 0.0

call ssymv('U', p, 1.0, A, p, x, 1, 0.0, ys, 1)

print *, 'Correct:', y
print *, 'ssymv:', ys

end program symvtest

$ gfortran-7.2 symvtest.f08 -o symvt2 -pthread /opt/OpenBLAS/lib/libopenblas.a

chris@elrod:~/Documents/programming/fortran$ ./symvt2
 Correct:   40.6064873       43.2426643       37.7620964       37.9279366       37.8264160       41.4897804       40.9279518       31.3577194       28.1611786       36.6945801       34.9506340       36.2780457    
 ssymv:   57.7142792       59.8994789       52.9540024       54.8316040       52.9050522       57.4723549       58.2451744       43.7106018       28.1611786       36.6945801       34.9506340       36.2780457

In Julia:

julia> A = randn(10,20) |> X -> X * X';

julia> x = rand(10);

julia> (A * x)' #calls gemv
1×10 RowVector{Float64,Array{Float64,1}}:
 11.538  7.90415  -1.03592  -3.84546  15.5033  3.37954  7.7948  4.26935  6.29426  5.57295

julia> Base.LinAlg.BLAS.symv('U', A, x)'
1×10 RowVector{Float64,Array{Float64,1}}:
 21.5629  15.7696  -4.14023  -10.6005  28.2271  5.50057  17.9761  6.60855  6.29426  5.57295

EDIT:
When building OpenBLAS it runs tests; excerpt:

OPENBLAS_NUM_THREADS=2 ./xdcblat3 < din3
 TESTS OF THE DOUBLE PRECISION LEVEL 3 BLAS

 THE FOLLOWING PARAMETER VALUES WILL BE USED:
   FOR N                   1     2     3     5     7     9
   FOR ALPHA             0.0   1.0   0.7
   FOR BETA              0.0   1.0   1.3

 ROUTINES PASS COMPUTATIONAL TESTS IF TEST RATIO IS LESS THAN   16.00

 COLUMN-MAJOR AND ROW-MAJOR DATA LAYOUTS ARE TESTED

 RELATIVE MACHINE PRECISION IS TAKEN TO BE  2.2D-16

 cblas_dgemm  PASSED THE TESTS OF ERROR-EXITS

 cblas_sgbmv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS ( 17284 CALLS)

 cblas_ssymv  PASSED THE TESTS OF ERROR-EXITS

 cblas_ssymv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_dgemm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS ( 17496 CALLS)
 cblas_ssymv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1729 CALLS)

 cblas_ssbmv  PASSED THE TESTS OF ERROR-EXITS

 cblas_ssbmv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  6913 CALLS)
 cblas_dgemm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS ( 17496 CALLS)

 cblas_dsymm  PASSED THE TESTS OF ERROR-EXITS

I assume the tests check that the answers are valid?
A little unsure what to make of this.

As a workaround I can copy values from the upper triangle into the lower triangle and use gemm, but I'd rather fix the problem.
Any idea what's going on?
Any suggestions of what I can look at, tests to run, etc?

[brada4]

Can you show julia sysinfo(true) to see openblas core type used?

[brada4]

Can you chech with bigger sample that first 8 (32 bYtes) values are corrupt but rest is fine?

[chriselrod]

Oops, the pattern definitely changes as size increases.
At n <= 8:
All answers are correct
At 8 < n <= 155:
rem(n,4) are correct , unless that remainder is zero in which case 4 are correct.
Easier to show:

julia> function check_symv(n)
           A = randn(n, 2n) |> X -> X * X';
           x = rand(n);
           gemv = A * x
           symv = Base.LinAlg.BLAS.symv('U', A, x)
           correct = gemv .≈ symv
           correct_count = sum(correct)
           eight = min(8,n)
           correct_in_first_8 = sum(@view(correct[1:eight]))
           n, correct_count, n - correct_count, correct_in_first_8, eight - correct_in_first_8
       end
check_symv (generic function with 1 method)

julia> check_symv.(1:30)
30-element Array{NTuple{5,Int64},1}:
 (1, 1, 0, 1, 0)  
 (2, 2, 0, 2, 0)  
 (3, 3, 0, 3, 0)  
 (4, 4, 0, 4, 0)  
 (5, 5, 0, 5, 0)  
 (6, 6, 0, 6, 0)  
 (7, 7, 0, 7, 0)  
 (8, 8, 0, 8, 0)  
 (9, 1, 8, 0, 8)  
 (10, 2, 8, 0, 8) 
 (11, 3, 8, 0, 8) 
 (12, 4, 8, 0, 8) 
 (13, 1, 12, 0, 8)
 (14, 2, 12, 0, 8)
 (15, 3, 12, 0, 8)
 (16, 4, 12, 0, 8)
 (17, 1, 16, 0, 8)
 (18, 2, 16, 0, 8)
 (19, 3, 16, 0, 8)
 (20, 4, 16, 0, 8)
 (21, 1, 20, 0, 8)
 (22, 2, 20, 0, 8)
 (23, 3, 20, 0, 8)
 (24, 4, 20, 0, 8)
 (25, 1, 24, 0, 8)
 (26, 2, 24, 0, 8)
 (27, 3, 24, 0, 8)
 (28, 4, 24, 0, 8)
 (29, 1, 28, 0, 8)
 (30, 2, 28, 0, 8)

Columns are:

1. n
2. total correct
3. total incorrect
4. number in first 8 that are correct
5. number in the first 8 that are incorrect

At 156 and above the pattern becomes more complicated:

julia> check_symv.(149:172)
24-element Array{NTuple{5,Int64},1}:
 (149, 1, 148, 0, 8)
 (150, 2, 148, 0, 8)
 (151, 3, 148, 0, 8)
 (152, 4, 148, 0, 8)
 (153, 1, 152, 0, 8)
 (154, 2, 152, 0, 8)
 (155, 3, 152, 0, 8)
 (156, 8, 148, 0, 8)
 (157, 1, 156, 0, 8)
 (158, 2, 156, 0, 8)
 (159, 3, 156, 0, 8)
 (160, 4, 156, 0, 8)
 (161, 5, 156, 0, 8)
 (162, 2, 160, 0, 8)
 (163, 3, 160, 0, 8)
 (164, 4, 160, 0, 8)
 (165, 5, 160, 0, 8)
 (166, 6, 160, 0, 8)
 (167, 3, 164, 0, 8)
 (168, 4, 164, 0, 8)
 (169, 5, 164, 0, 8)
 (170, 6, 164, 0, 8)
 (171, 7, 164, 0, 8)
 (172, 4, 168, 0, 8)

As for sysinfo, is this what you wanted?

julia> versioninfo()
Julia Version 0.7.0-DEV.2767
Commit 8fb8db1 (2017-12-05 18:20 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: AMD Ryzen Threadripper 1950X 16-Core Processor
  WORD_SIZE: 64
  BLAS: libopenblas (ZEN)
  LAPACK: liblapack
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, znver1)
Environment:

Or

julia> versioninfo()
Julia Version 0.6.1
Commit 0d7248e* (2017-10-24 22:15 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: AMD Ryzen Threadripper 1950X 16-Core Processor
  WORD_SIZE: 64
  BLAS: libopenblas (ZEN)
  LAPACK: liblapack
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, generic)

Core type is ZEN.

EDIT:
A few more ranges. If there are any that would be particularly diagnostic, let me know:

julia> check_symv.(1020:1040)
21-element Array{NTuple{5,Int64},1}:
 (1020, 16, 1004, 0, 8)
 (1021, 17, 1004, 0, 8)
 (1022, 18, 1004, 0, 8)
 (1023, 19, 1004, 0, 8)
 (1024, 16, 1008, 0, 8)
 (1025, 17, 1008, 0, 8)
 (1026, 18, 1008, 0, 8)
 (1027, 19, 1008, 0, 8)
 (1028, 20, 1008, 0, 8)
 (1029, 17, 1012, 0, 8)
 (1030, 18, 1012, 0, 8)
 (1031, 19, 1012, 0, 8)
 (1032, 20, 1012, 0, 8)
 (1033, 21, 1012, 0, 8)
 (1034, 22, 1012, 0, 8)
 (1035, 23, 1012, 0, 8)
 (1036, 24, 1012, 0, 8)
 (1037, 21, 1016, 0, 8)
 (1038, 22, 1016, 0, 8)
 (1039, 23, 1016, 0, 8)
 (1040, 24, 1016, 0, 8)

julia> check_symv.(2040:2060)
21-element Array{NTuple{5,Int64},1}:
 (2040, 56, 1984, 0, 8)
 (2041, 53, 1988, 0, 8)
 (2042, 54, 1988, 0, 8)
 (2043, 55, 1988, 0, 8)
 (2044, 56, 1988, 0, 8)
 (2045, 53, 1992, 0, 8)
 (2046, 54, 1992, 0, 8)
 (2047, 55, 1992, 0, 8)
 (2048, 56, 1992, 0, 8)
 (2049, 53, 1996, 0, 8)
 (2050, 54, 1996, 0, 8)
 (2051, 55, 1996, 0, 8)
 (2052, 56, 1996, 0, 8)
 (2053, 53, 2000, 0, 8)
 (2054, 54, 2000, 0, 8)
 (2055, 55, 2000, 0, 8)
 (2056, 56, 2000, 0, 8)
 (2057, 53, 2004, 0, 8)
 (2058, 54, 2004, 0, 8)
 (2059, 55, 2004, 0, 8)
 (2060, 56, 2004, 0, 8)

julia> check_symv.(4080:5100) #Meant to do 4080-4100.
1021-element Array{NTuple{5,Int64},1}:
 (4080, 116, 3964, 0, 8)
 (4081, 113, 3968, 0, 8)
 (4082, 114, 3968, 0, 8)
 (4083, 115, 3968, 0, 8)
 (4084, 112, 3972, 0, 8)
 (4085, 113, 3972, 0, 8)
 (4086, 114, 3972, 0, 8)
 (4087, 115, 3972, 0, 8)
 (4088, 116, 3972, 0, 8)
 (4089, 117, 3972, 0, 8)
 (4090, 114, 3976, 0, 8)
 (4091, 115, 3976, 0, 8)
 (4092, 116, 3976, 0, 8)
 (4093, 117, 3976, 0, 8)
 (4094, 114, 3980, 0, 8)
 (4095, 115, 3980, 0, 8)
 (4096, 116, 3980, 0, 8)
 (4097, 117, 3980, 0, 8)
 (4098, 118, 3980, 0, 8)
 (4099, 119, 3980, 0, 8)
 (4100, 116, 3984, 0, 8)
 (4101, 117, 3984, 0, 8)
 ⋮                      
 (5080, 152, 4928, 0, 8)
 (5081, 153, 4928, 0, 8)
 (5082, 150, 4932, 0, 8)
 (5083, 151, 4932, 0, 8)
 (5084, 152, 4932, 0, 8)
 (5085, 153, 4932, 0, 8)
 (5086, 154, 4932, 0, 8)
 (5087, 155, 4932, 0, 8)
 (5088, 156, 4932, 0, 8)
 (5089, 157, 4932, 0, 8)
 (5090, 154, 4936, 0, 8)
 (5091, 155, 4936, 0, 8)
 (5092, 152, 4940, 0, 8)
 (5093, 153, 4940, 0, 8)
 (5094, 154, 4940, 0, 8)
 (5095, 151, 4944, 0, 8)
 (5096, 152, 4944, 0, 8)
 (5097, 153, 4944, 0, 8)
 (5098, 154, 4944, 0, 8)
 (5099, 155, 4944, 0, 8)
 (5100, 152, 4948, 0, 8)

At small sizes, symv benchmarked at over 10 times slower than gemv with this setup, besides producing the wrong answer.

[martin-frbg]

There is something wrong with the fortran testcase it seems, you definitely need to uncomment the initialization statements for ys, else every second number is random uninitialized memory contents.
Haswell,Nehalem and Atom builds run on Kaby Lake produce numbers that differ in the last two digits only. Are your numbers obtained with a recent build of OpenBLAS ? The first snapshots with ZEN
support from late March, early April happened to be severely broken, see #1147

[chriselrod]

I changed beta in the symv call to 0.0, and also added the ys = 0.0 in case some uninitialized entries are treated as NaN or Infs.

Just confirmed -- I'm on the develop branch:

$ git pull
Already up-to-date.
$ git checkout develop
Already on 'develop'
Your branch is up-to-date with 'origin/develop'.

Just to confirm, after building, I ran sudo make install -- does this properly replace the old install, which would have been build a couple months ago (and I do not recall which version I had checked out at the time)?
If not, what would I have to do to correctly update?

EDIT:
Should I try building with TARGET=HASWELL ?
Update: Build fails when I run make with TARGET=HASWELL; lots of undefined references:

<snip>
linktest.c:(.text.startup+0x8490): undefined reference to `LAPACKE_sgeqpf'
linktest.c:(.text.startup+0x8497): undefined reference to `LAPACKE_sgeqpf_work'
linktest.c:(.text.startup+0x849e): undefined reference to `LAPACKE_zgeqpf'
linktest.c:(.text.startup+0x84a5): undefined reference to `LAPACKE_zgeqpf_work'
collect2: error: ld returned 1 exit status
Makefile:143: recipe for target '../libopenblas_haswellp-r0.3.0.dev.so' failed
make[1]: *** [../libopenblas_haswellp-r0.3.0.dev.so] Error 1
make[1]: Leaving directory '/home/chris/Documents/programming/OpenBLAS/exports'
Makefile:90: recipe for target 'shared' failed
make: *** [shared] Error 2

[chriselrod]

Using the lower triangle instead:

julia> function check_symv(n, Uplo = 'U')
           A = randn(n, 2n) |> X -> X * X';
           x = rand(n);
           gemv = A * x
           symv = Base.LinAlg.BLAS.symv(Uplo, A, x)
           correct = gemv .≈ symv
           correct_count = sum(correct)
           eight = min(8,n)
           correct_in_first_8 = sum(@view(correct[1:eight]))
           n, correct_count, n - correct_count, correct_in_first_8, eight - correct_in_first_8
       end
check_symv (generic function with 1 method)

julia> check_symv.(1:20, 'L')
20-element Array{NTuple{5,Int64},1}:
 (1, 1, 0, 1, 0)  
 (2, 2, 0, 2, 0)  
 (3, 3, 0, 3, 0)  
 (4, 4, 0, 4, 0)  
 (5, 5, 0, 5, 0)  
 (6, 6, 0, 6, 0)  
 (7, 7, 0, 7, 0)  
 (8, 8, 0, 8, 0)  
 (9, 8, 1, 8, 0)  
 (10, 8, 2, 8, 0) 
 (11, 8, 3, 8, 0) 
 (12, 8, 4, 8, 0) 
 (13, 8, 5, 8, 0) 
 (14, 8, 6, 8, 0) 
 (15, 8, 7, 8, 0) 
 (16, 8, 8, 8, 0) 
 (17, 8, 9, 8, 0) 
 (18, 8, 10, 8, 0)
 (19, 8, 11, 8, 0)
 (20, 8, 12, 8, 0)

julia> check_symv.(149:172, 'L')
24-element Array{NTuple{5,Int64},1}:
 (149, 8, 141, 8, 0) 
 (150, 8, 142, 8, 0) 
 (151, 8, 143, 8, 0) 
 (152, 8, 144, 8, 0) 
 (153, 8, 145, 8, 0) 
 (154, 12, 142, 8, 0)
 (155, 12, 143, 8, 0)
 (156, 12, 144, 8, 0)
 (157, 12, 145, 8, 0)
 (158, 16, 142, 8, 0)
 (159, 16, 143, 8, 0)
 (160, 16, 144, 8, 0)
 (161, 16, 145, 8, 0)
 (162, 16, 146, 8, 0)
 (163, 16, 147, 8, 0)
 (164, 16, 148, 8, 0)
 (165, 16, 149, 8, 0)
 (166, 16, 150, 8, 0)
 (167, 16, 151, 8, 0)
 (168, 16, 152, 8, 0)
 (169, 16, 153, 8, 0)
 (170, 16, 154, 8, 0)
 (171, 16, 155, 8, 0)
 (172, 16, 156, 8, 0)

[chriselrod]

I asked on gitter if anyone with a Ryzen processor could reproduce, and someone kindly responded with: https://pastebin.com/hJX8gF6T
They could not.

I assumed they didn't manually build OpenBLAS from source, either using a precompiled Julia binary (comes linked with OpenBLAS), or built Julia (which automatically downloads and builds OpenBLAS).
Because of this I tried checkout the last tagged version of OpenBLAS:

$ make clean
$ git checkout v0.2.20
$ make CC=gcc-7.2 FC=gfortran-7.2 COMMON_OPT=-O3 -j32
$ sudo make install

and now I get:

$ gfortran-7.2 -O3 symvtest.f08 -o symvt2 -pthread /opt/OpenBLAS/lib/libopenblas.a
$ ./symvt2
 Correct:   30.9002171       32.3528900       29.4461060       29.0806026       24.1383610       31.0188465       29.0695744       30.7085075       32.8042488       28.7348518       33.4618492       34.3150177    
 ssymv:   30.9002151       32.3528900       29.4461060       29.0806026       24.1383610       31.0188465       29.0695744       30.7085056       32.8042488       28.7348518       33.4618492       34.3150177    

So, now my problem is solved.

However, I am leaving the issue open because the development version of OpenBLAS's symv is broken on Rzyen.

[martin-frbg]

Can you confirm that your build of "develop" had actually been installed in the correct location ? There have not been any commits since 0.2.20 that targetted the Ryzen architecture, if anything I would suspect the recent code cleanup that removed a number of (obviously) unused variables. However the target is supposed to be identical to Intel "Haswell" and so far I am unable to reproduce your problem on Intel. Ideally we would need someone with a Ryzen system to do a "git bisect" between 0.2.20 and today if there is actually a problem with the current development version.

[chriselrod]

Okay:

$ git clone git://github.com/xianyi/OpenBLAS.git
$ cd OpenBLAS
$ git checkout develop #already on develop
$ make CC=gcc-7.2 FC=gfortran-7.2 COMMON_OPT=-O3 -j32
$ sudo make install PREFIX=/opt/OpenBLAS-dev
$ gfortran-7.2 -O3 symvtest.f08 -o symvt -pthread /opt/OpenBLAS-dev/lib/libopenblas.a
$ ./symvt # Linked to the development version
 Correct:   38.3103065       28.6310291       31.0854874       38.3372383       27.0824089       29.8740845       24.2437725       28.0576820       28.3724155       33.8617172       37.3014297       31.5492420    
 ssymv:   52.3679199       38.7304001       43.0292168       51.6194725       37.2317848       41.5859642       34.4001999       38.4238358       28.3724155       33.8617172       37.3014297       31.5492420
$ gfortran-7.2 -O3 symvtest.f08 -o symvtv2.2 -pthread /opt/OpenBLAS/lib/libopenblas.a
$ ./symvtv2.2 # Linked to OpenBLAS v0.2.20, which is installed at /opt/OpenBLAS
 Correct:   32.1122055       30.0320606       38.3960342       36.7253914       39.5436707       41.6092072       42.0634842       40.0813179       42.7693214       39.7569427       40.9749603       33.1785774    
 ssymv:   32.1122055       30.0320606       38.3960342       36.7253914       39.5436707       41.6092072       42.0634842       40.0813141       42.7693214       39.7569427       40.9749603       33.1785774

Let me know if you'd like me to try anything else.

[martin-frbg]

Could you try with "git checkout 5f402b7" please ? That would roll back your checkout of "develop" to directly before the only commit that touched the ssymv_U kernel file. (I still do not understand how that commit - removing the declarations of four unused variables - would cause such problems, unless there is some memory corruption occuring that was previously mitigated by the padding)

[brada4]

Im compartite in wondering about padding-looking declarations i removed around

[chriselrod]

After git checkout 5f402b7 and re-making, still get the wrong answer:

$ gfortran-7.2 -O3 symvtest.f08 -o symvt -pthread /opt/OpenBLAS-dev/lib/libopenblas.a
$ ./symvt
 Correct:   28.5529785       22.0841026       27.2444382       23.6138706       22.2497787       21.6344509       22.6846790       21.3639870       27.3313675       23.4497471       24.7438240       22.2422047    
 __ssymv:   43.7869606       34.2717743       42.4019775       36.6149216       34.4411087       34.6225929       35.2519379       33.8367157       27.3313675       23.4497471       24.7438240       22.2422047
$ ./symvt
 Correct:   24.1835136       27.7404785       24.7421417       28.3680420       23.6532784       26.2750969       22.8979492       22.0192585       22.0544701       25.9655113       29.9678802       25.3623199    
 __ssymv:   28.8016720       35.4656982       30.4767952       35.6221695       29.4691963       31.3436108       28.4602337       26.6515293       22.0544701       25.9655113       29.9678802       25.3623199

=/
While I don't get why removing unused variables would change anything, it would also have been nice to have a lead. I'll try reverting to different commits tonight to pin down which is the first to break it.

[martin-frbg]

Thank you very much for your help. git bisect will probably speed up the process a bit, as it will automatically determine which version to checkout next to narrow down the search. (Do git bisect start followed by git bisect good 5dde4e6 and git bisect bad 5f402b7 - this will fetch the commit in the middle between 0.2.20 and what you just tried. Build and test this, then depending on outcome enter git bisect good or git bisect bad until git responds with "xxx is the first bad commit". git bisect reset followed by git checkout will then return your checked-out copy to the current "develop" state)

[martin-frbg]

If you do not have the time for pinning it down exactly, testing a single version from end of August should narrow it down considerably I think.