add the polybench testcases to the test folder

[jamesthejellyfish]

migrates the polybench testcases over to PyDSL. All files in tests/polybench/benchmarks are largely unchanged, the main addition is the test_polybench.py file. A bash script is also added so that a user to generate the polybench .so files if they provide a Polybench directory, and then those will be used instead of the golden files for checking against, if they exist.

[Balint-R]

I count 29 dots. There should be 30?

[Balint-R]

What is this? From CompilationSettings, this seems to be something only used for PolyCTarget?

[Balint-R]

It's probably more appropriate to merge this into tests/helper.py?

[Balint-R]

Ah ok. These are just default values. Maybe write a one-line comment about that.

[Balint-R]

Actually @jamesthejellyfish can you give me a high level overview of how these testcases work, e.g. where do you get the input from, what parts are new and what parts are there just so you don't have to modify the existing implementation? Like the general thing of passing arguments as command line arguments is largely due to compatibility with llvm-lit?

[jamesthejellyfish]

Actually @jamesthejellyfish can you give me a high level overview of how these testcases work, e.g. where do you get the input from, what parts are new and what parts are there just so you don't have to modify the existing implementation? Like the general thing of passing arguments as command line arguments is largely due to compatibility with llvm-lit?

Sure thing. Basically the inputs are defined within each function based on what is used in the upstream polybench init_arrays implementation. And I was trying to minimize the need to alter the existing implementations which is why the arguments are passed as command line arguments, yes. I don't know if that was an llvm-lit compatibility issue or just was something that Xingyu decided on though. The actual implementations have been almost completely untouched atm (I think I just got rid of the llvm-lit comments at the top of each file)

[jamesthejellyfish]

I count 29 dots. There should be 30?

we don't test against the gauss-filter testcase. Not exactly sure why. I remember writing a guassian filter kernel in PyDSL... I assume that it just wasn't relevant to the other work we were doing so they didn't bother adding it to the test suite

[Balint-R]

No newline at end of file.

[hw-yuexy]

I count 29 dots. There should be 30?

we don't test against the gauss-filter testcase. Not exactly sure why. I remember writing a guassian filter kernel in PyDSL... I assume that it just wasn't relevant to the other work we were doing so they didn't bother adding it to the test suite

gauss-filter is not a polybench testcase. The one we are missing from the 30 is nussinov, which needs affine.if python binding that's added in llvm-20. The gauss-filter kernel written by James long time ago was to test PyDSL's expressability on a CV related kernel, the syntax is probably outdated, so I think it's ok to leave it.

[jamesthejellyfish]

I count 29 dots. There should be 30?

we don't test against the gauss-filter testcase. Not exactly sure why. I remember writing a guassian filter kernel in PyDSL... I assume that it just wasn't relevant to the other work we were doing so they didn't bother adding it to the test suite

gauss-filter is not a polybench testcase. The one we are missing from the 30 is nussinov, which needs affine.if python binding that's added in llvm-20. The gauss-filter kernel written by James long time ago was to test PyDSL's expressability on a CV related kernel, the syntax is probably outdated, so I think it's ok to leave it.

ah okay I do vaguely remember that with nussinov. I was looking at this documentation here: https://sourceforge.net/p/polybench/wiki/Home/ and it includes gauss-filter as a polybench testcase which is why I assumed we were missing that one

[hw-yuexy]

I count 29 dots. There should be 30?

we don't test against the gauss-filter testcase. Not exactly sure why. I remember writing a guassian filter kernel in PyDSL... I assume that it just wasn't relevant to the other work we were doing so they didn't bother adding it to the test suite

gauss-filter is not a polybench testcase. The one we are missing from the 30 is nussinov, which needs affine.if python binding that's added in llvm-20. The gauss-filter kernel written by James long time ago was to test PyDSL's expressability on a CV related kernel, the syntax is probably outdated, so I think it's ok to leave it.

ah okay I do vaguely remember that with nussinov. I was looking at this documentation here: https://sourceforge.net/p/polybench/wiki/Home/ and it includes gauss-filter as a polybench testcase which is why I assumed we were missing that one

the downloaded polybench package has floyd-warshall but does not have gauss-filter... I didn't even know the polybench's documentation has this mistake..

[Balint-R]

Actually @jamesthejellyfish can you give me a high level overview of how these testcases work, e.g. where do you get the input from, what parts are new and what parts are there just so you don't have to modify the existing implementation? Like the general thing of passing arguments as command line arguments is largely due to compatibility with llvm-lit?

Sure thing. Basically the inputs are defined within each function based on what is used in the upstream polybench init_arrays implementation. And I was trying to minimize the need to alter the existing implementations which is why the arguments are passed as command line arguments, yes. I don't know if that was an llvm-lit compatibility issue or just was something that Xingyu decided on though. The actual implementations have been almost completely untouched atm (I think I just got rid of the llvm-lit comments at the top of each file)

Why does GitHub not have proper reply feature for comments directly in the PR...

Ok, so you are saying I should only review helpers.py, generate_polybench_folder.sh, and test_polybench.py, since changing something in all 29 of the other files is a lot of effort, and thus we should model them as we can't easily change them and must work with their existing interface?

Ok so the input is not stored in text files or generated by an external program, but rather it is the code that looks like this (?):

PyDSL/tests/polybench/benchmarks/lu.py

Lines 79 to 85 in 92d41c2

	# init array
	for i in range(n):
	for j in range(i + 1):
	a[i, j] = -(j % n) / n + 1
	for j in range(i + 1, n):
	a[i, j] = 0.0
	a[i, i] = 1.0

Something I am still confused about is, what is generate_polybench_folder.sh. I have not run it and I have no Polybench_so_files directory (why is that uppercase btw?), yet all 29 tests still pass.
Is this for the user to run if they want to compare against actual polybench impl?
Maybe write a README.md in tests/polybench about how that works (also a general description about how these tests work, but not too detailed, only what a user running these tests might care about).

Also it is still hard for me to figure out some things about how the functions are called, mainly what command line arguments are passed in and thus what branches are taken. E.g. do we pass in c_test? It seems yes, since I found the relevant code in test_polybench.py, but this could still be explained better in a textual form (it's hard to trace this kind of stuff by reading the code, since I start by reading something like lu.py, then have to read helpers.py to see how arguments are parsed, then have to read test_polybench.py to see what arguments are passed in, then have to check the directory structure to see what directories/files exist/don't exit). Basically it would be easier to understand this if there was some text describing the high level control flow of how we run these testcases (e..g "We don't generate Polybench_so_files for pytest. test_polybench goes through all the files in polybench dir and calls the Python functions in them (or executes them as a subprocess?). We pass in these arguments. Inputs are generated in this manner and outputs are checked for correctness by comparing to the text files."). Maybe since this is not something that concerns the user too much, write a docstring for test_polybench that explains how it works?

gauss-filter is not a polybench testcase. The one we are missing from the 30 is nussinov, which needs affine.if python binding that's added in llvm-20. The gauss-filter kernel written by James long time ago was to test PyDSL's expressability on a CV related kernel, the syntax is probably outdated, so I think it's ok to leave it.

Okay, we don't have to add it while we are on LLVM 19 then.

[jamesthejellyfish]

Ok, so you are saying I should only review helpers.py, generate_polybench_folder.sh, and test_polybench.py, since changing something in all 29 of the other files is a lot of effort, and thus we should model them as we can't easily change them and must work with their existing interface?

That's not exactly what I'm saying. I modeled my approach to try and change the 29 files as little as possible
since not changing the 29 files is less effort, but if you think that it would be better for us to change the implementation I'm
not saying that we shouldn't necessarily, just that I tried my best to leave them mostly as-is.

Ok so the input is not stored in text files or generated by an external program, but rather it is the code that looks like this (?):

PyDSL/tests/polybench/benchmarks/lu.py

Lines 79 to 85 in 92d41c2

	# init array
	for i in range(n):
	for j in range(i + 1):
	a[i, j] = -(j % n) / n + 1
	for j in range(i + 1, n):
	a[i, j] = 0.0
	a[i, i] = 1.0

yes exactly. And this is a python implementation that mirrors what is done in the polybench C files.

Something I am still confused about is, what is generate_polybench_folder.sh. I have not run it and I have no Polybench_so_files directory (why is that uppercase btw?), yet all 29 tests still pass. Is this for the user to run if they want to compare against actual polybench impl? Maybe write a README.md in tests/polybench about how that works (also a general description about how these tests work, but not too detailed, only what a user running these tests might care about).

generate_polybench_folder.sh is meant for a user to use the actual polybench implementation to compare against, it takes every C implementation in polybench, alters the C file to remove the "static" from the kernel_* function (something we hardcoded before), compiles them, and then stores all the .so files in Polybench_so_files directory. If the Polybench_so_files directory doesn't exist, then the testcases are tested against the golden files, but if it is, then the testcases run against the polybench c implementations. I can write a README.md giving an overview of this, yes.

Also it is still hard for me to figure out some things about how the functions are called, mainly what command line arguments are passed in and thus what branches are taken. E.g. do we pass in c_test? It seems yes, since I found the relevant code in test_polybench.py, but this could still be explained better in a textual form (it's hard to trace this kind of stuff by reading the code, since I start by reading something like lu.py, then have to read helpers.py to see how arguments are parsed, then have to read test_polybench.py to see what arguments are passed in, then have to check the directory structure to see what directories/files exist/don't exit). Basically it would be easier to understand this if there was some text describing the high level control flow of how we run these testcases (e..g "We don't generate Polybench_so_files for pytest. test_polybench goes through all the files in polybench dir and calls the Python functions in them (or executes them as a subprocess?). We pass in these arguments. Inputs are generated in this manner and outputs are checked for correctness by comparing to the text files."). Maybe since this is not something that concerns the user too much, write a docstring for test_polybench that explains how it works?

we pass in c_test only if the Polybench_so_files exists. But yes, I will write a docstring and also mention usage briefly in the README