Errors while setting up a new nightly runner

[Naman-Priyadarshi]

After following the steps given in the Readme.md, I got an error in step 10 i.e. Validate and build environments using asv check.

---

---

---

---

Versions:
asv = 0.4.2
python = 3.7.1
conda = 4.10.3
virtualenv 20.14.0

[Naman-Priyadarshi]

Sorry for pasting all the screenshots, I couldn't understand the error properly.

[kandersolar]

Hi @Naman-Priyadarshi, thanks for filing this issue. Right now we run the benchmarks in two environements: one with python 3.6 and on with 3.8. It looks like creating the 3.6 environment is what's failing on your computer. Our benchmark server runs linux, so I'm not entirely surprised that it doesn't work on windows. One quick fix just to get things working on your computer might be to disable the 3.6 environment by deleting/commenting out these lines in the configuration file: https://github.com/pvlib/pvlib-python/blob/master/benchmarks/asv.conf.json#L115-L125

Of course we should think about better fixes for the longer term. A couple ideas:

• Fiddle with the package versions to find a combination that doesn't fail on Windows
• Stop benchmarking on python 3.6 (it's reached EOL anyway), and replace it with a newer version like 3.9 or 3.10

[Naman-Priyadarshi]

I removed the lines to disable the 3.6v environment but there's a new error now
i.e.

[kandersolar]

Hmm, I'm not sure about that one. It kind of looks like there's some artifact of an old build left over. Can you locate that file and delete it? Restarting your computer might help too, if it's something silly like an old process didn't close that file properly. Otherwise I'm not sure what to suggest.

[Naman-Priyadarshi]

I located the file and deleted it, but I'm still getting the same error. (Restarted my computer too)

[Naman-Priyadarshi]

Should I try setting up the benchmark in linux?

[Naman-Priyadarshi]

Update: asv check ran successfully in Linux and gave the following output:

· Discovering benchmarks
·· Creating conda environment for conda-py3.8-ephem-h5py-numba-numpy-pandas-scipy..............................................
·· Uninstalling from conda-py3.8-ephem-h5py-numba-numpy-pandas-scipy
·· Building 5cb695d2 <master> for conda-py3.8-ephem-h5py-numba-numpy-pandas-scipy..
·· Installing 5cb695d2 <master> into conda-py3.8-ephem-h5py-numba-numpy-pandas-scipy.
· Checking benchmarks
·· No problems found.

But after running the next step i.e. asv dev there's an error

(also I have some suggestions for Readme.md)

[kandersolar]

Good progress! Looks like the example benchmarks you're running are written for python 2.x, but you're running on 3.7, so no big surprise that it's failing. I guess maybe you copied the example benchmarks from here? Since the failures are in the benchmark code itself, it seems like ASV itself is working correctly, which is great! FYI I just sent in this PR to update the asv docs to not use those old 2.x examples anymore: airspeed-velocity/asv#1207

Anyway, instead of using those dummy example benchmarks, I guess the next step is to use the real pvlib benchmarks. If you have the pvlib-python repository next to the pvlib-benchmarks repository, I think all you need to do is modify the configuration file to point to the real benchmarks location like how the nightly job script does it:

https://github.com/pvlib-benchmarker/pvlib-benchmarks/blob/68c11a57eb2619790ab0a797d52e427391c22983/cronjob.sh#L29-L32

So it should probably look like this:

"repo": "../pvlib-python",
...
"benchmark_dir": "../pvlib-python/benchmarks/benchmarks",

I have some suggestions for Readme.md

A PR with improvements would be most welcome! I originally wrote those instructions for myself in case we needed to switch machines at some point in the future. I didn't really think anybody else would read them, and I've not had any reason to update them until now :)

[Naman-Priyadarshi]

The Benchmarks ran successfully! Thanks @kanderso-nrel