Unable to reproduce result on HalfCheetah-v2

[quanvuong]

I am unable to obtain the result as reported in the paper on the openai environment HalfCheetah-v2. The commit used to obtain this result is 1f6147c, which isn't too long ago. The result is averaged over 5 random initial seeds.

Do you know what might be causing this issue? Thank you!

I am able to obtain the result as reported (or close to it) in the paper on the remaining environments, posted here for reference.

[hartikainen]

Thanks a lot for reporting this! This is not expected, and it's not immediately obvious to me what might cause it. Just to make sure: are all these figures produced with the default values in the variants.py? How many seeds are in each figure?

[quanvuong]

yup, the figures are produced with the default values in variants.py.

The results in each figure are averaged over 5 random initial seeds.

[hartikainen]

Thanks! I'll try to look into this soon.

[hartikainen]

Hey, I just ran HalfCheetah from the latest master, and unfortunately cannot reproduce the problems you're seeing. Here are the results I see across 6 seeds:

The results are a tiny bit worse than what we report in our paper [1], maybe because of the upgrade from mujoco 1.5 to 2.0.

Maybe there's something different in your environment? Could you post the output of pip freeze? Also, which mujoco version are you using?

[1] https://arxiv.org/pdf/1812.05905.pdf

[quanvuong]

Thank you for your comment!

I will rerun master and let you know how it goes here.

[quanvuong]

Thank you for your effort in maintaining the repo. It has been super useful and illuminating to read the code. Also, thank you for rerunning the master and helps me get to the bottom of my issue. Please find my results below:

Running the master on 5th May, which corresponds to commit 1f6686d does not reproduce the reported results. This is because in my runs, 1 out of 5 is significantly worse than the other runs. The performance graph is shown below.

However, you might not be able to reproduce this result by setting the initial seeds to the values in my runs. This is because of this issue #80.

After I also set the seed of the environment and fix the value for the initial seeds in the runs, I obtained similar results where some of the runs are significantly worse than others.

The output of pip freeze can be found in the file below

output.txt

I am using mujoco 1.50 for Linux.

[hartikainen]

Thanks for the detailed response! This still seems a little odd since I'm not able to get any seeds failing like that.

One difference I see in your pip freeze output is the mujoco-py version. You have mujoco-py==1.50.1.68 whereas we recently upgraded to mujoco-py==2.0.2.0. Could you try running pip install -U gym mujoco-py and installing mujoco 2.0, and see if that solves the issue?

[Nicolinho]

I also observed very similar results as @quanvuong did (including runs that converged at ~2k return). I used mujoco 2.0 and mujoco-py 2.0.2.0.

[hartikainen]

@Nicolinho would you mind pasting the output of your pip freeze and conda list here?

[Nicolinho]

@hartikainen
pip_freeze.txt
conda_list.txt

[hartikainen]

Hey @quanvuong and @Nicolinho, thanks a lot for providing the information. I spent a bit of time debugging this today. I couldn't find anything obvious that could be wrong, but was eventually able to reproduce this issue on one of my machines where 2/4 seeds failed. The weird thing is that still on other machines, the cheetah runs completely fine; I ran a total of 30 seeds on 3 different machines and none of them had the issue. 10 of those seeds are found from a comment in the latest PR: #85 (comment).

I'll try to dig deeper soon and let you know if I find anything.

[hartikainen]

Looks like the policy for some reason flips around in the beginning and thus gets stuck to a local minimum:

[quanvuong]

It was so satisfying to finally know why this happened. Thank you for the investigative work!

[szrlee]

Hey, I just ran HalfCheetah from the latest master, and unfortunately cannot reproduce the problems you're seeing. Here are the results I see across 6 seeds:

The results are a tiny bit worse than what we report in our paper [1], maybe because of the upgrade from mujoco 1.5 to 2.0.

Maybe there's something different in your environment? Could you post the output of pip freeze? Also, which mujoco version are you using?

[1] https://arxiv.org/pdf/1812.05905.pdf

Hi @hartikainen , for the curve of one single trial, do you use smoothed curve?
My reproduction is not very stable. This is plotted by tensorboard with smooth=0.6. Each curve is from one random seed.

Thanks!

[hartikainen]

@szrlee yeah, my figures were smoothed with viskit's default smoothing.