Playing around with the current pytest sources after merging #1621, I figured out that my example `py.test --cli-option ../../test.log` does still not work if you do `touch ../../test.log` before executing pytest. Sorry for finding this out only after the merge :(