Running Pytest2 locally is failing

[dabacon]

./continuious_integration/check.sh is failing on my and another developers box.  

____________________________ test_run_simulator_run ____________________________

    @mock.patch.multiple(cirq.SimulatesSamples, _run=mock.Mock())
    def test_run_simulator_run():
        simulator = cirq.SimulatesSamples()
        expected_measurements = {u'a': [[1]]}
        simulator._run.return_value = expected_measurements
        circuit = mock.Mock(cirq.Circuit)
        param_resolver = mock.Mock(cirq.ParamResolver)
        expected_result = cirq.TrialResult(repetitions=10,
                                           measurements=expected_measurements,
                                           params=param_resolver)
        assert expected_result == simulator.run(circuit=circuit,
                                                repetitions=10,
                                                param_resolver=param_resolver)
        simulator._run.assert_called_once_with(circuit=circuit,
                                               repetitions=10,
>                                              param_resolver=param_resolver)

cirq/sim/simulator_test.py:40: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

_mock_self = <Mock id='140181326865360'>, args = ()
kwargs = {'circuit': <Mock spec='Circuit' id='140181723293584'>, 'param_resolver': <Mock spec='ParamResolver' id='140181723291728'>, 'repetitions': 10}
self = <Mock id='140181326865360'>
msg = "Expected 'mock' to be called once. Called 2 times."

    def assert_called_once_with(_mock_self, *args, **kwargs):
        """assert that the mock was called exactly once and with the specified
        arguments."""
        self = _mock_self
        if not self.call_count == 1:
            msg = ("Expected '%s' to be called once. Called %s times." %
                   (self._mock_name or 'mock', self.call_count))
>           raise AssertionError(msg)
E           AssertionError: Expected 'mock' to be called once. Called 2 times.

.test_virtualenv_py2/local/lib/python2.7/site-packages/mock/mock.py:947: AssertionError

[Strilanc]

What version of python 2? python2 --version

[Strilanc]

I've managed to reproduce this. It's strange. It doesn't happen when you run ./check/pytest2, but it does for ./continuous_integration/check.sh. Also, it seems to have something to do with the tests running twice instead of once. I'm pretty sure the qubit sets not agreeing is due to the measurement being appended during the first run, then still being present during the second.

So, it appears that we're running into something analogous to the "do not use mutable values as default arguments", but in pytest itself?

For example, this:

@pytest.mark.parametrize(
    'x',
    [
        []
    ]
)
def test_random_same_matrix(x):
    x.append(1)
    assert x == [1]

Fails with this error:

E       assert [1, 1, 1] == [1]

I'll try to narrow down the exact set of things needed to make this happen. Current guess is "cache invalidation bug in pytest".

[Strilanc]

Figured it out. It's because of pytest-dev/pytest#4322 and also because the python 2 test checking code was using '' (which pytest interprets as 'run all tests') as a "no parameter" command line parameter instead of None.