test_socket failures on Mac OS X

[ncoghlan]

BPO	12958
Nosy	@brettcannon, @ronaldoussoren, @ncoghlan, @ned-deily, @bitdancer, @voidspace

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2011-09-19.10:26:46.561>
created_at = <Date 2011-09-11.05:40:43.125>
labels = ['type-bug', 'library']
title = 'test_socket failures on Mac OS X'
updated_at = <Date 2014-02-01.10:21:12.638>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2014-02-01.10:21:12.638>
actor = 'ned.deily'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2011-09-19.10:26:46.561>
closer = 'python-dev'
components = ['Library (Lib)']
creation = <Date 2011-09-11.05:40:43.125>
creator = 'ncoghlan'
dependencies = []
files = []
hgrepos = ['67']
issue_num = 12958
keywords = []
message_count = 14.0
messages = ['143854', '143860', '143863', '144241', '144248', '144257', '144264', '159694', '187635', '187637', '187642', '187643', '187646', '209884']
nosy_count = 8.0
nosy_names = ['brett.cannon', 'ronaldoussoren', 'ncoghlan', 'baikie', 'ned.deily', 'r.david.murray', 'michael.foord', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue12958'
versions = ['Python 3.3', 'Python 3.4']

[ncoghlan]

Several of the new socket tests introduced in bpo-6560 are failing on the Mac OS X buildbots. These appear to be due to platform problems rather than anything in the test suite or the new sendmsg/recvmsg code, but it would be good if a developer on Mac OS X could take a closer look at them.

To avoid polluting the buildbot results, the affected tests will be flagged as expected failures when run on Mac OS X (this isn't in the repo yet, but once it is, the tests will be marked with "anticipate_failure" in the test_socket code).

[ncoghlan]

First attempt didn't quite work - the FD passing tests somehow seem to be reporting both 'ERROR' *and* 'expected failure', which is causing the test overall to remain red.

http://www.python.org/dev/buildbot/all/builders/AMD64%20Snow%20Leopard%202%20custom/builds/41/steps/test/logs/stdio

[ncoghlan]

Ah, I believe I see why the expected failure isn't working properly - the failing testFDPass* tests are causing the subsequent tear down code to also fail.

[ncoghlan]

OK, I've now looked into *why* the socket tests are throwing errors in tearDown, and it has to do with the way the threaded client/server tests in test_socket are set up.

Specifically, ThreadableTest uses tearDown to reraise any exception raised in the client thread, and these are therefore outside the scope of the "expectedFailure" suppression in unittest.

Now that I've tracked this down, it would be fairly straightforward to fix this specifically within test_socket.ThreadableTest by appropriately adjusting the definition of ThreadableTest.clientRun to discard exceptions encountered in tests flagged as expected failures.

However, I'm wondering if that's the right thing to do. Perhaps it would make more sense to change unittest itself so that "expectedFailure" also suppresses tearDown errors. It doesn't seem all that unusual for a known failing test to also cause problems for the tearDown code.

Added Michael to the nosy list to ask for his advice/opinion. In the meantime, I'll work on a patch that adjusts ThreadableTest directly.

[voidspace]

See bpo-10548. There is some resistance to expectedFailure masking errors in setUp/tearDown as these aren't the place where you would normally expect the expected failure...

[ncoghlan]

OK, I'll just deal with the problem directly in test_socket then.

It looks like my latest attempt (suppressing unittest._ExpectedFailure in test_socket.ThreadableTest.clientRun) did the trick, so I'll push the updated tests some time this evening:
http://www.python.org/dev/buildbot/all/builders/AMD64%20Snow%20Leopard%202%20custom/builds/44/steps/test/logs/stdio

[python-dev]

New changeset a4e4facad164 by Nick Coghlan in branch 'default':
Close bpo-12958 by flagging expected failures in test_socket on Mac OS X
http://hg.python.org/cpython/rev/a4e4facad164

[python-dev]

New changeset e64bec91ac91 by Richard Oudkerk in branch 'default':
Issue bpo-14669: Skip multiprocessing connection pickling test on MacOSX
http://hg.python.org/cpython/rev/e64bec91ac91

[ronaldoussoren]

Not sure what changed, but these tests are reported as failures when I run 'make test' and not as expected failures.

[bitdancer]

Ronald: in 3.4 only? If so I wonder if the subtests patch had an unintended side effect...I'm pretty sure it touched the expected failure machinery somehow.

[ronaldoussoren]

Yes, only on 3.4:

======================================================================
ERROR: testFDPassSeparate (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/ronald/Projects/python/rw/default/Lib/test/test_socket.py", line 261, in _tearDown
    raise exc
  File "/Users/ronald/Projects/python/rw/default/Lib/test/test_socket.py", line 273, in clientRun
    test_func()
  File "/Users/ronald/Projects/python/rw/default/Lib/test/test_socket.py", line 2644, in _testFDPassSeparate
    array.array("i", [fd1]))]),
  File "/Users/ronald/Projects/python/rw/default/Lib/test/test_socket.py", line 1790, in sendmsgToServer
    *(args + self.sendmsg_to_server_defaults[len(args):]))
OSError: [Errno 22] Invalid argument

(And simular for the other 3 marked socket tests)

[ncoghlan]

It's entirely possible the hack in test_socket (where it catches unittest._ExpectedFailure to suppress the issues in the tearDown code after the test fails) were invalidated by the subtest changes.

[ronaldoussoren]

FWIW I've filed a bug report about the behavior of sendmsg with Apple, the RADAR number is 13716133.

[ned-deily]

FYI, the OS X platform bugs causing the three "send timeout" test failures were fixed in OS X 10.7 causing those test cases to have previously silently skipped "unexpected successes". See bpo-20474 for more details.