MemoryError from xmlgen for a big number of tests

[ghost]

Hi

I've not executed this test for a long time, maybe 6 month, but it was running and not failing. The PC was updated multiple times with Win10 updates.
I've updated all pytest packages to make this problem disappear, but still it is failing somewhere I can't find

This is my pytest packages:

This is pytest version 3.4.2, imported from c:\python27\lib\site-packages\pytest.pyc
setuptools registered plugins:
  pytest-xdist-1.22.2 at c:\python27\lib\site-packages\xdist\looponfail.py
  pytest-xdist-1.22.2 at c:\python27\lib\site-packages\xdist\plugin.py
  pytest-progress-1.2.1 at c:\python27\lib\site-packages\pytest_progress.py
  pytest-metadata-1.6.0 at c:\python27\lib\site-packages\pytest_metadata\plugin.py
  pytest-html-1.16.1 at c:\python27\lib\site-packages\pytest_html\plugin.py
  pytest-forked-0.2 at c:\python27\lib\site-packages\pytest_forked\__init__.pyc
  pytest-base-url-1.4.1 at c:\python27\lib\site-packages\pytest_base_url\plugin.py

The number of tests are 13392 and it is always fails in between 7000 - 9000, running multiple instances

Error from the pytest:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "c:\python27\lib\site-packages\_pytest\main.py", line 100, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "c:\python27\lib\site-packages\_pytest\main.py", line 138, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "c:\python27\lib\site-packages\pluggy\__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "c:\python27\lib\site-packages\pluggy\__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "c:\python27\lib\site-packages\pluggy\__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "c:\python27\lib\site-packages\pluggy\callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "c:\python27\lib\site-packages\pluggy\callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "c:\python27\lib\site-packages\pluggy\callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "c:\python27\lib\site-packages\xdist\dsession.py", line 116, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "c:\python27\lib\site-packages\xdist\dsession.py", line 139, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "c:\python27\lib\site-packages\xdist\dsession.py", line 250, in worker_testreport
INTERNALERROR>     self.config.hook.pytest_runtest_logreport(report=rep)
INTERNALERROR>   File "c:\python27\lib\site-packages\pluggy\__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "c:\python27\lib\site-packages\pluggy\__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "c:\python27\lib\site-packages\pluggy\__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "c:\python27\lib\site-packages\pluggy\callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "c:\python27\lib\site-packages\pluggy\callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "c:\python27\lib\site-packages\pluggy\callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "c:\python27\lib\site-packages\pytest_progress.py", line 158, in pytest_runtest_logreport
INTERNALERROR>     TerminalReporter.pytest_runtest_logreport(self, report)
INTERNALERROR>   File "c:\python27\lib\site-packages\_pytest\terminal.py", line 284, in pytest_runtest_logreport
INTERNALERROR>     self._tw.write(letter)
INTERNALERROR>   File "c:\python27\lib\site-packages\py\_io\terminalwriter.py", line 224, in write
INTERNALERROR>     write_out(self._file, markupmsg)
INTERNALERROR>   File "c:\python27\lib\site-packages\py\_io\terminalwriter.py", line 369, in write_out
INTERNALERROR>     fil.write(msg)
INTERNALERROR>   File "c:\python27\lib\site-packages\colorama\ansitowin32.py", line 40, in write
INTERNALERROR>     self.__convertor.write(text)
INTERNALERROR>   File "c:\python27\lib\site-packages\colorama\ansitowin32.py", line 141, in write
INTERNALERROR>     self.write_and_convert(text)
INTERNALERROR>   File "c:\python27\lib\site-packages\colorama\ansitowin32.py", line 169, in write_and_convert
INTERNALERROR>     self.write_plain_text(text, cursor, len(text))
INTERNALERROR>   File "c:\python27\lib\site-packages\colorama\ansitowin32.py", line 174, in write_plain_text
INTERNALERROR>     self.wrapped.write(text[start:end])
INTERNALERROR> IOError: [Errno 0] Error
Traceback (most recent call last):
  File "c:\python27\lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "c:\Python27\Scripts\pytest.exe\__main__.py", line 9, in <module>
  File "c:\python27\lib\site-packages\_pytest\config.py", line 59, in main
    return config.hook.pytest_cmdline_main(config=config)
  File "c:\python27\lib\site-packages\pluggy\__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "c:\python27\lib\site-packages\pluggy\__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\python27\lib\site-packages\pluggy\__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "c:\python27\lib\site-packages\pluggy\callers.py", line 201, in _multicall
    return outcome.get_result()
  File "c:\python27\lib\site-packages\pluggy\callers.py", line 77, in get_result
    _reraise(*ex)  # noqa
  File "c:\python27\lib\site-packages\pluggy\callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "c:\python27\lib\site-packages\_pytest\main.py", line 131, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "c:\python27\lib\site-packages\_pytest\main.py", line 125, in wrap_session
    exitstatus=session.exitstatus)
  File "c:\python27\lib\site-packages\pluggy\__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "c:\python27\lib\site-packages\pluggy\__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\python27\lib\site-packages\pluggy\__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "c:\python27\lib\site-packages\pluggy\callers.py", line 196, in _multicall
    gen.send(outcome)
  File "c:\python27\lib\site-packages\_pytest\terminal.py", line 469, in pytest_sessionfinish
    outcome.get_result()
  File "c:\python27\lib\site-packages\pluggy\callers.py", line 77, in get_result
    _reraise(*ex)  # noqa
  File "c:\python27\lib\site-packages\pluggy\callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "c:\python27\lib\site-packages\_pytest\junitxml.py", line 459, in pytest_sessionfinish
    time="%.3f" % suite_time_delta, ).unicode(indent=0))
  File "c:\python27\lib\site-packages\py\_xmlgen.py", line 54, in unicode
    return u("").join(l)
MemoryError

[pytestbot]

GitMate.io thinks possibly related issues are #641 (MemoryError when writing junitxml for large test suite), #643 (Show number of assertions in test results), #667 (Make tests using random numbers more reproducible), #2936 (-k cannot handle strings containing spaces that start with a number ), and #271 (IOError when writing --junitxml report when run via xdist, sometimes).

[RonnyPfannschmidt]

is this a 32 bit python or a 64 bit python?

[ghost]

I am using 32bit python on Win10
platform win32 -- Python 2.7.14, pytest-3.4.2, py-1.5.2, pluggy-0.6.0
plugins: xdist-1.22.2, progress-1.2.1, metadata-1.6.0, html-1.16.1, forked-0.2, base-url-1.4.1

[ghost]

It might be I found what is a problem the xml unit (junitxml) was flooded with a huge amount of data 8000*1000 at least line of text +20%. I guess if the xml can hold such big amount of text. Disabling xmlunit (junitxml) or reducing the output sorts the problem

[RonnyPfannschmidt]

@andreyv1978 i see, a 32 bit process on windows can at most use 2gb of memory, 3gb in special setups,
so i presume that easyly hit the limit as the junitxml plugin currently stores the sorted report data in text fragments,
while pytests also keeps full reports, i beleive we should elevate this in 2 ways

a) have junitxml write out the reports as they complete
b) have items and reports drop more details/logging data in some way thats accessible/reloadable

a should be straightforward, b is tricky

for your use-case a) might already be sufficient since you seem to do ok without junitxml

[ghost]

yes, for me I can minimize output that writes to junitxml like 1 line per test, which will not exceed about 200000 + 25% lines in total

[nicoddemus]

Created #3328 and #3329 to track the ideas here, so I'm closing this for now given @andreyv1978 has found a workaround.