Truncated MemoryError exception message when importing too-large .py

[dhalbert]

This is very odd:

1. Get the .py version of the CPX library: https://github.com/adafruit/Adafruit_CircuitPython_CircuitPlayground/releases/download/1.2.0/adafruit-circuitpython-circuitplayground-py-1.2.0.zip
2. Unzip it and put the adafruit_circuitplayground directory at the top level of CIRCUITPY
3. In the repl, try to import it. It's too big to compile, as expected. But note that the Memory Error: is incomplete.

Adafruit CircuitPython 2.2.1 on 2018-01-23; Adafruit CircuitPlayground Express with samd21g18
>>> from adafruit_circuitplayground.express import cpx
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: 
>>> 

4. Now move the adafruit_circuitplayground directory into the lib directory, and restart the board.
5. Import it again in the REPL:

Adafruit CircuitPython 2.2.1 on 2018-01-23; Adafruit CircuitPlayground Express with samd21g18
>>> from adafruit_circuitplayground.express import cpx
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: memory allocation failed, allocating 338 bytes

Note the differences in the exception message.

I've tried to make a simpler test case and haven't succeeded yet. T module's position on sys.path makes a difference. But it should not.

[dhalbert]

The truncated Memory Error: message occurs as far back as 2.0.0, but it's OK in 1.0.0.

[dhalbert]

The placement in sys.path is not always the issue, and may be a red herring. Here are more oddities, testing with various versions.
In this case the library is in lib/. If I make a typo on the first import, then the correct import does not do the printf expansion:

Adafruit CircuitPython 3.0.0-alpha.1-109-g7c6adaa66-dirty on 2018-02-01; Adafruit CircuitPlayground Express with samd21g18
>>> 
>>> from adafruit_circuitplayground import cpx
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: no module named 'adafruit_circuitplayground.cpx'
>>> from adafruit_circuitplayground.express import cpx
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: memory allocation failed, allocating %u bytes

Adafruit CircuitPython 3.0.0-alpha.1-109-g7c6adaa66-dirty on 2018-02-01; Adafruit CircuitPlayground Express with samd21g18
>>> from adafruit_circuitplayground.express import cpx
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: memory allocation failed, allocating 338 bytes

I suspect some free-before-use problem or smashed register or similar.

[tannewt]

I don't think we need to fix this in 2.x. I'd only fix crashes in 2.x at this point. Lets shoot to have this done as polish on 3.0. Please change the milestone if you agree.

[dhalbert]

I'll debug it in 3.0. If it's a serious memory problem I'll backport.

[tannewt]

Sounds good!

…

On Thu, Feb 1, 2018 at 10:28 AM Dan Halbert ***@***.***> wrote: I'll debug it in 3.0. If it's a serious memory problem I'll backport. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#569 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADNqfF3US6gv5tOaZCngIt8F9IJHl4xks5tQgJigaJpZM4R0_i9> .

[jallwine]

I'm seeing similar weirdness developing my own python library. Subtle, seemingly innocuous changes trigger a MemoryError exception. I don't have a solid test case, but in an attempt to track things down I've put a print(gc.mem_free() / 1024) my main code.py file and started commenting out swaths of code in the library that it's importing. It'll print >10K of memory left in one run, but uncommenting a msg = "" all of sudden triggers a MemoryError exception. It's repeatable for a few runs, but then works fine. Seems like something insidious is going on...

[deshipu]

That's effects of memory fragmentation, most likely — you don't only need to have that much memory free, you have to have it all in one chunk to allocate it.

[jallwine]

That makes sense, is there a recommended process for limiting fragmentation? Reseting the board more often?

[deshipu]

Reset works, of course. There also has been some awesome work in #547 to improve the situation.

[jallwine]

Awesome! Seems that this is worth a mention in a troubleshooting page somewhere. This isn't something that's regularly encountered in your every day python development.

[dhalbert]

There's no memory state saved between resets, so it starts from scratch after each reset. #547 will help a lot. For now, two things to do are perhaps to load larger modules first, and compile (mpy-cross) as many modules as you can, so you don't have to compile to bytecodes on chip.

[jallwine]

Thanks @dhalbert. Could you clarify what a reset is? Pressing the reset button? Power cycling the board? The soft reboot that happens after saving your code.py or library file?

[tannewt]

@jallwine @dhalbert means the soft reload that happens on save.

As @deshipu points out, we have improvements in master (future 3.0.0) that could help a bunch.

[deshipu]

I just hit the "MemoryError: memory allocation failed, allocating %u bytes" error on MicroPython, so I think this is actually an upstream bug.

[deshipu]

So I looked at the code that actually creates the exceptions, and it seems that this is not a bug, this is a feature: https://github.com/adafruit/circuitpython/blob/master/py/objexcept.c#L417-L435

Basically, if there is not enough memory to format the string, it will stay un-formated, and if there is no room to even allocate the string, the exception will have no string at all.

[dhalbert]

@deshipu Thanks for doing the research on this. I thought it might be lack of memory but didn't think it would be intentional. I wonder if we should have a fixed string to return that indicates lack of memory. @tannewt

[tannewt]

The behavior now is fine with me. It is useful knowing the size of the allocation that fails when its possible to. They can all be caught through MemoryError so they should be essentially equivalent.

Anyone have a different suggestion?

[tannewt]

I'm ok with how this works now and no one objected. :-)