I2C Bus error leaves board unrecoverable without power down

[Panometric]

In my example, in a loop reading the device as fast as possible, and doing other unrelated things..

while 1:
     acceleration = cpx.acceleration
    .....

You can get an error:

Traceback (most recent call last):
  File "code.py", line 63, in <module>
  File "adafruit_circuitplayground/circuit_playground_base.py", line 261, in acceleration
  File "adafruit_lis3dh.py", line 159, in acceleration
  File "adafruit_lis3dh.py", line 328, in _read_register
  File "adafruit_lis3dh.py", line 327, in _read_register
  File "adafruit_bus_device/i2c_device.py", line 82, in readinto
OSError: [Errno 5] Input/output error

Press any key to enter the REPL. Use CTRL-D to reload.soft reboot

Once this occurs, the I2C device bus is stuck because the device is holding the bus. The board never recovers unless you power it down. Every restart just issues a RuntimeError. This could happen on any I2C device.

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
Traceback (most recent call last):
  File "code.py", line 3, in <module>
  File "adafruit_circuitplayground/__init__.py", line 29, in <module>
  File "adafruit_circuitplayground/express.py", line 75, in <module>
  File "adafruit_circuitplayground/express.py", line 72, in __init__
  File "adafruit_circuitplayground/circuit_playground_base.py", line 110, in __init__
RuntimeError: SDA or SCL needs a pull up

The industry standard response to this is to bit-bang single clocks onto SCLK until the bus is released by the slave. This should be done at start, or even on error to recover the bus silently. It usually requires disconnecting the peripheral temporarily, using the SCLK os GPIO and then reconnecting.

[ladyada]

yeah this is a thing that happened with CODAL & the LIS3DH. very odd that this chip is particularly wierded out sometimes!
@dhalbert see email thread "I2C lockup" with peli and folks

[dhalbert]

Deferring this to after 5.0.0. It's a very good idea, but we want to vet it against a bunch of I2C devices first, since there may be a few that are pathological.

CODAL fix:
https://github.com/lancaster-university/codal-samd/blob/cplay_master_i2c_hack/src/ZI2C.cpp#L10

Another discussion:
https://www.raspberrypi.org/forums/viewtopic.php?t=241491

[creston-bob]

In a project using the Microchip PIC32MZ microcontroller I discovered that the first two versions (v1 & v2) have a hardware bug in the silicon that causes the I2C to hang occasionally. I dug into the matter and found the errata sheets indicated that v1 & v2 had this flaw but it was apparently fixed in v3. After trying out all Microchip's suggestions for fixing the hung I2C from within my program code (and failing), I eventually concluded that the ONLY possible fix was a power reset (which works reliably). I have also discovered that without doing anything, eventually the I2C will recover by itself but it can take many hours before that happens. As a solution, I have an Arduino board listening to the data stream coming from the PIC to detect the problem and rectify it. So how do I detect the I2C malfunction? Simple ... I've programmed a PIC32 onboard timer interrupt that checks the SCL and SDA pins after every transaction ... if the pins are both high (idle mode) then the I2C transaction was successful but if either pin is low then we have a malfunction and the integer 1 is appended to the data being sent to the server. The integer is sliced off the data after being checked to see it it's a 1 or 0. This scheme works extremely well but has made me wary of I2C malfunctions from any silicon. I suspect that's what is happening in this particular circumstance and this is how you fix it.

[creston-bob]

I forgot to mention that the Arduino board I mentioned in my previous comment energizes a relay to power reset the PIC32MZ board. The power is connected through the relay's NC contact which is opened for a few seconds to cause the power reset.

[kevinjwalters]

I just got this by pressing reset button on a CLUE (alpha) then trying to load clue object:

Adafruit CircuitPython 5.0.0-rc.0 on 2020-02-26; Adafruit CLUE nRF52840 Express                                                    with nRF52840
>>>
>>>
>>>
>>> import board
>>> from adafruit_clue import clue
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "adafruit_clue.py", line 886, in <module>
  File "adafruit_clue.py", line 172, in __init__
RuntimeError: SDA or SCL needs a pull up
>>>
>>>
>>>
soft reboot

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
Traceback (most recent call last):
  File "code.py", line 46, in <module>
  File "adafruit_clue.py", line 886, in <module>
  File "adafruit_clue.py", line 172, in __init__
RuntimeError: SDA or SCL needs a pull up



Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 5.0.0-rc.0 on 2020-02-26; Adafruit CLUE nRF52840 Express with nRF52840
>>>

[kevinjwalters]

Another example of a CLUE in forums being discussed with @caternuson: Adafruit Forums: clue_display_sensor_data.py not working. Reading from the accelerometer (LSM6DS33) could be implicated. Same RunTimeError after the problem:

    Adafruit CircuitPython 5.0.0 on 2020-03-02; Adafruit CLUE nRF52840 Express with nRF52840
    >>>import board
    >>> i2c = board.I2C()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    RuntimeError: SDA or SCL needs a pull up