"could not complete raw paste" on SeeedStudio XIAO-ESP32C3

[aivarannamaa]

CircuitPython version

Adafruit CircuitPython 9.0.0-alpha.5 on 2023-11-15; Seeed Studio XIAO ESP32C3 with ESP32-C3FN4

also 8.2.8 and 8.1.0

Code/REPL

> mpremote cp data.json :

Behavior

Similar issue was discussed in the context of Thonny (#7158), but the problem is reproducible also with mpremote (version 1.21.0).

Tried CP 8.2.8 and 9.0.0a5 on macOS Sonoma, Windows 11 and Ubuntu 23.10, also tried CP 8.1.0 on Ubuntu. All these tests failed in similar way -- mpremote would start writing the file but after a while wouldn't get raw paste confirmation from the device (it times out while trying to read a single byte).

I tried to upload this file: https://raw.githubusercontent.com/thonny/thonny/397679cd8f7eb11b20ba9075c694337f5aace54b/data/circuitpython-variants-esptool.json

When I cancelled the wait I got tracebacks like this (the first line shows the mpremote command line I used):

% mpremote cp data.json :
cp data.json :
 ... copying   6% [#-------------------]^CTraceback (most recent call last):
  File "/Users/aa/python_stuff/thonny/venv312/bin/mpremote", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/main.py", line 528, in main
    handler_func(state, args)
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/commands.py", line 159, in do_filesystem
    state.transport.filesystem_command(
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/transport_serial.py", line 476, in filesystem_command
    op(src2, dest2, progress_callback=progress_callback)
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/transport_serial.py", line 420, in fs_put
    self.exec("w(" + repr(data) + ")")
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/transport_serial.py", line 285, in exec
    ret, ret_err = self.exec_raw(command, data_consumer=data_consumer)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/transport_serial.py", line 271, in exec_raw
    self.exec_raw_no_follow(command)
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/transport_serial.py", line 249, in exec_raw_no_follow
    return self.raw_paste_write(command_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/transport_serial.py", line 225, in raw_paste_write
    data = self.read_until(1, b"\x04")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/transport_serial.py", line 118, in read_until
    data = self.serial.read(min_num_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/serial/serialposix.py", line 565, in read
    ready, _, _ = select.select([self.fd, self.pipe_abort_read_r], [], [], timeout.time_left())
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

Description

No response

Additional information

• mpremote can transfer the file without problems if I'm using MicroPython 1.21 (https://micropython.org/download/ESP32_GENERIC_C3) on the same device.
• Similar problem has been reported by Thonny users also for Wemos Lolin C3 Mini with ESP32-C3FH4 (CircuitPython upload file fail thonny/thonny#2930)

[aivarannamaa]

The same device (SeeedStudio XIAO-ESP32C3) is having problems also when trying to write files via the plain raw-REPL mode or via plain paste mode. In paste mode the echo starts going slightly off after a while, in raw mode the OK confirmation does not appear.

Nevertheless, I don't think the problem is specific to writing files. I changed the file writing procedure to a no-op, but made sure that the amount of serial communication stayed the same. The problem still persisted.

[bill88t]

Oh, the ESP32-Cx serial implementation is at fault here, no doubt about that.
The same flush desync/incomplete bug from what I can tell.

[aivarannamaa]

The problem persists in CP 9.0b0.

I wonder why is this issue in the "Long term" milestone? Is it too hard to be handled sooner or is it considered low priority?

[tannewt]

I wonder why is this issue in the "Long term" milestone? Is it too hard to be handled sooner or is it considered low priority?

Thanks for retesting! It's a low priority because we only have one C3 board from Adafruit. It is hard to know difficulty until after it has been debugged and fixed.

[berkut0]

I have a cheap board from the Chinese market and couple my own with mini modules.

Can confirm that 8.2.10 is installable and usable, but any 9+ versions have serious problems to work on ESP32-C3 (or just with Thonny, not completly sure)

I can describe the problem this way: by selecting different boards you can get about three states. Or it works and you can write to the console, but it hangs and reboots. Or it does not connect through Thonny. Or it is not recognised by the system at all as a normal device.

update: it is almost impossible to upload files on board with any version of CP:

adafruit-circuitpython-bundle-8.x-mpy-20231207\lib\adafruit_display_text_init_.mpy (4850 bytes)
PROBLEM IN THONNY'S BACK-END: Exception while handling 'upload' (thonny.plugins.micropython.mp_back.ProtocolError: Could not complete raw paste).
See Thonny's backend.log for more info.
You may need to press "Stop/Restart" or hard-reset your CircuitPython device and try again.

[RetiredWizard]

This is not a solution by any means but if you're keen on getting something working.... I have been able to use these board by uploading files using a serial transfer program similar to the way mpremote works. It's a little unfinished as I got it working "well enough" and left it at that. The UI is a bit weird but once you get used to it, works ok.

https://github.com/RetiredWizard/Serial-File-Manager

[SeanTheITGuy]

I have used ampy and the web file.manager without issue on c3 devices. With 8.2 through 9.1-beta.

[RetiredWizard]

Nice, I knew about ampy (https://learn.adafruit.com/micropython-basics-load-files-and-run-code/install-ampy) once but completely forgot about it.

[UnexpectedMaker]

I've also been experiencing this on my TinyC6 and had to use a modified version of the Serial-File-Manager by @RetiredWizard to be able to communicate with my C6 boards running CP when under test on my test jigs.

It would be great if we could work out what this is as it seems to be specific to the Espressif serial UART setup in their C3/C6 chips, so will likely effect the H2, C61 and C5.

[berkut0]

There is a non-zero chance that @aivarannamaa can fix something in Thonny and it will work. But it would be good to understand exactly what the problem is. There seem to be solutions that work. (But I suspect that the problem lies with Ampy itself)

I like the web file manager solution, thanks a lot for the tip, I'll give it a try

[RetiredWizard]

Since the C3 was on my mind, I pulled one out for a project I was working on yesterday. I did use ampy to get the initial settings.toml file loaded, but once it was connected to WiFi, I used the Web Workflow for everything. The File Browser to transfer files, the Internet based full code editor and associated internet based serial interface (not the CP hosted serital console) and everything went very smoothly. I was even able to use the Web Workflow feature of Circup to check and update the libraries on the device. It was an enjoyable workflow to use although I don't believe, other than Circup for the libraries the Web Workflow can be used to transfer files via a script (yet?).

Using Web Workflow is definitely another option to consider until the current serial UART isses are resolved.

[berkut0]

Can confirm that Web Workflow can be a sort of workaround.
BUT! I am having trouble getting into the editor. It just writes to me:

Uh oh! It looks like you may be offline. You can go to the file browser if you'd like to work with files without connecting to the internet.

I'm not sure how to use CircUp in the case of Web Workflow, it only tries to detect serial connections. Would be great if you could share how you achieve this.

[UnexpectedMaker]

I've been doing some thorough testing of Ampy with my TinyC6 on my test jig setup - and I can confirm the issue is still present, but it's it succeeds more often than it fails.

90% of the time it works and I can PUT files on the TinyC6.
70% of the time I can RUN a file on the device

The rest of the times a RUN hangs and a PUT fails, or the device locks up serial comms after a task completes.

This is way better than what I have been experiencing with mpremote and shell which both fail > 80% of the time, but I'd love to work out why this is happening, so all tools can work 100% of the time.

I'm unable to use a web-worlflow for my use case - it needs to be USB based serial comms.

[berkut0]

As there is some variety of very valuable in terms of makers chips, perhaps it is worth considering creating a new thread topic? Or maybe change the name to this one?

[UnexpectedMaker]

As there is some variety of very valuable in terms of makers chips, perhaps it is worth considering creating a new thread topic? Or maybe change the name to this one?

Well, the C3 and C6 both use the same UART peripheral, so whatever works or doesn't work for one, will be the same for the other, and all new chips from Espressif that use the same peripheral, like the C5, C61 and I believe the H2 + H4 will also have the same issues, so I think it's relevant token this here and not part multiple threads.

[RetiredWizard]

I'm not sure how to use CircUp in the case of Web Workflow, it only tries to detect serial connections. Would be great if you could share how you achieve this.

circup --host xx.xx.xx.xx --pasword xxxx update

BUT! I am having trouble getting into the editor. It just writes to me:

It may not be applicable to you, but I found the C3 board I was using had less WiFi range than most boards I use and the Web Editor would sometimes give me that error when I wasn't close enough to the router.

EDIT: I just re-read your message and the error you indicated actually suggests that your host machine can't reach the internet to get the full code editor and not a C3 WiFi range issue. In early testing I remember getting that message but I don't remember why it was happening. I haven't seen it recently.

[berkut0]

@RetiredWizard The problem lies in my browser settings. They're too restrictive. I've created a special profile to work with C3, although it's a rather controversial solution.

[dhalbert]

Research this in esp-idf issues. See if increasing our buffer size would help.

[RetiredWizard]

The larger buffer may help, but it looks to me like there's an XOFF/XON functionality missing. Once the buffer is full, the data seems to continue to flow and just get lost.

I was trying to paste code into the REPL (after pressing CTRL-E) on an ESP32 Sunton board today and had to break the copy/pastes into smaller blocks because attempting to copy the whole file, about 24 lines of code, didn't work. The first few lines seem to paste fine but then some of the characters were dropped from the remaining text. The display board may exaggerate the issue because the speed of the REPL appears to be reduced dramatically when it's being displayed on the attached display.

[dhalbert]

USB-serial converters don't need to literally send XON/XOFF in-band, usually, because they are able just to stall the USB transfer to do flow control. However, that is apparently not enabled or something on these C3 builds. I haven't looked at the code yet.

[dhalbert]

Does anyone have a simpler reproduction case that involves just pasting into the REPL or similar, and not using a file manager? I tried pasting long lines, and they seem fine with 9.0.3 on the QT Py ESP32-C3.

[RetiredWizard]

Did you mean ESP32-S3? I think this issue shows up on chips that don't support the Circuitpy drive, i.e. ESP32, ESP32-C3, ESP32-C6.

[dhalbert]

Did you mean ESP32-S3? I think this issue shows up on chips that don't support the Circuitpy drive, i.e. ESP32, ESP32-C3, ESP32-C6.

No, sorry, that was a typo, and is fixed to C3.

[bill88t]

For me, plain ESP32 has been rock solid. I have only had issues on ESP-Cx.

[RetiredWizard]

I've tried with both a QTPY Esp32-C3 and a SeeedXIAO-ESP32C3 to paste directly into the REPL after pressing CTRL-E, and the smaller pastes ~1.5k surprised me by working fine. When I tried pasting a larger python script (about 12.5k) the paste failed and left the board in a weird state. It didn't recognize Control keys and each key press seemed to process a bit of the paste buffer.

I also tried an ItsyBitsy ESP32 and the paste into the REPL (CTRL-E) failed on the smaller 1.5K file but didn't leave the board in the strange state so perhaps there is a different issue on the ESP32.

[dhalbert]

The USB-serial-jtag peripheral on the C3/C6 seems to have almost no configurable options. The doc seems to imply it will do USB stalling until its FIFO is read: https://docs.espressif.com/projects/esp-idf/en/stable/esp32c3/api-guides/usb-serial-jtag-console.html#data-buffering. But that is a rather high-level description of what is going on.

[RetiredWizard]

Does the USB-to-Serial bridge chip mentioned here refer to a chip on these development boards?

This is in contrast to using a USB-to-Serial (UART) bridge chip, which always ingests the bytes and sends them to a (possibly not listening) ESP32-C3.

If so, is this saying there is no flow control between these chips and the C3?

[UnexpectedMaker]

Does the USB-to-Serial bridge chip mentioned here refer to a chip on these development boards?

This is in contrast to using a USB-to-Serial (UART) bridge chip, which always ingests the bytes and sends them to a (possibly not listening) ESP32-C3.

If so, is this saying there is no flow control between these chips and the C3?

There are no external chips. The Serial JTAG bridge is inside the C3/C6. It's a native peripheral.

[dhalbert]

Chapter 30 in https://www.espressif.com/sites/default/files/documentation/esp32-c3_technical_reference_manual_en.pdf describes the peripheral. It is fairly autonomous in what it does.

[nbercher]

FYI, working from a Linux PC and connected with screen to the board, I copy files this way:

• text files: I rely on using home-made Python code based on sys.stdin.read(1) and copy-paste to the REPL reading so.

  • copy: cat file.py | xclip -i
  • paste: mouse button-2 click in my screen window

• binary files: I rely on https://github.com/RetiredWizard/Serial-File-Manager from @RetiredWizard (thank you so much!)

For text files, I have noticed inconsistent behaviour of some other similar solutions I have tried: they worked very well at times and then stopped to work, it seems the serial connection is not stable in my setup, but I couldn't manage to find why so far.

[dhalbert]

Could folks retry with 9.1.0? We fixed some things that improved C6 serial, and C3 serial might also be better.

[RetiredWizard]

I tried both the ~1K and ~12K pastes into a Ctrl-E repl and both worked perfectly on the QTPY ESP32-C3. I then used Thonny to upload about 180 files (~800K) without issue. I also used mpremote to copy a 50K file without issue.

Using 9.1.0, the ItsyBitsy ESP32 still won't paste even the smaller 1K file properly but perhaps that's a separate issue.

[bablokb]

the ItsyBitsy ESP32 still won't paste even the smaller 1K file properly

@RetiredWizard: do you have any further information? I also have problems with the paste-mode on ESP32 (not ESP32-xx) chips. When CTRL-E pasting even only a few lines of code, some lines in between are just swallowed and I usually end up with invalid code in the REPL. Even with "normal paste" I have to be very careful and anything larger than 64 characters fails (it just seems to hang). And btw, I also have other problems with the REPL on ESP32, see #9362. Might also be related to the serial connection.

[nbercher]

I will give a try soon to 9.1.0.

For those interested and using Linux host PC you could try this to read pasted contents and write files on the REPL side (and pasting as mentioned here)

import sys

def mkfile(output_filepath, line_sep='\n'):
    """Write a text file to local filesystem grabbing contents from
    stdin.

    POSIX version suitable for Unix/Linux.

    """
    with open(output_filepath, 'wt') as f:
        f.write(_posix_input())

def _posix_input():
    """"Grab text from stdin.

    POSIX version suitable for Unix/Linux.

    """
    s = ''
    try:
        while True:
            in_char_ = sys.stdin.read(1)
            s += in_char_
            # Provide real-time echo to the user as input() would do:
            print(in_char_, end="")
    except KeyboardInterrupt as err: # C-d is not received as EOF so that we use C-c instead.
        pass
    return s

Then you run mkfile(filepath) on the REPL and you are ready to paste.
Terminate your file with Ctrl-C.

[RetiredWizard]

@bablokb I'm getting more convinced that the esp32 paste issue is unrelated to this one and we should use #9362 or open a new issue. It looks to me like this C3 issue has been resolved and if confirmed by others should be closed 😁

[berkut0]

@RetiredWizard
Yes, I can confirm that the problem with working with Thonny has disappeared. I was using stable version 9.1.0 (and the C3 board from the comments above).

But so far the research is very superficial.

The overheating of the chip (which seemed to indicate a deeper problem) has also disappeared.

upd: I've uploaded a lot of files without any problems. Looks like the problem has been fixed.

[dhalbert]

Based on @berkut0's and others' testing I'm going to close this. @aivarannamaa if you still see this we can reopen it.