Remote code upload/WebREPL

[Cyphrz]

We really need the ability to upload code remotely, a webserver or maybe port WebREPL over from micropython.

My current idea is to host my code.py on a server, and have boot.py fetch the latest version on boot.

Otherwise any code changes requires a hookup to PC, and my project is out of reach without a ladder so any code changes are a pain.

[askpatrickw]

@Cyphrz Check out the discussion in this issue #3777

1. It has a PR for OTA Support of the entire CP image
2. It mentions an MicroPython FTP update library you might port and\or that you could update your local code via a CP script over HTTP.

If you're interested in other more IoT Cloud Infrastructure type approaches, ping me up on Discord.

BUT all this is doable.

[aivarannamaa]

IMO WebREPL (together with its file transfer protocol) is much more useful than FTP, because it gives you the REPL. In Thonny, the user experience is the same no matter whether you connected over serial or over WebREPL (at least in principle. ESP32 has some quirks but I hope these get resolved soon).

@tannewt, have you considered WebREPL for CircuitPython?

[Cyphrz]

I'm not a fan of using FTP for this, as it's more complicated especially if you want it to be secure. Since Circuit python is designed for beginners it seems like having a web interface is more friendly.

I agree WebREPL is the best first step as it's already part of Micro, so it in theory should be less effort?

[tannewt]

@tannewt, have you considered WebREPL for CircuitPython?

Not recently. I looked at it years ago. My impressions/feelings are:

1. I've never had a good experience provisioning over WiFi and subsequently finding a device. S2 support does simplify the first connect issue though because you could use USB.
2. When I looked last, MicroPython didn't have strong lifecycle management of WebREPL and soft reboots broke things.
3. CircuitPython has moved to a "Network in the VM" model which means network is shutdown after code finishes. It would be a lot of work to break this out of the VM so that you can interact over WiFi when the VM is done.

Ultimately, this work is very far down my personal priority list. I'd rather put Adafruit-funded dev effort into editing files over BLE so folks can edit from their phones and tablets. If someone wanted to take this on I'd be happy to give guidance but I suspect MicroPython-like WebREPL is a lot of work.

I think a simpler solution is to have the user code manage updating itself. #3454 adds the ability to reload to a non-code.py file but return to another file on error. This should be enough to dynamically get a new code.py running. There is a related PR to read the last exception as well.

I'm not a fan of using FTP for this, as it's more complicated especially if you want it to be secure. Since Circuit python is designed for beginners it seems like having a web interface is more friendly.

I suspect a RESTful API is probably the most familiar API for webdevs to use to read and write files.

I agree WebREPL is the best first step as it's already part of Micro, so it in theory should be less effort?

I suspect not. The outer layers of CircuitPython (what we call the supervisor) have diverged quite a bit from MicroPython because we share it across all ports. (We do this so they all work the same way.)

[aivarannamaa]

I see that updating the code can be achieved with simpler means than WebREPL. It is exactly the wireless interactive REPL I'm missing.

I'd rather put Adafruit-funded dev effort into editing files over BLE so folks can edit from their phones and tablets

#1010, right? Reading this I found #3707, which mentioned "repl over BLE", which is as interesting as WebREPL.

In general, I'm suprised that given the educational goals of CircuitPython, the interactivity (both in entering commands and observing debug output) is still considered optional. I understand that there may be problems upgrading from Notepad to a dedicated tool (and making sure you have right USB drivers and such), but if you need to pair your device anyway, then it would be really nice if you could have a dialogue with it.

Stopping now, because I noticed that according to the title this issue is mostly about code upload :)

[tannewt]

#1010, right? Reading this I found #3707, which mentioned "repl over BLE", which is as interesting as WebREPL.

Yup! #1010 is the related issue.

In general, I'm suprised that given the educational goals of CircuitPython, the interactivity (both in entering commands and observing debug output) is still considered optional.

It's optional for a number reasons:

1. Requiring an IDE increases the complexity and time to get started.
2. You can be interactive with an output on the board and a text editor without serial (such as blinky.) Being interactive with the board is more engaging than serial printing anyway.
3. You can copy example code to the device without an IDE. (Folks know how to use USB drives.)
4. The REPL isn't very useful because the code isn't saved. That's why we start with code.py.

I understand that there may be problems upgrading from Notepad to a dedicated tool (and making sure you have right USB drivers and such), but if you need to pair your device anyway, then it would be really nice if you could have a dialogue with it.

Right now, we don't need pairing because we only support USB boards now. With BLE-only boards we'll assume one is using a tablet or phone with it. In that ecosystem, app and permission settings are much more standardized. It still will be more complicated than the usb drive though.

[rafaelaroca]

Hi!!

Just to add another possible motivation / reason for WebREPL.

BIPES (https://bipes.net.br/) project relies on WebREPL, which is great for wireless interactive debugging, programming, uploading and downloading files and interacting with MicroPython shell. It would be great with CircuitPython also!

We have about 3000 users now, and many users reported flexibility and ease with BIPES + WebREPL, all using a web browser.

By the way, we already use BIPES + CircuitPython on ESP32-S2 using Web Serial API. Works great!

https://bipes.net.br/beta2serial/ui/

In that way, WebREPL would be amazing!

thanks

Rafael

[tannewt]

Thank you for the discussion! We've now add a BLE workflow and I've laid out what I see for a web workflow in #6174. Let's continue discussion there.