Allow custom bootloaders for ESP-IDF projects

[marchingband]

I am looking at https://github.com/adafruit/tinyuf2/tree/master/ports/espressif
I want to use it to add a uf2 bootloader to my project.
Looking at the Adafruit Matrix S3, i can see how they do it with Arduino, but my project uses esp-idf.
Is there any way I can accomplish this?
If not, please consider this a feature request.
thanks.

[robertlipe]

That's really not a problem for platformIO to solve, IMO. (Even if it were, as long as Espressif and PlatformIO aren't playing nice, this won't go much of anywhere.) Your request might get further as a request in https://github.com/espressif/idf-extra-components, but it's easy to see them leaving this to third parties.

All the code you need is right in that directory. "All" it does is load a normal program at a slightly funny memory address (the end of flash instead of the beginning), initialize the USB stack to pretend like it's a disk drive, then watch for a magic file to show up and when it does, scribble it into flash and reboot.

Of course, the support nightmare is that if someone does a flash erase or the loaded program is too long and clobbers the bootloader (surely it checks for that) or the program thinks it knows too much about flash layout and tries to use those blocks for spiffs or something, the bootloader gets clobbered. This isn't tragic as the bootloader is just another application; it's just one that has to be loaded via the equivalent of esptool instead of a copy to a disk. This means that everything talking to these boards actually needs to know both how to load the utf program (copy a file) and the old way of loading. I've seen the Adafruit groups on that board (I have one) and it's a bit of a support nightmare. I think Rpi2040 does a similar loader. In the ESP32-S3 (and related) you now have the "real" boot code in ROM and this becomes kind of a second-stage bootloader.

They've also had runins with OSes (ahem, MacOS) at least twice rolling out updates that "broke" their boards because the filesystem is a DOS type (FAT12) that's so ancient that devs forget to test 2MB "memory sticks" in a world of 2TB flash drives.

You don't even really need to integrate their source if you don't NEED to change the bootloader. You an just take a binary from that project or CircuitPython https://esp32s3.com/install_uf2.html or whatever (there are a few such projects on GitHub) and flash it as described. Your device should now be a "disk drive" and you can copy away.

I guess you might need an extra stage in your build that takes whatever binary you're generating and converts that into a .uf2 file. There are surely examples for creating .hex or .srec files or whatever in platformio so you shouldn't have to blaze any real trails here.

The hard work is already done here. You can integrate those parts.

Good luck and have fun!

[marchingband]

@robertlipe I have this working well, the tools needed from platformio to integrate would be access to esptool.py to burn the bootloader and partition table, and optionally offer some prebuilt bootloader/partition-table pairs.
I did add a script that uses uf2conv.py to output a uf2 binary on each build.

Several of the fears you express miss the mark. The partition table prevents anything overwriting the bootloader, which is a 3rd stage bootloader, not 2nd.

This does not belong with idf-extra-components, it is platform agnostic for Arduino/IDF, and fits perfectly inside PIO, they already have the OTA framework setup for ESP, this would be very similar.

So this is a feature request. Happy to make a full report of my work to build and use the uf2 bootloader if anyone is interested in adding support to PIO. I am not a python guy, so am not the person for this job.

[Jason2866]

@marchingband If you want/need this feature you have probably do it yourself. There is no development/support for new espressif MCUs and features. See #1225 this is not only relevant for Arduino. It is hitting every new stuff and features where espressif is involved.
Looking a bit into it shouldn't be to complicated to integrate. At the end replacing/adding the bootloader with a pio advanced script. We do similar to add a factory firmware additionaly to the built main firmware. Done with Arduino but it is litterally the same way how to achieve with IDF. The script doing the main work can be seen here https://github.com/arendst/Tasmota/blob/development/pio-tools/post_esp32.py

[marchingband]

@Jason2866 as I said above I have it working well. I do not need the feature but it would be a good feature. It took me many hours to implement, and it's a powerful addition. I am not a python developer so I am not the right person to provide easy/flexible PIO integration.

I listed the needs above, only a few things. I would not know where to begin personally, for example, to provide esptool.py access from a pio command.

[Jason2866]

@marchingband Providing your "hacky" solution would show whats needed to be done. I or maybe others consider try to integrate it fully in Platformio when knowing what! to do.

[marchingband]

The steps I followed are:
- INSTALL ESP_IDF
- BUILD BOOTLOADER
git clone git@github.com:adafruit/tinyuf2.git
cd tinyuf2
git submodule update --init --recursive
cd ports/espressif
get_idf
edit /boards/adafruit_feather_esp32s3/sdkconfig and /boards/adafruit_feather_esp32s3/board.h
I commented out the LED and GPIO stuff, because I like the 2 other ways to trigger the bootloader:

1. press reset then immediately press and hold boot.
2. calling this function from code:

void reboot_to_uf2(void)
{
  // Check out esp_reset_reason_t for other Espressif pre-defined values
  enum { APP_REQUEST_UF2_RESET_HINT = 0x11F2 };

  // call esp_reset_reason() is required for idf.py to properly links esp_reset_reason_set_hint()
  (void) esp_reset_reason();
  esp_reset_reason_set_hint((esp_reset_reason_t)APP_REQUEST_UF2_RESET_HINT);
  esp_restart();
}

In these files you can also set the name of the USB device and the name of the drive as it shows up.
You can also select the partition file, which is important for the various flash sizes.
I was actually not able to produce a working bootloader for 8MB flash, and didn't have time to debug, so I am not sure the reason, and just went with 4MB.
make BOARD=adafruit_feather_esp32s3 all
cd _build/adafruit_feather_esp32s3
esptool.py --chip esp32s3 merge_bin -o merged-flash.bin --flash_mode dio --flash_freq 80m --flash_size 4MB 0x8000 partition_table/partition-table.bin 0xe000 ota_data_initial.bin 0x0 bootloader/bootloader.bin 0x2d0000 tinyuf2.bin

- FLASH BOOTLOADER
esptool.py --chip esp32s3 -p /dev/cu.usbmodem1101 -b 115200 --before=default_reset --after=hard_reset write_flash --flash_mode dio --flash_freq 80m --flash_size 4MB 0x0 merged-flash.bin

- PRODUCE THE UF2 file
I copied uf2conv.py and uf2families.json to my project directory.
And wrote this script

Import("env")

APP_BIN = "$BUILD_DIR/${PROGNAME}.bin"

def make_uf2(source, target, env):
    env.Execute("python3 ./uf2conv.py $BUILD_DIR/firmware.bin -c -b 0x00 -f ESP32S3 -o uf2_firmware.bin")

# Add a post action that runs uf2conv.py to create a uf2 file
env.AddPostAction(APP_BIN , make_uf2)

Then I have extra_scripts = post:make_uf2.py in my platform.ini

@Jason2866 let me know if that makes sense, or if you hit any snags. I did my best to document the process, but I may not have done a perfect job.

I am not sure the best way to integrate this into pio. Perhaps it is to produce a set of booloaders, for some common partition tables and flash sizes. Possibly have a special command to flash the bootloader, and another to output a .uf2 after a compilation. It would be nice to be able to designate the drive name, but I am not sure how that would be possible. It would of course be nice to designate the partition file as well, but I have no idea if PIO can call make and have this whole routine available somehow, to produce a custom bootloader per user. The user would need to include the uf2 partition of course.

Anyhow I am here to answer questions.

[marchingband]

here is my boards.h

/*
 * The MIT License (MIT)
 *
 * Copyright (c) 2020 Ha Thach (tinyusb.org) for Adafruit Industries
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 * THE SOFTWARE.
 */

#ifndef ADAFRUIT_FEATHER_ESP32S3_H_
#define ADAFRUIT_FEATHER_ESP32S3_H_

//--------------------------------------------------------------------+
// Button
//--------------------------------------------------------------------+

// Enter UF2 mode if GPIO is pressed while 2nd stage bootloader indicator
// is on e.g RGB = Purple. If it is GPIO0, user should not hold this while
// reset since that will instead run the 1st stage ROM bootloader
#define PIN_BUTTON_UF2        0

// GPIO that implement 1-bit memory with RC components which hold the
// pin value long enough for double reset detection.
#define PIN_DOUBLE_RESET_RC   34

//--------------------------------------------------------------------+
// LED
//--------------------------------------------------------------------+

// GPIO connected to Neopixel data
// #define NEOPIXEL_PIN          33

// #define NEOPIXEL_POWER_PIN    21
// #define NEOPIXEL_POWER_STATE  1

// // Brightness percentage from 1 to 255
// #define NEOPIXEL_BRIGHTNESS   0x10

// // Number of neopixels
// #define NEOPIXEL_NUMBER       1

// // LED for indicator and writing flash
// // If not defined neopixel will be use for flash writing instead
// #define LED_PIN               13
// #define LED_STATE_ON          1

//--------------------------------------------------------------------+
// USB UF2
//--------------------------------------------------------------------+

#define USB_VID                  0x239A
#define USB_PID                  0x011B
#define USB_MANUFACTURER         "Adafruit"
#define USB_PRODUCT              "Feather ESP32-S3"

#define UF2_PRODUCT_NAME         USB_MANUFACTURER " " USB_PRODUCT
#define UF2_BOARD_ID             "ESP32-S3-Feather-revC"
#define UF2_VOLUME_LABEL         "FTHRS3BOOT"
#define UF2_INDEX_URL            "https://www.adafruit.com/product/5477"

// Use favicon
#define TINYUF2_FAVICON_HEADER   "favicon_adafruit_256.h"

#endif

and my sdkconfig:

# Board Specific Config

# Partition Table
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions-4MB.csv"

# Serial flasher config
CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y

[marchingband]

The attempt at 8MB was:

make BOARD=adafruit_qtpy_esp32s3 all
cd _build/adafruit_qtpy_esp32s3
esptool.py --chip esp32s3 merge_bin -o merged-flash.bin --flash_mode dio --flash_freq 80m --flash_size 8MB 0x8000 partition_table/partition-table.bin 0xe000 ota_data_initial.bin 0x0 bootloader/bootloader.bin 0x410000 tinyuf2.bin
esptool.py --chip esp32s3 -p /dev/cu.usbmodem1101  -b 115200 --before=default_reset --after=hard_reset write_flash --flash_mode dio --flash_freq 80m --flash_size 8MB 0x0 merged-flash.bin

And the bootloader simply didn't work. I think some basic debugging would get to the bottom of it but I ran out of patience.

[marchingband]

The beauty of this tool is that I can now ship firmware updates to clients who may not be technically inclined, or may not have PIO installed, etc. Likewise I can ship firmware to end users. It is incredibly powerful and was worth the effort. I just hope others can someday access it without the work of going through this lengthly and fairly complex process.

[Jason2866]

The description of the workflow is fine. Thx for the write up. I will have for sure a look at. Since more time is the thing you can't buy, sadly idk when.

[robertlipe]

Thanx for the head start. This isn't a high priority for me, but it's something I've kicked around wanting. My python is also terrible. But in open source, things like this sometimes go from someone getting something working and then others descending on it and making it awesome. So that first step gets penciled in with a head start on a bunch of footwork for the next of us that wants to run (walk) with it. We appreciate tht.

…

On Sun, Aug 11, 2024 at 8:52 PM Andrew March ***@***.***> wrote: The beauty of this tool is that I can now ship firmware updates to clients who may not be technically inclined, or may not have PIO installed, etc. Likewise I can ship firmware to end users. It is incredibly powerful and was worth the effort. I just hope others can someday access it without the work of going through this lengthly and fairly complex process. — Reply to this email directly, view it on GitHub <#1373 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD37LCVH3SNNIBCANTD3ZRAINTAVCNFSM6AAAAABHKE3JH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBSHE4TIMJRGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>