How Flash Memory Partitioning Works in ESP32

The organization of Flash memory in the ESP32 strongly reminds me of how hard drives on PCs worked during the MBR era. There is also a bootloader, a partition table, and, of course, the data partitions themselves.

And regardless of the flash size, the partitioning rules remain the same.

Usually, the partition table starts at the address 0x8000 and takes up 4 KB. However, in some cases, a larger bootloader might be needed, which requires shifting the partition table further down.

The structure of a single entry in the partition table looks like this:

struct PartitionEntry 
{
    uint8_t magic;         // Magic number for entry validation
    uint8_t type;          // Partition type
    uint8_t subtype;       // Partition subtype
    uint8_t reserved;      // Reserved
    uint32_t offset;       // Partition offset in flash memory
    uint32_t size;         // Partition size
    char name[16];         // Partition name
    uint32_t flags;        // Partition flags
};

However, you shouldn’t strictly rely on it, as it may change in different versions of the framework. For reliable operation, it is better to use the API functions (from the esp_partition.h header file):

  • esp_partition_find( sp_partition_type_t type, esp_partition_subtype_t subtype, const char label ) - starts the search for partitions. You can specify a particular type, subtype, and partition name. Alternatively, you can pass ESP_PARTITION_TYPE_ANY, ESP_PARTITION_SUBTYPE_ANY, NULL, to get an iterator for reading information about all available partitions.

  • esp_partition_get(esp_partition_iterator_t iterator) — allows you to get detailed information about the partition using the iterator.

  • esp_partition_iterator_t esp_partition_next(esp_partition_iterator_t iterator) — moves to the next element. When the list ends, the iterator will be NULL.

  • esp_partition_iterator_release(esp_partition_iterator_t iterator); — frees the memory allocated for the iterator.

An example of reading partitions in ESP-IDF at runtime (by the way, these same functions work similarly in Arduino for ESP32):

    const esp_partition_t *partition = nullptr;
    esp_partition_iterator_t it = esp_partition_find(
        ESP_PARTITION_TYPE_ANY,
        ESP_PARTITION_SUBTYPE_ANY,
        NULL);
    while (it != NULL)
    {
        partition = esp_partition_get(it);
        partition = esp_partition_get(it);
        printf("Partition: %s", partition->label);
        printf(", Address: 0x%lx", partition->address);
        printf(", Size: %ld", partition->size);
        printf(" bytes, type: %d \n", partition->type);
        it = esp_partition_next(it);
    }
    esp_partition_iterator_release(it);

Here is my output:

Partition: nvs, Address: 0x9000, Size: 24576 bytes, type: 1 
Partition: phy_init, Address: 0xf000, Size: 4096 bytes, type: 1 
Partition: factory, Address: 0x10000, Size: 1048576 bytes, type: 0

To better understand this output, let’s briefly look at the main types of partitions in the ESP32 ecosystem and their purposes:

Partition Name Purpose
nvs Intended for storing Wi-Fi logins and passwords, calibration data, and other user settings.
factory, otaXX, appXX Partitions for storing the firmware itself (there can be several of them). This is the foundation for the OTA (Over-The-Air) update mechanism.
otadata Contains information for the bootloader about exactly which firmware version (from which partition) should be booted right now. Used when multiple versions are present.
spiffs,littlefs,fat Partitions for storing user data in the respective file systems.
coredump Partition for saving a memory dump during critical errors. Later, this data can be used for debugging and finding bugs in the firmware code.

If we analyze my output through the lens of this table, one detail stands out: exactly 1 MB is allocated for the main firmware (factory). Meanwhile, there is a whole 4 MB available on the flash chip. Why does this happen, and how can it be fixed to use all the memory? Let’s figure it out.

How to change the partitioning in ESP-IDF?

ESP-IDF allows configuring a project using various tools — there is a graphical configurator for VSCode, a console-based menuconfig, and finally, nothing stops you from opening the sdkconfig configuration file in your favorite text editor and modifying it directly.

Where can you find these configurators?

  1. The graphical one can be found in the SDK Configuration editor.



  2. The console one can be launched with the idf.py menuconfig command (in VSCode, this is done via ESP-IDF: New Terminal).

It has identical sections and settings as the graphical configurator. Specifically, we can configure:

  • Partition table type - you can select a standard layout (for example, Single factory app, no OTA or Factory app, two OTA). Or you can even specify your own CSV file.

  • A custom offset for the partition table.

Let’s try to create and use our own CSV file with the partition descriptions. In the root of the project, I create a mypartitions.csv file and allocate 3 MB for the firmware:

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     ,        0x6000,
phy_init, data, phy,     ,        0x1000,
factory,  app,  factory, ,        3M,

The Offset column can be omitted; in that case, it will be calculated automatically.

Next, I launch the console configurator and select the Custom partition table CSV option. After that, an additional option appears in the menu where you can specify the name of this file. We enter it, then press s to save the configuration, and then q to exit.

Let’s see how this was written to sdkconfig:

#
# Partition Table
#
# CONFIG_PARTITION_TABLE_SINGLE_APP is not set
# CONFIG_PARTITION_TABLE_SINGLE_APP_LARGE is not set
# CONFIG_PARTITION_TABLE_TWO_OTA is not set
# CONFIG_PARTITION_TABLE_TWO_OTA_LARGE is not set
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="mypartitions.csv"
# default:
CONFIG_PARTITION_TABLE_FILENAME="mypartitions.csv"
# default:
CONFIG_PARTITION_TABLE_OFFSET=0x8000
# default:
CONFIG_PARTITION_TABLE_MD5=y

Now we need to completely erase the microcontroller’s flash memory. This can be done with the idf.py erase-flash command. If you have the serial monitor open at this time, you will get an error — so the monitor must be closed beforehand.

After that, I launch the project build and flashing process — and I get an error:

Partitions tables occupies 3.1MB of flash (3211264 bytes) which does not fit in configured flash size 2MB. 
Change the flash size in menuconfig under the 'Serial Flasher Config' menu.

By default, the maximum flash size is configured as 2 MB. This needs to be fixed: I go to Serial Flasher Config -> Flash Size, select 4 MB, and save the configuration. I run the build and flashing process again.

Now the test code outputs the following information about the partitions:

Partition: nvs, Address: 0x9000, Size: 24576 bytes, type: 1 
Partition: phy_init, Address: 0xf000, Size: 4096 bytes, type: 1 
Partition: factory, Address: 0x10000, Size: 3145728 bytes, type: 0

Everything worked out, we now have a whole 3 MB for the firmware.

How to change the partitioning in a PlatformIO + ESP-IDF project

PlatformIO supports project development based on both the Arduino and ESP-IDF frameworks.

For an ESP-IDF project, flash partitioning settings are changed directly in the platformio.ini configuration file. To use a custom partition table, simply add the board_build.partitions parameter:

[env:lolin32]
platform = espressif32
board = lolin32_lite
framework = espidf
board_build.partitions = mypartitions.csv
board_upload.flash_size = 4MB

In the same config file, you can also specify the total available flash memory size using the board_upload.flash_size parameter.

Important note: even if you previously configured a different partition layout or flash size via the console menuconfig, PlatformIO will ignore them and forcefully override the configuration with the values from platformio.ini. Therefore, when working in this environment, platformio.ini is the primary configuration file.

How to change the partitioning in a PlatformIO + Arduino project

For the Arduino framework, the configuration in PlatformIO is absolutely identical to what we discussed for ESP-IDF. This means we similarly use the board_build.partitions and board_upload.flash_size parameters in the platformio.ini file to specify the path to our custom CSV file and the total flash memory size.

Instead of creating your own file, you can also choose one of the standard partition layouts already prepared by the developers. They can be found in the PlatformIO packages directory at the following path: .platformio/packages/framework-arduinoespressif32/tools/partitions.

Here is an example of the layouts available in my PlatformIO:

app3M_fat9M_16MB.csv
app3M_fat9M_fact512k_16MB.csv
app3M_spiffs9M_fact512k_16MB.csv
bare_minimum_2MB.csv
boot_app0.bin
default_16MB.csv
default_8MB.csv
default_ffat_8MB.csv
default_ffat.csv
default.csv
ffat.csv
huge_app.csv
large_fat_32MB.csv
large_ffat_8MB.csv
large_littlefs_32MB.csv
large_spiffs_16MB.csv
large_spiffs_8MB.csv
max_app_8MB.csv
min_spiffs.csv
minimal.csv
no_ota.csv
noota_3g.csv
noota_3gffat.csv
noota_ffat.csv
rainmaker.csv

Note: if no layout is explicitly specified in the platformio.ini configuration file, PlatformIO will use the default.csv configuration by default during the build process.

References

Official Espressif Documentation: Partition Tables

Comments

Post a Comment