save large amount of data on ESP32

Question

I have a project where I need to save a large amount of bytes (maximum of 1080000) on my ESP32(NodeMCU). And it does not have to be permanent.

Is it possible and what is the best way to do it.

Answer 1

It is quite common to store data on the ESP32's flash. The method you should choose depends on the data size, number of files, and how frequently it is written.

In any case, you have to set up a partition for the data (separate from the app firmware). https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/partition-tables.html

If you just need to write a single file (and only appending new data to the end), you could write raw data to the flash partition using the partition API (link above), but that would not allow you to rewrite parts of the data or erase parts of it. You can only erase the entire data partition and start-over (BTW, erasing a block of flash takes a little while (few ms?) and uses a bit of energy).

I'd recommend using a file-system, since ESP-IDF makes it pretty easy if you follow an example:

ESP-IDF provides NVS storage for small variables that are frequently read/written. https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/nvs_flash.html

And FAT filesystem for large files that are less frequently written. https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/fatfs.html

And there are other filesystems supported like SPIFFS LittleFS, ESPFS, each with unique benefits and limitations.