ESP32 S3 USB MSC can't do more than 238kb storage?

[fyrebug]

I'm attempting to do data logging with the internal flash of the esp32 s3 but unsure how to correctly edit the values of the MSC example that can be found here
https://github.com/espressif/arduino-es ... USBMSC.ino

I'm trying to get this example working larger than 238kb. the example works fine up to 238kb and is detected as an external USB drive I can read and write to, but only as 238kb.
I am using an esp32 S3 with 8mb flash and 2mb PSRAM. (N8R2).
Preferably I'd like 1mb or higher to be usable as storage.

I understand I can use an external SD card. I would prefer not to use an SD card. I would like to get this working.

when editing

Code: Select all

static const uint32_t DISK_SECTOR_COUNT = 30 * 2*8; 

30 seems to be the highest number it will accept before it won't compile.

when trying to compile with a larger number (more kb storage), I just get "[Redacted]/bin/ld.exe: DRAM segment data does not fit."

any ideas what I need to edit? Does the additional PSRAM not help?

Thanks

[lbernstone]

The disk (msc_disk) is defined as a static global object. The PSRAM is not available at boot to build a global (for the compiler- it needs to be initialized before use). You need to change the code so that the initial definition just leaves the msc_disk as uint8_t*. Then you build out the disk in setup using ps_alloc.

Code: Select all

#ifndef ARDUINO_USB_MODE
#error This ESP32 SoC has no Native USB interface
#elif ARDUINO_USB_MODE == 1
#warning This sketch should be used when USB is in OTG mode
void setup() {}
void loop() {}
#else
#include "USB.h"
#include "USBMSC.h"

USBMSC MSC;

#define FAT_U8(v)          ((v) & 0xFF)
#define FAT_U16(v)         FAT_U8(v), FAT_U8((v) >> 8)
#define FAT_U32(v)         FAT_U8(v), FAT_U8((v) >> 8), FAT_U8((v) >> 16), FAT_U8((v) >> 24)
#define FAT_MS2B(s, ms)    FAT_U8(((((s) & 0x1) * 1000) + (ms)) / 10)
#define FAT_HMS2B(h, m, s) FAT_U8(((s) >> 1) | (((m) & 0x7) << 5)), FAT_U8((((m) >> 3) & 0x7) | ((h) << 3))
#define FAT_YMD2B(y, m, d) FAT_U8(((d) & 0x1F) | (((m) & 0x7) << 5)), FAT_U8((((m) >> 3) & 0x1) | ((((y) - 1980) & 0x7F) << 1))
#define FAT_TBL2B(l, h)    FAT_U8(l), FAT_U8(((l >> 8) & 0xF) | ((h << 4) & 0xF0)), FAT_U8(h >> 4)

static const uint32_t DISK_SECTOR_COUNT = 500 * 8;   // 8KB is the smallest size that windows allow to mount
static const uint16_t DISK_SECTOR_SIZE = 512;      // Should be 512
static const uint16_t DISC_SECTORS_PER_TABLE = 1;  //each table sector can fit 170KB (340 sectors)

static uint8_t* msc_disk[DISK_SECTOR_COUNT];

const char README_CONTENTS[] = "This is tinyusb's MassStorage Class demo.\r\n\r\nIf you find any bugs or get any questions, feel free to file an\r\nissue at github.com/hathach/tinyusb";

const uint8_t sector0[512] = //------------- Block0: Boot Sector -------------//
  {                                                        // Header (62 bytes)
   0xEB, 0x3C, 0x90,                                       //jump_instruction
   'M', 'S', 'D', 'O', 'S', '5', '.', '0',                 //oem_name
   FAT_U16(DISK_SECTOR_SIZE),                              //bytes_per_sector
   FAT_U8(1),                                              //sectors_per_cluster
   FAT_U16(1),                                             //reserved_sectors_count
   FAT_U8(1),                                              //file_alloc_tables_num
   FAT_U16(16),                                            //max_root_dir_entries
   FAT_U16(DISK_SECTOR_COUNT),                             //fat12_sector_num
   0xF8,                                                   //media_descriptor
   FAT_U16(DISC_SECTORS_PER_TABLE),                        //sectors_per_alloc_table;//FAT12 and FAT16
   FAT_U16(1),                                             //sectors_per_track;//A value of 0 may indicate LBA-only access
   FAT_U16(1),                                             //num_heads
   FAT_U32(0),                                             //hidden_sectors_count
   FAT_U32(0),                                             //total_sectors_32
   0x00,                                                   //physical_drive_number;0x00 for (first) removable media, 0x80 for (first) fixed disk
   0x00,                                                   //reserved
   0x29,                                                   //extended_boot_signature;//should be 0x29
   FAT_U32(0x1234),                                        //serial_number: 0x1234 => 1234
   'T', 'i', 'n', 'y', 'U', 'S', 'B', ' ', 'M', 'S', 'C',  //volume_label padded with spaces (0x20)
   'F', 'A', 'T', '1', '2', ' ', ' ', ' ',                 //file_system_type padded with spaces (0x20)

   // Zero up to 2 last bytes of FAT magic code (448 bytes)
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,

   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,

   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,

   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,

   //boot signature (2 bytes)
   0x55, 0xAA
  };
  
  const uint8_t sector1[] =  //------------- Block1: FAT12 Table -------------//
  {
    FAT_TBL2B(0xFF8, 0xFFF), FAT_TBL2B(0xFFF, 0x000)  // first 2 entries must be 0xFF8 0xFFF, third entry is cluster end of readme file
  };
  
  const uint8_t sector2[512] =  //------------- Block2: Root Directory -------------//
  {
    // first entry is volume label
    'E', 'S', 'P', '3', '2', 'S', '2', ' ', 'M', 'S', 'C',
    0x08,                                                                                                                 //FILE_ATTR_VOLUME_LABEL
    0x00, FAT_MS2B(0, 0), FAT_HMS2B(0, 0, 0), FAT_YMD2B(0, 0, 0), FAT_YMD2B(0, 0, 0), FAT_U16(0), FAT_HMS2B(13, 42, 30),  //last_modified_hms
    FAT_YMD2B(2018, 11, 5),                                                                                               //last_modified_ymd
    FAT_U16(0), FAT_U32(0),

    // second entry is readme file
    'R', 'E', 'A', 'D', 'M', 'E', ' ', ' ',  //file_name[8]; padded with spaces (0x20)
    'T', 'X', 'T',                           //file_extension[3]; padded with spaces (0x20)
    0x20,                                    //file attributes: FILE_ATTR_ARCHIVE
    0x00,                                    //ignore
    FAT_MS2B(1, 980),                        //creation_time_10_ms (max 199x10 = 1s 990ms)
    FAT_HMS2B(13, 42, 36),                   //create_time_hms [5:6:5] => h:m:(s/2)
    FAT_YMD2B(2018, 11, 5),                  //create_time_ymd [7:4:5] => (y+1980):m:d
    FAT_YMD2B(2020, 11, 5),                  //last_access_ymd
    FAT_U16(0),                              //extended_attributes
    FAT_HMS2B(13, 44, 16),                   //last_modified_hms
    FAT_YMD2B(2019, 11, 5),                  //last_modified_ymd
    FAT_U16(2),                              //start of file in cluster
    FAT_U32(sizeof(README_CONTENTS) - 1)     //file size
  };

static int32_t onWrite(uint32_t lba, uint32_t offset, uint8_t *buffer, uint32_t bufsize) {
  Serial.printf("MSC WRITE: lba: %lu, offset: %lu, bufsize: %lu\n", lba, offset, bufsize);
  memcpy(msc_disk[lba] + offset, buffer, bufsize);
  return bufsize;
}

static int32_t onRead(uint32_t lba, uint32_t offset, void *buffer, uint32_t bufsize) {
  Serial.printf("MSC READ: lba: %lu, offset: %lu, bufsize: %lu\n", lba, offset, bufsize);
  memcpy(buffer, msc_disk[lba] + offset, bufsize);
  return bufsize;
}

static bool onStartStop(uint8_t power_condition, bool start, bool load_eject) {
  Serial.printf("MSC START/STOP: power: %u, start: %u, eject: %u\n", power_condition, start, load_eject);
  return true;
}

static void usbEventCallback(void *arg, esp_event_base_t event_base, int32_t event_id, void *event_data) {
  if (event_base == ARDUINO_USB_EVENTS) {
    arduino_usb_event_data_t *data = (arduino_usb_event_data_t *)event_data;
    switch (event_id) {
      case ARDUINO_USB_STARTED_EVENT: Serial.println("USB PLUGGED"); break;
      case ARDUINO_USB_STOPPED_EVENT: Serial.println("USB UNPLUGGED"); break;
      case ARDUINO_USB_SUSPEND_EVENT: Serial.printf("USB SUSPENDED: remote_wakeup_en: %u\n", data->suspend.remote_wakeup_en); break;
      case ARDUINO_USB_RESUME_EVENT:  Serial.println("USB RESUMED"); break;

      default: break;
    }
  }
}

void setup() {
  Serial.begin(115200);
  Serial.setDebugOutput(true);
  for (uint x=0; x<DISK_SECTOR_COUNT; x++) {
     msc_disk[x] = (uint8_t*)ps_malloc(DISK_SECTOR_SIZE * sizeof(uint8_t));
     if (!msc_disk[x]) {
      Serial.println("Unable to allocate memory");
      return;
     }
  }

  memcpy(msc_disk[0], sector0, 512);
  memcpy(msc_disk[1], sector1, 6);
  memcpy(msc_disk[2], sector2, 512);
  memcpy(msc_disk[3], README_CONTENTS, 512);

  USB.onEvent(usbEventCallback);
  MSC.vendorID("ESP32");       //max 8 chars
  MSC.productID("USB_MSC");    //max 16 chars
  MSC.productRevision("1.0");  //max 4 chars
  MSC.onStartStop(onStartStop);
  MSC.onRead(onRead);
  MSC.onWrite(onWrite);

  MSC.mediaPresent(true);
  MSC.isWritable(true);  // true if writable, false if read-only

  MSC.begin(DISK_SECTOR_COUNT, DISK_SECTOR_SIZE);
  USB.begin();
  Serial.print("Total PSRAM:");
  Serial.println(ESP.getPsramSize());
  Serial.print("Free PSRAM: ");
  Serial.println(ESP.getFreePsram());

}

void loop() {
  delay(-1);
}
#endif /* ARDUINO_USB_MODE */

[fyrebug]

Thanks lbernstone!
this is starting to work, however,
I'm confused about this line

Code: Select all

static const uint16_t DISC_SECTORS_PER_TABLE = 1;  //each table sector can fit 170KB (340 sectors)

if I alter it to account for a larger total disk size than 170kb the default readme.txt file becomes a garbled unreadable 4gb file.
I assumed though if I alter the total size of the disk though I need to add more table sectors?

If I don't alter that line though then no matter what value I use for this line

Code: Select all

static const uint32_t DISK_SECTOR_COUNT = 260 * 8;   // 8KB is the smallest size that windows allow to mount

The disk always reads as 170kb. however, I can format it in windows to be larger. but that seems like something is wrong as it's not the correct size right away. I assume files would get corrupted and I'm doing something very wrong.

[lbernstone]

I don't care to reverse engineer FAT12, so if your workflow allows it, then formatting from the PC is 100% legit, and reduces your firmware size.
Otherwise, your best bet is to build the disk image you want externally (truncate, mkfs.vfat), then xxd dump the first 512 bytes (sector0.h is for 2MB). Put that output as the definition for sector0. Then inspect the rest of the hexdump to see where you need to add the additional table keys (aka sector1). For 2MB, that is at msc_disk[1] and [4].

[fyrebug]

Wow, thank you lbernstone. this is a great answer.
I don't mind formatting from the PC at all. I was just worried that it was going to lead to corrupted data as maybe the reported size of 1mb and useable size weren't going to match (possibly still stuck at 170k or something due to that variable).
It sounds like you're saying that's not the case. A one time format after uploading the firmware seems like a trivial price to pay and it's not a problem at all.

I really appreciate you taking the time to help me out with this! Thank you again.