ESP32 Forum

Posted: **Tue May 28, 2024 8:55 am**

Hi, first time posting after a fair bit of reading as a guest. I've done a fairly thorough search of ideas here and in other resources and not been able to find a solution or test to try.

I've written an open 3D world renderer in Arduino C (I appreciate Arduino is really C++ but code is struct-based rather than true objects) and it works after a lot of effort. However I've run into a problem with 'large' worlds with around 10k triangles.

I build the world in a workflow from Blender via GoogleScript into const arrays of indices and vertex floats (these may be thousands of elements each and my 'Vec3f' type is 3 floats hence 12 bytes) to a .h file that is compiled into the code. It works as it should upto nearly 9k triangles. With a larger world I got the 'dangerous relocation' error and so added the '-text-section-literals' flag to c, cpp and s extra flags in Arduino platform file. That changed the error to L32R operand being out of range. The longcalls flag is set as default in Arduino options.

I largely understand the assembly language theory here but don't see why a 32bit processor and toolchain can't deal with this. It is frustrating to have this as a compile time error that doesn't feel to be due to poor coding, especially as the error message doesn't indicate which out of range literal is causing the problem!

My code allows world maps to be broken up which reduces the size of each one but same problem.

Any ideas would be gratefully received.

I have thought about quitting Arduino but ultimately whatever UI I use they have ESP IDF under the hood so can't see an instant win.

Thanks!

Posted: **Wed May 29, 2024 1:12 am**

That is odd. How do you declare those const arrays? In general, can you share your project?

Posted: **Wed May 29, 2024 10:00 pm**

I don't have an explanation of what's going wrong in your case, but, as an alternative, you may want to consider storing the world data in a dedicated data partition. This way, it cannot in any way interfere with building your application, and you can change/update the world data on the chip without having to rebuild the application.

Posted: **Wed May 29, 2024 10:47 pm**

Sorry to sound like a broken record, but we really need to see how you are mapping the disk data into your structures.
Flash sectors are 4096 bytes, 32-bit aligned. It will be best if you access them as such if you read the disk raw.
C-style const arrays (eg, from xxd) are uint8_t arrays. I'm not entirely sure how the mapper is aligning ro-data, but if it is translating your 8-bit array into 32-bit aligned memory, that might be taking a lot more space than you expect. I use const arrays all the time to deliver web assets (.js libraries) up to 100KB, so it isn't a problem inherent to a large array.
Post very minimal code that shows how you are using/casting the data for your model, and that generates the error if possible so we can get a better idea of what is going on.

Posted: **Sun Jun 09, 2024 7:24 pm**

Thanks, I've not managed to set a minimal fail code as yet.

World is set with arrays like this to set vertex coordinates, which is the largest array:

Code: Untitled.c Select all

const Vec3f cvertices[NVERTS + 1] = { {0.f,0.f,0.f},// Dummy vertex to allow OBJ standard indexing



{40.000000f, 0.000000f, 90.000000f}, {50.000000f, 0.000000f, 90.000000f}, {40.000000f, 0.000000f, 80.000000f},

{50.000000f, 0.000000f, 80.000000f}, {42.000000f, 0.000000f, 88.000000f}, {44.801487f, 10.000000f, 85.198532f},

{42.000000f, 0.000000f, 82.000000f},...

with similar arrays for faces and colour attributes:

Code: Untitled.c Select all

const uint16_t cnvertices[1170] = {

2, 3, 1, 2, 4, 3, 8, 10, 12,

20, 13, 16, 19, 16, 15, 14, 19, 15,

17, 14, 13, 43, 26, 41, 21, 27, 22,

27, 23, 22, 35, 33, 28, 29, 26, 25,...

As I've a system for overlaying worlds these are referenced from a structure. I'd think that as I'm not loading a literal directly the full 32 bit address could be pulled from the WorldLayout without the limitations of bit depth in the L32R instruction:

Code: Untitled.c Select all

struct WorldLayout

{

  // The constants of number of triangles etc aren't needed as they are fixed via chunks

  const uint32_t type;            // what type of layout is this?

  const uint32_t frames;          // how mnay frames are in a flip book style animation

  const uint32_t nvertices_num;   // number of vertices per flip  - must be constant

  const uint32_t frame_time;      // the number of ms betwen each frame in a flick book

  Vec3f const * const vertices;         // Pointer to an array of Vec3f vertices - these are all const ptrs to const values

  uint16_t const * const nvertices;     // Pointer to an array of indices to the vertex coordinates

  uint16_t const * const texel_verts;   // Pointer to an array of indices to texture UV coordinates

  Vec2f const * const vts;              // Pointer to an array of UV coordinates

  faceMaterials const * const palette;  // Pointer to an array of faceMaterials that's a palette

  uint16_t const * const attributes;    // Pointer to an array that selects from the palette per face 

  ChunkFaces const * const TheChunks;   // Pointer to the array of lists of faces per chunk

  const ChunkArr ChAr;            // The arrangement of chunks used in this layout

};



const WorldLayout world[NUMBER_OF_LAYOUTS]=

{

{

  LO_PLAIN,

  0, // No value needed

  0,

  0,

  cvertices,

  cnvertices,

  ctexel_verts,

  cvts,

  cpalette,

  cattributes,

  cTheChunks,

  20,70,4,3,10

}

};

The data is then sent to the projector and raster as pointer:

Code: Untitled.c Select all

        // set the pointer to the layout in use

        this_world_ptr = &world[worlds]; // ERROR SITE?

and utlised in phrases such as:

Code: Untitled.c Select all

const Vec3f& v0 = layo_ptr->vertices[layo_ptr->nvertices[idx * 3]];

The code works spot on until the vertex array approaches 64k bytes, as mentioned that is equivalent to around 8500 triangles. I'd understand it better if it failed ABOVE 64k!

I appreciate that I could work from a filesystem but at present it seemed simpler to stay in DROM rather than parse from a file-like arrangement into PSRAM arrays. I did try to copy the arrays into PSRAM but same assembler error.

Posted: **Fri Jun 14, 2024 8:33 pm**

You don't have to use files or a filesystem; you can flash any data (structure) you want into a data partition and then mmap it into the CPU's address space...

Posted: **Tue Jun 18, 2024 3:47 pm**

Thanks, I'll take a look at that, might be simpler than fatfs although that has the merit that it can be built on the host from 'normal' files.

I have managed to reproduce the error from a fairly minimal program. I took the arrays and then made repeated small routines to memcpy to malloc'd PSRAM. As I approached the full set of arrays L32R error appeared. It isn't due to a specific array as I could swap them around/not build in and still get the same error. It was reading the const arrays not the malloc'd space as I kept that the same.

BTW, I moved the #include into app_main() and got even more of the L32R errors.

Posted: **Tue Jun 18, 2024 9:06 pm**

The partition API also has convenience functions for mmapping, e.g. esp_partition_mmap(...). But yes, you'll have to write the (binary) data in the format you want yourself. You may be able to modify the scripts you're using to generate the C arrays to output binary; or you could compile the generated arrays into a little C program to run on the PC which dumps the binary data from the arrays into a file. If you're into Python, you can also do it in Python, somewhat like e.g. the IDF's gen_crt_bundle.py does.

Posted: **Thu Oct 24, 2024 7:33 am**

A few months later....

I took the plunge and set up partitions for my 3D world and textures independent of the code. I rewrote my world 'compiler' in node.js to take a .obj from Blender and make a structured binary that is self contained. As you say, 'finding' the partitions is easy from IDF. Of course, I needed some initialisation to copy pointers in the partition and more importantly correct them for their mapped destinations.

I now have a world > 1MB in its partition and can update the world, textures or code independently except when I've altered the binary structure.

Definitely the right way to go and it also shows that ESP32S3 can handle long range pointers as you'd expect. It seems that the compiler is perhaps not so good at doing that (or I've misunderstood the various flags).

Posted: **Sat Mar 22, 2025 4:58 pm**

Another 6 months later... and all recent work in the IDF rather than Arduino

Using partitions for world data / textures / sound has worked well and I've been able to set up multiple layer worlds at 2MB. Pointers haven't been a problem at all, reading the binary data and making large arrays and C++ vectors has been trouble-free. I'm just trying to reuse code on ESP32P4 host and the partition aspect seems to be working fine. This suggests that I have used the APIs correctly. So the 'correct' approach works but I do still wonder why mega-constants doesn't.

ESP32 Forum

Fairly large (>64k) const arrays

Fairly large (>64k) const arrays

Re: Fairly large (>64k) const arrays

Re: Fairly large (>64k) const arrays

Re: Fairly large (>64k) const arrays

Re: Fairly large (>64k) const arrays

Re: Fairly large (>64k) const arrays

Re: Fairly large (>64k) const arrays

Re: Fairly large (>64k) const arrays

Re: Fairly large (>64k) const arrays

Re: Fairly large (>64k) const arrays