memcpy somehow zeroing out bytes

rohansingh
Posts: 8
Joined: Mon May 14, 2018 2:07 pm

memcpy somehow zeroing out bytes

Postby rohansingh » Tue Dec 03, 2019 1:39 am

Has anyone else run into an issue with memcpy somehow replacing bytes with null characters? I seem to be "losing" bytes during a copy.

For example, I have a zero-initialized buffer with a capacity of 512 bytes, with 390 bytes filled with non-null data. I attempt to copy 32 additional bytes onto the buffer like this:

Code: copy.c Select all


  // preserve original contents of *out* for debugging
char* preMemcpy = malloc(512);
memcpy(preMemcpy, out, 390);

// copy new data into *out*
memcpy(out + 390, newData, 32);

After that, I have a breakpoint that gets hit if there are any null characters in the out buffer. The breakpoint is hit sometimes, but not always.When the breakpoint is hit, I'm using the debugger to inspect everything.

The last time this happened, out had 388 bytes of data that matched preMemcpy (instead of 390 bytes), followed by two null characters, followed by all 32 bytes from newData. So somehow, two bytes were "deleted".

What's even more alarming is that this is happening in the bytes prior to the block provided to memcpy. Even though I'm specifying out + 390, it's bytes 388 and 389 that are being replaced by null characters.

This is happening in memory that should be byte-addressable, since it's allocated using:

Code: Select all

heap_caps_malloc(byte_count, MALLOC_CAP_SPIRAM|MALLOC_CAP_8BIT)
Has anyone seen anything like this? Any ideas on how to troubleshoot it further?

Angus
Espressif staff
Espressif staff
Posts: 2344
Joined: Sun May 08, 2016 4:11 am

Re: memcpy somehow zeroing out bytes

Postby Angus » Tue Dec 03, 2019 2:06 am

Hi Rohan,

Do any of the memory areas in a single memcpy() call overlap?

Could you please let us know the addresses of the 3 pointers preMemcpy, out, newData?


Angus

rohansingh
Posts: 8
Joined: Mon May 14, 2018 2:07 pm

Re: memcpy somehow zeroing out bytes

Postby rohansingh » Tue Dec 03, 2019 4:21 am

There is definitely no overlap. I also tried replacing memcpy with memmove and hit the same issue.

This actually only occurs intermittently. I'll try to catch it in debug mode again tomorrow and I'll post the affected memory addresses.

rohansingh
Posts: 8
Joined: Mon May 14, 2018 2:07 pm

Re: memcpy somehow zeroing out bytes

Postby rohansingh » Tue Dec 03, 2019 7:33 pm

OK, it looks like I can get this to recur pretty frequently when I put the device under a lot of load. This code is part of a loop that copies incoming MQTT messages into a destination buffer, up to 32 bytes at a time.

This time, the out buffer started with 611 bytes of data, zero-padded to 1024 bytes total. We attempted to copy 32 bytes from newData to out + 131. All 32 bytes were copied, but the 3 bytes of out prior to the copy point were zeroed out.

Could you please let us know the addresses of the 3 pointers preMemcpy, out, newData?

I can do you better, here are the entire contents of each.

Code: preMemcpy Select all


// preMemcpy + 1024
Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
3ffec9d0: 7B 22 61 70 70 49 44 22 3A 22 47 79 {"appID":"Gy
3ffec9e0: 41 47 34 50 6F 39 4B 4A 7A 42 32 63 39 37 44 7A AG4Po9KJzB2c97Dz
3ffec9f0: 79 48 4E 59 6B 38 61 58 65 35 55 75 46 68 57 42 yHNYk8aXe5UuFhWB
3ffeca00: 55 65 57 64 61 44 48 54 6F 22 2C 22 73 63 72 65 UeWdaDHTo","scre
3ffeca10: 65 6E 73 22 3A 5B 22 69 56 42 4F 52 77 30 4B 47 ens":["iVBORw0KG
3ffeca20: 67 6F 41 41 41 41 4E 53 55 68 45 55 67 41 41 41 goAAAANSUhEUgAAA
3ffeca30: 45 41 41 41 41 41 67 43 41 49 41 41 41 41 74 2F EAAAAAgCAIAAAAt/
3ffeca40: 2B 6E 54 41 41 41 41 75 30 6C 45 51 56 52 34 6E +nTAAAAu0lEQVR4n
3ffeca50: 4F 79 57 77 51 34 44 49 51 68 45 61 39 50 2F 2F OyWwQ4DIQhEa9P//
3ffeca60: 32 56 36 4D 44 57 45 46 42 77 33 62 6D 5A 4E 35 2V6MDWEFBw3bmZN5
3ffeca70: 70 30 38 44 4F 41 55 73 50 74 2B 48 59 34 4D 73 p08DOAUsPt+HY4Ms
3ffeca80: 4A 45 42 4E 6A 4C 41 52 67 62 59 79 41 43 62 44 JEBNjLARgbYyACbD
3ffeca90: 79 49 79 73 33 35 6F 72 65 47 61 61 64 51 51 44 yIys35oreGaadQQD
3ffecaa0: 49 32 5A 2B 66 43 69 33 41 4B 2B 6A 44 2F 58 6D I2Z+fCi3AK+jD/Xm
3ffecab0: 76 75 69 41 74 64 48 43 43 78 51 39 79 31 4C 69 vuiAtdHCCxQ9y1Li
3ffecac0: 79 65 66 47 2F 68 62 50 70 54 5A 30 2B 74 66 71 yefG/hbPpTZ0+tfq
3ffecad0: 71 58 68 67 58 61 67 34 2F 4E 6D 5A 63 49 51 44 qXhgXag4/NmZcIQD
3ffecae0: 33 47 52 63 36 72 5A 41 39 4C 54 62 49 4B 52 48 3GRc6rZA9LTbIKRH
3ffecaf0: 63 43 6A 41 74 41 4F 49 44 33 64 39 6D 67 73 4D cCjAtAOID3d9mgsM
3ffecb00: 6A 65 51 33 53 7A 38 57 70 54 62 4C 2F 38 50 2B jeQ3Sz8WpTbL/8P+
3ffecb10: 41 63 37 6D 2F 69 75 36 55 76 69 51 78 34 48 2F Ac7m/iu6UviQx4H/
3ffecb20: 74 4B 4A 69 75 4F 2F 68 57 53 41 6A 51 79 77 6B tKJiuO/hWSAjQywk
3ffecb30: 51 45 32 4D 73 44 6D 65 41 50 66 41 41 41 41 2F QE2MsDmeAPfAAAA/
3ffecb40: 2F 39 55 61 49 63 47 46 6D 45 6E 73 41 41 41 41 /9UaIcGFmEnsAAAA
3ffecb50: 41 42 4A 52 55 35 45 72 6B 4A 67 67 67 3D 3D 22 ABJRU5ErkJggg=="
3ffecb60: 2C 22 69 56 42 4F 52 77 30 4B 47 67 6F 41 41 41 ,"iVBORw0KGgoAAA
3ffecb70: 41 4E 53 55 68 45 55 67 41 41 41 45 41 41 41 41 ANSUhEUgAAAEAAAA
3ffecb80: 41 67 43 41 49 41 41 41 41 74 2F 2B 6E 54 41 41 AgCAIAAAAt/+nTAA
3ffecb90: 41 41 70 30 6C 45 51 56 52 34 6E 4F 79 57 77 51 AAp0lEQVR4nOyWwQ
3ffecba0: 37 44 49 41 78 44 59 64 72 2F 2F 33 4A 32 51 45 7DIAxDYdr//3J2QE
3ffecbb0: 4F 49 71 6F 6B 6A 55 64 78 57 66 71 63 65 6E 4D OIqokjUdxWfqcenM
3ffecbc0: 4B 44 5A 4F 75 6E 50 42 77 4A 73 4A 45 41 47 77 KDZOunPBwJsJEAGw
3ffecbd0: 6D 77 6B 51 41 62 43 62 44 35 49 69 45 7A 61 77 mwkQAbCbD5IiEzaw
3ffecbe0: 2B 31 56 6A 77 54 56 76 56 41 7A 35 6A 5A 57 4F +1VjwTVvVAz5jZWO
3ffecbf0: 34 73 6C 32 42 63 5A 6E 7A 32 4D 39 64 56 54 65 4sl2BcZnz2M9dVTe
3ffecc00: 78 6F 6F 64 52 42 48 6B 31 38 59 67 46 6B 2B 54 xoodRBHk18YgFk+T
3ffecc10: 56 33 2F 58 39 56 71 6E 6D 67 47 57 67 67 37 35 V3/X9VqnmgGWgg75
3ffecc20: 32 61 75 4F 2F 4A 79 59 65 5A 4E 53 42 33 65 74 2auO/JyYeZNSB3et
3ffecc30: 62 42 79 41 7A 67 56 00 00 00 00 00 00 00 00 00 bByAzgV.........
3ffecc40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecc50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecc60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecc70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecc80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecc90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffeccb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffeccc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffeccd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffeccf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecd10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecd20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecd30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecd40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecd50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecd60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecd70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecd80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecd90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecda0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecdb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecdc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3ffecdd0: 00 00 00 00 ....

// out + 1024
Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
3f813970: 7B 22 61 70 70 49 44 22 {"appID"
3f813980: 3A 22 47 79 41 47 34 50 6F 39 4B 4A 7A 42 32 63 :"GyAG4Po9KJzB2c
3f813990: 39 37 44 7A 79 48 4E 59 6B 38 61 58 65 35 55 75 97DzyHNYk8aXe5Uu
3f8139a0: 46 68 57 42 55 65 57 64 61 44 48 54 6F 22 2C 22 FhWBUeWdaDHTo","
3f8139b0: 73 63 72 65 65 6E 73 22 3A 5B 22 69 56 42 4F 52 screens":["iVBOR
3f8139c0: 77 30 4B 47 67 6F 41 41 41 41 4E 53 55 68 45 55 w0KGgoAAAANSUhEU
3f8139d0: 67 41 41 41 45 41 41 41 41 41 67 43 41 49 41 41 gAAAEAAAAAgCAIAA
3f8139e0: 41 41 74 2F 2B 6E 54 41 41 41 41 75 30 6C 45 51 AAt/+nTAAAAu0lEQ
3f8139f0: 56 52 34 6E 4F 79 57 77 51 34 44 49 51 68 45 61 VR4nOyWwQ4DIQhEa
3f813a00: 39 50 2F 2F 32 56 36 4D 44 57 45 46 42 77 33 62 9P//2V6MDWEFBw3b
3f813a10: 6D 5A 4E 35 70 30 38 44 4F 41 55 73 50 74 2B 48 mZN5p08DOAUsPt+H
3f813a20: 59 34 4D 73 4A 45 42 4E 6A 4C 41 52 67 62 59 79 Y4MsJEBNjLARgbYy
3f813a30: 41 43 62 44 79 49 79 73 33 35 6F 72 65 47 61 61 ACbDyIys35oreGaa
3f813a40: 64 51 51 44 49 32 5A 2B 66 43 69 33 41 4B 2B 6A dQQDI2Z+fCi3AK+j
3f813a50: 44 2F 58 6D 76 75 69 41 74 64 48 43 43 78 51 39 D/XmvuiAtdHCCxQ9
3f813a60: 79 31 4C 69 79 65 66 47 2F 68 62 50 70 54 5A 30 y1LiyefG/hbPpTZ0
3f813a70: 2B 74 66 71 71 58 68 67 58 61 67 34 2F 4E 6D 5A +tfqqXhgXag4/NmZ
3f813a80: 63 49 51 44 33 47 52 63 36 72 5A 41 39 4C 54 62 cIQD3GRc6rZA9LTb
3f813a90: 49 4B 52 48 63 43 6A 41 74 41 4F 49 44 33 64 39 IKRHcCjAtAOID3d9
3f813aa0: 6D 67 73 4D 6A 65 51 33 53 7A 38 57 70 54 62 4C mgsMjeQ3Sz8WpTbL
3f813ab0: 2F 38 50 2B 41 63 37 6D 2F 69 75 36 55 76 69 51 /8P+Ac7m/iu6UviQ
3f813ac0: 78 34 48 2F 74 4B 4A 69 75 4F 2F 68 57 53 41 6A x4H/tKJiuO/hWSAj
3f813ad0: 51 79 77 6B 51 45 32 4D 73 44 6D 65 41 50 66 41 QywkQE2MsDmeAPfA
3f813ae0: 41 41 41 2F 2F 39 55 61 49 63 47 46 6D 45 6E 73 AAA//9UaIcGFmEns
3f813af0: 41 41 41 41 41 42 4A 52 55 35 45 72 6B 4A 67 67 AAAAABJRU5ErkJgg
3f813b00: 67 3D 3D 22 2C 22 69 56 42 4F 52 77 30 4B 47 67 g==","iVBORw0KGg
3f813b10: 6F 41 41 41 41 4E 53 55 68 45 55 67 41 41 41 45 oAAAANSUhEUgAAAE
3f813b20: 41 41 41 41 41 67 43 41 49 41 41 41 41 74 2F 2B AAAAAgCAIAAAAt/+
3f813b30: 6E 54 41 41 41 41 70 30 6C 45 51 56 52 34 6E 4F nTAAAAp0lEQVR4nO
3f813b40: 79 57 77 51 37 44 49 41 78 44 59 64 72 2F 2F 33 yWwQ7DIAxDYdr//3
3f813b50: 4A 32 51 45 4F 49 71 6F 6B 6A 55 64 78 57 66 71 J2QEOIqokjUdxWfq
3f813b60: 63 65 6E 4D 4B 44 5A 4F 75 6E 50 42 77 4A 73 4A cenMKDZOunPBwJsJ
3f813b70: 45 41 47 77 6D 77 6B 51 41 62 43 62 44 35 49 69 EAGwmwkQAbCbD5Ii
3f813b80: 45 7A 61 77 2B 31 56 6A 77 54 56 76 56 41 7A 35 Ezaw+1VjwTVvVAz5
3f813b90: 6A 5A 57 4F 34 73 6C 32 42 63 5A 6E 7A 32 4D 39 jZWO4sl2BcZnz2M9
3f813ba0: 64 56 54 65 78 6F 6F 64 52 42 48 6B 31 38 59 67 dVTexoodRBHk18Yg
3f813bb0: 46 6B 2B 54 56 33 2F 58 39 56 71 6E 6D 67 47 57 Fk+TV3/X9VqnmgGW
3f813bc0: 67 67 37 35 32 61 75 4F 2F 4A 79 59 65 5A 4E 53 gg752auO/JyYeZNS
3f813bd0: 42 33 65 74 62 42 79 41 00 00 00 52 50 51 44 47 B3etbByA...RPQDG
3f813be0: 54 50 66 69 65 78 77 4A 31 33 6E 2F 34 66 77 48 TPfiexwJ13n/4fwH
3f813bf0: 2F 55 32 79 7A 36 4A 61 4B 38 34 00 00 00 00 00 /U2yz6JaK84.....
3f813c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813c60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813c80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813c90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813ca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813cb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813cc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813cd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813ce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813cf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813d10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813d20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813d30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813d40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
3f813d70: 00 00 00 00 00 00 00 00 ........

// newData + 32
Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
3f80dd20: 52 50 51 44 RPQD
3f80dd30: 47 54 50 66 69 65 78 77 4A 31 33 6E 2F 34 66 77 GTPfiexwJ13n/4fw
3f80dd40: 48 2F 55 32 79 7A 36 4A 61 4B 38 34 H/U2yz6JaK84

The corrupted bytes in out start near 3f813bd0.

rohansingh
Posts: 8
Joined: Mon May 14, 2018 2:07 pm

Re: memcpy somehow zeroing out bytes

Postby rohansingh » Tue Dec 03, 2019 7:36 pm

@ESP_Angus, do you think this is due to the PSRAM cache issue reported here:
https://github.com/espressif/esp-idf/issues/2892

Looking around, I found a similar report of string corruption here:
https://github.com/espressif/esp-idf/issues/3006

I'm also noticing that I can't repro the issue if I use MALLOC_CAP_INTERNAL.

Angus
Espressif staff
Espressif staff
Posts: 2344
Joined: Sun May 08, 2016 4:11 am

Re: memcpy somehow zeroing out bytes

Postby Angus » Tue Dec 03, 2019 10:55 pm

Yes, I think you're right that this is a variant of those bugs.

Just for one more data point, what ESP-IDF version are you using?

PeterR
Posts: 621
Joined: Mon Jun 04, 2018 2:47 pm

Re: memcpy somehow zeroing out bytes

Postby PeterR » Tue Dec 03, 2019 11:27 pm

HI Angus,
The link is a detailed read & I am not sure what the corrolary is (&| is the jury still out?).

I use PSRAM. What should I do to ensure proper operation?
Is there a full problem statement (rather than discussion)?

Thanks
& I also believe that IDF CAN should be fixed.

Sprite
Espressif staff
Espressif staff
Posts: 10618
Joined: Thu Nov 26, 2015 4:08 am

Re: memcpy somehow zeroing out bytes

Postby Sprite » Wed Dec 04, 2019 3:45 am

The current state is that we arrived at a proper fix for this issue, but it's not in mainline yet. We're working on that. For the time being, you can try the toolchain that is linked in the discussion; that should solve the issue as well.

rohansingh
Posts: 8
Joined: Mon May 14, 2018 2:07 pm

Re: memcpy somehow zeroing out bytes

Postby rohansingh » Wed Dec 04, 2019 12:53 pm

Yes, I think you're right that this is a variant of those bugs.

Just for one more data point, what ESP-IDF version are you using?
I'm using a fork of v3.3. It is just v3.3 with two commits backported onto it from:
https://github.com/espressif/esp-idf/issues/3592

Here is the exact branch that I am using:
https://github.com/tidbyt/esp-idf/commits/master

For what it's worth, I found that the issue is much more frequent without the linker flag for the PSRAM issue workaround. So it does all seem related.

Who is online

Users browsing this forum: Bing [Bot], ChatGPT-User, PerplexityBot, PetalBot, Qwantbot, trendictionbot and 3 guests