memcpy somehow zeroing out bytes

rohansingh
Posts: 8
Joined: Mon May 14, 2018 2:07 pm

memcpy somehow zeroing out bytes

Postby rohansingh » Tue Dec 03, 2019 1:39 am

Has anyone else run into an issue with memcpy somehow replacing bytes with null characters? I seem to be "losing" bytes during a copy.

For example, I have a zero-initialized buffer with a capacity of 512 bytes, with 390 bytes filled with non-null data. I attempt to copy 32 additional bytes onto the buffer like this:

  1.   // preserve original contents of *out* for debugging
  2.   char* preMemcpy = malloc(512);
  3.   memcpy(preMemcpy, out, 390);
  4.  
  5.   // copy new data into *out*
  6.   memcpy(out + 390, newData, 32);

After that, I have a breakpoint that gets hit if there are any null characters in the out buffer. The breakpoint is hit sometimes, but not always.When the breakpoint is hit, I'm using the debugger to inspect everything.

The last time this happened, out had 388 bytes of data that matched preMemcpy (instead of 390 bytes), followed by two null characters, followed by all 32 bytes from newData. So somehow, two bytes were "deleted".

What's even more alarming is that this is happening in the bytes prior to the block provided to memcpy. Even though I'm specifying out + 390, it's bytes 388 and 389 that are being replaced by null characters.

This is happening in memory that should be byte-addressable, since it's allocated using:

Code: Select all

heap_caps_malloc(byte_count, MALLOC_CAP_SPIRAM|MALLOC_CAP_8BIT)
Has anyone seen anything like this? Any ideas on how to troubleshoot it further?

ESP_Angus
Posts: 1897
Joined: Sun May 08, 2016 4:11 am

Re: memcpy somehow zeroing out bytes

Postby ESP_Angus » Tue Dec 03, 2019 2:06 am

Hi Rohan,

Do any of the memory areas in a single memcpy() call overlap?

Could you please let us know the addresses of the 3 pointers preMemcpy, out, newData?


Angus

rohansingh
Posts: 8
Joined: Mon May 14, 2018 2:07 pm

Re: memcpy somehow zeroing out bytes

Postby rohansingh » Tue Dec 03, 2019 4:21 am

There is definitely no overlap. I also tried replacing memcpy with memmove and hit the same issue.

This actually only occurs intermittently. I'll try to catch it in debug mode again tomorrow and I'll post the affected memory addresses.

rohansingh
Posts: 8
Joined: Mon May 14, 2018 2:07 pm

Re: memcpy somehow zeroing out bytes

Postby rohansingh » Tue Dec 03, 2019 7:33 pm

OK, it looks like I can get this to recur pretty frequently when I put the device under a lot of load. This code is part of a loop that copies incoming MQTT messages into a destination buffer, up to 32 bytes at a time.

This time, the out buffer started with 611 bytes of data, zero-padded to 1024 bytes total. We attempted to copy 32 bytes from newData to out + 131. All 32 bytes were copied, but the 3 bytes of out prior to the copy point were zeroed out.

ESP_Angus wrote: Could you please let us know the addresses of the 3 pointers preMemcpy, out, newData?

I can do you better, here are the entire contents of each.

  1. // preMemcpy + 1024
  2.   Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F  
  3. 3ffec9d0:             7B 22 61 70 70 49 44 22 3A 22 47 79       {"appID":"Gy
  4. 3ffec9e0: 41 47 34 50 6F 39 4B 4A 7A 42 32 63 39 37 44 7A   AG4Po9KJzB2c97Dz
  5. 3ffec9f0: 79 48 4E 59 6B 38 61 58 65 35 55 75 46 68 57 42   yHNYk8aXe5UuFhWB
  6. 3ffeca00: 55 65 57 64 61 44 48 54 6F 22 2C 22 73 63 72 65   UeWdaDHTo","scre
  7. 3ffeca10: 65 6E 73 22 3A 5B 22 69 56 42 4F 52 77 30 4B 47   ens":["iVBORw0KG
  8. 3ffeca20: 67 6F 41 41 41 41 4E 53 55 68 45 55 67 41 41 41   goAAAANSUhEUgAAA
  9. 3ffeca30: 45 41 41 41 41 41 67 43 41 49 41 41 41 41 74 2F   EAAAAAgCAIAAAAt/
  10. 3ffeca40: 2B 6E 54 41 41 41 41 75 30 6C 45 51 56 52 34 6E   +nTAAAAu0lEQVR4n
  11. 3ffeca50: 4F 79 57 77 51 34 44 49 51 68 45 61 39 50 2F 2F   OyWwQ4DIQhEa9P//
  12. 3ffeca60: 32 56 36 4D 44 57 45 46 42 77 33 62 6D 5A 4E 35   2V6MDWEFBw3bmZN5
  13. 3ffeca70: 70 30 38 44 4F 41 55 73 50 74 2B 48 59 34 4D 73   p08DOAUsPt+HY4Ms
  14. 3ffeca80: 4A 45 42 4E 6A 4C 41 52 67 62 59 79 41 43 62 44   JEBNjLARgbYyACbD
  15. 3ffeca90: 79 49 79 73 33 35 6F 72 65 47 61 61 64 51 51 44   yIys35oreGaadQQD
  16. 3ffecaa0: 49 32 5A 2B 66 43 69 33 41 4B 2B 6A 44 2F 58 6D   I2Z+fCi3AK+jD/Xm
  17. 3ffecab0: 76 75 69 41 74 64 48 43 43 78 51 39 79 31 4C 69   vuiAtdHCCxQ9y1Li
  18. 3ffecac0: 79 65 66 47 2F 68 62 50 70 54 5A 30 2B 74 66 71   yefG/hbPpTZ0+tfq
  19. 3ffecad0: 71 58 68 67 58 61 67 34 2F 4E 6D 5A 63 49 51 44   qXhgXag4/NmZcIQD
  20. 3ffecae0: 33 47 52 63 36 72 5A 41 39 4C 54 62 49 4B 52 48   3GRc6rZA9LTbIKRH
  21. 3ffecaf0: 63 43 6A 41 74 41 4F 49 44 33 64 39 6D 67 73 4D   cCjAtAOID3d9mgsM
  22. 3ffecb00: 6A 65 51 33 53 7A 38 57 70 54 62 4C 2F 38 50 2B   jeQ3Sz8WpTbL/8P+
  23. 3ffecb10: 41 63 37 6D 2F 69 75 36 55 76 69 51 78 34 48 2F   Ac7m/iu6UviQx4H/
  24. 3ffecb20: 74 4B 4A 69 75 4F 2F 68 57 53 41 6A 51 79 77 6B   tKJiuO/hWSAjQywk
  25. 3ffecb30: 51 45 32 4D 73 44 6D 65 41 50 66 41 41 41 41 2F   QE2MsDmeAPfAAAA/
  26. 3ffecb40: 2F 39 55 61 49 63 47 46 6D 45 6E 73 41 41 41 41   /9UaIcGFmEnsAAAA
  27. 3ffecb50: 41 42 4A 52 55 35 45 72 6B 4A 67 67 67 3D 3D 22   ABJRU5ErkJggg=="
  28. 3ffecb60: 2C 22 69 56 42 4F 52 77 30 4B 47 67 6F 41 41 41   ,"iVBORw0KGgoAAA
  29. 3ffecb70: 41 4E 53 55 68 45 55 67 41 41 41 45 41 41 41 41   ANSUhEUgAAAEAAAA
  30. 3ffecb80: 41 67 43 41 49 41 41 41 41 74 2F 2B 6E 54 41 41   AgCAIAAAAt/+nTAA
  31. 3ffecb90: 41 41 70 30 6C 45 51 56 52 34 6E 4F 79 57 77 51   AAp0lEQVR4nOyWwQ
  32. 3ffecba0: 37 44 49 41 78 44 59 64 72 2F 2F 33 4A 32 51 45   7DIAxDYdr//3J2QE
  33. 3ffecbb0: 4F 49 71 6F 6B 6A 55 64 78 57 66 71 63 65 6E 4D   OIqokjUdxWfqcenM
  34. 3ffecbc0: 4B 44 5A 4F 75 6E 50 42 77 4A 73 4A 45 41 47 77   KDZOunPBwJsJEAGw
  35. 3ffecbd0: 6D 77 6B 51 41 62 43 62 44 35 49 69 45 7A 61 77   mwkQAbCbD5IiEzaw
  36. 3ffecbe0: 2B 31 56 6A 77 54 56 76 56 41 7A 35 6A 5A 57 4F   +1VjwTVvVAz5jZWO
  37. 3ffecbf0: 34 73 6C 32 42 63 5A 6E 7A 32 4D 39 64 56 54 65   4sl2BcZnz2M9dVTe
  38. 3ffecc00: 78 6F 6F 64 52 42 48 6B 31 38 59 67 46 6B 2B 54   xoodRBHk18YgFk+T
  39. 3ffecc10: 56 33 2F 58 39 56 71 6E 6D 67 47 57 67 67 37 35   V3/X9VqnmgGWgg75
  40. 3ffecc20: 32 61 75 4F 2F 4A 79 59 65 5A 4E 53 42 33 65 74   2auO/JyYeZNSB3et
  41. 3ffecc30: 62 42 79 41 7A 67 56 00 00 00 00 00 00 00 00 00   bByAzgV.........
  42. 3ffecc40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  43. 3ffecc50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  44. 3ffecc60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  45. 3ffecc70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  46. 3ffecc80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  47. 3ffecc90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  48. 3ffecca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  49. 3ffeccb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  50. 3ffeccc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  51. 3ffeccd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  52. 3ffecce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  53. 3ffeccf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  54. 3ffecd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  55. 3ffecd10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  56. 3ffecd20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  57. 3ffecd30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  58. 3ffecd40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  59. 3ffecd50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  60. 3ffecd60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  61. 3ffecd70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  62. 3ffecd80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  63. 3ffecd90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  64. 3ffecda0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  65. 3ffecdb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  66. 3ffecdc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  67. 3ffecdd0: 00 00 00 00                                       ....
  68.  
  69. // out + 1024
  70.   Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F  
  71. 3f813970:                         7B 22 61 70 70 49 44 22           {"appID"
  72. 3f813980: 3A 22 47 79 41 47 34 50 6F 39 4B 4A 7A 42 32 63   :"GyAG4Po9KJzB2c
  73. 3f813990: 39 37 44 7A 79 48 4E 59 6B 38 61 58 65 35 55 75   97DzyHNYk8aXe5Uu
  74. 3f8139a0: 46 68 57 42 55 65 57 64 61 44 48 54 6F 22 2C 22   FhWBUeWdaDHTo","
  75. 3f8139b0: 73 63 72 65 65 6E 73 22 3A 5B 22 69 56 42 4F 52   screens":["iVBOR
  76. 3f8139c0: 77 30 4B 47 67 6F 41 41 41 41 4E 53 55 68 45 55   w0KGgoAAAANSUhEU
  77. 3f8139d0: 67 41 41 41 45 41 41 41 41 41 67 43 41 49 41 41   gAAAEAAAAAgCAIAA
  78. 3f8139e0: 41 41 74 2F 2B 6E 54 41 41 41 41 75 30 6C 45 51   AAt/+nTAAAAu0lEQ
  79. 3f8139f0: 56 52 34 6E 4F 79 57 77 51 34 44 49 51 68 45 61   VR4nOyWwQ4DIQhEa
  80. 3f813a00: 39 50 2F 2F 32 56 36 4D 44 57 45 46 42 77 33 62   9P//2V6MDWEFBw3b
  81. 3f813a10: 6D 5A 4E 35 70 30 38 44 4F 41 55 73 50 74 2B 48   mZN5p08DOAUsPt+H
  82. 3f813a20: 59 34 4D 73 4A 45 42 4E 6A 4C 41 52 67 62 59 79   Y4MsJEBNjLARgbYy
  83. 3f813a30: 41 43 62 44 79 49 79 73 33 35 6F 72 65 47 61 61   ACbDyIys35oreGaa
  84. 3f813a40: 64 51 51 44 49 32 5A 2B 66 43 69 33 41 4B 2B 6A   dQQDI2Z+fCi3AK+j
  85. 3f813a50: 44 2F 58 6D 76 75 69 41 74 64 48 43 43 78 51 39   D/XmvuiAtdHCCxQ9
  86. 3f813a60: 79 31 4C 69 79 65 66 47 2F 68 62 50 70 54 5A 30   y1LiyefG/hbPpTZ0
  87. 3f813a70: 2B 74 66 71 71 58 68 67 58 61 67 34 2F 4E 6D 5A   +tfqqXhgXag4/NmZ
  88. 3f813a80: 63 49 51 44 33 47 52 63 36 72 5A 41 39 4C 54 62   cIQD3GRc6rZA9LTb
  89. 3f813a90: 49 4B 52 48 63 43 6A 41 74 41 4F 49 44 33 64 39   IKRHcCjAtAOID3d9
  90. 3f813aa0: 6D 67 73 4D 6A 65 51 33 53 7A 38 57 70 54 62 4C   mgsMjeQ3Sz8WpTbL
  91. 3f813ab0: 2F 38 50 2B 41 63 37 6D 2F 69 75 36 55 76 69 51   /8P+Ac7m/iu6UviQ
  92. 3f813ac0: 78 34 48 2F 74 4B 4A 69 75 4F 2F 68 57 53 41 6A   x4H/tKJiuO/hWSAj
  93. 3f813ad0: 51 79 77 6B 51 45 32 4D 73 44 6D 65 41 50 66 41   QywkQE2MsDmeAPfA
  94. 3f813ae0: 41 41 41 2F 2F 39 55 61 49 63 47 46 6D 45 6E 73   AAA//9UaIcGFmEns
  95. 3f813af0: 41 41 41 41 41 42 4A 52 55 35 45 72 6B 4A 67 67   AAAAABJRU5ErkJgg
  96. 3f813b00: 67 3D 3D 22 2C 22 69 56 42 4F 52 77 30 4B 47 67   g==","iVBORw0KGg
  97. 3f813b10: 6F 41 41 41 41 4E 53 55 68 45 55 67 41 41 41 45   oAAAANSUhEUgAAAE
  98. 3f813b20: 41 41 41 41 41 67 43 41 49 41 41 41 41 74 2F 2B   AAAAAgCAIAAAAt/+
  99. 3f813b30: 6E 54 41 41 41 41 70 30 6C 45 51 56 52 34 6E 4F   nTAAAAp0lEQVR4nO
  100. 3f813b40: 79 57 77 51 37 44 49 41 78 44 59 64 72 2F 2F 33   yWwQ7DIAxDYdr//3
  101. 3f813b50: 4A 32 51 45 4F 49 71 6F 6B 6A 55 64 78 57 66 71   J2QEOIqokjUdxWfq
  102. 3f813b60: 63 65 6E 4D 4B 44 5A 4F 75 6E 50 42 77 4A 73 4A   cenMKDZOunPBwJsJ
  103. 3f813b70: 45 41 47 77 6D 77 6B 51 41 62 43 62 44 35 49 69   EAGwmwkQAbCbD5Ii
  104. 3f813b80: 45 7A 61 77 2B 31 56 6A 77 54 56 76 56 41 7A 35   Ezaw+1VjwTVvVAz5
  105. 3f813b90: 6A 5A 57 4F 34 73 6C 32 42 63 5A 6E 7A 32 4D 39   jZWO4sl2BcZnz2M9
  106. 3f813ba0: 64 56 54 65 78 6F 6F 64 52 42 48 6B 31 38 59 67   dVTexoodRBHk18Yg
  107. 3f813bb0: 46 6B 2B 54 56 33 2F 58 39 56 71 6E 6D 67 47 57   Fk+TV3/X9VqnmgGW
  108. 3f813bc0: 67 67 37 35 32 61 75 4F 2F 4A 79 59 65 5A 4E 53   gg752auO/JyYeZNS
  109. 3f813bd0: 42 33 65 74 62 42 79 41 00 00 00 52 50 51 44 47   B3etbByA...RPQDG
  110. 3f813be0: 54 50 66 69 65 78 77 4A 31 33 6E 2F 34 66 77 48   TPfiexwJ13n/4fwH
  111. 3f813bf0: 2F 55 32 79 7A 36 4A 61 4B 38 34 00 00 00 00 00   /U2yz6JaK84.....
  112. 3f813c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  113. 3f813c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  114. 3f813c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  115. 3f813c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  116. 3f813c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  117. 3f813c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  118. 3f813c60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  119. 3f813c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  120. 3f813c80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  121. 3f813c90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  122. 3f813ca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  123. 3f813cb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  124. 3f813cc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  125. 3f813cd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  126. 3f813ce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  127. 3f813cf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  128. 3f813d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  129. 3f813d10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  130. 3f813d20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  131. 3f813d30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  132. 3f813d40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  133. 3f813d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  134. 3f813d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
  135. 3f813d70: 00 00 00 00 00 00 00 00                           ........
  136.  
  137. // newData + 32
  138.   Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F  
  139. 3f80dd20:                                     52 50 51 44               RPQD
  140. 3f80dd30: 47 54 50 66 69 65 78 77 4A 31 33 6E 2F 34 66 77   GTPfiexwJ13n/4fw
  141. 3f80dd40: 48 2F 55 32 79 7A 36 4A 61 4B 38 34               H/U2yz6JaK84

The corrupted bytes in out start near 3f813bd0.

rohansingh
Posts: 8
Joined: Mon May 14, 2018 2:07 pm

Re: memcpy somehow zeroing out bytes

Postby rohansingh » Tue Dec 03, 2019 7:36 pm

@ESP_Angus, do you think this is due to the PSRAM cache issue reported here:
https://github.com/espressif/esp-idf/issues/2892

Looking around, I found a similar report of string corruption here:
https://github.com/espressif/esp-idf/issues/3006

I'm also noticing that I can't repro the issue if I use MALLOC_CAP_INTERNAL.

ESP_Angus
Posts: 1897
Joined: Sun May 08, 2016 4:11 am

Re: memcpy somehow zeroing out bytes

Postby ESP_Angus » Tue Dec 03, 2019 10:55 pm

Yes, I think you're right that this is a variant of those bugs.

Just for one more data point, what ESP-IDF version are you using?

PeterR
Posts: 217
Joined: Mon Jun 04, 2018 2:47 pm

Re: memcpy somehow zeroing out bytes

Postby PeterR » Tue Dec 03, 2019 11:27 pm

HI Angus,
The link is a detailed read & I am not sure what the corrolary is (&| is the jury still out?).

I use PSRAM. What should I do to ensure proper operation?
Is there a full problem statement (rather than discussion)?

Thanks

ESP_Sprite
Posts: 2914
Joined: Thu Nov 26, 2015 4:08 am

Re: memcpy somehow zeroing out bytes

Postby ESP_Sprite » Wed Dec 04, 2019 3:45 am

The current state is that we arrived at a proper fix for this issue, but it's not in mainline yet. We're working on that. For the time being, you can try the toolchain that is linked in the discussion; that should solve the issue as well.

rohansingh
Posts: 8
Joined: Mon May 14, 2018 2:07 pm

Re: memcpy somehow zeroing out bytes

Postby rohansingh » Wed Dec 04, 2019 12:53 pm

ESP_Angus wrote:
Tue Dec 03, 2019 10:55 pm
Yes, I think you're right that this is a variant of those bugs.

Just for one more data point, what ESP-IDF version are you using?
I'm using a fork of v3.3. It is just v3.3 with two commits backported onto it from:
https://github.com/espressif/esp-idf/issues/3592

Here is the exact branch that I am using:
https://github.com/tidbyt/esp-idf/commits/master

For what it's worth, I found that the issue is much more frequent without the linker flag for the PSRAM issue workaround. So it does all seem related.

Who is online

Users browsing this forum: boarchuz, MSN [Bot] and 19 guests