FreeRTOS unexpected notifications / notify values

[lotharyx]

Working with ESP32-C3-MINI-1U.

A communication receive task sits on an xTaskNotifyWait until a wire protocol task notifies it that something has come in on the wire. Various bits get set in the notify value to indicate what has arrived.

More often than not, at application startup, the communication task acts as though it's received a notification, and the notification value is garbage. I've partially mitigated the problem by adding these two lines at the start of the task that is getting the garbage notifications:

Code: Select all

xTaskNotifyStateClear(xTaskGetCurrentTaskHandle());
ulTaskNotifyValueClear(xTaskGetCurrentTaskHandle(), UINT32_MAX);

This seems to have alleviated phantom notification states; however, I still see garbage in the high 16 bits of the notification value, as logged by this code:

Code: Select all

uint32_t notify_value = 0;
rc = xTaskNotifyWait(SEND_SIGNAL_NOTIFICATION_MASK, SEND_SIGNAL_NOTIFICATION_MASK, &notify_value, TX_Q_POLL_PERIOD);
if (rc == pdTRUE && notify_value != 0) {
   ESP_LOGI("hlc_tx", "Notified, value 0x%08lx", notify_value);
   // ...

I see output such as:

Code: Select all

I (319481) hlc_tx: Notified, value 0x3fcb1000

None of my code ever sets any of the high 16 bits on that task. The displayed notification value should be 0x00001000.

I've looked through the ESP-IDF port of FreeRTOS code dealing with creating tasks, and it looks like it zeros out the entire task control block correctly, so I don't think it's a simple case of uninitialized data. I'm at a loss to explain the behavior, though, and am inclined to think there's a bug in ESP's FreeRTOS implementation.

Has anyone else encountered inexplicable notification states or values? Did you discover a solution or workaround?

[MicroController]

More code needed.
How do you set the bits/send the notification?

and am inclined to think there's a bug in ESP's FreeRTOS implementation

Unlikely.
May be some memory corruption. Or a synchronization issue in the use of the task handle. Or both. Or something else.

Btw,

Code: Select all

... xTaskNotifyWait(SEND_SIGNAL_NOTIFICATION_MASK, ...)

may make you miss/lose notifications, namely all notifications generated while the task is not blocked in xTaskNotifyWait().

[lotharyx]

inclined to suspect a bug in ESP's FreeRTOS

Unlikely.

I'll readily admit I sometimes write some very bad code, but in this case it's a port of an existing, functional codebase from an STM32 project that does not suffer the same problem. Of course, with heap corruption, seemingly unrelated code can make all the difference.

Here's an example of a notification sent to this task:

Code: Select all

xTaskNotify(task_data->tx_task, HLC_PTT_NOTIFY_BIT_CTS, eSetBits);

That is the pattern I use throughout (this is a communication protocol; I use notifications to communicate certain state information among tasks, e.g., notify the transmit task that the wire task received a Clear-To-Send signal). My calls to xTaskNotify always use the eSetBits operation, and none of them ever set bits 14 - 30. Here's the list of HLC_PTT_NOTIFY_ symbols:

Code: Select all

 enum {
     HLC_PTT_NOTIFY_BIT_CTS = 0x01,
     HLC_PTT_NOTIFY_BIT_nCTS = 0x02,
     HLC_PTT_NOTIFY_BIT_CRC_ERROR = 0x04,
     HLC_PTT_NOTIFY_BIT_ACK = 0x08,
     HLC_PTT_NOTIFY_BIT_CRC_READY = 0x10,
     HLC_PTT_NOTIFY_BIT_SEND_CTS = 0x100,
     HLC_PTT_NOTIFY_BIT_SEND_nCTS = 0x200,
     HLC_PTT_NOTIFY_BIT_SEND_BAD_CRC = 0x400,
     HLC_PTT_NOTIFY_BIT_SEND_ACK = 0x800,
     HLC_PTT_NOTIFY_BIT_XMIT_WTG = 0x1000,
     HLC_PTT_NOTIFY_BIT_SEND_RETRAN = 0x2000,
     HLC_PTT_NOTIFY_BIT_SIG_SYNC  = 0x80000000,
 };
 

That's it; I haven't redacted anything from that set. I use only those symbols as arguments to xTaskNotify; there are no magic numbers or variables used.

Another puzzle piece is that sometimes the garbage value is 0xa5a5a5a5, which looks very much like some sort of canary value. But other times it's something different, so that could be a red herring.

Btw,

Code: Select all

... xTaskNotifyWait(SEND_SIGNAL_NOTIFICATION_MASK, ...)

may make you miss/lose notifications, namely all notifications generated while the task is not blocked in xTaskNotifyWait().

That's not how I interpret the FreeRTOS docs about pending notifications (emphasis added):

• ulBitsToClearOnEntry

Any bits set in ulBitsToClearOnEntry will be cleared in the calling RTOS task's notification value on entry to the xTaskNotifyWait() function (before the task waits for a new notification) provided a notification is not already pending when xTaskNotifyWait() is called. For example, if ulBitsToClearOnEntry is 0x01, then bit 0 of the task's notification value will be cleared on entry to the function.Setting ulBitsToClearOnEntry to 0xffffffff (ULONG_MAX) will clear all the bits in the task's notification value, effectively clearing the value to 0.

If a notification is already pending (i.e., sent while the task was not blocked in its call to xTaskNotifyWait), ulBitsToClearOnEntry bits are not cleared. So I am not at risk of missing notifications (unless the pending state was cleared in the interim).

In any case, my next step is to try to set a watch on that piece of memory to see when it gets written with the garbage value. Since my last post, I added code to verify that the notification value is indeed 0x00000000 right after the task starts up, so something is definitely writing the garbage to that location. Maybe there's a buffer overflow in another module I created. Once I nail down the cause, I'll update this thread with what I discovered.

[lotharyx]

*facepalm* And I'm an idiot. A lucky idiot. Sometimes it just takes talking it out to get my head around the right concept.

There is one place where my code does use a variable to hold the argument sent to xTaskNotify, that I had forgotten about. The intent of that variable is to defer certain actions until something else completes. That variable was uninitialized, so when the time came to examine it to see if a deferred notification was needed, it held garbage and that garbage was sent to xTaskNotify.

Apparently I had been getting lucky with the other platform, and that heap space always ended up being zero at application start.

Now that I'm properly initializing that variable, the problem is gone.

I will go put on the Cone of Shame and ponder my life choices.

[MicroController]

Now that I'm properly initializing that variable, the problem is gone.

Glad you got it working

Btw, 0xa5 is indeed the 'canary' FreeRTOS initializes the stack with.

[MicroController]

If a notification is already pending (i.e., sent while the task was not blocked in its call to xTaskNotifyWait), ulBitsToClearOnEntry bits are not cleared. So I am not at risk of missing notifications.

You're right!
Looking into the FreeRTOS sources confirms that ulBitsToClearOnEntry isn't used at all if there's already a notification pending when xTaskNotifyWait() is entered.