Bug Watch: FreeRTOS meets FPU
In this article, we will discuss, a data corruption issue that we encountered while using FreeRTOS, how the usage of the FPU might be the culprit and a potential fix for the data corruption issue.
Background
The project involves developing an upgrade software for an automotive cockpit module, that houses an Audio DSP SoC. The audio dsp board is built around a dual-core SHARC+ and ARM Cortex-A5 SoC from Analog Devices. This board interfaces a SPI flash module over Quad SPI, and the ARM Cortex-A5 core runs a FreeRTOS-based application compiled using the GCC toolchain.
The current project, involves implementation of software upgrade over Ethernet. The upgrade is done from the application code. The upgrade is initiated from a host SoC running QNX, which sends a sequence of control and data messages to transfer the firmware image to the target device. Once the download completes, a verification step is run on the received image to confirm integrity. The verification calculates SHA256 on the downloaded image and compares it with the expected SHA256, and the result is sent back to QNX.
Problem
While the verification was successful in almost all cases, we encountered one instance where the verification failed after a download. To better understand the issue and how we proceeded with debugging it, the data flow is described below.
The data message from QNX is received by the Ethernet driver, and hands it over to the TCP/IP stack. The TCP/IP stack processes the message and hands it over to the SOME/IP layer. The SOME/IP layer then processes the message and hands it over to the Upgrade service. The Upgrade service ultimately invokes the Flash Driver to the write data to Flash.
Few interface points have been tapped to help debug the issue. We have a packet capture running on the QNX Ethernet interface, and logging is done by the Upgrade service for each write. We also have dump of the Flash contents, after the verification failed.
The flash contents were compared with the original file contents. It was found that a section of 8 bytes did not match the original file contents. Now that we know the flash contents are indeed incorrect, we need to identify at what point in the flow the data was corrupted.
--- orig.hex 2023-12-02 11:24:22.186054578 +0530
+++ flash.hex 2023-12-02 11:24:15.606274731 +0530
@@ -55705,7 +55705,7 @@
000d9fc0 73 73 65 5f 72 65 66 5f 6c 61 74 65 6e 63 79 5f |sse_ref_latency_|
000d9fd0 65 73 74 00 70 4d 61 69 6e 00 00 00 00 00 00 00 |est.pMain.......|
000d9fe0 70 50 61 72 61 6d 73 00 70 52 65 73 75 6c 74 00 |pParams.pResult.|
-000d9ff0 70 4d 61 69 6e 2d 3e 43 42 4d 00 00 00 00 00 00 |pMain->CBM......|
+000d9ff0 70 4d 61 69 6e 2d 3e 43 01 11 11 11 44 c8 18 a2 |pMain->C....D...|
000da000 70 4d 61 69 6e 2d 3e 43 42 58 00 00 00 00 00 00 |pMain->CBX......|
000da010 70 4d 61 69 6e 2d 3e 43 42 4d 50 00 00 00 00 00 |pMain->CBMP.....|
000da020 70 4d 61 69 6e 2d 3e 43 42 58 50 00 00 00 00 00 |pMain->CBXP.....|Analyzing the packet trace, the bytes that was transmitted by QNX, had the
correct data. The following Wireshark screenshot shows the packet that
transmitted the data. At offset 0130, the packet is shown containing the string
esult.pMain->CBM..., which was the corrupted portion of string in the flash.
The ADSP flash driver performs a read after every write and compares the data, and logs the result. It is very unlikely that any corruption would happen within the Flash.
So the corruption should have happened within the ADSP software. But what could have caused the corruption?
There are a whole lot of possibilities, given that this is a firmware based system, without memory protection: wild pointers, buffer overruns, stack overflow, heap corruption, etc. After systematically eliminating most of these possibilities we zeroed in on a potential issue, the unsuspecting FPU.
FPU stands for Floating-Point Unit. It is used by the processor to perform floating-point operations. When the floating-point unit is used in a FreeRTOS system, FreeRTOS should save and restore the FPU register state during a context switch. If this is not done, the FPU registers will be corrupted after a task switch.
The FPU registers are not only used to perform floating-point operations, they are also used to perform vector operations. These vector operations are also called SIMD (Single Instruction Multiple Data) instructions. Vector operations allow us to perform, multiple arithmetic and logical operations in parallel. These are generally used for accelerating 3D graphics, audio and video processing.
As it happens, the seemingly harmless memcpy() and friends use the vector
operations to accelerate copying of data in Cortex-A-based processors. The
following listing shows the disassembly of memcpy(). The instructions that
start with a v are vector instructions, and d<n> and s<n> are actually
floating-point registers.
memcpy Disassembly a200a680 <memcpy>:
[...clip...]
a200a808: 15cca000 strbne sl, [ip]
a200a80c: e49da004 pop {sl} ; (ldr sl, [sp], #4)
a200a810: e12fff1e bx lr
a200a814: ed913b00 vldr d3, [r1]
a200a818: ed914b10 vldr d4, [r1, #64] ; 0x40
a200a81c: ed915b20 vldr d5, [r1, #128] ; 0x80
a200a820: ed916b30 vldr d6, [r1, #192] ; 0xc0
a200a824: ed917b40 vldr d7, [r1, #256] ; 0x100
a200a828: ed910b02 vldr d0, [r1, #8]
a200a82c: ed911b04 vldr d1, [r1, #16]
a200a830: ed912b06 vldr d2, [r1, #24]
a200a834: e2811020 add r1, r1, #32
a200a838: e25aad0a subs sl, sl, #640 ; 0x280
a200a83c: ba000055 blt a200a998 <memcpy+0x318>
a200a840: ed8c3b00 vstr d3, [ip]
a200a844: ed913b00 vldr d3, [r1]
a200a848: ed8c0b02 vstr d0, [ip, #8]
a200a84c: ed910b02 vldr d0, [r1, #8]
a200a850: ed8c1b04 vstr d1, [ip, #16]
a200a854: ed911b04 vldr d1, [r1, #16]
a200a858: ed8c2b06 vstr d2, [ip, #24]
a200a85c: ed912b06 vldr d2, [r1, #24]
a200a860: ed8c3b08 vstr d3, [ip, #32]
a200a864: ed913b48 vldr d3, [r1, #288] ; 0x120
a200a868: ed8c0b0a vstr d0, [ip, #40] ; 0x28
a200a86c: ed910b0a vldr d0, [r1, #40] ; 0x28
a200a870: ed8c1b0c vstr d1, [ip, #48] ; 0x30
a200a874: ed911b0c vldr d1, [r1, #48] ; 0x30
a200a878: ed8c2b0e vstr d2, [ip, #56] ; 0x38
a200a87c: ed912b0e vldr d2, [r1, #56] ; 0x38
a200a880: ed8c4b10 vstr d4, [ip, #64] ; 0x40
a200a884: ed914b10 vldr d4, [r1, #64] ; 0x40
a200a888: ed8c0b12 vstr d0, [ip, #72] ; 0x48
a200a88c: ed910b12 vldr d0, [r1, #72] ; 0x48
a200a890: ed8c1b14 vstr d1, [ip, #80] ; 0x50
a200a894: ed911b14 vldr d1, [r1, #80] ; 0x50
a200a898: ed8c2b16 vstr d2, [ip, #88] ; 0x58
a200a89c: ed912b16 vldr d2, [r1, #88] ; 0x58
a200a8a0: ed8c4b18 vstr d4, [ip, #96] ; 0x60
a200a8a4: ed914b58 vldr d4, [r1, #352] ; 0x160
a200a8a8: ed8c0b1a vstr d0, [ip, #104] ; 0x68
a200a8ac: ed910b1a vldr d0, [r1, #104] ; 0x68
a200a8b0: ed8c1b1c vstr d1, [ip, #112] ; 0x70
a200a8b4: ed911b1c vldr d1, [r1, #112] ; 0x70
a200a8b8: ed8c2b1e vstr d2, [ip, #120] ; 0x78
a200a8bc: ed912b1e vldr d2, [r1, #120] ; 0x78
a200a8c0: ed8c5b20 vstr d5, [ip, #128] ; 0x80
a200a8c4: ed915b20 vldr d5, [r1, #128] ; 0x80
a200a8c8: ed8c0b22 vstr d0, [ip, #136] ; 0x88
a200a8cc: ed910b22 vldr d0, [r1, #136] ; 0x88
a200a8d0: ed8c1b24 vstr d1, [ip, #144] ; 0x90
a200a8d4: ed911b24 vldr d1, [r1, #144] ; 0x90
a200a8d8: ed8c2b26 vstr d2, [ip, #152] ; 0x98
a200a8dc: ed912b26 vldr d2, [r1, #152] ; 0x98
a200a8e0: ed8c5b28 vstr d5, [ip, #160] ; 0xa0
a200a8e4: ed915b68 vldr d5, [r1, #416] ; 0x1a0
a200a8e8: ed8c0b2a vstr d0, [ip, #168] ; 0xa8
a200a8ec: ed910b2a vldr d0, [r1, #168] ; 0xa8
a200a8f0: ed8c1b2c vstr d1, [ip, #176] ; 0xb0
a200a8f4: ed911b2c vldr d1, [r1, #176] ; 0xb0
a200a8f8: ed8c2b2e vstr d2, [ip, #184] ; 0xb8
a200a8fc: ed912b2e vldr d2, [r1, #184] ; 0xb8
a200a900: e28cc0c0 add ip, ip, #192 ; 0xc0
a200a904: e28110c0 add r1, r1, #192 ; 0xc0
a200a908: ed8c6b00 vstr d6, [ip]
a200a90c: ed916b00 vldr d6, [r1]
a200a910: ed8c0b02 vstr d0, [ip, #8]
a200a914: ed910b02 vldr d0, [r1, #8]
a200a918: ed8c1b04 vstr d1, [ip, #16]
a200a91c: ed911b04 vldr d1, [r1, #16]
a200a920: ed8c2b06 vstr d2, [ip, #24]
a200a924: ed912b06 vldr d2, [r1, #24]
a200a928: ed8c6b08 vstr d6, [ip, #32]
a200a92c: ed916b48 vldr d6, [r1, #288] ; 0x120
a200a930: ed8c0b0a vstr d0, [ip, #40] ; 0x28
a200a934: ed910b0a vldr d0, [r1, #40] ; 0x28
a200a938: ed8c1b0c vstr d1, [ip, #48] ; 0x30
a200a93c: ed911b0c vldr d1, [r1, #48] ; 0x30
a200a940: ed8c2b0e vstr d2, [ip, #56] ; 0x38
a200a944: ed912b0e vldr d2, [r1, #56] ; 0x38
a200a948: ed8c7b10 vstr d7, [ip, #64] ; 0x40
a200a94c: ed917b10 vldr d7, [r1, #64] ; 0x40
a200a950: ed8c0b12 vstr d0, [ip, #72] ; 0x48
a200a954: ed910b12 vldr d0, [r1, #72] ; 0x48
a200a958: ed8c1b14 vstr d1, [ip, #80] ; 0x50
a200a95c: ed911b14 vldr d1, [r1, #80] ; 0x50
a200a960: ed8c2b16 vstr d2, [ip, #88] ; 0x58
a200a964: ed912b16 vldr d2, [r1, #88] ; 0x58
a200a968: ed8c7b18 vstr d7, [ip, #96] ; 0x60
a200a96c: ed917b58 vldr d7, [r1, #352] ; 0x160
a200a970: ed8c0b1a vstr d0, [ip, #104] ; 0x68
a200a974: ed910b1a vldr d0, [r1, #104] ; 0x68
a200a978: ed8c1b1c vstr d1, [ip, #112] ; 0x70
a200a97c: ed911b1c vldr d1, [r1, #112] ; 0x70
a200a980: ed8c2b1e vstr d2, [ip, #120] ; 0x78
a200a984: ed912b1e vldr d2, [r1, #120] ; 0x78
a200a988: e28cc080 add ip, ip, #128 ; 0x80
a200a98c: e2811080 add r1, r1, #128 ; 0x80
a200a990: e25aad05 subs sl, sl, #320 ; 0x140
[...clip...]So, even if the tasks do not perform any floating-point operation, and do not use the FPU registers, the optimized C library functions might be internally using vector instructions, that use the FPU registers. And hence it is essential to save and restore the state of the FPU registers, during a context switch.
But the default FreeRTOS configuration does not save and restore FPU registers,
across context switch. This is because saving and restoring FPU state has
additional overhead, and unless the system really uses the FPU, this
additional overhead is unwarranted. This could have resulted in the FPU
registers getting corrupted, during a memcpy() resulting in the corrupted data
being stored in the Flash.
Solution
Fortunately, FreeRTOS already supports FPU context save and restores. It just
needs to be enabled through a configuration setting
configUSE_TASK_FPU_SUPPORT.
#define configUSE_TASK_FPU_SUPPORT 2
If the macro is set to 2, then the floating-point context will be automatically
saved and restored across context switch. In addition, if the memcpy() and
friends are to be used within interrupt handlers, then
vApplicationFPUSafeIRQHandler() should be defined and used in the interrupt
handler. And when the interrupt handler is invoked, FreeRTOS will automatically
save and restore FPU context, when executing the interrupt handler.
With this configuration in place, the FPU context is correctly saved and restored across context switches and interrupt handlers, eliminating the source of corruption. The issue has not recurred since applying the fix, confirming that the FPU context mismatch was the root cause.
Concluding Notes
In a complex system like a Cortex-A5, there are lots of opportunities for optimization. This can come up as a surprise when using FreeRTOS, which was designed primarily for micro-controller environments. Usage of the FPU registers for performing copy is one such case. Next time, when using FreeRTOS on a system, this needs to be checked and enabled as required.
This bug has reinforced the fact that people working on a firmware / RTOS based system, need to have a broad understanding of all the software and hardware components in the system. This is even more so, with increasingly complex SoCs and micro-controllers being used in embedded-systems. If you’re exploring FreeRTOS and looking for expert engineering support, reach out to us at sales@zilogic.com to discuss your requirements and explore how we can help.