The Unknown & Hidden Behaviors of Real-Time Systems

Start on day one and measure what your system is doing, don’t assume you know what it is doing. By doing this, you’ll find that you have far fewer headaches to deal with and will also be far more efficient in developing your software and avoiding defects.

Hardware designers have it easy. They are working with a physical component that can be examined, weighed, and measured. If the hardware doesn’t work or has an issue, hardware designers can probe the electrical signals and visualize them using an oscilloscope. When I started developing embedded software almost two decades ago, it always bothered me that I couldn’t see what was actually happening in the microcontroller. I had to cross my fingers and hope that the software was executing as I intended. Sure, the LED’s blinked, the sensors seemed to be sampled appropriately, and the system didn’t immediately burst into flames or crash. But was the software actually executing the way I had designed or was it doing something unexpected that I could not detect?

Engineering is not about guessing but for decades that is exactly how embedded software developers have developed their software. Engineering uses scientific principles to design, implement, and verify the design. Measurements are important if a design team is to be successful and efficient. The problem is that most development teams that I encounter take no real measurements of their software or really understand how it is behaving on their target processor despite there being the technologies available to do so.

Let me give a very simple example about how developers really have no idea what their software is doing without tracing their application. I’ll use myself as an example. I was recently developing some lab materials for my real-time operating systems course using FreeRTOS. I was using the STM32F429I-DISCO board with its associated STM32CubeMX software to create a simple example where a start-up task initializes the system, spawns a few tasks, and then deletes itself. It was a simple application that included three tasks; Blinky, Led_Red, and Led_Green.

Since I was using vendor software, I decided that I would setup Percepio Tracealyzer to make sure my three tasks ran as expected. With the trace tool included, I knew there would now be a fourth task, TzCtrl, that would transmit the systems event data when my tasks were idle. The application code did nothing that interesting except blink a few LEDs for test code and I was expecting that the system idle time would be 99%. With the TzCtrl task and the FreeRTOS Idle task, I was also expecting to have a total of five tasks in my trace. After receiving the trace data, I was shocked to discover several interesting behaviors in my application that I would never have expected.

First, my application did not have five tasks but instead had eight tasks. Second, my application which was only blinking some LEDs was not 99% idle but instead was closer to 80% idle. What on Earth was going on here? This was a very simple application that was using the STM32CubeMx tool to generate some start-up code for the STM32F429I-DISCO board and then I added just a few tasks and the Tracealyzer recorder. Why was my system utilization so bad and where did these extra tasks come from?

Jacob Beningo, ESC, Embedded Systems Conference, Real-Time Systems, software, RTOS, CPU

Figure 1 — This is an example trace taken using Percepio Tracealyzer showing the number of tasks in the application and a brief snippet showing those tasks executing.

The answer to this question has multiple parts. First, behind the scenes, the STM32CubeMx tool created several tasks that were hidden from me as a developer. These tasks included a starter default task which was a high priority, high frequency task that did nothing, and a USB task designed to maintain a USB host connection. These two tasks combined were using nearly 15% CPU utilization and since my application was not using USB, this was completely wasted cycles but also the default behavior for the software. I never added USB support to my application, it was provided by the tool as a default unknown to me.

Second, I found that approximately 3% CPU utilization was being used by a Blinky task that had actually been deleted and should not have been executing at all. I would have thought that a deleted task would be deleted and not use any CPU cycles. The trace showed me that assumption was incorrect. It turns out that on FreeRTOS 9.0.0, a deleted or suspended task will still context switch and execute briefly during every RTOS tick. I never would have expected such behavior in a million years if I had not traced it and seen it with my own eyes.

Measuring, tracing, visualizing, and plotting events and data from embedded software is critical to understanding how the application code is actually executing. Had I just looked at my system, seen the LEDs blinking correctly and declared victory, I never would have known that my software was not executing the way that I had intended. Once I took the measurements though, I could analyze and ask myself the right questions that allowed me to then go in and adjust my code so that I didn’t have extra, unwanted tasks, running amok in my application. Instead, I was able to discover these tasks, remove them, and tune the unexpected behavior from the RTOS I was using. In the end, my base application was executing with 99.7% idle time and was ready for me to start building upon a solid and well-known foundation.

Jacob Beningo, ESC, Embedded Systems Conference, Real-Time Systems, software, RTOS, CPU

Figure 2 — The base application example behavior after it had been optimized and the originally unknown tasks had been removed from the application.

This leads me to ask you the reader, “Do you know how your system is executing”? Do you have hidden and unknown execution behaviors which are slowing down your system and affecting response times? I see teams start implementing debugging techniques and tracing their systems when problems start to arise but isn’t that too late? Start on day one and measure what your system is doing, don’t assume you know what it is doing. By doing this, you’ll find that you have far fewer headaches to deal with and will also be far more efficient in developing your software and avoiding defects.

Jacob Beningo is an embedded software consultant who currently works with clients in more than a dozen countries to dramatically transform their businesses by improving product quality, cost and time to market. He has published more than 200 articles on embedded software development techniques, is a sought-after speaker and technical trainer and holds three degrees which include a Masters of Engineering from the University of Michigan. Feel free to contact him at jacob@beningo.com, at his website www.beningo.com/, and sign-up for his monthly Embedded Bytes Newsletter.

ESC, Embedded Systems ConferenceJoin Jacob Beningo at ESC Boston!
Join Jacob Beningo for three can't-miss sessions at ESC Boston, April 18-19, 2018. He will be on-hand live at the Boston Convention and ExhibitionCenter, where he will be discussing: "Transitioning Embedded Software From C to C++," "Jump Starting Code Development to Minimize Software Bugs," and "Verifying and Debugging Real-Time Systems Using Deep Insight Analysis."

Comments (0)

Please log in or register to post comments.
  • Oldest First
  • Newest First
Loading Comments...