5 Must-Have Tests for Robust Embedded Systems

There are more than 100 different types of tests you could perform on your embedded devices. Here's how to develop your own testing strategy.

Jacob Beningo

June 10, 2024

7 Min Read
Embedded Systems Testing
Figure 1: An example testing pyramid (triangle) for embedded systems. Jacob Beningo

Every developer wants to develop robust systems. No one wants to point to a product that behaves poorly and say, “I built that.” Many factors come into play when designing and implementing a robust embedded system. One less glamorous factor is how you test your software and system. 

In today’s post, we will look at the five must-have tests every embedded system should have.

The Testing Triangle

If you were to quickly survey the internet, you’d quickly find that there are more than 100 different types of tests you can perform on your embedded devices. In fact, there are so many types of tests, you could burn through all your company's resources to test your system and still not be done. 

Obviously, we don’t want that to happen. You must constrain the amount and types of testing that you do on your devices. Creating a testing triangle is one method to ensure you at least get the minimum types of tests you need. 

A testing triangle, also called the test automation pyramid, is a concept used to describe the optimal distribution of different types of automated tests you should have. If you adhere to the testing triangle, then you’ll generally have a well-rounded test strategy. A common example of a testing triangle can be seen above in Figure 1. 

Related:3 Techniques to Simulate Firmware

You’ll notice that this testing triangle has a foundation built on unit testing. It then moves upwards to cover other types of tests, such as integration testing, system testing, and several others that we will discuss shortly. 

While all these tests are common, you may find that there are additional types of tests you need to add to the triangle for your particular device. For example, if you are developing an IoT product, you’d probably want to add security testing.

An important goal of the testing triangle is to automate as many of these tests as possible. A robust embedded system will be continuously tested. While you need to identify the core types of tests in your triangle, you’ll also need to make sure that these tests can be automated as much as possible. 

Let’s take a closer look at some of these tests and see which five tests are absolutely essential for every embedded device.

Must-Have Test #1: Unit Testing

Unit tests form the foundation of any testing strategy. They are designed to test the smallest building blocks of your application. They usually target individual functions and verify that the function under test does what it is supposed. 

As you can imagine, unit tests are critical because if the individual functions and components don’t work as expected, how can you ever expect them to work when integrated with the rest of the system? In fact, if they don’t test at the unit level when you encounter an issue, you’ll have no clue where to start looking for the problem! 

Related:5 Embedded Software Trends to Watch in 2024

Unit tests for embedded devices can be a bit tricky. Embedded software doesn’t just have application code but also code that touches hardware. As a team, you must decide how far you take your unit tests. For example, you have several choices like:

  • Unit test only your application code.

  • Unit test everything but use mocks for hardware interactions.

  • Unit test even on target hardware.

That last one can be a little more difficult, and many developers think it can’t be done. It can; you must put extra work into your test harness to deploy a small number of tests to your hardware and then implement a method for retrieving the results. It’s not trivial, but it can be done. 

You should have unit tests for your application code at a minimum. A good hardware abstraction layer will help determine where the application meets your hardware. 

Must-Have Test #2: Integration Testing

Once you have unit tests that pass and know that your foundation is working as expected, you can start testing those components when they work together. Integration tests combine two or more components together to ensure that they work correctly together. 

Related:5 Tips for Becoming a Successful Embedded Software Architect

Integration tests help you figure out whether there are any strange interactions between your components. You might find that they work well in isolation, but defects could appear when they try to work together. Usually, this is caused by poorly documented requirements or complex interactions that couldn’t be foreseen. 

Unfortunately, integration tests are lacking in the embedded systems industry. Teams have gotten better at unit testing, but they often skip integration and go straight to system testing. Integration testing provides you with an intermediate mechanism to ensure the software is working before you start system-level testing. 

Must-Have Test #3: System Testing

Testing at the system level involves running your embedded device as a fully integrated system to evaluate its compliance with specified requirements. System testing involves both functional and non-functional testing, which aims to verify the system’s behavior, performance, etc. 

It’s important to note that these tests often focus on proving reliability in real-world scenarios and environments. You're not necessarily trying to prove the system works under all conditions, just those conditions in which the device is expected to operate. 

At the system level, there are several ways that you can test the system:

  • Clear box testing

  • Opaque box testing

Clear box testing involves testing an application's internal workings rather than its functionality. The tester selects inputs to exercise paths through the code and determines the appropriate outputs. This type of testing requires knowledge of the system's internal logic. As you can imagine, it requires input from the designers and developers to make the system testing successful. 

Opaque box testing is a software testing method that examines an application's functionality without peering into its internal structures or workings. This type of testing focuses on the software system's input and output and is based on requirements and specifications. Developers are not required to do this type of testing. Instead, a quality assurance team can design and run the tests based on the provided requirements.

The ultimate goal is to prove the system works correctly under the desired environmental and user conditions. 

Must-Have Test #4: Performance Testing

I personally like to list performance testing as a must-have test. If you are designing an embedded product, real-time performance is always important. Can your system meet its deadlines? If you add features in the future, are there enough CPU cycles to handle them? Is your CPU maxed out?

Performance testing is done to determine a system's speed, responsiveness, and stability under a particular workload. I find that performance testing is often neglected. While it’s not as foundational as unit testing, it’s still critical to ensure that the customer has a good experience with your product. 

I highly recommend that teams profile their systems’ performance throughout the development cycle to ensure they are on the right track. If a feature gets added that uses more CPU than expected, it can be optimized quickly rather than at the end of the cycle. 

Must-Have Test #5: User Acceptance Testing

The last must-have test, and perhaps the one that matters the most, is user acceptance testing. User acceptance testing determines whether a system satisfies the business needs and requirements and is typically performed by end-users or clients before the system goes live.

If the customer doesn't buy into the device, then your engineering team doesn’t have a problem, the whole company has a problem! For this reason, it’s a good idea to include user acceptance testing. In fact, I’d highly encourage you to get your customer involved early and often! 

Yes, customers do change their minds and cause some requirements churn and volatility, but at the end of the day, they are the ones we’re trying to make happy. If you can get their feedback throughout the development cycle, the chances of a successful product and launch will dramatically increase.

Taking Your Next Steps

Testing can be a black hole if you aren’t careful. Whether you are fully testing your system today or just trying to add testing, leveraging the testing triangle can help you develop a road map for your tests.

There’s a simple process you can follow from here:

  1. Identify the types of tests you need to succeed.

  2. Create your testing triangle.

  3. Build out the test harnesses you need.

  4. Automate as much of your testing as possible.

  5. Test frequently (and don’t forget about regression testing).

Testing isn’t simple, but if you start moving with these simple steps, you’ll be walking in the right direction. From there, the devil is just in the details. 

About the Author(s)

Jacob Beningo

Jacob Beningo is an embedded software consultant who currently works with clients in more than a dozen countries to dramatically transform their businesses by improving product quality, cost and time to market. He has published more than 300 articles on embedded software development techniques, has published several books, is a sought-after speaker and technical trainer and holds three degrees which include a Masters of Engineering from the University of Michigan.

Sign up for the Design News Daily newsletter.

You May Also Like