Several years ago, I was working on a project for a client that involved an embedded Linux machine. I was planning to travel over the weekend, but given the project timeline it was critical that progress still be made. The prototype device would not travel well, so we decided to connect it to the internet so that I could easily access it remotely.

Friday evening the system was connected to the web, and when I returned Sunday evening I discovered that Chinese hackers had managed access to the system and had added it to a botnet. The meantime to compromise (MTTC) for this system turned out to be relatively short – less than 48 hours. While the system was as easily recovered as it was hacked, there are several points in this little incident that I think can provide extremely good lessons for embedded systems developers. 

Lesson #1 – Know your system's meantime to compromise (MTTC)

Developers are probably familiar with common terms such as meantime to failure (MTTF) or meantime to repair (MTTR), but system security can also be looked at from a meantime to compromise (MTTC) perspective. (I’ve also heard this referred to as meantime to breach, or MTTB).

MTTC can be defined as the average time it takes an attacker to breach, or gain access, to the system and obtain the desired assets. Those assets could be data, or they could the system's firmware or the ability to remotely control the device.

Obviously, the goal is to make the MTTC large enough that would be attackers decide that the time and resources required to breach the system are not worth the payout, and therefore their time and effort. This requires that developers understand three key points:

In fact, you’ll find that these three points are included in the Arm Platform Security Architecture (PSA) and in documents that describe how to perform a security threats analysis.

Developers also need to ensure that they balance the MTTC to adequately protect their system and clients as well as the development costs and schedules.  

Lesson #2 – Always have secondary security measures

Considerable effort can be used to try to maximize a system's MTTC, but embedded systems, especially IoT connected systems, face an interesting challenge. They may be connected to the internet indefinitely, which means that even if the MTTC is very large, say three years, a system may still be vulnerable because it may be in use for a decade or more and can be attacked 24/7 using automation.

In order to try and keep the MTTC for connected devices reasonable, it may make sense to have secondary security measures in place, as is often done for physical security. For example, a physical safe is often rated based on how long it would take someone with knowledge of the safe to successfully breach it. You may find that on a home fire safe for important documents, the ratings are anywhere from a few hours down to about 15 minutes. Even sophisticated safes may not have ratings beyond several hours. This is why there are often secondary security measures such as security cameras or security guards, who make their rounds in timeframes that are less than the rated breach time. 

The same idea can be applied to an embedded system. If you understand your system's MTTC, you can place security watchdogs that periodically scrub memory, restart the system, or perform other activities designed to thwart any software that has started to find its way through the system. This is by no means a new concept, many space systems clients I work with will often have an automated hardware reset in order to ensure long lasting reliability and recover the spacecraft in the event that it has locked up or experienced some unexpected event.

RELATED ARTICLES:

Lesson #3 – Hackers are always faster than you think

Right before we placed our Linux system on the internet, if you had asked any of us how long it would take for the system to be breached, no one would have given an answer less than 48 hours. In fact, I know that we all thought exposing the system for 48 hours would be fine. With so many systems on the internet, how could this one little node be found amongst the billions of devices in 48 hours? Even if it was found, surely it would still take considerable time to apply known exploits and access the system.

This was all optimistic thinking!

There are programs running 24/7 that are probing, searching, and testing the defenses of systems looking for vulnerabilities. It can be quite enlightening to examine your home or businesses router logs and statistics. You’ll undoubtedly find several attacks per day that are thwarted or blocked. Hackers have the tools to find your devices quickly, and if you have not followed security best practices when you developed your system, your data, your customers data, and your intellectual property will be at risk. As an optimistic engineer out to change the world, you can’t be optimistic when it comes to protecting your embedded systems.

Conclusions

There are so many companies that are now connecting their products to the internet and, unfortunately, there are many that are taking an optimistic approach to security. Companies often rush to market, leaving security as something that can be added later. It’s critical that developers understand their system's MTTC. In fact, it would be fantastic if this was something that had to be advertised on the box of every product. Wouldn’t you be interested to know what the MTTC is on that new smart lock you just bought and assume is securing your home?

The only way we are going to usher in a secure IoT is to develop our security strategies at the start of our development efforts, use isolation to partition our software, understand our MTTC, and put in place secondary security measures designed to augment our first lines of defense. Yes, it will require more time and money, but isn’t the effort worth protecting your customers and your own intellectual property?

Jacob Beningo is an embedded software consultant who currently works with clients in more than a dozen countries to dramatically transform their businesses by improving product quality, cost and time to market. He has published more than 200 articles on embedded software development techniques, is a sought-after speaker and technical trainer, and holds three degrees which include a Masters of Engineering from the University of Michigan. Feel free to contact him at [email protected], at his website, and sign-up for his monthly Embedded Bytes Newsletter.