What Embedded Developers Need to Know About Open Source

Microsoft's recent acquisition of GitHub, the largest repository of open-source software on the Internet, was only the latest in a trend that has been happening in open source software since about 2016. Specifically, large companies and even governments have begun to embrace open source in their workflows. In 2016, the Obama administration even established a Federal Source Code policy for government agencies to improve access to their source code.

But this new level of adoption also comes with hidden pitfalls. In its 2017 Open Source Support Report, Rogue Wave Software—a provider of software development tools and libraries as well as embedded components—found that in 98% of audits of its clients' software, the source code contained open source code of which the company was not necessarily aware.

While this may not sound immediately problematic, it does carry a host of potential legal and security issues that could come back and haunt developers. With embedded development finally embracing open source, engineers working on embedded applications and devices now have to be wary of these issues as well.

Ahead of his talk at ESC Minneapolis, “Are Open Source Software & Embedded Software Development on a Collision Course,” Rod Cope, CTO of Rogue Wave Software, spoke with Design News about what open source really means for embedded development. He also discussed how open source code ends up in devices without anyone noticing and what every embedded engineer needs to know before bringing open source into a project:

Design News: When we're talking about open source, are there specific or unique considerations for embedded developers?

Rod Cope: There are two main ones. One is related to the legal aspects and the licensing of the open source itself. I've done a lot of talks over the last five or six years to embedded audiences—in particular at ESC and other conferences. And this is one thing that keeps coming up because embedded developers aren't always aware that they're using open source.

In some cases, they say they're not using any open source. Then, we'll ask questions and find out they actually are using Python or BusyBox or something like that and it is open source. Then, developers find out some of the open source code they are using comes with licenses that require them to make the source code publicly available. Or they're compiling or binding or linking with their own code or using things like GPL [General Public License], which would mean they'd have to open source all of their code. So there's just not a great awareness of that. There are some compliance issues certainly and people might run into trouble if they're not aware of how the open source code is licensed and what they need to do to comply.

The other aspect is related to security. I think that's a big one that is becoming more and more popular. People making embedded devices put code on a device, they ship it out, and they think it's a one time effort. They don't think that it has to be maintained in the sense that the software, if it hasn't been updated and patched for security issues, is basically just a huge source of vulnerabilities and could lead to all kinds of problems.

Developers really need to have the mindset that security is not a one time thing. It's an ongoing thing. So how do we offer remote updates or ways to facilitate changes in any device—certainly, ones running Linux or anything sophisticated on them? That's got to be taken into account. But a lot of times, it's not.

DN: So you're saying your company does these audits and you find out that developers aren't aware that they're using open source in some cases. How does that happen?

RC: [laughs] You think it'd be pretty straightforward, right? Somebody had to place the open source in the code base. But a lot of times, developers think, 'Well, I'm just looking to solve the problem, so let me search Google...Okay, here's some code that does what I need. Let me just download that and stick it in.' And a lot of times, developers—especially more junior developers—aren't familiar with open source. They think code is code, and it doesn't matter how they get it—whether they write it themselves or download it.

So a lot of times, the code is grabbed from open source because it's a lot faster than writing it on your own. It already works, has already been tested, and developers don't think about telling anybody. So a lot of times, that will happen across teams—especially in a decent-sized organization, where the developers don't necessarily know all the components that are being brought into a device. Especially if you are looking at something more sophisticated, like if it runs Linux or is a car entertainment system or something like that, a decent amount of software can be convoluted and nobody really looks at it. Before you know it, there are a bunch of different components being put in where only one person knows about each of them. And suddenly, you're shipping something with a lot of open source in it. And some of those licenses can run into trouble, depending on the way you use them.

A lot of times, managers don't really know what's in there either. They kind of ask the engineers and they send out an e-mail and say, 'Hey, please update the spreadsheet with what you're using.' Of course, hardly any developers ever respond to those emails. If they do respond, they respond with whatever they happen to remember. So yes, it is more common than not that we find a lot of stuff that people don't really know that they're using. It's actually surprising when we talk to people and find that they actually knew most of what they have. That's almost never been the case.

DN: Using open source can save a lot of time and energy by keeping developers from having to write all of their code from scratch. But is that really the only advantage of open source? How do you talk to people about other reasons they should be using it?

RC: Well certainly, that's the big advantage. We live in an accelerating world, where getting products out to market faster is becoming more and more important over time. It used to be you had years. Now, you've got months. You really can't afford to write all the code yourself and be competitive anymore, because you need so many features and such a great user experience. You don't have time to write it all. You have to use something. You have to use libraries from third parties. Otherwise, you can't go fast enough. I think there's nothing wrong with that as long as you know you're doing it right.

RELATED ARTICLES:

If you are going to use software, make sure you know how it's licensed and if it's open source. The trick is to find out not just how that one component that you think you're downloading is open source, but all the dependencies that component can drag with it. A lot of times, especially for a significant open-source package, it's kind of like an iceberg: You only see what's above the water. You say, 'I'm downloading this thing, but that requires five others.' And two of those require three other things. And before you know it, you've got this whole pyramid of stuff, and all those things can be licensed differently and have different compliance requirements.

Finally, all those components in that dependency hierarchy also could have security implications. They need to be updated and patched et cetera. Where do you get help and support for all that stuff if you're responsible for a product? You expect your developers to be working on your devices or components or software, not be experts in supporting open source. You've got to make sure you also have that side of the equation taken care of. What if there's another Heartbleed or Shellshock? What do you do?

DN: So then, for developers, should they be educating their engineering teams already in place on these issues? Or is there perhaps a place for teams to start adding individuals who can function as experts on open source?

RC: Certainly, different companies do different things. I think in most cases, it makes more sense to just give the team some basic training and some help as opposed to bringing in dedicated staff. The reason is that it gets too expensive. It's not like you need that expertise every single day.

You have to have simple rules. I always tell people: If you put your open source policy in a Word document, you might as well just hide it and set it on fire because no developer is ever going to look at it. So you've got to have something really simple, like three quick bullet points outlining what is okay, not okay, or should be removed based on licensing.

Keep it simple for developers. If you make it a bunch of hoops they have to jump through, they'll just skip the process, download what they want, and you're back to the initial problem again.

DN: Your report talks about a lot of major companies and even some parts of the government embracing open source in the last few years. What do you attribute this shift to?

RC: I've been doing this for 16 years and I've seen open source go from: 'What it is that? Never heard of it,' to 'Open source is scary and bad. Keep it out or you're fired,' to 'Okay, we'll consider using open source, but we're suspicious,' to 'Open source is okay. In fact, we prefer it.' Even some major Wall Street firms are saying, 'We actually require you to use open source unless you can prove that you need something else instead.'

So the pendulum has swung all the way back the other way. A lot of that is because, with the type of market we talked about earlier, you don't have time to not use open source. You can't write all this code yourself.

The other side of that is open source is just more familiar now. So it's not as scary to legal folks in the corporation and for senior management. We've kind of been through Red Hat Linux, and a lot of the major components out there were safe. Now, it's kind of ubiquitous and we are not as scared.

Open source has really permeated and so embedded is the last frontier, I think. Gartner has said that open source is already in 99 percent of the Fortune or the Global 2000 data centers now.

DN: But with that 99 percent, are they all necessarily compliant and up to date with everything like they should be?

RC: No way. Not even close. But they are using it, right? I guess that's the first step and then you've got to kind of get your house in order.

DN: If embedded is the last frontier for open source, the question you have to ask is: What took embedded so long?

RC: Well, I think it's the legal licensing side again. I think people who didn't know about open source looked at it and got nervous.

There was an incident with Best Buy a number of years ago. They actually had to pull a couple of million dollars' worth of DVD players off the shelves because there was an open source in there that some downstream supplier of a downstream supplier stuck in. Nobody found out about it until after the DVD player had already been shipped, so they had to do a recall. And that's really expensive, right? So I think things like that kind of scared people.

Also, at least in the past, embedded tended to be more about hardware than software. I think that's changing quite a bit. Now, hardware tends to be a little bit more of a commodity and the software is where people differentiate their product in a lot of cases.

DN: Microsoft recently acquired Github. When you see things like this, do you find it encouraging? A lot of people are concerned that we're seeing corporations take over open source, something that runs sort of contradictory to the rebel/outsider spirit around a lot of open source.

RC: I'm not really concerned. It further legitimizes open source and says, 'Yes, this is the way to do software.' We're not going to go back. Open source is not a fad or something. It's not going to go away. This is the future of software.

I think if you talk to younger developers today, they grew up just knowing that open source is software. I don't think it was much of a shock that the commercial entities are getting behind it. Red Hat is doing three billion dollars a year selling their operating system.

DN: Speaking of younger developers: Do you see this as a generational shift because now, you're having more developers coming into the field that are more used to open source?

RC: In general, yes. New developers tend to bring major shifts in paradigms. Older developers tend to like tradition.

DN: In a way, the story of open source is almost the story of the Internet, right? There was a time when open source enthusiasts were rather isolated. But now, the Internet has connected communities and allowed a lot more information (and code) to be easily shared.

RC: I agree. You know, if you talk about the younger developers coming up, a lot of the way they learn about how software works is to just look at open-source projects and see how they're run. These are the kinds of things that you had to learn by word of mouth back in the day or read a manual or a magazine or something. It was very kind of hit or miss. Now, there's a lot of amazing training and best practices available that very quickly get adopted by wide swaths of communities. When somebody comes up with a new way to do something, it spreads like wildfire.

DN: So for all these embedded developers that are finally jumping on the open source train, are there general principles or best practices?

RC: Some of the basics are to first make sure the code you're using actually is open source. There's a lot of code out there that is open source-ish, but if you actually look at the license, it's not. And someone might come back to you asking for money.

The other big point is to realize that bit rot truly is a thing. If you download software, it may be fine today. But if you look at that same software in a year, somehow it's degraded and has become insecure and buggy. This is just the nature of software. So make sure that whatever you pick has a good, thriving community behind it that is responsive to security issues and is actively putting out fixes and patches and has migration paths for you so you're not going to get left behind.

If your code is going into a toy app that you're going to write over the weekend, then support isn't something you necessarily care about. But if your code is being shipped on a million devices or put into something critical, you have to make sure you can get support on a timeline that works for you.

DN: But at the same time, you can't completely lean on the community, right?

RC: The one thing I always try to point out to people is: Once you've downloaded open-source code, whatever you bring in, you own it in the sense that you're responsible for what's put into your embedded device. From your customers' point of view, it came from you regardless of whether there's open source in it or not. You own all of it, and that means you have to proactively keep up with security and fixes and patches and things like that. You can't just kind of sit back and say, 'The community will tell me if something goes wrong.' You're not subscribing to the open source community. You have to proactively reach out and check, so make that part of your process around adopting open source.

*This interview has been edited for clarity.

Chris Wiltz is a senior editor at Design News covering emerging technologies, including VR/AR, AI, and robotics.