All engineers think they know the truth about embedded product development. It's hard. It's time-consuming. It's expensive.
But that's only part of it. Unfortunately, the hard truth about embedded development needs to absorbed, not intellectually, but viscerally, through late-night work sessions, missed deadlines, and lost sleep. And the hard truth is this: Embedded development is big and slippery, easy to underestimate, and costly enough to break the budgets of the uninitiated.
Experienced embedded developers and consultants know this. They've seen teams work for years on embedded projects that were supposed to take months. They've seen development costs balloon by factors of 10 or more. They've seen whole departments give up on embedded projects that grew to encompass 40 or more engineers and software developers.
Wind River's Platform for Android served in an in-vehicle entertainment system built by Clarion. (Source: Wind River)
"I saw one company that was five years late and $40 million into their design, using a home-brewed operating system and an elaborate new programming language, and I had to recommend that they cancel the project," Jack Ganssle, an embedded consultant and founder of the Ganssle Group, told us. "And when I made that recommendation, the engineers were almost giddy. Their frustration level was so high that they were hoping to hear something like that." Ultimately, the company took Ganssle's advice and threw out the project.
War stories like this one are common, largely because embedded development is filled with pitfalls that confound even the best engineers. And that's especially true for mechanical engineers, who are coming late to the embedded world and often don't have a scrap of programming experience to fall back on.
Great job, Chuck, highlighting the common misperceptions and challenges associated with embedded development. The whole notion that the development effort is predicated more around software design, not hardware design, is a huge cultural barrier for many engineering organizations which may lack a deep pool of expertise in that area and for years, have priortized and emphasized non-software related development.
The tools are also a huge issue. You talk about the need to invest in development tools specifically around writing embedded code. There also needs to be an investment in tools that integrate the embedded development effort with the rest of the product design effort, both mechanical and electrical components. If all the work is done in silos, you can run into intercompatibility problems and design snafus late in the cycle when it is expensive to make changes.
Great article, Chuck. You have hit many of the major pinch points in embedded development.
I like your first point "It's all about the software" and your second - "Software is a relatively new field... and we're really still figuring it out". The difficulty arises from Moore's law -- with every 18-months the capability of each component doubles as we continue to innovate.
Five years late with their product? That's more than three generations out of date before the team launched their first product.
If the development team decided to use traditional "requirements-based" project management, they need to freeze the requirements and build to the original specification. But as most teams find out quickly, even if they don't change requirements for new technology, the requirements change because of unanticipated difficulties and incorrect assumptions made by the original designers.
May I suggest an initial "Zeroth Point" on your list: "0. Use an Agile Software Development Method". Once the team develops using Agile methods, the subsequent components on your list are still important, but can be altered with less difficulty. The smart-phone industry is one of the best examples of successful Agile development. Pushing software updates to the devices, new models with new features arriving often, and using standard ports and interfaces so upgraded devices can be swapped out with relative ease are slowly becoming the norm.
Charles, really good article and lots of good points!!
I know and have been on those projects that seem to keep going and going and begging for someone to put them down. Many good projects have over run budgets or just failed due to poorly written or no requirements or requirement creeping or engineer's free give aways.
If I had a .01 for every time I've heard an engineer say I can do this better and it would be nice to have this extra feature and just write it in without really understanding it is out of scope and most likely budget, I would be a millionaire.
I remember having to fight management to purchase good lab equipment to test our designs. It cost the company more money and the designers unnecessary time due to fighting with management to get what we needed to get the job done.
Very good point, Bill. The smartphone industry HAD to learn agile software development to survive, given their rate of progression of their products. Others need to learn to follow their lead.
Great point about agile development practices, William. I'm hearing more and more about those as I start to dive into cloud-based design tools and mobile apps. It's definitely a mind set change. Hopefully, it's a practice being woven into new curriculum to help engineers get their arms around how to do it adeptly.
Beth: As a former hardware guy, I feel comfortable saying this: Those of us who were educated in mechanical/electrical in the 1980s or earlier are have more trouble bridging that cultural barrier that you refer to. We've been trained to think in terms of hardware, and it's a tough habit to break. Maybe it's a bit of an ego thing, but it's tough to say, "Yes, software is the most important part of our product."
"Software is a relatively new field -- only about 50 or 60 years old -- and we're really still figuring it out,"
I think this is one of the best points in the article. This is a relatively unrecognized fact in the computer world. There is a lot about software that we are still trying to figure out.
ChasChas: It can't really be boiled down any further than to say that writing and de-bugging code is a very slow, tedious, complex process and many products have hundreds of thousands, or even over a million, lines of code. As RogerD accurately points out here, the numbers cited here refer more to larger projects. Still, the stories I've heard seem to indicate that many, many teams don't have a full appreciation for the scale of the software portion, and that misunderstanding (or lack of understanding) gets them into trouble. As for your reference to eccentric behavior by programmers, we'll need some deeper insight from some of our readers on that one.
I went back to my last embedded project and tallied the hours and code, and the result was 50 lines of assembly-level code per 8 hour day, and this includes the algorithm development. This was a small medical device (2100 lines of code not including LUTs) where I did both the H/W and S/W. The S/W was real-time in nature due to a feedback controller.
So I think there are several caveats to the 200 lines per month number. It's no doubt accurate on large projects with a random assortment of programmers, and I find that just the problem of having multiple entities involved bogs the process down. In my example I had intimate knowledge of the H/W since I designed it as well, an advantage never afforded the typical project. But a factor critical to assess is the motivation and technical ability of the individuals involved rather than the use of a fixed line/day estimate; in fact I'd argue that using the smallest (carefully chosen) S/W staff possible has a significant benefit toward minimizing the development time.
I couldn't have said it better. I completely agree that the smallest possible software team will usually minimize development time.
I have seen a one man "team" design, debug, and program a very high performance FPGA/DSP/Microcontroller based motion control and data acquisition system in about a year (hardware and software). I doubt if a whole team could have done it in five years. A government funded team would probably never have finished it. I'm not saying that anybody could have done what he did, but he was the right man for the job, and adding more people to the mix would have only slowed him down.
Unless a software project can be very distinctly divided and conquered, the fewer programmers the better.
To point #1 (all about SW) truer words were never spoken. A recent contract assignment involved placement of a Standardized (COTS) transceiver on a motherboard. One staff meeting discussion entertained the topic of eliminating the COTS transceiver in favor of a direct chipset embedded solution. Easier for the EE's; easier for the ME's. But the SW guys hit the ceiling, citing months of recoding development. All the points of your article are great checkpoints for whole teams and especially program managers to post on their walls for continuous awareness.
One additional thought: when a project takes any COTS module and places it onto a product host PCB, any Agency Approvals (FCC, UL, CE, etc.) are streamlined because the COTS module already "grand-fathers" the host product's approval process. On the contrary, embedding the solution eliminates that short-cut and you must face the full scrutiny of any Agency. Plan on adding at least another 8-12 weeks before approvals are granted.
Great article Charles. I am one of those mechanical types and have very limited experience with software, embedded or otherwise. This field fascinates me and I am certainly appreciative of your article highlighting the difficulties with the technology. Your comments about the time-consuming efforts and costs to produce the code were revelations. Revelations. My experience in programming is with C++, Pascal and Visual Basic, which are basically "learning-types" of software. If I may, I write an educational blog published through WordPress; i.e. www.cielotech/wordpress.com. Would you mind giving me permission to reference your article in an upcoming blog discussing embedded systems? I think my readers would also be fascinated by the subject. Many thanks, Bob J.
"The average programmer writes about 200 lines of code per month. At that rate, a staff of 50 would need 100 months -- more than eight years -- to write a million lines of code."
Knowing the author is simplifying for the sake of brevity, it may not be obvious to some that total lines of 'good' project code per time unit is never linear in the number of people devoted to the task. There is definitely a point of diminishing returns, and a point at which adding more people does the project a disservice by making the overall task unwieldy, if not outright unmanageable. Microsoft used to blame IBM for ruining OS2 because non-technical managers relied on the 'masses of asses' principle as a means of (erroneously) getting it done faster, then ran roughshod over the coders when the simple arithmetic did not prove axiomatic. The people who were a party to the overall 'vision' at the outset can become disconnected from what is actually emerging, as new people are added at the back end to expedite certain tasks or address new requirements. Moreover, the newcomers may have a completely different view of what the goal posts look like. If you start out with a few people who all know 'C' well then, for example, marketing decides the thing needs Android, bringing in Java experts who've never seen a pointer in their life may cause the team to split into two camps, and they may end up competing more often than working together.
Regarding operating systems, I agree they should be avoided wherever prudent and possible. For some projects however, there's no getting away from it. For example, if you're targeting a high end MPU like a Cortex A8 or above, you NEED an OS else you'll get bogged down in the minutiae of writing drivers etc. The first rule of thumb is you should abandon all rules of thumb. The second might be if you're using a MCU like a MSP430 or a Cortex-M0 to M3, you can probably get away without an OS. As stated in the article, concurrency beyond all but the simplest of requirements dictates you need an OS to manage access to inter-process, shared objects.
Thanks to the author for making a software guy feel important for an afternoon. It's time to go home so my teenager can ruthlessly burst that delicate bubble.
A new book by Thomas Edison's great-grandniece takes on the notion that he was a lone-wolf inventor and replaces it with an image of a man who ascribed great value to the ideas of colleagues.
In response to rising interest in autonomous vehicles, the federal government has called upon states not to authorize operation of self-driving cars, except for the purpose of testing.
With LEDs dropping in price virtually every year, automakers have begun employing them, not only on luxury vehicles, but on entry-level models, as well.
Using almost 200 light-emitting diodes in the front and back of the new 2014 CTS, Cadillac designers are showing how LEDs can change the character of a vehicle.
From Dell / Intel® New Paradigms in Design Work Scott Hamilton, vertical market strategist for Dell Precision workstations, 5/2/2013 5
Early in my career, I worked as a draftsman and remember the days of drawing on vellum with numbered pencils and Mylar with plastic lead. This was a fun experience in the sense that I ...
I've been using workstations for more than 10 years and love finding ways to get more performance from my system. With demanding professional applications that require more power each ...
A lasting memory from my first job as an engineer in an auto assembly plant is standing on hard concrete at six in the morning, vending-machine coffee clutched in hand, listening to ...
For industrial control applications, or even a simple assembly line, that machine can go almost 24/7 without a break. But what happens when the task is a little more complex? That’s where the “smart” machine would come in. The smart machine is one that has some simple (or complex in some cases) processing capability to be able to adapt to changing conditions. Such machines are suited for a host of applications, including automotive, aerospace, defense, medical, computers and electronics, telecommunications, consumer goods, and so on. This radio show will show what’s possible with smart machines, and what tradeoffs need to be made to implement such a solution.
To save this item to your list of favorite Design News content so you can find it later in your Profile page, click the "Save It" button next to the item.
If you found this interesting or useful, please use the links to the services below to share it with other readers. You will need a free account with each service to share an item via that service.
Tweet This
23 
