We have a Munchkin condensing boiler configured for natural gas that, with routine maintenance, ran reliably for three years. One morning, it stopped heating, and the controller display showed a code of F13.
The code on the display meant that the blower in this boiler wasn't spinning fast enough to safely open the gas valve and begin ignition. The problem was that, according to the diagnostics display on the controller, it actually was spinning at least 200RPM faster than the lowest speed for ignition. If I reset the controller, it would begin ignition, heat the water, satisfy the demand, and then shut down with that mysterious error code again. Sometimes, it would inexplicably shut down before demand was met.
I decided that I needed to make sure all the sensors were working as they should. One by one, I tested them to make sure the controller responded appropriately. The controller reacted pretty much as I would have expected. While doing this, I noticed that many key parts weren't made by Munchkin. The blower was made by EBM Pabst of Germany; the gas solenoid was made by Dungs of Denmark; the controller was made by S.I.T. Controls of Italy; the rest was apparently assembled in the US. This thing reminded me of the Tower of Babel story in the Bible.
EBM Pabst has a website with lots of data about its blowers. I had presumed that the blower was under some sort of control because, when I unplugged it while powered up, it went to full speed. From EBM Pabst, I found out that if I grounded the PWM speed control line, it should stop and go to a standby mode. I tried that, and the blower didn't shut down. Now I had reason to spend hundreds on a new blower. After replacing the blower and checking for leaks, I put it to the test. Everything worked properly again.
Looking back, I have to wonder whether the folks at S.I.T. Controls considered this failure mode. The F13 code was being triggered because the blower speed was free-running, not because it was under-speed, as the error code documentation suggested. The controller also wasn't consistent about why or how it tripped, leaving everyone wondering what the controller was balking at.
I give EBM Pabst low marks for quality control. S.I.T. Controls gets failing marks for erratic firmware behavior, and poor Internet presence, besides. However, Munchkin completely flunked because of its manual.
Despite detailed performance data, the boiler manuals did not have a Control Narrative section. The Control Narrative explains in gory detail exactly what the automation does in response to all stimuli in every automation state that it could have. It is a key deliverable to clients. As a registered controls engineer myself, I wouldn't dream of designing a potentially dangerous device such as a gas boiler without explicit documentation of this sort.
Without that narrative, even experienced, reputable service firms have little idea what the unit is supposed to be doing, why an error code might be there, or what tests one might use to diagnose it. It is not enough to failsafe on devices like this. Customers or their designated repair firms deserve a reasonable explanation of what kept a device from working.
This entry was submitted by Jacob Brodsky and edited by Rob Spiegel.
Tell us your experiences with Monkey-designed products. Send stories to Rob Spiegel for Made by Monkeys.
Seems as the world grows more and more digital, the way to diagnose problems is to read a code and check the manual. Which is OK if the manual is useful and/or detailed. It seems that in this instance it was not. I wonder if the control manufacturer was vague on purpose so that the customer has to rely on their service. Or they just like to cut costs and leave the manual writing to a non-technical employee.
As a car guy, the OBD2 diagnostics is useful, but I find the service techs start replacing parts until the error code goes away. In some instances, the error code is unrelated to the actual problem. Change enough parts and the problem is then "fixed".
Jacob, well at least all the countries involved use the same alphabet. Imagine if there were Asian parts there as well. This is not a hit on Asian parts, just a comment on languages.
Letting an Italian company do the control system seems a little suspect. Oh, well.
In reality, your automobile is very similar to this setup. Most of the major subassemblies are made by different manufacturers. That is why there are so many microprocessors in a car. Since microcontrollers are so inexpensive, each vendor uses their own rather than trying to integrate a software routine into a central control computer, which would be feasible. Thus, even the temperature guage in your car probably has its own microcontroller.
The issue, as you have pointed out is integration. That includes specification as well as testing. Seems it was not done well in this case.
This just in: Many OEM system builders, e.g. boilers, automobiles & trucks, copier/printers, residential & commercial appliances etc., use components from other manufacturers, e.g. EBM Pabst, one of a couple of blower/fan specialists. True Babel would be every manufacturer trying to reinvent the wheel.
Speaking of OBDII, if your tech is just swapping parts to address OBD codes you need to find a tech who better understands OBDII and how the various systems & components interact; while remarkably effective as engine control systems, (like modern ultra high efficiency boiler systems) they're not trivial systems! The CEL/MIL code(s) is/are only a starting point, systematic diagnosis is also required.
For those interested in OBDII -- MA RMV publishes a quarterly newsletter for inspection stations & OBDII emissions techs w/ some interesting case studies of particularly difficult OBDII cases. http://www.vehicletest.state.ma.us/inspection_newsletter.html
I agree. However, finding good techs that diagnose issues is getting harder as dealerships are about making money. Service is not defined by fixing your vehicle cheap, but by fixing it with a profit!
Myself, I read the codes and diagnose the issue similar to the author of this article. Get the meter out, look up specifications from the OEM of the sensors and actuators, try to figure out why the computer/controller displays an error code. There has been several Made by Monkey posts that seem to share this theme. The science of troubleshooting systems is becoming a lost skill. We read a code, look it up in the book (or online) then proceed to fix the code. Even when we find the bad component, we replace it and all is good. But why did the component fail? I know age, vibration, wear contribute, but what was the failure? I know in this article, the blower was replaced, but why did the blower fail? Electrical surge, worn, vibration (fatigue), design flaw?
By the way, like the link. Another good resource to learn how to troubleshoot more effectively.
Sounds like the manual is one of several culprits in this story. I find far too many service manuals are written as an afterthought, as if no one cared in the first place.
I am no luddite. I design control systems for industrial processes. The thing that got me started down this garden path was because I was busy, and my wife called upon three different, reputable service firms to fix this boiler. The first one simply reset the unit. The second gave me the equivilant of an intelligent shrug, saying that "we don't service these things any more" and the third told me that he really thought I had a bad blower (though he couldn't explain why), but that he really wanted to replace the whole unit to the tune of nearly $11k.
These firms were not populated by fools. They were all knowlegable, reputable, and capable people who wanted to do the right thing. What they lacked is any way to reason through this problem. My primary concern is that with this tower of Babel going on inside this boiler, there may not have been a way that Munchkin themselves could have known that the F13 code documentation was wrong. And it turns out that Munchkin is not alone in this. Many firms build their products out of the pieces and parts from other companies.
There were other hints that the controls board from S.I.T. Controls had been adapted from other projects that didn't really resemble this one. There were mysterious registers with instructions not to change them. There were hints of an external modbus control system, but no indications of what the interface might look like or what each register might do.
This is the hazard of modern controls: unless you know and define exactly what a system is supposed to do at every step of the way and what stimuli they're expecting, there is no way to know that anything you have is in specification or that it is even broken.
In contrast, one of the projects I worked on where I'm employed is a filter backwash system. We have backwash control narratives with 16 steps. We carefully document each of these steps for the operators. If a step gets stuck there will be an alarm and the operators can know exactly how to respond to get things moving again.
The lack of any such documentation in this application shows that this company was too sloppy to care, and they're hardly alone in this. In the last several years I had problems with a dryer, an oven, and a cooktop. The common thread in all of these products is a missing set of documents to explain how the thing is supposed to do its job. It's no wonder modern "technicians" are reduced to board swapping --it's all they can do.
The only reason I was able to fix this boiler was because I reasoned my way through the entire process narrative. In effect I had to reverse engineer this thing. Expecting a technician to spend time doing that is, well, optimistic. Were it not for the motive of saving over $10k, I don't think I would have bothered either.
I'm old enough to recall the days when every radio or TV set had a service manual and a schematic diagram pasted on the inside of the unit. Owners were expected to be able to replace tubes and even conduct minor repairs. We need something similar for control systems, or there will be many more disgusted customers like me. Had I been less aware of this issue, I'd have spent $11k on another boiler with yet another tower of Babel in it. As efficiency demands increase, complexity increases. We need better diagnostics. We can not afford to replace such appliances just because nobody can figure out how they're supposed to work.
I can't disagree with you that there have been and always will be third party assemblies of one sort or another in every complex system. And in and of itself, this is not a problem as long as the assembly inputs and outputs are documented and the narrative for the control system is well defined.
The problem is that neither are commonplace. So now technicians have become board swappers and sensor shotgun experts. And I'll bet at least 1/2 to 3/4 of what they replace was working fine and did not need to be replaced.
Your remark about Italian controls struck a nerve. Bavelloni makes CNC controlled cutters. The documentation is there but often puzzling and parts supplied from Italty very expensive, even though some are off the shelf items.
The thing that had me most scratching my head was their choice of wire colors.
To most of the world, power wiring that is green implies frame ground. In their machines, it often means the 24 volt supply. There is an obvious world of difference in troubleshooting between the two.
Stephen & GTOLover: While I agree in principle with your comments, I think one factor that has been overlooked in these blog entries as well as other similar ones is that it is extremely probable that the EBM PABST blower was a proprietary unit, and therefore not well documented in the public domain. This is not a new phenomenon. Manufacturers for eons have been coding common parts w/ their own internal numbers forcing the issue of returning to them for replacement items. IF one thinks back the most blatant example of this is the semiconductor application sector. How many common diodes, transistors, integrated circuits have had unique numbers applied? No doubt, countless! For example, I have a bin box full of MOTOROLA TO92 transistors marked "9706". These ARE EXACTLY equivalent to their catalogued part, MPSA64 (P-N-P Darlington). In my lengthy professional career, I've been responsible for the specification of many parts, from capacitors to transformers, to "you name it" that were functionally EXACTLY the same as catalog items, but my employers sought to bolster their Repair Depts. with "proprietary" components. This works especially well when the product that you're marketing is sold throughout the world. By the same token, I'm sure that company executives "overseas" use this "tool" also.
One has no further to look than to NTE Corporation. They've built an extremely successful business by cross referencing & second sourcing a wealth of electronic components, from simple diodes to transistors to integrated circuits to mechanical relays, to ..........
Mr. Brodsky: I wouldn't hold my breath IF I were you for this climate of quick exchange to be surmounted by a more robust policy of providing manuals, troubleshooting guides, etc. I suspect that there has been more than one high-level corporate meeting involving the attorneys urging their companies to go in the opposite direction for fear of some hypothetical liability claim. That, to me, IS the saddest comment of the current state of affairs.
While the code explanation may not have included this exact situation, it sounds like it was directing any plug and play service response to replacing the blower. And that was the correct solution, wasn't it? Or did I miss something in the story?
From Dell / Intel® New Paradigms in Design Work Scott Hamilton, vertical market strategist for Dell Precision workstations, 5/2/2013 3
Early in my career, I worked as a draftsman and remember the days of drawing on vellum with numbered pencils and Mylar with plastic lead. This was a fun experience in the sense that I ...
I've been using workstations for more than 10 years and love finding ways to get more performance from my system. With demanding professional applications that require more power each ...
A lasting memory from my first job as an engineer in an auto assembly plant is standing on hard concrete at six in the morning, vending-machine coffee clutched in hand, listening to ...
A quick look into the merger of two powerhouse 3D printing OEMs and the new leader in rapid prototyping solutions, Stratasys. The industrial revolution is now led by 3D printing and engineers are given the opportunity to fully maximize their design capabilities, reduce their time-to-market and functionally test prototypes cheaper, faster and easier. Bruce Bradshaw, Director of Marketing in North America, will explore the large product offering and variety of materials that will help CAD designers articulate their product design with actual, physical prototypes. This broadcast will dive deep into technical information including application specific stories from real world customers and their experiences with 3D printing. 3D Printing is
To save this item to your list of favorite Design News content so you can find it later in your Profile page, click the "Save It" button next to the item.
If you found this interesting or useful, please use the links to the services below to share it with other readers. You will need a free account with each service to share an item via that service.
Tweet This
6 
