Seems as the world grows more and more digital, the way to diagnose problems is to read a code and check the manual. Which is OK if the manual is useful and/or detailed. It seems that in this instance it was not. I wonder if the control manufacturer was vague on purpose so that the customer has to rely on their service. Or they just like to cut costs and leave the manual writing to a non-technical employee.
As a car guy, the OBD2 diagnostics is useful, but I find the service techs start replacing parts until the error code goes away. In some instances, the error code is unrelated to the actual problem. Change enough parts and the problem is then "fixed".
This just in: Many OEM system builders, e.g. boilers, automobiles & trucks, copier/printers, residential & commercial appliances etc., use components from other manufacturers, e.g. EBM Pabst, one of a couple of blower/fan specialists. True Babel would be every manufacturer trying to reinvent the wheel.
Speaking of OBDII, if your tech is just swapping parts to address OBD codes you need to find a tech who better understands OBDII and how the various systems & components interact; while remarkably effective as engine control systems, (like modern ultra high efficiency boiler systems) they're not trivial systems! The CEL/MIL code(s) is/are only a starting point, systematic diagnosis is also required.
For those interested in OBDII -- MA RMV publishes a quarterly newsletter for inspection stations & OBDII emissions techs w/ some interesting case studies of particularly difficult OBDII cases. http://www.vehicletest.state.ma.us/inspection_newsletter.html
I agree. However, finding good techs that diagnose issues is getting harder as dealerships are about making money. Service is not defined by fixing your vehicle cheap, but by fixing it with a profit!
Myself, I read the codes and diagnose the issue similar to the author of this article. Get the meter out, look up specifications from the OEM of the sensors and actuators, try to figure out why the computer/controller displays an error code. There has been several Made by Monkey posts that seem to share this theme. The science of troubleshooting systems is becoming a lost skill. We read a code, look it up in the book (or online) then proceed to fix the code. Even when we find the bad component, we replace it and all is good. But why did the component fail? I know age, vibration, wear contribute, but what was the failure? I know in this article, the blower was replaced, but why did the blower fail? Electrical surge, worn, vibration (fatigue), design flaw?
By the way, like the link. Another good resource to learn how to troubleshoot more effectively.
I can't disagree with you that there have been and always will be third party assemblies of one sort or another in every complex system. And in and of itself, this is not a problem as long as the assembly inputs and outputs are documented and the narrative for the control system is well defined.
The problem is that neither are commonplace. So now technicians have become board swappers and sensor shotgun experts. And I'll bet at least 1/2 to 3/4 of what they replace was working fine and did not need to be replaced.
While the code explanation may not have included this exact situation, it sounds like it was directing any plug and play service response to replacing the blower. And that was the correct solution, wasn't it? Or did I miss something in the story?
The error code was confusing. The blower appeared to be under some sort of control because when I unplugged the control lines it went to full speed (as it was supposed to). The service technicians wanted to replace both the controller at the blower to the tune of close to $1k. They lacked any diagnostics to isolate which was broken.
The bottom line: was it not working because the controller was defective, a sensor was defective, or the blower was defective? I do not like to replace parts without knowing with some certainty that it was actually defective.
Yes, I know, from a labor and overall cost perspective, given the lack of information on what the embedded system was supposed to do, their approach is cheaper.
Jacob, well at least all the countries involved use the same alphabet. Imagine if there were Asian parts there as well. This is not a hit on Asian parts, just a comment on languages.
Letting an Italian company do the control system seems a little suspect. Oh, well.
In reality, your automobile is very similar to this setup. Most of the major subassemblies are made by different manufacturers. That is why there are so many microprocessors in a car. Since microcontrollers are so inexpensive, each vendor uses their own rather than trying to integrate a software routine into a central control computer, which would be feasible. Thus, even the temperature guage in your car probably has its own microcontroller.
The issue, as you have pointed out is integration. That includes specification as well as testing. Seems it was not done well in this case.
Your remark about Italian controls struck a nerve. Bavelloni makes CNC controlled cutters. The documentation is there but often puzzling and parts supplied from Italty very expensive, even though some are off the shelf items.
The thing that had me most scratching my head was their choice of wire colors.
To most of the world, power wiring that is green implies frame ground. In their machines, it often means the 24 volt supply. There is an obvious world of difference in troubleshooting between the two.
I am no luddite. I design control systems for industrial processes. The thing that got me started down this garden path was because I was busy, and my wife called upon three different, reputable service firms to fix this boiler. The first one simply reset the unit. The second gave me the equivilant of an intelligent shrug, saying that "we don't service these things any more" and the third told me that he really thought I had a bad blower (though he couldn't explain why), but that he really wanted to replace the whole unit to the tune of nearly $11k.
These firms were not populated by fools. They were all knowlegable, reputable, and capable people who wanted to do the right thing. What they lacked is any way to reason through this problem. My primary concern is that with this tower of Babel going on inside this boiler, there may not have been a way that Munchkin themselves could have known that the F13 code documentation was wrong. And it turns out that Munchkin is not alone in this. Many firms build their products out of the pieces and parts from other companies.
There were other hints that the controls board from S.I.T. Controls had been adapted from other projects that didn't really resemble this one. There were mysterious registers with instructions not to change them. There were hints of an external modbus control system, but no indications of what the interface might look like or what each register might do.
This is the hazard of modern controls: unless you know and define exactly what a system is supposed to do at every step of the way and what stimuli they're expecting, there is no way to know that anything you have is in specification or that it is even broken.
In contrast, one of the projects I worked on where I'm employed is a filter backwash system. We have backwash control narratives with 16 steps. We carefully document each of these steps for the operators. If a step gets stuck there will be an alarm and the operators can know exactly how to respond to get things moving again.
The lack of any such documentation in this application shows that this company was too sloppy to care, and they're hardly alone in this. In the last several years I had problems with a dryer, an oven, and a cooktop. The common thread in all of these products is a missing set of documents to explain how the thing is supposed to do its job. It's no wonder modern "technicians" are reduced to board swapping --it's all they can do.
The only reason I was able to fix this boiler was because I reasoned my way through the entire process narrative. In effect I had to reverse engineer this thing. Expecting a technician to spend time doing that is, well, optimistic. Were it not for the motive of saving over $10k, I don't think I would have bothered either.
I'm old enough to recall the days when every radio or TV set had a service manual and a schematic diagram pasted on the inside of the unit. Owners were expected to be able to replace tubes and even conduct minor repairs. We need something similar for control systems, or there will be many more disgusted customers like me. Had I been less aware of this issue, I'd have spent $11k on another boiler with yet another tower of Babel in it. As efficiency demands increase, complexity increases. We need better diagnostics. We can not afford to replace such appliances just because nobody can figure out how they're supposed to work.
Good job figuring this out. To echo another's post, did you do a post mortem on the blower?
When replacing expensive components at home, I always try to fix the old one to keep as a spare to avoid future expense (takes a little extra time, not always successful, but I usually learn something about the equipment I'm maintaining).
As to the tower of babel, remember that this is a global economy. People usually buy strictly on price, globally. This include's "primes" like the boiler nameplate. It's hard for a company to justify the cost of fully understanding all the features of each component, or of writing a well designed troubleshooting guide. If the product outlasts the warantee period, they have an adequate design. Sad, but it's the state of the times (and of the technology).
I really wanted to do a post mortem on the blower, but it was assembled in a way that made removal of the control board very difficult.
To make a long story short, the motor control circuit board actually penetrates inside the motor. The screws holding this circuit board to the motor were obscured by the impeller casing and casing was not possible to remove until the impeller was removed. The impeller was press-fit using components that I did not have the tools to remove.
My best guess is that one or more of the power devices in the H-Bridge shorted out. This might cause the motor to be less controllable, without actually failing.
As an aside, I would like to note that the world's market for power devices has always been fraught with cheap imitations that are not as robust. In earlier days I used to find fake 2N3055-type bipolar power transistors in failed power supplies. From what I've read in various electronics magazines, these bogus devices continue to find their way in to too many distribution channels.
This goes to show just how difficult it can be to build a reliable product. And where profit margins are very narrow, most people simply do not want to be bothered with this level of diligence.
I wonder how many times we have inadvertently installed fake parts into our systems. It is embarrassing to find a good design failing for no good reason. We suspect many things- circuit boards, wiring, etc. without thinking that little IC or transistor could actually have been defective to begin with. Unnerving, at least.
Stephen & GTOLover: While I agree in principle with your comments, I think one factor that has been overlooked in these blog entries as well as other similar ones is that it is extremely probable that the EBM PABST blower was a proprietary unit, and therefore not well documented in the public domain. This is not a new phenomenon. Manufacturers for eons have been coding common parts w/ their own internal numbers forcing the issue of returning to them for replacement items. IF one thinks back the most blatant example of this is the semiconductor application sector. How many common diodes, transistors, integrated circuits have had unique numbers applied? No doubt, countless! For example, I have a bin box full of MOTOROLA TO92 transistors marked "9706". These ARE EXACTLY equivalent to their catalogued part, MPSA64 (P-N-P Darlington). In my lengthy professional career, I've been responsible for the specification of many parts, from capacitors to transformers, to "you name it" that were functionally EXACTLY the same as catalog items, but my employers sought to bolster their Repair Depts. with "proprietary" components. This works especially well when the product that you're marketing is sold throughout the world. By the same token, I'm sure that company executives "overseas" use this "tool" also.
One has no further to look than to NTE Corporation. They've built an extremely successful business by cross referencing & second sourcing a wealth of electronic components, from simple diodes to transistors to integrated circuits to mechanical relays, to ..........
Mr. Brodsky: I wouldn't hold my breath IF I were you for this climate of quick exchange to be surmounted by a more robust policy of providing manuals, troubleshooting guides, etc. I suspect that there has been more than one high-level corporate meeting involving the attorneys urging their companies to go in the opposite direction for fear of some hypothetical liability claim. That, to me, IS the saddest comment of the current state of affairs.
I would have to agree with you Curmudgeon. The only way that such diagrams would be included is by regulation.
I also have to say that there were parts of this Munchkin manual that had to have been written with an attorney breathing down the neck of the technical writer. The first chapters of the installation manual were so littered with bold face large type warnings of dire consequences that it made reading the rest of the manual far more difficult than it had to be.
This is what I call the "Step Ladder Warning Disease." If you plaster too many cautions, warnings, and bold faced admonitions in a manual, few will read it and most of the public will continue to do the stupid things outlined in the warnings. Frankly, I have seen pilot operation handbooks (for aircraft), and even firearm manuals with fewer warnings than this monstrosity.
The solution is to undestand that this is indeed a gas boiler. Yes, natural gas can be explosive. There is also heat and condensate involved here. If an installer didn't know that, it is very unlikely a bold faced warning will teach them anything.
Attorneys need to understand that writing a manual that people will actually read and comprehend is an art form. The more they inject liability disclaimers in to the manual, the less likely it will be that anyone will be motivated to read the silly thing.
Some day I would like to see a conference between technical writers, engineers, and attorneys on how to build a better manual, because the state of the art right now isn't serving anyone well.
As an OLD Motorola alumnus, I'm exceptionally familar with the "M" seires parts. There's a legit reason for this: these parts have their own proprietary specs, and are SELECTIONS from the parent product line. The spec differences are (usually) narrow range of beta, and (often) leakage current, noise figure, etc. The "generic' replacement part (like NTE) will mostly work, but design margins may be violated and therefore performance degraded. In my 13 "batwing" years I probably set up several dozen of these P/Ns myself. So of course if you characterize a specific M9706, it will fit all the parameters of the MPSA64; however, the reverse is NOT true for other MPSA64s! Given your comment on "proprietary unit" I'm surprised that you didn't recognize that those "M" parts were too!
There is another side to this story that has not been brought up. I've worked in various industries for OEM suppliers and been on numerous system integration teams. For at least the past decade I've seen far too many OEM customers go with suppliers who had never worked on any component like that required for the system, and who subsequently screwed up the programs so severely that they had to be delayed (to the point of missing market windows in some cases!) or cancelled. The reason for the bad sourcing decision? Low bidder, without regard for qualifications! Perhaps DN needs a new blog series called "Managed by Morons."
That is far too often the case. There have been a number of threads recently about component substitutions causing failures. I also blame high turnaround. It seems that three years has become the normal tenure, instead of working until retirement. It seems that as soon as a product is out the door, that the product's designers, and all of their knowledge, are out the door too. The next thing you know Purchasing starts to complain about the sourcing on a component, and Senior Management pressures Engineering to accommodate the change without the time (read as cost) required to fully understand the change.
I worked for one organization that allowed Manufacturing Engineering to specify changes on a design without the approval of Product Development. Worse yet, Manufacturing Engineering didn't even have a EE on staff, so they just followed the recommendations of Purchasing. Purchasing even routed contested ECNs during lunch if one of the primary signatures was against their change. I know it sounds like the premise for a sitcom, but they actually made medical devices.
At least your boiler had a manual, although it did not provide the answers you were seeking. I am daily infuriated by CAD programs that come without manuals. These are not cheap give aways like those free things I used to run across, but even thousands per seat does not come with a manual. They come with a web address for a fixed amount of time and then a, charge-by-th-hour service to answer questions I used to be able to find in the operator's manuals that came with AutoCad, or whatever program was the flavor of the day at wherever I was working.
Then add the fact that the help at the other end of the line can neither speak nor understand English.
Robots that walk have come a long way from simple barebones walking machines or pairs of legs without an upper body and head. Much of the research these days focuses on making more humanoid robots. But they are not all created equal.
The IEEE Computer Society has named the top 10 trends for 2014. You can expect the convergence of cloud computing and mobile devices, advances in health care data and devices, as well as privacy issues in social media to make the headlines. And 3D printing came out of nowhere to make a big splash.
For industrial control applications, or even a simple assembly line, that machine can go almost 24/7 without a break. But what happens when the task is a little more complex? That’s where the “smart” machine would come in. The smart machine is one that has some simple (or complex in some cases) processing capability to be able to adapt to changing conditions. Such machines are suited for a host of applications, including automotive, aerospace, defense, medical, computers and electronics, telecommunications, consumer goods, and so on. This discussion will examine what’s possible with smart machines, and what tradeoffs need to be made to implement such a solution.