The food packing machines we built were designed to include a 12kW servo motor. The drive was equipped with a large external braking resistor that probably was not even required in our application, but the drive manufacturer insisted it could not be eliminated. Some time earlier, a predecessor had realized that these resistors can get hot if they are really used in the application, so they built a Plexiglas guard around the resistor.
Everything went fine for several years until eventually we had several dozen of these machines out in the field. Then, one day, we got the call no machine builder ever wants to hear: "Hey, your machine caught fire last night and burned up." The investigation soon led to the regen resistor and its flammable plastic guard as being a likely suspect for causing the fire. But we had never seen one of these resistors even get warm in operation, and we had had no problems with this servo in years.
The mystery continued until the same customer almost had a repeat of the same event, but the operator managed to kill the power when he started to notice the hot smell of melting Plexiglas in time to prevent the fire. Curiously, even though the plastic guard had melted when the machine was restarted, it ran flawlessly, and extensive testing revealed no bad parts.
Upon further investigation, we discovered that just prior to the event, the operator had mistakenly bumped an E-Stop button, which stopped the machine, and he quickly reset the E-stop circuit and continued to run. It turns out that these drives had always had the same feature, but no one had chanced on just the right sequence of operations to activate it. The drive has large capacitors in it and the manufacturer decided that, since a braking resistor is used with these drives, it would just dump the juice from the capacitors into the braking resistor when the drive was shut down.
A relay connected the braking resistor directly across the drive's DC bus on command from the drive's controller. The controller logic went something like this: “When the main power goes off, connect the braking resistor to the DC bus. Leave it connected until the bus voltage falls to a safe level.” What was lacking was the logic that should have said, “If the main power comes back on, open that relay, even if the DC bus voltage has not dropped to a safe level yet.” Fact is, it never will fall to a safe level with 40A of 480V three-phase power supplying the DC bus.
The drive manufacturer balked at correcting the design defect in the drive, arguing that the drives had been used with no problems for years. Then they suggested that we retrofit every one of these machines in the field with controls that would prevent reset of an E-stop for at least 45 seconds, and they agreed to pay for the parts to do that. Needless to say, this was unacceptable to most everyone. Our customer was a multinational company with an army of lawyers and a collection of color glossy photos of the smoking remnants of their $250k control panel, so about a year later a control revision was issued on the drive.
This entry was submitted by Kim L. Ground and edited by Rob Spiegel.
Tell us your experiences with Monkey-designed products. Send stories to Rob Spiegel for Made by Monkeys.
@gbergman: If we had a machine that could be restarted by simply pulling out the E-stop butoon, it would be taken offline until maintenance was able to correct the situation. It is reminds me of days of old when you could just pull a gearshift from Park to Drive without depressing the brake.
1) The E-stop circuit was fully compliant with good design practice in that it required two operations (release the E-stop button and press the 'reset' button) in order to resume machine operation. Our design failure here was to have the E-stop button located where it might get accidently operated.
2) I think that some of the manufacturer's reluctance to modify the drive was that they probably had the replacement or upgraded model already pretty far along the development pathway or even in early stages of production. (i.e. they already had a plan to drop that product from their line) In any event the offending model was dropped from their product line no more than another year after the fix was issued.
3) I do not mention the manufacturer because:
a) against my employer's policy for us peons to discuss vendor relations with a specific vendor
b) I happen to like the product of this vendor and find them to be generally producing top quality products and to be very responsive to our needs. Describing a single foul-up without any of the good experiences we have had with this company might prejudice others against them unfairly. We continue to use this manufacturer's product with good results.
4) Yes, plexiglas was a bad idea for the guard, and I pointed that out to those responsible for that part of the design when I first noticed its use. But it is hard to argue that a change is necessary when dozens of machines have been operating with the current design for years and no problems.
This past history of good (or bad) results can be troublesome. For instance, we built a machine a few years back for a Korean company. They have 380V 60Hz power, which is a very unusual standard (normally it would be 50Hz on a 380V system). At the time we were not very concerned about the line frequency issue as many of our pumps are dual rated 230/460 60Hz or 380/400 50Hz. But now they want to buy a new machine of very similar design. In checking the specs on a new blower for this machine the blower manufacturer tells us that this blower absolutely will fail if used on 380V 60Hz. So in checking back with the pump manufacturer, they now say the same thing - 380/50Hz is ok, or 480/60Hz is ok, but their motors will definitely fail if used on 380 60Hz. (This in spite of the fact that almost 3 dozen of these pumps in sizes between fractional and 10 HP have operated on the first machine without problems for several years.) SO, what to do on the new machine? Install transformers or VFDs to correct the voltage or frequency to what the pump manufacturer says is required or build the same machine as before and hope for the best? We may be getting off lucky because the voltage on the first machine is higher than nominal (which is not an unusual occurrance), and we have requested voltage measurements on the supply, but suppose the supply is high enough to let us get by - then what happens in a few years as the load on that substation rises and the supply voltage drops to the nominal value ? On the other hand if we install transformers or VFDs on the new machine then we leave ourselves open to the customer wanting an explanation of why the two machines are different, which could possibly lead to retrofitting the first machine at our expense. I have no trouble knowing what the right thing to do is - build the machine in the best possible way and consequences be damned. But that is a hard sell to management who are counting on saving a lot of engineering cost by releasing a clone of the earlier design, again especially in light of the fact that the first machine which has these 'design defects' has functioned flawlessly up to this point.
Sometimes, as may be inferred from the start of the article, a company realizes a potential problem and applies a solution to avoid it. Unfortunately, in this case the fix was incompletely thought out and introduced the observed problem. In other cases, I am sure we have all heard of the solution to one problem introducing new failure modes never considered because the no one went back to analyze the altered system. Each of these problems is caused by either people rushing a project out the door before thinking through and verifying that the final proposal satisfies all foreseeable issues. Only the largest companies have the luxury of getting "fresh eyes" on a design before release but the danger is that a single engineer/designer can get "finish line fever" or get to celebrating or even marrying their "brilliant" solution to a problem. In this case, the servo manufacturer did not incompletely analyse their logic for "unanticipated data" AND the resistor was probably not spec'd to handle continuous power anyway.
A year to drag thru a solution is an unbearably long time, but I tend to believe that either the manufacturer of the servo system had to work out internally a "divorce" from their idea or they were afraid lawyers would point to the release of the update as an admission of fault and open them to other claims. Add the cost of releasing an update/retrofit, plus the "loss of reputation" and you can see why they might stonewall for some time. As I said, a year is a long time, especially if they are still producing the offending product model.
I do have a slight issue with the plexi guard over the resistor though. The guard was obviously added in the understanding that the resistor WOULD get objectionably warm and to protect workers from burns, but unless a ventilation method was designed in, heat would not dissipate efficiently. The logic bug in the drive programming was the source of the eventual failure but an enclosing guard probably accentuated it, especially one made of a fusible, flammable material. Another example of incomplete analysis of a fix even if you can't really lay the blame on the guard designer for not realizing that the current would not stop to end further energy input to the resistor.
Good question, Chuck. Some of these problems may begin to be alleviated as simulation software gains traction. Product design can be validated before the product is manufactured and shipped to customers.
Hmmm...The drive manufacturer's logic seems to be, "We'll address problems after they happen, not before." Whatever happened to, "an ounce of prevention is worth a pound of cure?"
Warren: I could not agree more, but I think that attitude may be generational. If I put something out in the shop that has an error, I have lost sleep, worked on my own time on kicked myself from one end of the office to the other until the problem was solved.. I have never had a boss who gave me the chewing out as bad as I gave myself. But too many times today there is an, "Oh well. Mistakes happen."
As far as management goes, I am fortunate to work for a small one-owner company in which the owner came out of the shop and still takes great pride in his work and the things that go out the door, with our name on them. The first thrust is always, "What does it take to fix the problem and satisfy the customer?" After the dust has settled there is plenty of time to afix blame and take permanent corrective action. You are correct about ruining a reputation. It make take years to build, seconds to destroy and decades to recover.
Plain and simple it was a design flaw on your company's part. E-Stops are exactly that an emergency stop. If the button is pressed the machine MUST stop. Not continue running if someone then pulls the button back out. A Plexiglass guard around the resistor, are you kidding? Then to blame the drive manufacturer for a poor system design......
From Dell / Intel® New Paradigms in Design Work Scott Hamilton, vertical market strategist for Dell Precision workstations, 5/2/2013 3
Early in my career, I worked as a draftsman and remember the days of drawing on vellum with numbered pencils and Mylar with plastic lead. This was a fun experience in the sense that I ...
I've been using workstations for more than 10 years and love finding ways to get more performance from my system. With demanding professional applications that require more power each ...
A lasting memory from my first job as an engineer in an auto assembly plant is standing on hard concrete at six in the morning, vending-machine coffee clutched in hand, listening to ...
A quick look into the merger of two powerhouse 3D printing OEMs and the new leader in rapid prototyping solutions, Stratasys. The industrial revolution is now led by 3D printing and engineers are given the opportunity to fully maximize their design capabilities, reduce their time-to-market and functionally test prototypes cheaper, faster and easier. Bruce Bradshaw, Director of Marketing in North America, will explore the large product offering and variety of materials that will help CAD designers articulate their product design with actual, physical prototypes. This broadcast will dive deep into technical information including application specific stories from real world customers and their experiences with 3D printing. 3D Printing is
To save this item to your list of favorite Design News content so you can find it later in your Profile page, click the "Save It" button next to the item.
If you found this interesting or useful, please use the links to the services below to share it with other readers. You will need a free account with each service to share an item via that service.
Tweet This
6 
