HOME  |  NEWS  |  BLOGS  |  MESSAGES  |  FEATURES  |  VIDEOS  |  WEBINARS  |  INDUSTRIES  |  FOCUS ON FUNDAMENTALS
  |  REGISTER  |  LOGIN  |  HELP
Blogs
Made by Monkeys

Servo Woes Fired Up the Machine

NO RATINGS
View Comments: Threaded|Newest First|Oldest First
Cadman-LT
User Rank
Platinum
Fixing a mistake
Cadman-LT   11/5/2012 2:28:59 PM
NO RATINGS
It always seems like no company wants to admit to, or even correct their mistake. Not a good thing.

Larry M
User Rank
Platinum
Re: Fixing a mistake
Larry M   11/5/2012 3:14:13 PM
Cadman-LT wrote: "It always seems like no company wants to admit to, or even correct their mistake. Not a good thing."

Well, stonewalling usually seems to occur on software and hardware products that have been in production for a long time. The original engineers and designers who understand the design have long since moved on, and the product engineering group is afraid to change anything. They will dig their heels in on the most trivial things to avoid making a change.

Notice that in this instance they wouldn't waive the unncessary resistor or accept the need for change when the multiple-button-press problem was found. Once you get attuned to looking for this behavior you will see it everywhere. <sigh>

Larry M
User Rank
Platinum
Re: Fixing a mistake
Larry M   11/5/2012 3:14:42 PM
Cadman-LT wrote: "It always seems like no company wants to admit to, or even correct their mistake. Not a good thing."

Well, stonewalling usually seems to occur on software and hardware products that have been in production for a long time. The original engineers and designers who understand the design have long since moved on, and the product engineering group is afraid to change anything. They will dig their heels in on the most trivial things to avoid making a change.

Notice that in this instance they wouldn't waive the unncessary resistor or accept the need for change when the multiple-button-press problem was found. Once you get attuned to looking for this behavior you will see it everywhere.

Cadman-LT
User Rank
Platinum
Re: Fixing a mistake
Cadman-LT   11/14/2012 12:21:32 PM
NO RATINGS
Larry, ok, but does that make it ok?

Larry M
User Rank
Platinum
Re: Fixing a mistake
Larry M   11/14/2012 1:28:00 PM
NO RATINGS
Of course not, but the first step in fixing behavioral problems like this is understanding why they occur.

Cadman-LT
User Rank
Platinum
Re: Fixing a mistake
Cadman-LT   2/12/2013 8:29:45 AM
NO RATINGS
That, I can agree with!

Rob Spiegel
User Rank
Blogger
Taking a year for a correction
Rob Spiegel   11/5/2012 2:30:35 PM
Given the potential liability that comes with a machine that catches fire, it's surprising that it took the drive company a year to deliver a complete solution. I would guess they lost some customers along the way.

OLD_CURMUDGEON
User Rank
Platinum
Re: Taking a year for a correction
OLD_CURMUDGEON   11/5/2012 3:15:25 PM
NO RATINGS
It's too bad that the folks who provide these tales don't mention offending companies by name so the rest of us out here in the real world will have some precautionary "ammunition" to avoid them.

Rob Spiegel
User Rank
Blogger
Re: Taking a year for a correction
Rob Spiegel   11/5/2012 4:33:32 PM
NO RATINGS
Hi Old_Curmudgeon. Many of our Made by Monkeys and Sherlock Ohms postings do identify the offending companies. This is especially true with the postings about cars. Sometimes brands are not mentioned because the blogger is concerned about the posting being libelous.

Charles Murray
User Rank
Blogger
Re: Taking a year for a correction
Charles Murray   11/6/2012 7:15:22 PM
NO RATINGS
Hmmm...The drive manufacturer's logic seems to be, "We'll address problems after they happen, not before." Whatever happened to, "an ounce of prevention is worth a pound of cure?"

Rob Spiegel
User Rank
Blogger
Re: Taking a year for a correction
Rob Spiegel   11/6/2012 11:40:34 PM
NO RATINGS
 Good question, Chuck. Some of these problems may begin to be alleviated as simulation software gains traction. Product design can be validated before the product is manufactured and shipped to customers.

MKsteel
User Rank
Iron
Re: Taking a year for a correction
MKsteel   11/7/2012 7:27:04 AM
NO RATINGS
Sometimes, as may be inferred from the start of the article, a company realizes a potential problem and applies a solution to avoid it. Unfortunately, in this case the fix was incompletely thought out and introduced the observed problem. In other cases, I am sure we have all heard of the solution to one problem introducing new failure modes never considered because the no one went back to analyze the altered system. Each of these problems is caused by either people rushing a project out the door before thinking through and verifying that the final proposal satisfies all foreseeable issues. Only the largest companies have the luxury of getting "fresh eyes" on a design before release but the danger is that a single engineer/designer can get "finish line fever" or get to celebrating or even marrying their "brilliant" solution to a problem. In this case, the servo manufacturer did not incompletely analyse their logic for "unanticipated data" AND the resistor was probably not spec'd to handle continuous power anyway.

A year to drag thru a solution is an unbearably long time, but I tend to believe that either the manufacturer of the servo system had to work out internally a "divorce" from their idea or they were afraid lawyers would point to the release of the update as an admission of fault and open them to other claims. Add the cost of releasing an update/retrofit, plus the "loss of reputation" and you can see why they might stonewall for some time. As I said, a year is a long time, especially if they are still producing the offending product model.

I do have a slight issue with the plexi guard over the resistor though. The guard was obviously added in the understanding that the resistor WOULD get objectionably warm and to protect workers from burns, but unless a ventilation method was designed in, heat would not dissipate efficiently. The logic bug in the drive programming was the source of the eventual failure but an enclosing guard probably accentuated it, especially one made of a fusible, flammable material. Another example of incomplete analysis of a fix even if you can't really lay the blame on the guard designer for not realizing that the current would not stop to end further energy input to the resistor.

GeorgeG
User Rank
Platinum
Re: Taking a year for a correction
GeorgeG   11/6/2012 10:17:57 AM
NO RATINGS
That's a prudent policy. Integrators frequently have to employ controls solutions imposed by their customers or even their own supply chain. One must tread very lightly. Also, major suppliers have a mix of good and not so good product. And it's not just drive manufacturers that can be delinquent. The fact that a certifying body will approve products that have functional safety deficiencies is also an issue. Ideally, integrators should be able to select approved product and have no worries assuming they follow the manufacturers instructions but, with servo drives, that is far from the case. One must always RTFM (at least twice). When the problem is systemic, few suppliers want to step up. Based on experience, I'd say truly safe stopping adds approximately $125 to the cost of a servo drive - seems expensive until OSHA catches up with you. However, users aren't entirely blameless: a careful reading of NFPA 70 & 79 and EN 60204-1 will provide good instruction in providing proper and sufficient protection for  motors and drives. Of course, in this specific instance, the user did not follow the supplier's instructions exactly (vis 45 second delay) and executed an incomplete risk analysis (vis prevention of restart and use of flammabile materials in controlgear).

Note, the use of transparent finger guards may be a necessary evil: it is a requirement that identification of devices and terminals by clearly visible and also a requirement that terminals and devices be touch safe. Metal covers typically require duplication of marking and must be polarized so that they can't be installed improperly and must be properly grounded.  Transparent materials have  obvious advantages; however, one must choose the correct material with adequate temperature tolerance and fire retardent properties (these should be standard check boxes in the annotated BOM).                                

KGround
User Rank
Iron
Re: Taking a year for a correction
KGround   11/9/2012 10:06:31 AM
NO RATINGS
A couple of points by way of clarification:

1) The E-stop circuit was fully compliant with good design practice in that it required two operations (release the E-stop button and press the 'reset' button) in order to resume machine operation.   Our design failure here was to have the E-stop button located where it might get accidently operated.

2) I think that some of the manufacturer's reluctance to modify the drive was that they probably had the replacement or upgraded model already pretty far along the development pathway or even in early stages of production. (i.e. they already had a plan to drop that product from their line) In any event the offending model was dropped from their product line no more than another year after the fix was issued.

3) I do not mention the manufacturer because:

a) against my employer's policy for us peons to discuss vendor relations with a specific vendor

b) I happen to like the product of this vendor and find them to be generally producing top quality products and to be very responsive to our needs.  Describing a single foul-up without any of the good experiences we have had with this company might prejudice others against them unfairly.  We continue to use this manufacturer's product with good results.

4) Yes, plexiglas was a bad idea for the guard, and I pointed that out to those responsible for that part of the design when I first noticed its use.   But it is hard to argue that a change is necessary when dozens of machines have been operating with the current design for years and no problems.

This past history of good (or bad) results can be troublesome. For instance, we built a machine a few years back for a Korean company. They have 380V 60Hz power, which is a very unusual standard (normally it would be 50Hz on a 380V system).  At the time we were not very concerned about the line frequency issue as many of our pumps are dual rated 230/460 60Hz or 380/400 50Hz.  But now they want to buy a new machine of very similar design.  In checking the specs on a new blower for this machine the blower manufacturer tells us that this blower absolutely will fail if used on 380V 60Hz. So in checking back with the pump manufacturer, they now say the same thing - 380/50Hz is ok, or 480/60Hz is ok, but their motors will definitely fail if used on 380 60Hz. (This in spite of the fact that almost 3 dozen of these pumps in sizes between fractional and 10 HP have operated on the first machine without problems for several years.) SO, what to do on the new machine? Install transformers or VFDs to correct the voltage or frequency to what the pump manufacturer says is required or build the same machine as before and hope for the best? We may be getting off lucky because the voltage on the first machine is higher than nominal (which is not an unusual occurrance), and we have requested voltage measurements on the supply, but suppose the supply is high enough to let us get by - then what happens in a few years as the load on that substation rises and the supply voltage drops to the nominal value ?   On the other hand if we install transformers or VFDs on the new machine then we leave ourselves open to the customer wanting an explanation of why the two machines are different, which could possibly lead to retrofitting the first machine at our expense.   I have no trouble knowing  what the right thing to do is - build the machine in the best possible way and consequences be damned. But that is a hard sell to management who are counting on saving a lot of engineering cost by releasing a clone of the earlier design, again especially in light of the fact that the first machine which has these 'design defects' has functioned flawlessly up to this point.  

 

 

warren@fourward.com
User Rank
Platinum
Putting off the fix
warren@fourward.com   11/5/2012 7:39:38 PM
NO RATINGS
As we engineers all know, we hate to find out we did something stupid or sloppy- or that something just slipped past us.  But I bet when the design engineers at the company found out what had happened, they were sure anxious to fix it.  But management, who is mostly interested in bottom lines and not quality, probably just tried to sweep it under the rug by offering a few cheap parts as a solution.  It is shame to ruin a reputation over something that could potentially cause a fire or injury and REALLY cost money in the long or short run.  Shame on them taking a year!  (I'm a Texan so I have to add "Bless their hearts.")

Tool_maker
User Rank
Platinum
Re: Putting off the fix
Tool_maker   11/6/2012 1:04:43 PM
NO RATINGS
Warren: I could not agree more, but I think that attitude may be generational. If I put something out in the shop that has an error, I have lost sleep, worked on my own time on kicked myself from one end of the office to the other until the problem was solved.. I have never had a boss who gave me the chewing out as bad as I gave myself. But too many times today there is an, "Oh well. Mistakes happen."

  As far as management goes, I am fortunate to work for a small one-owner company in which the owner came out of the shop and still takes great pride in his work and the things that go out the door, with our name on them. The first thrust is always, "What does it take to fix the problem and satisfy the customer?" After the dust has settled there is plenty of time to afix blame and take permanent corrective action. You are correct about ruining a reputation. It make take years to build, seconds to destroy and decades to recover.

TJ McDermott
User Rank
Blogger
The behavior is by design
TJ McDermott   11/6/2012 12:16:22 AM
If the capacitors do not discharge through the regen resistor, the drive stands a good chance of blowing those capacitors by overcharging them through too-rapid e-stop power-cycling.

A number of drive manufacturers state in their operations manuals that drives should not be power cycled more than once a minute.

Dumping the DC bus through the resistor is a good way to have numerous power cycles in a short period of time.  If the resistor got that hot, it might be undersized for drive, even if the packaging machine doesn't normally use it.  A large regen resistor is normally protected by a large perforated metal enclosure, not plastic.

 

GeorgeG
User Rank
Platinum
Re: The behavior is by design
GeorgeG   11/6/2012 9:40:59 AM
NO RATINGS
Been there, done that. Often, the root cause of this problem is that the safety logic power of the drive is not separately powered from the control power - consequently, its safety logic is flawed. Quite a few drives have this particular feature but quite a few more don't even have an emergency stopping capability with or without an external braking resistor and some have an internal resistor which is adequate for normal operation but burns up during an E-Stop if a correctly sized external resistor is not connected. When you talk to drive manufacturers about functional safety issues, they will often tell you that they are only responsible for electrical safety and that the machine builder is responsible for functional safety, although there is only so much that you can do external to the drive. One big name vendor supplies an add-on  E-Stopping solution which overheats  motor windings necessitating a delayed restart to avoid cooking the motor. Even many drives with a certification, prove to be not certified to machinery or robotic standards, which is more than a little problem. While most machine safety standards require control voltages to be reduced to a nominally safe level after a brief period of time (see EN 60204-1 or NFPA 79), quite a few drives don't do this. In this case, which is not uncommon, the drive manufacturer implements a bargain basement solution. I'm sure most integrators are familiar with reset timers - this makes the servo drive happy ... the end user, not so much. Also, since everything in the control cabinet is supposed to have IP2x or IPxxb enclosure, it's surprising how many drive manufacturers do not supply properly enclosed resistors, as in this case. Obviously, a custom built enclosure is going to be questionable and more expensive than a commercial solution. But there are a few good guys in the business who manage category 1 stops sensibly and who provide properly sized and packaged regen and braking resistors. BTW, any power device should be packaged in materials with a good UL 94 or equivalent flammability rating (this should have been a routine design check).            

gbergman
User Rank
Silver
Machine control design flaw
gbergman   11/6/2012 11:40:16 AM
NO RATINGS
Plain and simple it was a design flaw on your company's part.  E-Stops are exactly that an emergency stop.  If the button is pressed the machine MUST stop.  Not continue running if someone then pulls the button back out.  A Plexiglass guard around the resistor, are you kidding?  Then to blame the drive manufacturer for a poor system design......

Tool_maker
User Rank
Platinum
Re: Machine control design flaw
Tool_maker   12/4/2012 1:04:52 PM
NO RATINGS
@gbergman: If we had a machine that could be restarted by simply pulling out the E-stop butoon, it would be taken offline until maintenance was able to correct the situation. It is reminds me of days of old when you could just pull a gearshift from Park to Drive without depressing the brake.

Partner Zone
More Blogs from Made by Monkeys
Made by Monkeys highlights products that somehow slipped by the QC cops.
Made By Monkeys highlights products that somehow slipped by the QC cops.
Made by Monkeys highlights products that somehow slipped by the QC cops.
Made by Monkeys highlights products that somehow slipped by the QC cops.
Made By Monkeys highlights products that somehow slipped by the QC cops.
Design News Webinar Series
10/7/2014 8:00 a.m. California / 11:00 a.m. New York
9/25/2014 11:00 a.m. California / 2:00 p.m. New York
9/10/2014 11:00 a.m. California / 2:00 p.m. New York
7/23/2014 11:00 a.m. California / 2:00 p.m. New York
Quick Poll
The Continuing Education Center offers engineers an entirely new way to get the education they need to formulate next-generation solutions.
Oct 20 - 24, How to Design & Build an Embedded Web Server: An Embedded TCP/IP Tutorial
SEMESTERS: 1  |  2  |  3  |  4  |  5  |  6


Focus on Fundamentals consists of 45-minute on-line classes that cover a host of technologies. You learn without leaving the comfort of your desk. All classes are taught by subject-matter experts and all are archived. So if you can't attend live, attend at your convenience.
Next Class: 10/28-10/30 11:00 AM
Sponsored by Stratasys
Next Class: 10/28-10/30 2:00 PM
Sponsored by Gates Corporation
Learn More   |   Login   |   Archived Classes
Twitter Feed
Design News Twitter Feed
Like Us on Facebook

Sponsored Content

Technology Marketplace

Copyright © 2014 UBM Canon, A UBM company, All rights reserved. Privacy Policy | Terms of Service