HOME  |  NEWS  |  BLOGS  |  MESSAGES  |  FEATURES  |  VIDEOS  |  WEBINARS  |  INDUSTRIES  |  FOCUS ON FUNDAMENTALS
  |  REGISTER  |  LOGIN  |  HELP
Blogs
Sherlock Ohms

The Case of the Dying AA Battery Cells

View Comments: Newest First|Oldest First|Threaded View
tonystewart
User Rank
Iron
Hipot Test Destroyed Power Supplies
tonystewart   7/22/2011 4:55:42 PM
I once designed a product for AVAYA which was a 1U x19" rack with an open framed OEM power supply.  Yields were good at first then later HIPOT testing started failing on random units at a 10% rate, which 10% higher than I expected since the supplier also did this test before shipping.

It supplied power for 16 AVAYA's VOIP phones in a distribution box with 10 watts for each device inline on RJ45 ports which ran at T1 or 1.544Mbps rates.  I was head of design @ C-MAC in Winnipeg, MB, Canada almost  10 yrs ago and responsible for the design of the Unit, Test Process and fault isolation.

I had help with limited resources arround the Corp with C-MAC in Kanata and local sheet metal suppliers doing the sheet metal detail design.  I chose 2 suppliers for 180W 1U open frame PSU's.  We had a contract to supply AVAYA in Denver 10K units in the 1st year and then the Power Supply Unit (PSU) started failing in Hipot testing of the Unit.

After way too many e-mails and phone calls to the supplier, they refused to believe their product could have such a high failure rate. I found out they had transferred the production from Cali. to Mexico and started to investigate myself the Root Cause of the failures.

Not only did the 1.5kV DC Hipot test fail but many units became non-functional after the test!  We started to test the units at Incoming and same result and some passed but were DOA later in FT (functional test).

I figured out what could cause such failures and decided to current limit the HIPOT tester with a high voltage series resistor because, even though it was designed for 2mA current limit, the capacitance in the tester was enough to create high current impulse if low impedance arccing occurred. Normally it is a slow rising outout to 1.5kV for 1 second and duration of 1 second and then shut off.  

THat successfully prevented PSU failures from catastrophic failure of arc discharge but 1.5 mA overcurrent still occurred on 10% of the units.

Then I examined the root cause of the failures and discovered poor assembly process methods in assembly with arcing and I high-lighted all the areas of the PSU where insulation gaps had to increase and insufficient Silicone or PU material had been applied.

THe General Mgr of this major Power Supply company in Cali, COULD NOT BELIEVE that his design team in Cali and the production team in Mexico could not detect the same problems in their facility and was blaming us for breaking his product.

Neither could I.  So I was determined to figure out why there was such a major discrecpancy in yield and catastrophic failure rates.

As a design engineer, I figured out what could cause such failure mode and concluded that they were testing it wrong. WHen I asked how they performed the test with DC high voltage input, it was OK. But when I asked how they terminated the outputs, they replied with outputs loaded but floating, wheras We tested with DC- output grounded open circuit.  

EUREKA.  So I demanded they test the same way our product was being tested  and then they started to see the same problems.... but never apologized.  My reasoning was that the increased insulation on the floating secondary side was enough to make their test pass and our test fail.  It was only a matter of 1 or 2 mm gap increase that made it pass but the design was not being assembled perfectly with enough insulating paste on exposed passive device leads after the production transfer. Details...Details... tisk tisk.  They never did apologize, but I'm sure they learnt their lesson.  Also the polarity of the HIPOT test might affect failure modes since the passive parts were connected to active parts. 

Normally Neutral and Line (hot) are shorted together and the high voltage input is applied between them and the ground wire.  But if the secondary is also floating this means any arccing between primary (AC) and secondary (DC) could still pass if the DC insulation breakdown was enough. But a positive DC Hipot arcing to ground would put a reverse spike on the DC output by means of +ve pulse going from ground to DC- output. 

Now the odds of a high potential spike on the power line are high in Florida, and can be as much as the meter 6KV suppressor, that is usually between Hot and Neutral and less likely between H+N and Ground unless their is a ground fault since Neutral is grounded at the Mains transformer.  HIPOT is intended to be a leakage to ground test. The noise input caps are chosen for 60Hz at the rated voltage to ground to prevent an unsafe leakage current but tested with an inpulse for line ingress on an ungrounded unit. If the DC output is grounded externally in an application for shielding or whatever reasons, then it is essential that it is tested in this manner with DC- output grounded for HIPOT.

Don't forget to current limit your probes just above the test failure threshold to prevent capacitor discharge on short.

How many PSU's HIPOT escapes have you seen?

!♥ Life is just an application of Ω's Law.  If you need design or production help, just ask. ♥!

Rob Spiegel
User Rank
Blogger
Re: THe Case of 70% Arrival Quality on $3K PCA's
Rob Spiegel   7/22/2011 3:53:13 PM
NO RATINGS
Tony, I'd love to use this as a Sherlock entry. If you're willing, please send along a brief bio and we'll get it up in the next two weeks.

rob.spiegel@ubm.com

tonystewart
User Rank
Iron
THe Case of 70% Arrival Quality on $3K PCA's
tonystewart   7/22/2011 3:48:29 PM
As Test Engineering Mgr at Unisys, we put out a lot of production "fires" or line stoppers every week.  One company in NJ designed and produced  the board and then it was transferred to a capable plant in Singapore, after the rights were purchased.  It an 8 channel high speed SMD disk control with dual host control and dual drive string control for redundancy and about the size of a large PC Motherboard but higher chip density and that was the early days of SMD in the '80s.  When we were shipping $100K per day of that product and dead on arrival rate of 30% for that PCA began after the new supplier qualified the process. Our plant manager was worried he might miss shipments for lack of good boards and we took quality very seriously.  The unit had extensive onboard diagnostics with a dedicated CPU and an inch thick of code on printouts to run the diagnostics which was great for detecting errors but difficult to isolate the fault. The faults were random and we could not afford to continue shipping defective boards back to Singapore for rework and continue getting poor arrival quality. So after a week of struggling with solutions, I surmised the issues were insufficient solder under the SMD pads and impossible to find with the naked eye with 10K solder joints and it only took 1 to fail.  THe devices did not always fail at first and some began to fail after a week.  

Our arrival quality on the boards was 70% and 60% on Final Test after a 24 hr burning on the assembled 6 ' x19" rack tower. Crisis time.

Solution: I demanded that the factory implement random vibration testing immediately running self test continuously.  Our Yields dropped from 70% to 10% on the boards and Final Test Yields using good boards then returned to normal high 90's. 

The vibration on the boards was gentle enough not damage good solder joints but fail defective joints.  I don't recall if the fault was solder balls, insufficient solder (micro voids), but these large boards must have experienced enough vibration in transport to cause the failures and once the factory implemented the vibration test, our incoming yields returned to normal high 90's.

HASS or Highy Accelerated Stress Screening is the most effective way to improve product reliability with environmental, electrical and mechanical stresses in an active test. In this case pseudo-random vibration up to 3Khz did the trick using piezo-electric controlled feedback on the fixture.  Other ways also work that are simpler but less accurate.

♥ If anyone needs testability help (DFT or yield improvement), this is one of my expertises after 10yrs in R&D and 10 yrs in Test Engineering and later Eng Mgr for a Contract Mfg later sold to Soletron then Flextronics.♥

Jack Rupert, PE
User Rank
Platinum
Great Analysis
Jack Rupert, PE   6/28/2011 1:53:41 PM
NO RATINGS
Great use of step by step and statistical analysis.  Still, it's a good thing your boss believed you!

Tim
User Rank
Platinum
Replacement Materials
Tim   6/27/2011 7:49:12 PM
NO RATINGS
This was an amazing amount of thorough detail which led to the final culprit.  The replacement cheaper material was the failure.  Sometimes changing vendors is not the best decision for the end customer.

Rob Spiegel
User Rank
Blogger
Re: Good analysis techniques!
Rob Spiegel   6/27/2011 12:18:44 PM
NO RATINGS
Great points, ivank2139.


Do you have any stories (cases) that involve discovering and solving problems during the test process? We would love to include another test story in the Sherlock Ohms blog.

Ivan Kirkpatrick
User Rank
Platinum
Good analysis techniques!
Ivan Kirkpatrick   6/27/2011 12:13:38 PM
NO RATINGS
I appreciate the very thorough and detailed analysis you used to find out what was wrong.  I consider a careful and methodical analysis of the test data the hallmark of good engineering and an absolute imperative when the pressure is on!

I once had a test engineer tell me there was more engineering in testing than in Development.  It took a few years for me to appreciate his point of view.

Another item is to stick with the tests when the schedule pressures start building up.  If one abandons their methodical plan and assumes something is going to go right, well, it might but it is better to have your tests in place so you can prove it and if something does go wrong then your tests will help you troubleshoot it.

This is especially true in Software when sometimes tests and the results are all you have to work with for a while.


 

Partner Zone
More Blogs from Sherlock Ohms
Sherlock Ohms highlights stories told by engineers who have used their deductive reasoning and technical prowess to troubleshoot and solve the most perplexing engineering mysteries.
Sherlock Ohms highlights stories told by engineers who have used their deductive reasoning and technical prowess to troubleshoot and solve the most perplexing engineering mysteries.
Sherlock Ohms highlights stories told by engineers who have used their deductive reasoning and technical prowess to troubleshoot and solve the most perplexing engineering mysteries.
Sherlock Ohms highlights stories told by engineers who have used their deductive reasoning and technical prowess to troubleshoot and solve the most perplexing engineering mysteries.
Sherlock Ohms highlights stories told by engineers who have used their deductive reasoning and technical prowess to troubleshoot and solve the most perplexing engineering mysteries.
Design News Webinar Series
7/23/2014 11:00 a.m. California / 2:00 p.m. New York
7/17/2014 11:00 a.m. California / 2:00 p.m. New York
6/25/2014 11:00 a.m. California / 2:00 p.m. New York
5/13/2014 10:00 a.m. California / 1:00 p.m. New York / 6:00 p.m. London
Quick Poll
The Continuing Education Center offers engineers an entirely new way to get the education they need to formulate next-generation solutions.
Aug 4 - 8, Introduction to Linux Device Drivers
SEMESTERS: 1  |  2  |  3  |  4  |  5  |  6


Focus on Fundamentals consists of 45-minute on-line classes that cover a host of technologies. You learn without leaving the comfort of your desk. All classes are taught by subject-matter experts and all are archived. So if you can't attend live, attend at your convenience.
Next Class: August 12 - 14
Sponsored by igus
Learn More   |   Login   |   Archived Classes
Twitter Feed
Design News Twitter Feed
Like Us on Facebook

Sponsored Content

Technology Marketplace

Copyright © 2014 UBM Canon, A UBM company, All rights reserved. Privacy Policy | Terms of Service