Yes, Al, it's important for Boeing to regain some confidence that's been lost as a result of this incident. This is going to be watched carefully for some time to come because of lithium-ion's prominent place in the auto industry.
Mr. Sinnitt's over-confidence rather pales in comparison to that shown by the confident second guessing exibited in replies here, by people not nearly as clued in as he is about the issue. Compare air passenger flight deaths/mile to automobile deaths/mile, and this begins to sound like hysteria, which we engineers pride ouselves as free of.
No other of the engineering mistakes mentioned really has much to with this one, the only thing they have in common is hubris. Fortunately, these battery issues came to the attention of Boeng early in the program without any deaths or injuries.
I'm OK with Boeings answers and fixes. They are not betting just the lives of passengers, but also betting the entire company, their livelihoods, their reputations. I'm guessing these were tested six ways to Sunday.
Airbus took technology risks with it's early planes, and also hit some bumps. Any plane is potentially life threatening, even after thousands upon thousands of hours. One can't say that any systems are safe beyond a doubt, only that they are safe beyond reasonable doubt. So what's reasonable? Perhaps that what this discussion should be about.
The problem seem to be that they don't know that each cell is a chemical bomb which ignites when the temperature gets high enough. It supplies its own oxidizer. The cells must be kept below the ignition temperture if they short internally or contained and kept from igniting adjacent cells. Keeping oxygen out may not solve the problem. They may need thermal separators between cells and/or some cell cooling. If their real solution is to just accept some battery meltdowns but contain them to save the plane they should say so. The "failure is impossible" PR does not inspire confidence.
@Ken E.: You'd think that the events of 2007-8 in the financial services industry would serve as a reminder that the fact that a large company is betting a lot of money on something is no guarantee that it is reasonable or safe.
I'm not an expert on battery chemistry, and I don't claim to be. But when Sinnitt claims that the problem has been solved while acknowledging that the root cause hasn't been determined... well, I've heard that before, in other contexts. In fact, I've heard words to that effect coming out of my own mouth in the past.
And I've found, through experience, that they are usually a prelude to failure.
I know Boeing has a lot of smart people, and I hope they've got this problem under control. But Sinnett's statement doesn't inspire a lot of confidence.
Charles noted that Boeing's done several things to prevent POSSIBLE failure modes. Adding insulation can't hurt, I suppose. The addition of locking fasteners implies that they weren't locking fasteners in the original design. I thought aerospace construction was all about locking fasteners, or locking wires on the fasteners?
You're right though, all of the new features of these batteries sounds like bandaids.
According to Snopes.com even as reports of the Titanic disaster began to reach America early in the morning of 15 April 1912, the Vice-President of the White Star Line in New York stated, without qualification, "We place absolute confidence in the Titanic. We believe that the boat is unsinkable."
And 100 years later, we have from Mike Sinnett, Boeing Co. Vice-President and Chief Engineer of the 787 program, "This enclosure keeps us from ever having a fire to begin with. That's the number one job of this enclosure. It eliminates the possibility of fire."
With that, Vice-President Mike Sinnett should turn in any Engineering credentials he claims to have. And to get them back, he needs to take a mandatory High School-level course in Statistics and Probability. This is the same road taken by NASA when the administrators stopped being Scientists and Engineers and turned 100% Politicians. This goes in the same category as Politicians who claim on the stump, "We have passed a law restricting everyone's liberty that will ensure that this tragedy will never happen again."
"History doesn't repeat itself, but it does rhyme." - Mark Twain.
Bill, thank you for making a comment that has made all of us think a little more deeply about this subject. In one short assessment, you've eloquently captured what many engineers felt but struggled to put into words. I first saw your comment on a Saturday morning and couldn't get the words out of head for the next several hours. Your comparison is right on the money, especially topped off by the perfect Mark Twain quote. Very well said.
Thanks for your very kind words, Chuck. It's too bad that my eloquence no longer has any effect on my wife... =]
"NEVER" is a strong pet peeve of mine with my students. Unless they want to hear me drone on about how long "never" and "ever" actually is and how quickly the environment can make their solutions obsolete, they learn to avoid using qualitative speech when quantitative speech is required.
Thanks for the chuckle, Jack. Your use of "ever" in your comment speaks directly to my sense of humor. I have not yet ordered any electrons on the topic of "ever/never", but your comment is strong motivation to grab a soap box and capture my rant. So I guess to answer your question of "Did you ever..." I guess that makes the answer "Yes. But not yet." =]
None of our lith-ion battery packs are for commercially aircraft... but we still have an engineer spending massive amounts of computer time analyzing heat propagation and ways to keep one cell going thermal from taking the rest with it. Design changes that mostly avoid active cooling are being researched.
It's what we will have to rely on until they find a way to have all that energy available per-cell but not have anything flammable in the pack...
@williamweaver made reference to NASA (the way attitudes brought to reality the Columbia accident). Mr. Sinnett in his teleconference proved Boeing is taking exactly the same path: he tries to justify the battery incidents by saying lots of battery incidents have already occurred, that battery incidents are somehow the norm rather than the exception. NASA did the same thing by accepting again and again insulating foam liberation events as normal when in fact ANY foam liberation violated launch specifications and should have been investigated.
Instead, such incidents were accepted as routine up until Columbia was fatally damaged. NASA had a rude reality rubbed in its nose when a projectile test showed exactly what happened to Columbia. almost 2 years of redesign later, to NASA's horror, on the very first return-to-flight launch a very large piece of foam liberated from the external tank. Luckily for them, it happened well after the time in which it could cause damage. NASA thought they had all bases covered, so they had to go back to the drawing boards again.
Mr. Sinnett has set himself and Boeing up for more failure with the words used in his teleconference.
The last two paragraphs in Charles' article are the most disturbing of all. Sinnett acknowledges Boeing does not yet know root cause of the battery incidents and may never know.
I'd like to use expletives in a response to Mr. Sinnett. NASA spent hundreds of millions of dollars in testing and redesign to protect the lives of the approximately 200 astronauts that flew after the Columbia accident. MILLIONS of airline passengers will fly on 787s, and they say they may never know the root cause?
That is unnacceptable. The two battery incidents occurred within weeks of each other. As @williamweaver noted, it is statistically certain it will happen again if they don't discover root cause. It should not be that difficult for Boeing and its battery supplier to run tests until they get a similar failure.
We're putting our lives in the hands of the FAA - let us hope they tell Boeing that the redesign is a good start, but they must find the root cause before the planes carry passengers again.
I live in metro-Seattle; this grounding is only going to hurt the economy here if they can't get the planes back in the air. But I do not want them flying based on what we've learned from Mr. Sinnett's teleconference.
@TJ McDermott: I agree. To say with certainty that you've eliminated a failure mode when you haven't even established the root cause is to invite calamity. On the other hand, if this is the kind of "thinking," or rather lack of thinking, that is going on at Boeing, then it's easier to understand how this problem occurred in the first place... or why the 787 program ended up so far over budget, overweight, and behind schedule. If you respond to every problem you encounter by adding additional features to make your product more "robust" rather than identifying and solving the root cause, you are guaranteed to make things more expensive, heavier, and more complex, but not necesesarily any less failure prone. In fact, the added complexity may make the system more failure prone.
"or rather lack of thinking, that is going on at Boeing" I would add 'management' right after this. Some of the engineers that I have come into contact with at Boeing definitely want to solve this issue and kill it, not cover it and pray!
But as I have said in prior Boeing posts, it will be about economics and politics that cover these choices. Remember Ray Lahood cannot figure out a miniscule budget cut, how is he going to "personnally" review corrective actions!
""or rather lack of thinking, that is going on at Boeing" I would add 'management' right after this."
I agree, but we have to be more speciffic: Avionics team and the full management.
Avionics is the source of heavy project problems and accounts sometimes for up to half of the staff in some companies....remember A380 problems with the cabling which pushed them 2 years off ...
I like this jet and I will fly with it. Component failures occurs alltime in planes- we just don't know about. For example , planes can fly without functional Auxiliary Power Unit for few days. This is the mentality of our time.
I want to add only the words of my professor Petre Augustin ( Aerolasticity):
"Man can work for mony or glory, you as engineers, will work allways for glory!"
No "root cause" = can't even start FMECA. But when there's so much commerce on the line, who's gonna hold the FAA to their OWN rules? I'm guessing in the final analysis the formal procedures will be "expedited" out of the way and once again the dice will be rolled - after all the "sticklers" like us who read this don't have the clout to stop it.
@TJ McDermott and @Dave Palmer, you have resolved a very complex, very expensive situation down to the "root cause"... not dealing with "root cause". Without jumping into a bunch of organizational jargon, this is the root cause of so many failures: We are suffering from a Leadership Vacuum. Managers we have. Managers we need. Managers are trained to enhance their talent in accepting a given system and optimizing the system to minimize risk. Too often that minimization of risk involves transferring the risk out of the boundaries of their system and into the surrounding systems. To TJ's example, foam liberation on the External Tank was minimized to an "acceptable level" as defined by the External Tank Managers until a large chunk of liberated foam was transferred over to the Obiter System. Similarly, the O-Ring sealing problem was managed by the Solid Rocket Booster team until the blow-by was transferred to the External Tank that was attached to Challenger.
"Kicking the Can Down The Road" is too often an acceptable risk-minimization solution utilized by Management in all types of mechanical, electrical, chemical, financial, educational, and political systems. Leadership seeks to find the Root Cause and reduce risk through redesign of the system, not through the bolting-on of contingency components that simply pass along the risk of failure to other components in the system...
You have raised the bar higher now. Leadership has slowed things down and even resulted in huge failures, like the Challenger "O" seal debcale.
I was one of the companies associated with the SRB program and everything was all based on $$$$ and savings. Well; as you are all aware Roger Boisjoly and his engineers gave compeling evidence on why not to launch; but management over rode their evidence; never tried firing an engine in such cold temperatures, so not data on how the "O"-rings would perform- Duh management 101 listen to the experts and not the Bean counters or Washington Politicos/Lobbyists.
The external tank was also an accident waiting to occur and again engineers had some good ideas, such as wrap the fuel tank with a Saran wrap type covering or place a protective burnable shield on the underbelly and wing edges so ice, frost chunks and foam would not do damage; again NO, costs too much.
As Willima stated "Kicking the Can Down The Road" is too often an acceptable risk-minimization solution utilized by Management in all types of mechanical, electrical, chemical, financial, educational, and political systems.
So the ROOT CAUSE ANALYSIS is a must not leave it up to a possible Acturial/Statistical alogorthim for RISK. FORD did on the PINTO and it cost them $1+ biilion in claims, attorney fees and fines etc at the end of the day and for what saving approximately $1.05 per fuel tank by gluing it the under body of the car, where it got ripped off in a rear end accident and spilt all of its fuel- Ford made approx 890,000 + vehicles in the Pinto range, do the math on "Kicking the Can Down The Road".
Great comment William, more people should feel like you and management should be hanging their collective heads in shame.
Well, the recent battery failures, indeed, did not result in an "aircraft fire." So the cocky lead project engineer can dream that with additional improvements they won't ever have a fire.
This reminds me of a statement by a once prominent political leader who in response to a grand jury question said, "It depends on what the meaning of the word 'is' is."
So, what's his definition of a fire? If it is limited to the inside of a battery cell and does not produce an open flame is that not a fire? If it escapes the battery cell but is contained by the case, is it not a fire? Is it a fire only if the aircraft beyond the battery case bursts into flames? Do you require a fire by any definition to cause serious damage or endanger the passengers and crew before you define it as an onboard fire?
Any form of thermal runaway, whether or not it is sustained by combustion is still very dangerous. Even if ignition temperatures are not reached, thermal damage to insulators and structural members, melting for example, might lead to passenger injury or loss of the aircraft.
When events are tracked from raw materials, through details, through subassemblies, through assemblies a Liaison Engineer is likely involved to disposition nonconformances. The Material Reiew Board (MRB) is the organisation which keeps everybody honest. In Duracell literally every issue will have some form of nonconformance....the raw zinc particle distribution may fluctuate...it's written up. The manganese dioxide powder density may fluctuate....it's written up...and so on. Critical battery assemblies follow very narrowly defined Frequency Distribution rules to ensure that indivdual cells provide a balanced output.
One issue which DOESN'T show up is dendrite growth within individual cells since even after a 5 days aging, that event is still microscopic in scale and not detectable within the standard testing procedures. If the root cause of Boeing's problem turns out to be INTERNALLY shorted cells due to dendrite growth all the corrective action featured will not stop the battery from malfunctioning...which begs the question...How vital is the battery system where flight control is concerned? Boeing has committed heavily to electrical systems as a weight saving for the 787 and the battery is part of that...Is there at least triple redundancy in the electrical system? And if so, what part does the battery play in that?
Talk to the Liaison Engineers...they're more involved in the day to day complications which arise during the manufacture of ANYTHING whether it's raw material, individual cells (in this case), assembly problems or even test aberrations.
Your comment is very relevant about material QC and specifications. I have worked with a number of Lithium Battery companies in the US and overseas in providing process mixing systems, all are built with 316 L SS processing parts-even though I pointed out the equipment will shed miniscule particles of the 316 l SS.
Even the BASE Materials used in the batteries are processed within steel contact surface systems, so mininscule metallic particles will always sluogh off as the equipment is used. Even the equipment I have supplied.
Remember 316 L SS is non-magnetic so cannot be removed magnetically, like 4130 or caste iron (used on roll mills-even though some are chromed).
The electrolyte is a liquid dispersion which can be filtered under pressure through an appropriate filter assy down to 0.5 microns and occasionally less for low viscosity fluids (0.02 microns). Some particles have been measured in the 2 to 8 micron range- easier to remove
It appears the electrolyte has been one of the sources of the heating occuring due to the build up of metallic particles agglomerating in sections of the cells. Where did they come from, is obvious the Base Raw Material processing and never considered a problem because no one looked deep enough to find them till these recent problems (particles from 21microns down too 0.75 microns have been observed).
Once agglomerated and held in place by a static load in each particle the battery is now prone to shorting, thus a thermal run occurs. How do they agglomerate, easy the electrolyte is a semi fluid so these particle migrate through gravity, vibration, angular tilt during flight, slight changes in internal viscosity due to heat/load-temperature and internal pressures as the battery is being cycled.
Is their a process method which can eliminate these metallic paricles, YES. But will require a total rethink and retooling from all suppliers of the base raw materials and the process mixing equipment being used to make the final battery assemblies. If these batteries are to continue as a source of power on aircraft
Just like in the electronics industry, "cleanliness is next to godliness" through each and every phase of production. Even a skin particle can ruin a semiconductor component. That why many of there process systems are specialized ceramics, coatings or mills, all the way through the operation.
Boeing have put more than 1,000 personnel on this problem and still do not have a ROOT CAUSE-whoa. Who forgot basic chemistry/engineering 101?
The sketch they have in this article does not even address the real problem and Scott you have placed your finger on it and so has Professor Don Sadoway of MIT.
They can vent for all there worth, but what about keeping it cool or cooler when such a condition arises if they are not willing to go back to the basic chemistry of the battery and clean that up first.
I mentioned above the number of Boeing personnel involved and I have spoken with a long time Boeing person who has some ideas but their are too many now on the table. Nothing at this time can distract them from getting the aircraft back into service and the above is Mike Sinnett's current answer, without a ROOT CAUSE ANLAYSIS at hand!
Maybe Boeing should ask FOR ASSISTANCE outside the company and also the Battery Manufcarurers should also do the same together with the Base Raw Material suppliers.... get clean materials and reduce any problems there and make a better containmant housing.
Some great post heres so I'll try to keep up. If there was only a fire in one of the failures why does it seem thar the only remeady is fire prevention and not heat. It would seem to me these insulators can get just as hot or make heat worse. Why has the battery itself not been look at. I have delt with several re-calls from manufactorer concerning Li ion batteries
Boeing is clearly in full PR mode. What if any statements by the FAA. I thought FAA was working on this problem as well. I would bet the reason they don't know what the original problem is because they were never able to duplicate in the shop, it is very difficult to duplicate real world in the lab so I am not entirely convinced.
There was a time when engineers had a good feel for the existence of unknowns in a design and introduced safety factors to reduce the chance of failure. The Boeing battery and Carnival cruise ships suggest that blind faith in computer outputs has eroded this essential art.
Reminds me of the scene in the "Avengers" movie. The pilot is told to get the flying carrier over water, after one of the engines fails. The pilot said that the navigational computer hasn't re-calibrated yet. Then Nick Fury states the common sense, "Is the sun rising? Turn it towards the sun and get us over water!". The implication that the pilot forgot that the sun always rises from the same direction!
I am as guiltly as any to rely on computer simulations and design theory to tell me everything is good. Sometimes real world common sense has to intervene. Like you stated, we need to get back to expecting the unknown and adding safety factors. I know for airplanes, safety should always trump cost and/or weight.
your goal should be to limit operation to a range you can carefully test for proper behavior. And you instrument the hell out of it to make sure operations stay in this range.
To this end, lowering the maximum voltage and increasing the minimum voltage sound like the most important changes that were made. Better control of operating temperature and other environmental factors is also called for - from what I've read it isn't clear that was done. (it may be because it's infeasible.)
Like all the other commenters here, I'm pretty uncomfortable with the apparent focus on trying to effectively manage a catastrophic failure should it occur. In my experience the sorts of changes you tend to make when you attempt this can actually end up making things worse. The obvious example in the present caseis that more insulation may prevent a fire from spreading but it can also cause more heat buildup.
But the really appalling thing is the overconfident language used in the announcement. It's an indication that the same sorts of problems that led to the Challenger disaster are also in play here. Specifically, this is what happens when there's a long management chain and the engineers at the bottom who are actually doing the work are qualifying their statements in various ways, but as information flows up the chain the qualifications are getting lost and the picture takes on an increasingly rosy glow.
Of course it's entirely possible that the changes Boeing has made will solve the problem. And I certainly don't have sufficient expertise in this specific problem domain to claim otherwise. But having spent 20+ years doing engineering of complex systems, a lot of done inside large organizations, what I see here is very troubling.
So let's see. We put the battery in a fireproof box with no oxygen. So now when it shorts it will get really hot and although it won't actually catch fire, the plane will lose electric power. Since just about everything on the 787 airliner runs on electricity, aren't we still screwed?
I was deeply involved at Lux Aviation, one of the pioneers of "trying to make lithium main ship batteries tamed". During 2008 and 2009 we had discovered the need to separate the cells 30AH Lithium Polymer TRI metal cells (7 in series) to prevent propagation of cells combustion. We determined that we needed a insulated space as wide or wider than the cell itself. This would have distroyed our envelope, not to mention the weight advantage our major aircraft manufacturer was using as the reason for the Lithiumbattery.
Next, we knew through testing that the cells' would produce their own oxygen after being approximately 120C so we tried a unique idea (we thought). We built a sealed case for the battery, then bled the earth's atmosphere out and replace it with pressurized Argon gas (which is inert) and then charged a cell inside the sealed box to point of ignition and as Sandia Labs already know----the damn thing still wooshed (as I call it) to approximately 50-70% of that which we would see in normal earth's atmosphere. Again, no cigar!
We finally came up with a sniffer that would detect the gas from the electrolyte that emitted from the perforated case of the cell, which "always" preceded ignition. This detector was used to immediately stop the charging WHICH WE ALWAYS FOUND AS THE REASON FOR BATTERY DISTRUCTION. We, in over 3 years of battery testing found a single case of a battery taking off in and of itself. This is consistant with what the laptop people and the EV people have experienced. The number we believe is still the "one chance in a million flight hours for cell probability. By the way, my name is on the patent for the "sniffer" as well as (5) other patents that came out of our extensive work in attempting to tame the beast. The trick is in prevention, detection and "redundant sensing having no chance for latent failures.
I think the rule of thumb in the auto industry, tekochip, is to keep the battery between 20% and 80%. Below 20% is too deep a discharge, above 80% risks overcharging. That's why some of the battery packs end up being so big.
The auxiliary power unit battery provides power to start the APU in ground and flight operations. The main battery provides power to start "selected electrical equipment." Boeing says that both batteries are used only for short periods of time to provide power when the engines aren't running. The 787 also has six primary electrical generators.
It does look like the folks at Boeing have come up with a whole lot of things to reduce the risks, with the biggest one being reducing the voltage. It does look like excessive charging is what leads to the worst problems, so perhaps they will be OK now.
As for what to do if the battery pack fails? Many planes can function to some degree using the engine powered alternators, while many others have ab APU, (Auxuilliary Power Unit) to serve as backup for the backups.
And Hey!, we live with dangerous thins all of the time. A race car engine may deliver 1000 horsepower, or more, and sit only inches from the driver. And mostly there is never a problem. And we all hear about it whenever there is a problem, don't we.
I read in AW&ST that Securaplane didn't test their charger with an actual battery. They used a simulated electrical load instead. That's fine IF the real load acts just like that described in the spec from Boeing. However, is was also reported that the actual power discharge of the battery was not at the constant rate which Boeing specified to Securaplane. In fact, the actual load had large changes and reversals in short periods of time. Obviously, the engineers at Securaplane could not be expected to design this charger by "guessing" what the load characteristics were like. I'm disturbed by the lack of information flow here. I bet the engineers at Securaplane were disturbed about this years ago.
What is the "root cause" of the failure which Boeing can learn from? Answer: The root cause is not paying more attention to the technical flow of information between tier 2 and 3 subcontractors. It's not going to happen if it's not contractually required.
Prevention is key and adding weight by embedding the cells in a heavier structure, does not scale well. If the same solution were used for the fuel, the plane might not get off the ground.
Prevention is key and requires mastery of the battery failure mode(s). But that takes time and testing in field conditions and this is seldom cheap or fast.
I am sympathetic to the need to get the planes flying again. But I won't be surprised when a battery problem happens again even if contained. I'll ride in the plane as the other advances are more than enough to mitigate this unsettled battery.
I would like to see more articles about high-energy battery risks and not limited to just lithium based chemistries.
Great idea, Bob. Lithium-ion is so frequently in the news that it's easy and timely to keep coming back to it. But other chemistries -- older and newer -- should be addressed with regard torisk, as well. Thanks.
Sinnett also said that although Oxygen is rereleased if the temperature gets high enough, testing revealed that the amount was insufficient to support combustion for anything but a short puff.
He also contended that neither plane experienced a fire since it was not possible. Further, the fire department report could not substantiate a fire. What was said was that a small area looked white on a thermal imager. It was claimed that the white area in question turned out to be an electrical connector on the outside that did suffer considerable damage.
Basically the whole "Thermal Runaway" issue sounds like a red herring. Too many experts out trying to make a buck if you ask me.
We recently contacted Boeing with details of a system that is able to prevent thermal runaway by monitoring the cells and detecting any physical instability including swelling or ballooning of one or more of the cells in a lithium-ion battery pack. By implementing this means of detection we were able to successfully develop a viable solution that is able to prevent the onset of thermal runaway and combustion before it ever occurs.
Unlike typical battery management systems which at best only monitor voltage, temperature, and current conditions of the battery, this new system is able to detect and protect at a far earlier stage from the potential dangers caused by the volatile nature of lithium-based battery cells by measuring and monitoring any minimal changes, in a 3 dimensional space, the physical dimensions of one or more of the battery cells within the pack. This dimensional deformation which leads to ballooning or swelling of any of the unstable cells within a battery pack is detected before Thermal Runaway and combustion of any cell occurs.
Whether they implement this solution or not is probably down to the accountants!
Using wireless chips and accessories, engineers can now extract data from the unlikeliest of places -- pumps, motors, bridges, conveyors, refineries, cooling towers, parking garages, down-hole drills and just about anything else that can benefit from monitoring.
With strong marketplace demand for qualified engineers across the board that currently outstrips the available supply, there may never be a better time for engineers and project managers to advance their careers and salaries. Whether those moves are successful in the short-term and long-term is likely to depend on how the transition from one job to the next is handled.
Focus on Fundamentals consists of 45-minute on-line classes that cover a host of technologies. You learn without leaving the comfort of your desk. All classes are taught by subject-matter experts and all are archived. So if you can't attend live, attend at your convenience.