Fixing New Orleans’ Flood Pumps

A lot of ink has been spilled over the huge hydraulic pumps that now contribute to New Orleans’ flood protection system. And rarely do the newspaper stories and blog posts fail to describe these pumps as “defective” and lay much of the blame on the pumps’ supplier, Moving Water Industries (MWI), and the U.S. Army Corps of Engineers. Were the pumps really flawed or did MWI and the Corps get a bad rap?

The answer isn’t as straightforward as the mainstream media accounts and blogs would have you believe. And it’s an answer that involves an untold story about the unexpected engineering consequences that can arise when engineering organizations have to design, build and deliver complex equipment in a hurry.

MWI’s axial flow hydraulic pumps, 40 in all, work in conjunction with flood gates put in place by the U.S. Army Corps of Engineers in the wake of Hurricane Katrina. Installed where the city’s three drainage canals meet Lake Pontchartrain, the gates swing closed during storms to prevent the lake’s storm surge from entering the city. But the closed gates also prevent rainwater in the canals from emptying into the lake. The pumps are supposed to move the water from the drainage canals into the lake, bypassing the closed flood gates. With their 60-inch impellers, these massive pumps need to move a lot of water: they have design flows of 105,000 GPM at 8.5 ft total dynamic head (TDH), 98,000 GPM at 12.2 ft TDH and 85,000 GPM at 16.8 ft TDH.

Word that the pumps might not work first started to spread in March 2007, when a May 2006 memo written by Corps engineer Maria Garzino surfaced on Fix The Pumps, a blog written by New Orleans mechanical engineer and Corps critic Matt McBride. Garzino’s memo warned of design and manufacturing flaws that could cause the pumps to fail and it triggered a wave of negative publicity for both MWI and the Corps. Making matters worse, newspaper articles and blog posts about the pumps also raised allegations that MWI won the $26.6 million pump contract because of political connections. It turns out MWI founder David Eller and Florida Gov. Jeb Bush had been business partners at one time.

Since March, new information has emerged that puts MWI in a different light. Some of it comes from the Government Accountability Office (GAO). In a report issued in May 2007, the GAO concludes MWI was “in the strongest competitive position to receive the contract for the pumping systems.” And the GAO report notes MWI was, in fact, the only bidder who had actually built such a big pump of this type.

But the GAO didn’t let MWI entirely off the hook. Its report details performance deficiencies that were ultimately fixed after delivery of the pumps. Likewise, the Army Corps of Engineers, which declined requests for an interview, conducted an internal technical review of the pumps in June 2007. Like the GAO account, the review highlights a collection of technical deficiencies. But in a cover memo that accompanied the internal technical review, Brig. Gen. Robert Crear, commander of the Corps district that includes New Orleans, said any problems had been fixed since the initial installation in June 2006 and the pumps were all “working well” as of May 2007.

Until now, MWI has made few public comments about the pump problems. “All the negative publicity was a nightmare, but we were involved in an on-going contract with a good customer, so we didn’t feel we could comment,” says Dana Eller, MWI’s executive vice president and an agricultural engineer who worked on the New Orleans pump project.

With the GAO report and Corps’ technical review now publicly available, Eller and MWI professional engineer Jim Endres provided Design News with new details about hydraulic pulsations that initially plagued the pumps and cast doubt on their soundness. They also explained MWI’s position on a controversial pump design change, one involving the design of the intake on the drive unit’s hydraulic pumps. Eller also made a strong case, in interviews and in a just-released 140-page technical document he sent to the Corps, that both the Garzino memo and the technical review contain information that just doesn’t hold water from an engineering standpoint.

Pulsating Pumps

The only technical problem with the pumps the MWI does not dispute involves hydraulic pulsations that cropped up only after the pumps had been installed in the field — but not during factory testing. These hydraulic pulsations climbed as high 500 psi, according to measurements cited by the Corps’ three-engineer internal technical review team. “This could have led to failure of the equipment,” their report states, after noting that normal pressure fluctuations in a system of this type range from about 100-200 psi.

The source of the fluctuations at first baffled both MWI and the Corps engineers, according to Eller. “It affected some pumps but not others. Sometimes it would come and go,” he says. “It was a real engineering mystery.” But it’s ultimately one they solved. Eller says some of the early speculation about these high-amplitude pulsations revolved around a “magnification effect” caused by hydraulic fluid flowing through the roughly 300 ft of piping that connect the pumps with their drive units. Or as Endres puts it, “the long plumbing lengths and mass of oil can magnify the pulsation beyond acceptable levels much like a trumpet magnifies sound.”

To find the source of the vibrations, MWI and the Corps looked at a huge variety of design factors. Endres lists the size of the hydraulic fluid tank, the length and diameter of the steel plumbing, the inlet suction strainers, the inside finish of hydraulic components, oil cleanliness, choice of hydraulic motor, crosstalk between pumps and motors and choice of relief valve as just some of the items investigated. “Neither the components nor the system as a whole was found to be deficient,” he says.

What all the analysis work did reveal, however, was a problem with the Rineer hydraulic motors used to drive the pump impellers. “Frequency scans showed pulsing frequencies predominately at or near the speed of the hydraulic motor-water pump shaft,” Endres recalls. “Although nothing was ever clear cut, the analysis began to focus on the hydraulic motors.”

Ultimately, the hydraulic motors required a relatively simple fix. A rebuild gave them stiffer rotor vane springs–128 per motor — and the pulsations simply went away. The spring upgrade in essence transformed the original motors, which were known as Type 61 motors, into a Type 62 motors that have a higher hydraulic pressure rating. After field testing of the upgraded motors, the Corps “determined that the hydraulic pulsation has been eliminated by replacing the internal working springs.” And Eller notes the new springs fit inside the same recess as the original springs, “so the upgrade was not particularly difficult.”

So why didn’t the original springs hold up? Both MWI and the Corps attribute the motor difficulties to unexpected over-torque conditions. In their report, the Corps investigators lay some of the blame with MWI, claiming they picked the Type 61 pump based on its design of previous large pumps that operated at lower head conditions and operating pressures. The Corps investigators note the motors chosen by MWI required 4325 ft-lb of torque with TDH at 18.2 ft. And that torque is close to the design limit of the Type 61 motor. “The selected equipment has very little safety margin and this could have been the cause of hydraulic pulsations and vibrations,” the investigators conclude.

Eller and Endres don’t dispute that the motors experienced excessive torque, but they argue the root cause can be found in the difference between the Corps’ design specifications and the field conditions out on the canals. They point out that MWI designed its pumps, as spelled out in the contract with the Corps, to start at a canal elevation of 4.0 ft and shut off at 2.0 ft. These elevations reflect flood conditions in the canals with the gates closed. And they point out the pumps were also designed with a higher TDH value than the Corps specifications called for — to account for higher static head that the pump would temporarily experience during the priming of a siphon on its discharge side. “Although this priming condition was not listed in the contract specifications, MWI prepared for it by using priming as the design maximum head and power criteria rather than the lower values listed in the specifications,” Endres says, noting the maximum design head for the pumps reaches 18.6 ft during priming and drops to a continuous 16.8 ft after priming. “After the pumps were delivered,” he says, “the Corps decided they needed to start the pumps near 0 ft canal elevation,” an elevation that represents non-flood conditions.

A few feet might not seem like a lot, but they makes a big difference in pump power and torque. That elevation change increases the priming head to 22.6 ft and translates to an increase in water horsepower of more than 21 percent (WHP= (Q gpm)(H ft)/3960). “This is a corresponding increase in engine power of well over 21 percent. And since everything had already been designed to provide as much water as possible for the original condi-tions, this change would severely overload the equipment,” Endres explains.

More Deficiencies?

Both the Garzino memo and the Corps internal technical review paint an unflattering portrait of MWI. The Garzino memo, which last month triggered a whistle-blower investigation by the U.S. Office of Special Counsel, details catastrophic failures of the pump’s hydraulic systems during tests at MWI’s factory. And the Corps’ internal review dings MWI too, citing problems with testing procedures and the hydraulic components chosen for the pumps. The review also states that actual pump capacity in field tests fell short of the design capacity by 2.5 to 4.6 percent.

The Corps, which withheld some of MWI’s payment, also required some remedial work on the pumps. This work included flipping the discharge elbow to make priming easier, rewelding some piping, replacing flexible hydraulic hoses and galvanized quick connects with rigid piping to prevent corrosion and replacing problematic cams inside the drive units’ Parker Hannifin Denison oil pumps.

Eller’s 140-page response to the Corps’ review makes a credible case that, with the exception of the very real pulsations, the pump problems have largely been overblown. And he characterizes some of the reported problems with hydraulic components as “minor.”

Engineers who have built hydraulic equipment might be inclined to agree with Eller’s assessment. Indeed, the biggest challenge that MWI and the Corps had to overcome does not appear to be deficient pump components but rather a deficiency in lead time. “People forget that this was an emergency project,” says Eller, who notes MWI delivered the first of its pumps in just 104 days. Custom-built pumps of this size typically take 1.5 years to design and build, while smaller pumps could take eight to 16 months. And in his June 2007 memo, Brig. Gen. Crear notes the urgency of protecting New Orleans after Katrina drove the Corps to get the huge pump stations up and running in as little six months, a feat that would normally take three to five years.

To produce the pumps so quickly, MWI added 70 skilled workers and switched to round-the-clock manufacturing. Eller says MWI also had about 15 engineers working on the project at any given time — including not just Eller but also his brother and sister.

Yet, even with all the extra design and manufacturing capacity, the lead times did affect MWI and the Corps by drastically compressing the amount of time available to design, test and troubleshoot these pumping systems. Getting pump capacity in place for the 2006 hurricane season required the Corps and MWI to test pumps in the field. “Meanwhile, our team of engineers worked with the manufacturer in the factory to adjust, retro-fit, improve the pumps in actual field conditions daily to assure that the pumps reached the required level of reliability,” according to Crear’s June 2007 memo.

Eller maintains the component failures and design issues would have been identified and remedied by MWI’s normal design optimization, testing and quality control practices given normal lead times. “Unfortunately, nothing about this project was normal,” he says.