While working on a satellite earth station, we had finished the majority of the commissioning and were doing a shakedown period prior to full operation. The so-called geostationary satellites actually execute a small, daily figure-eight path and slight drift, which is no problem for small dishes on the house for TV, but when you’re using a 90-foot dish to handle huge volumes of telephone traffic, that means the dish has to be in "autotrack mode" 24/7.
We realized over a few days that we kept finding the tracking servo system in "standby mode." Consequently, the satellite would slowly drift out of the narrow beam, and the channel noise alarms would sound. Eventually we realized that the problem happened every day at precisely 11:30 a.m.
After several days of posting engineers at various places to observe what actually happened on the site at 11:30 a.m., someone spotted a momentary blink on a lubrication control panel warning lamp. Tracing the interlock signals then led us to the main oil pressure switch for the elevation lead screw gearbox and a 24-hour time switch set to 11:30 a.m. Once a day, the time switch opened a solenoid valve to spray oil over the exposed surface of the lead screw for a few minutes, and due to rather inept design of the pipework, all the oil drained out during idle periods.
When the solenoid valve opened, the main manifold pressure dropped until the pipes filled up as far as the spray nozzles. The pressure drop tripped the main servo system for an apparent lubrication fault. A swift adjustment of the pressure switch setting and installation of check valves to prevent the oil draining back solved the problem.
This entry was submitted by Rod Hine and edited by Rob Spiegel.
Rod Hine, MA (Cantab) MIET MBCS, graduated from Churchill College in Cambridge, England. He worked in satellite communications, meteorological telecomms, and then general automation, machine tools, and industrial control systems.
Tell us your experience in solving a knotty engineering problem. Send stories to Rob Spiegel for Sherlock Ohms.
Intermittent or momentary faults can be very hard to find. On some machines, where possible, I added a latch to the alarm. Other times, you just had to watch for the error. One of the machines that I worked on had an air blow for spindle cleaning during tool change. If the air pressure dropped below the switch setting the machine Emergency-Stopped on low air. As soon as it stopped and shut off the air blow, the pressure built back up above the switch setting, and the error automatically reset . Machine operators were notorious for not paying attention, so the complaint was usually "the machine just stopped". Often the fix was just replacing the air supply hose with a shorter, larger diameter hose.
GlennA, I was working for a company that made simulators at one time. I was involved in a R&D project to track errors and predict faults. It was very interesting and an early application of AI in a real-time system. We acutally logged every fault, even intermittent ones. I learned about this working on satellites, actually. Having an indicator that just blinked is not really useful, you need to keep that information.
Alarm history is vital when there is more than a handful of alarms to track. On the other hand, having a hundred alarms to sort through can be just as difficult as the intermittent one.
TJ McDermott; Yes, too many alarm messages can also be a problem. The ABB IRB 6400 S4c controller had several message lists, and a general list. And some alarms would generate a secondary message, or even a tertiary message; e.g. on a servo fault, each of the 6 axes would reply with a shutdown message. It took some time to sort through the alarms to get to the primary error.
You have to remember that this was around 1968 and control system design was moving from traditional relay technology to use of early (and expensive!) integrated circuits. It just wasn't possible to latch every one of the hundreds of possible fault conditions and with hindsight, use of hard-wired printed circuit logic with IC's (mostly 7400 family in flat-packs) was smaller but no better than the relay version. Only with the arrival of the microprocessor and true Programmable Logic Controller or PLC would all this be resolved sensibly.
I'm surprised this wasn't discovered during testing of the lubrication system. It's not intermittent, and there wasn't anything odd about 11:30, it's just that the lubrication system caused the machine to fail. It makes me think that the lubrication system was tested outside of the machine, or when it was at idle.
Rod, we had a similar experience while working with the rocket launch division. During the testing phase, when time counter is incremented from 23.59 to 00.00, the vernier engines stops fueling the booster rockets and hence there is change in thrust pressure. We had done all sort of testing to find out why this fuel injection missing happens at that particular time and finally found that a small bug in the Micro controller program. According to the program 00.00 is the resetting cycle, so whenever the clock reaches 00.00, it reads as a reset instruction.
Tekochip, it seems that integration testing had not happened. By component wise all the devices or parts may work fine and by integration testing we can find how the system is behaving in a complex atmosphere or in collaborative mode.
Now you have put your finger on the root of the problem! The gearboxes and lubrication systems had been supplied and installed by a sub-contractor and the only "official" interface to our control system was the oil pressure signal. The timeswitch didn't appear on any of our system diagrams and was fed from a spur from the antenna domestic mains, installed on an ad hoc basis by a sub-contract "sparks" at the request of the gearbox supplier. That's how it got as far as it did - we were as surprised as anyone else when we found it. Later earth-stations projects paid much more attention to such integration issues and consequently the commissioning threw up fewer and fewer surprises. We still had some nasty experiences though, one in particular from an air-conditioning system. But that's another story...
From Dell / Intel® New Paradigms in Design Work Scott Hamilton, vertical market strategist for Dell Precision workstations, 5/2/2013 3
Early in my career, I worked as a draftsman and remember the days of drawing on vellum with numbered pencils and Mylar with plastic lead. This was a fun experience in the sense that I ...
I've been using workstations for more than 10 years and love finding ways to get more performance from my system. With demanding professional applications that require more power each ...
A lasting memory from my first job as an engineer in an auto assembly plant is standing on hard concrete at six in the morning, vending-machine coffee clutched in hand, listening to ...
A quick look into the merger of two powerhouse 3D printing OEMs and the new leader in rapid prototyping solutions, Stratasys. The industrial revolution is now led by 3D printing and engineers are given the opportunity to fully maximize their design capabilities, reduce their time-to-market and functionally test prototypes cheaper, faster and easier. Bruce Bradshaw, Director of Marketing in North America, will explore the large product offering and variety of materials that will help CAD designers articulate their product design with actual, physical prototypes. This broadcast will dive deep into technical information including application specific stories from real world customers and their experiences with 3D printing. 3D Printing is
To save this item to your list of favorite Design News content so you can find it later in your Profile page, click the "Save It" button next to the item.
If you found this interesting or useful, please use the links to the services below to share it with other readers. You will need a free account with each service to share an item via that service.
Tweet This
12 
