Even for a lunch hour, the development lab was unusually quiet. The systems we had been working on for most of the last year sat wrapped on pallets waiting to be shipped. Their cooling fans and hard drives were silent. The test equipment, computers. and other gear filling the lab were all dark. The HVAC system, relieved of its cooling burden, had shutdown. Even our technician, who would normally be eating lunch at his workbench while listening to the radio, had decided to dine out.
The small custom engineering group I worked for was a couple of days away from delivering a new network monitoring system for a major telecommunications carrier. It was the early '90s and our customer wanted to improve the efficiency and reliability of their network in what had become a very competitive market for telecommunication services. Part of that strategy was to unify and centralize the monitoring and control of the transmission equipment which made up their network. Our systems, composed of VME chassis stuffed with various processor cards, were to be located in the customer's network switch centers at the end of the transmission links. These devices would convert the various proprietary management protocols used by the transmission equipment into a standard form for the higher level network management system.
I was in the lab that noontime testing some final tweaks to what had been a late addition to the initial product delivery. To allow the local technicians at the switch centers to monitor and control the transmission equipment, our systems provided a "craft" terminal interface. Our customer wanted this terminal capability extended to the repeater or regenerator stations along the transmission links. This feature had been scheduled for a later release but, with only a few months to go, the customer wanted it included in the initial delivery. This new deadline required an accelerated development effort for the remote terminal hardware and associated software.
Usually, a field service technician at a regenerator station would plug a handset into a jack on the transmission equipment to talk with a technician sitting at the maintenance terminal of the end-point equipment. Our group developed an approach to add a "craft" terminal capability to the regenerator stations by using multi-drop modems connected to this voice interface. The field terminals were based on a ruggedized PC-AT module, developed for an industrial control product, with a new mezzanine card holding the video monitor interface and the serial interface to the modem. To support multiple terminals and assure reliable communication, the High-Level Data Link Control (HDLC) serial protocol was used between the remote terminals and the end-point management system. HDLC is a frame-based protocol with each frame having starting and ending flags, addressing information and a CRC over its contents. HDLC is also a polling protocol with the master, our end-point management system, initiating all transactions with the slaves, the remote terminals.
I designed the overall software architecture and wrote the user interface code, the diagnostic test scheduler code, and part of the diagnostic test code. To meet our reliability requirements, the terminals needed to automatically perform a complete suite of internal diagnostic tests at least once a day. Execution of these tests could also be commanded locally from the user interface. A consultant was hired to write the HDLC protocol code and the device driver for the serial interface. As a secondary task, this consultant also wrote the diagnostic code to test the serial interface device.
Satisfied with my changes to the software, I was ready to release the code to our manufacturing people to burn into the units. But before I powered-down my test system, I decided, on a whim, to run the diagnostics. That is when I heard the soft but distinct metallic click of a small relay being energized. A characteristic of the voice interface on the transmission gear was that the technician needed to enable it or go "off-hook" in order to talk. Included on the mezzanine card was a electromechanical relay to implement what would be done with a jumper wire by the technician. The serial interface diagnostic test was inadvertently activating the "off-hook" relay. If fielded, this problem could cause random, intermittent communication failures between our management systems and the remote terminals. With neither the needed equipment nor the time, we would never have discovered this problem during testing in our lab. Given the wide geographic distribution of the remote terminals, it would have been a difficult problem to isolate in the field and costly to fix.
Our consultant had based the interface diagnostic test code on his serial device driver in which the relay control code was embedded. In the limited time before the delivery deadline, we scrambled to correct the code, test the changes, and burn the code into the systems. This experience reinforced for me the hazards of blindly copying a design from one application to another, and reminded me of the importance of isolating, whenever possible, different functions performed by a piece of software. This experience also made me appreciate any breaks in the development rush. A brief lull in activity revealed a serious problem that had been literally masked by the noise and bustle of the engineering effort.
Design engineers and professionals, the West Coastís most important design, innovation, and manufacturing event, Pacific Design & Manufacturing, is taking place in Anaheim, Feb. 10-12, 2015. A Design News event, Pacific Design & Manufacturing is your chance to meet qualified suppliers, get hands-on access to the latest technologies, be informed from a world-class conference program, and expand your network. (You might even meet a Design News editor.) Learn more about Pacific Design & Manufacturing here.
Schooled in electrical engineering at Lehigh University, Jason Evans stumbled into embedded firmware development. He has worked in radar, telephony, secure communications, networking, and industrial control as a developer and architect. His interests include the new, the novel and the cool.