By Jim Baker, Contributing Writer. By scrutinizing the dump of a digital output point table, engineers find important clues.
I used to be a commissioning engineer for a SCADA company and spent much of my time traveling through the Middle East and Asia commissioning substations and running training courses. (SCADA - for Supervisory Control and Data Acquisition — relates to a system that collects data from sensors on the plant floor or other remote locate and sends this data to a central computer for processing.)
I was once on my way to a Middle Eastern country to commission some of our RTUs (remote terminal units) at several substations when my boss asked me to drop into another installation on the way. One of our engineers was having trouble at a site and could not solve a particularly disturbing problem. The symptom was random operations of control outputs. Not something that is welcomed in a 330kv substation! This particular RTU installation had redundant RTUs and many peripheral devices. The RTU also had some embedded ladder logic. It was a quite complex setup and the project manager was getting a bit nervous about the delays in commissioning.
I met with our commissioning engineer who was a recent graduate and fairly new in the organisation, and we went through his quite complex configuration, It looked fine to me. All the firmware application versions matched up and the embedded ladder logic didn’t appear to be doing anything untoward. There were no apparent errors in the point mapping. We started up the RTU and sure enough, a whole bunch of output relays operated. It was an impressive site watching all these output LEDs turn on like a wave running down the equipment racks.
I had never seen this type of fault before and could not imagine what would cause such a catastrophic result in what was an extremely reliable RTU. I thought about it for a while and decided to look at the digital output point table in memory. This particular RTU had one byte per output point and so it was fairly easy to determine the state of the point from a dump of the table. What I found was a very strange array of seemingly random point states.
I was looking at the table in hex trying to work out why the points would be in this state and it suddenly struck me - this looks like ASCII! I decoded it as though it was ASCII and the result was the text of a logon message for one of the embedded RTU applications. I rang my colleague back at the office and asked him if there were any known errors with this version of the application. “Yes” he said, “Writes randomly to memory”. I had discovered the problem. Probably an uninitialized pointer in the original code. After installing a new build of the applications with the new version of the culpable application all worked perfectly and we had a very happy project manager.
Jim Baker has a degree in Electronic Engineering from Curtin University in Perth and has been involved in SCADA as a development/commissioning/training engineer for over 20 years.