A robotic control system goes slightly haywire and a chance remark by an engineer helps solve the case
By Brian Kowald, Contributing Wrtier
Several years ago, I received a call in the office from a coworker in the field servicing our video on-demand system at a local hotel in the Dallas TX area.
The system was a robotic tape handing system that played videos requested by guests. The problem seemed to be that the system couldn’t “home” itself. After power up, the system would drive each axis toward the “home” position until a sensor indicated that the extent of travel was reached.
After that the system used encoders to position itself.In this case the indicator wasn’t seen before the mechanical stop (the side of the cabinet) was reached. I knew how it all worked because I had written the microcontroller firmware to run it.
“Try it again”, I said. “Watch closely and tell me what happens.” WHAM - the xy mechanism hit the side of the cabinet! It was pretty loud too, as I could hear it over the phone. I said that I couldn’t tell anything else over the phone and that I would grab a spare board and head out there, a thirty to forty minute drive depending on traffic. I said I was on my way and the tech said “Hurry up - I’m freezing my ‘you-know-what’ off out here.” Only he didn’t say “you-know-what.”
I arrived sometime later, looked the system over, decided that since nothing was broken so far the system could take at least one more pounding, I moved pulled the transporter away from “home” and issued the command to home it. Low and behold - it worked perfectly. I said “See - all it took was pressing the right button”. (My motto is never pass up a chance for a little good-natured ribbing).
I tested the system a couple of more times and it didn’t fail. Pausing to think, I went over the conditions that led to a failure. They system failed earlier in the day, but then started working. What changed? Then I remembered the coworker complaining about how cold he was.
The system was in an unheated space and the temperature was probably in the upper twenties or low thirties at night, but warmed up nicely as the Texas sun came out. I knew the circuitry of the control board, since I had written the software, so I suspected what the problem was.
I called the hardware engineer who had designed the board and told him, “Hey - your board doesn’t like the cold.” Turns out the board had hardware debounce on it, and the capacitors were sized for a fairly long value. As the temperature dropped, the debounce got even longer, until the microcontroller couldn’t see the home indicator before the transport mechanism ran into the side of the cabinet. “Just cut off the capacitors,” the board designer said, and gave me the numbers. I snipped them and the problem didn’t occur again.
The system was good until summer when things starting heating up, and other failures started, but that’s another story.
Contributing Writer Brian Kowald graduated from Texas A&I University in Kingsville Texas (Now Texas A&M Kingsville) in 1983 with a degree in Electrical Engineering with an emphasis in computer systems, and worked for Texas Instruments, then Synergy Broadcast Systems for over twenty years. He is currently a senior software developer for Reveal Data Corporation, an e-discovery support firm. He lives in the mountains of Western North Carolina.