It usually takes one event of long and painful troubleshooting for a particular failure mode to stick in one's mind forever. I was building a hall effect tester for high gauss devices and needed an IEEE current source to drive a coil. I have been GPIB programming for years - should not have been a problem to establish communication, although this particular piece of equipment was not by a well known manufacturer and the electronics seemed rather antiquated (which was fine for this particular application) with dip switches that needed to be set manually. I spent hours trying to get my program to talk to the current source but it would not respond. I went through their manual with a fine tooth comb, triple checking all of the dip switch settings. On a whim, I decided to invert the switch settings - ones to zeroes and zeroes to ones. The system started communicating. The company reps wound up taking us out for a steak dinner after I told them about the error in the manual. I learned never to trust documentation when troubleshooting - learn to look outside of the box even if it doesn't necessarily make sense. This is a great article because it reminds us to share these stories - you never know what you might run across.
I had to WRITE the program without the OS or hardware.
And what fun it was with only a two page description of the interface card.
Onsite, I didn't have access to any debuggers since it was a realtime environment. I was really happy that I had taken the time to write all the status results to a ring buffer instead of the more usual read and forget, otherwise I wouldn't have had a clue as to why it was failing, nor documentation to beat over the head of the hardware guy.
I never understood why people couldn't 'fess up to their mistakes. Over the years I've had to figuratively press people up against a wall to get them to admit the fault could have been theirs. I wasn't looking to place blame, call me silly, I expect problems, but I needed solutions and they would be hiding information which made the problems much harder to fix.
@TJ: Even though you remember the stories, it will vanish with you once you leave the organization. So documenting or transferring the knowledge is something which should take place if it has to go from one mind to another
Curt, I liked your article and your approach to solving the problem. What really helps to debug hardware, especially in the field, is that knowledge base of actual experiences. So, my suggestion is, keep exchanging those stories. This column in Desing News is a great place to get such information as well.
Focus on Fundamentals consists of 45-minute on-line classes that cover a host of technologies. You learn without leaving the comfort of your desk. All classes are taught by subject-matter experts and all are archived. So if you can't attend live, attend at your convenience.