Time of day proves to be a crucial clue when engineers investigate a cranky compiler
By Charles Glorioso, Contributing Writer
Some years ago, I was in a group that was developing a graphics program with a very sophisticated user interface. The development was being done on minicomputers, and the code was complicated enough that a compile took many hours. The programmers hit on the idea of arranging their work so they could start the compile as they left for the night with the goal of it being finished when the returned next morning.
However, the compile always crashed. It never completed successfully. They soon got into the habit of starting the compile, going to dinner and then returning to check on it. A full 100% of the time they found that the compile had crashed while they were at dinner. If they then started the compile again, it would always complete successfully by morning. Over a month, it always crashed while they were at dinner, and always completed successfully when restarted.
They spent a great deal of time trying to find the cause of the problem. They tried code from different people. They tried to find something different in the compiler or the state of the machine between the two runs. They tried different minicomputers. They even tried moving the minicomputer to different locations and power sources within the building. None of these helped them identify the cause of the failure. Compiles during the day always worked. Compiles started late in the evening always worked, but compiles started to run over the dinner hour always crashed.
Finally, they asked the hardware group for assistance. We covered much of the same ground they had, including checking the line voltage to the minicomputer. Nothing solved the mystery. Eventually, in desperation, we located a recording voltmeter and connected it to the input power line to the minicomputer.
What we found when we recorded for several nights was that the line voltage drifted up from about 4 pm to about 7 pm, at which time there was a sudden drop in line voltage of about 20 volts. During the step there was also a lot of noise and a brief period of zero volts. When we hung around one evening and watched, the minicomputer reset itself during the zero volt interval.
When we called the power company to ask about this mysterious behavior, we were told that the step down transformer for the industrial park where our building was located had multiple taps. A clock was used to adjust the taps in order to keep the line voltage within specifications as the power usage changed at the start and end of the work day.
We bought a Ferro resonant line isolation transformer for the minicomputer, and the programmers never had a crash overnight again.
Contributing Writer Charles Glorioso has a BSEE from Purdue and an MSEE from Illinois Institute of Technology.He has over 40 years experience in electronics design and management for industrial and consumer products.Employers have included Teletype Corporation, Cadence, The Exploratorium, at least 6 companies which no longer exist, and most recently Davis Instruments. Charles retired earlier this year after six years as Director of Engineering at Davis Instruments, and is now working there part time on special projects.