Been there, done that. I use the LFW (Last Fiddled With, or at least thats the "nice" version of it) method in evaluating problems. How many times have we all heard "oh yes, I did so-and-so, but that had nothing to do with this problem"?
And not just o-scopes, but in my old age I take no readout as the word of God. A very recent example is one of the staff engineers told me an eight watt air to ground radio was outputting 1 watt and had a 1/4watt reflection using a Bird watt meter. It came back from the repair facility no trouble found. I measured it myself with an attenuator and spectrum analyzer and found full power. He used the wrong slug in the meter!
Also I get a kick out of the guys here who use 1 hertz resolution on a 500MHz signal, then have the nerve to tell me "the oscillator is off frequency by 1234Hz." I pull out the spectrum analyzer spec on freq drift/accuracy. I get the "Gee, I never thougt about that." answer. Same with the LSD on digital meters. Read the accuracy specs!
Had a non-tech guy yesterday come in an he reached out to touch a circuit card I just fixed. I told him "Don't touch that or you'll get hurt." He asked "Will it electrocute me?" I replied, "No, its low voltage, but if you break it I WILL HURT YOU!"!
I love my digital instruments, but have to learn their limitations. Great story, one that I have lived countless times over the past (almost) half century.
Good point about analog scopes. I cherish my older Tektronix 465B, which I bought on Ebay years ago and had calibrated and a few components replaced. Love that scope. Recently I had to help troubleshoot a signal source. At first it looked OK, but when I stepped through the time/division-switch settings it was obvious the oscillator had noise on the signal. Turned out that the power-supply filter was bad and let 120-Hz noise modulate the oscillator output. I find if I don't look at signals on all of the frequency settings I can miss problems that don't appear when I "think" the problem exists elsewhere.
As a Calibration Tech for AMD in the beginning of my career, I had seen these problems easily with my analog scopes.
When I worked at Cray, I saw and dealt with SAMPLE and HOLD issues at the GHz level. Issues with the plug-ins for the 7104 to work with greater than 1 GHZ levels were always a problem. Intermittent and harmonic static had to be captured and NO trigger was fast enough!
Today, I ONLY keep analog scopes. The noise may show up as a blur, but that is still better than the scope " proving " that no noise exists...
That " Sample and Hold " issue is the ugly secret that today's scope makers ( including Tektronix ) won't talk about.
The real irony: It cost Cray ~$60k for the 7104 & the plug-ins. That was in 1980s $.
It costs around $1k for the same stuff today. Just spend another $1k to get it recalibrated. New is not always better. Old warhorses still kick @$$...
I've already collected my stuff. Have at it on eBay....
A classical example of how misleading digital scopes can be.
An analog scope wouldn't have hidden the problem--you'd have had a broad stripe across the display at low sweep speeds. When using digital scopes, I leave them in "envelope" mode most of the time, for just that reason.
One of the first things that jumped out at me was the comment about being "in tolerance." Sometimes I find people that are looking to make a change to this or that for whatever their personal reason/goal is and all they care about is if the parts are "in tolerance". What happened to wanting to make sure the end product was of good quality and made as cost effective as possible?
Looks like the solution came from not taking anything for granted. Over and over with these Sherlock Ohms problems and solutions, the answer comes when the most simple (and thus invisible) assumptions are challenged.
Sometimes knowing too much and exploring too many options masks the simplicity of the real problem. By asking the basic questions and doing the due diligence in terms of the general state and maintenance history of the oscillator, you would have been on the trail of the problem much earlier. But you're absolutely right that it's always easier to come to this conclusion in hindsight. How come we never learn??!!!
Focus on Fundamentals consists of 45-minute on-line classes that cover a host of technologies. You learn without leaving the comfort of your desk. All classes are taught by subject-matter experts and all are archived. So if you can't attend live, attend at your convenience.