The Case of the Semi-Proficient Measurement

I manage a proficiency test program for calibration labs. This consists of sending an artifact to a laboratory and asking them to measure it, then seeing if their answer is correct, taking measurement uncertainty into account.

One test involves measuring the angle deviation of a pair of fixed optical wedges, one with a nominal 50 arc-second deviation, and the other with a 10-second deviation. This is a fairly simple test, but the calibration procedure they use for the proficiency test is old and specifies obsolete equipment. The idea is that to do the test, they'll use their more modern equipment, and they will have to adapt the procedure accordingly to make a successful measurement. That is, they'll have to understand what they're doing, rather than just blindly follow a step-by-step recipe.

That's the theory, anyway. In practice, it turns out that some labs kept those old museum pieces in the back of the store room and dusted them off to do the proficiency test. "Hey, that's what the procedure called for..."

One of the labs running the proficiency test that did this used its approximately forty-year-old manual autocollimator. Its result for the 50-second wedge was right on the money, but for the 10-second wedge, it was way off. I called the lab and talked with the technician. He seemed experienced with the measurement equipment and generally seemed to know what he was doing. I couldn't find any fault with his technique based on his description. I asked him to send me his raw data.

To do the test according to the procedure, they make five separate runs of the wedge deviation angle measurement, and they average the five results for the final reported value. The readings for the 50-second wedge showed a normal scatter of a few tenths of an arc-second in the readings, but for the 10-second wedge, the readings were all over the place!

They varied by a couple of seconds or so. There isn't much that can go wrong with a fixed optical wedge, which is one of the reasons it makes a good proficiency test artifact. It's just a piece of glass with an angle between its two aperture faces, mounted in a metal frame. I asked him to check if the wedge prism had come loose in its holding cell. To cause this much variation, it should be rattling like something you'd give a baby to play with. He reported back, "No, it's solid as a rock."