Plan Multiple Comparisons Before Searching PCB Test Data for Differences

A large PCB test dataset offers many opportunities to find an apparently unusual difference. Comparing every station, lot, parameter, and time window increases the chance of highlighting a chance pattern. Define the comparison family and the decision purpose before treating one small probability value as strong evidence.

NIST’s Bonferroni guidance discusses simultaneous confidence statements for a preselected finite set of comparisons. It is one approach to multiplicity, not a universal instruction to apply the same correction to every engineering analysis.

Write the primary questions first

Identify the product or process decision, the outcome measurements, and the comparisons that directly address it. State whether the study compares candidates with a reference, every candidate with every other candidate, or a specific planned contrast. Keep that plan with the dataset so a later reviewer can distinguish it from subsequent exploration.

Record additional exploratory searches separately. Exploring the data is useful for generating hypotheses, but a finding discovered after examining many alternatives should not be presented as though it were the sole question planned before the experiment.

Keep the comparison structure visible

Do not hide unsuccessful comparisons and report only the favorable one. Preserve enough analysis history to show the scope of the search. This is especially important when several transformations, exclusion rules, or subgroup boundaries were tried.

Interpret magnitude and uncertainty together

1. Check data quality and the sampling structure.
2. Use a method appropriate to the planned comparison family.
3. Report effect estimates and relevant uncertainty.
4. Compare the magnitude with engineering requirements.
5. Confirm important exploratory findings independently.

A statistically detectable difference may be too small to matter to the product, while an inconclusive result may still leave a practically important difference plausible. Neither conclusion is resolved by selecting a convenient significance threshold after seeing the data.

Keep correlated measurements and repeated observations in the model where appropriate. Multiplicity handling does not repair an analysis that incorrectly treats repeated readings from one board as independent boards. The sample structure and comparison structure both need attention.

The final report should make clear which claims were planned, which arose during exploration, and what further evidence is required. That transparency supports useful engineering decisions without turning every interesting dashboard fluctuation into a claimed process improvement or supplier difference.