A criterion to evaluate the influence of the so-called plasticity effect on the final outcome of the incremental hole-drilling technique (IHD), for measuring residual stresses induced by mechanical surface treatments, is proposed here. In practice, it is currently accepted that residual stresses can be accurately determined by IHD if the residual stress level does not exceed about 60% of the material’s yield strength. However, this criterion is not appropriate when IHD is used to measure residual stresses in work-hardened surface layers, since the yield strength of these layers is very difficult to determine. The proposed criterion takes into account the strain-hardening effect and the local yield strength of work-hardened surface layers, using the concept of normalized hardness variation. The criterion was validated experimentally and numerically for shot-peening residual stress measurements.