Your application requires a very sensitive force measuring device. A load cell sensitive enough for such low amplitude of vibration can work or you could also resort to using other types of sensing techniques but I think they might be costly. You could learn more about the force sensing techniques and related concepts through here: https://tacunasystems.com/knowledge-base/an-overview-of-load-cells/

very helpful document...looking for it for a long time.

I am assuming that you are homing Z the use of a G30 command in homez.g and in the Z homing phase of homeall.g. There seems to be a distinction of round 0.7mm in the mattress peak or gantry top at the XY function at which you probe for Z homing, in contrast to the XY positions at which you probe for leadscrew levelling.