One will bother you only if you have a karmic account with him. No one can bother you without your signature (karmic cause). All this is due solely to your signature.