The mind is not the one that harasses you, it is the attachment-abhorrence that harasses you. It is because of the attachment-abhorrence that one has memory.