I've personally had to learn this the hard way, but when you don't want to be a scapegoat, fighting against it by giving people actual reasons to fear and dislike you isn't the answer. That's on reason Shepherd's Haven exists. We're safer because we typically keep to ourselves, and no one will bother us here.