The moment you put a stop to people taking advantage of you and disrespecting you, is when they define you as difficult, selfish or crazy. Manipulators hate boundaries.