What should I do if I often find myself together with people whom I don’t fit in with or whose company I actually don’t like?

Lama Ole’s answer:

Then wish them all the best, leave, and do what you are interested in. Build up your own power-field somewhere else. Humans emanate certain energies; some match well and others don’t. If others are disturbed by what you are doing, then part company with them in a beyond-personal way—because you know that they also want to have happiness and to avoid suffering.

Act according to the situation. If you think someone is treating you badly, then you can teach him a lesson. You have to be able to stand up for yourself. But your reaction should always be relative to the situation. So please do not crack a nut with a sledgehammer.