When dealing with someone who knows the things that annoy you it can make for a downright testy relationship at times. You must always be aware of the fact that they know the things that will set you off in a moments-time and be careful when dealing with them from a place of frustration.
Now knowing your triggers will help you to become more aware during times when people are trying to antagonize you. Usually during the course of a conversation going wrong or heated debate and/or discussion they will constantly refer to those things that annoy you. This is the time to exercise restraint in not falling prey to their attempts. I know it can be rough because the more you try to avoid becoming agitated in the conversation; the more they make certain inferences to draw you nearer to your boiling point. You try to ignore them and yet they will continue to make comments and insinuations to further antagonize you to that place of discomfort especially when you're trying not to get angry.
Knowing your triggers and the things that agitate you will assist you in not being drawn into unnecessary conflict. When you sense someone attempting to take you to a negative place and you have identified what they're trying to do, either end the conversation or change topics. But by all means don't be drawn into the negativity. Over time you will find that most people who try to use insults and what they perceive to be your short comings to annoy you as well. Most of the time these people have their own issues that they're dealing with internally and for whatever reason, if they're having a bad day they want you to have one too.
Sometimes you can rise above it but sometimes you just can't. I find that it's best to limit your contact with people who behave in this manner. Or sometimes you have to sever ties with these people altogether because you will find yourself being on the receiving end of their malice comments and bad behavior more often than not. It's not always in your personal relationships either. This issue can arise in business as well,especially if there is some kind of inferiority involved. You may have that business associate who is just looking for an axe to grind and then you show up. "Game on,".
Surprisingly though depending on the person and the caliber of your relationship you can actually speak to them about the behavior they're displaying and bring it to their attention so that they can correct this issue. In some instances they aren't even aware that they're doing these things to you and if you don't say anything they will continue to do the same things over and over again.
The best thing to do here is to know yourself and the people you are dealing with. Essentially this will help you to navigate through these situations and also help you teach others to deal with their own issues as well.