Rub you up the wrong way

Altered image of a baby patting an annoyed cat
To be rubbed up the wrong way means that someone's behavior annoys you slightly, particularly small habits or remarks.

The phrase comes from brushing the fur of an animal against its natural direction. The motion creates resistance and irritation. The image later became a metaphor for social friction.

His sarcastic comments always rub you up the wrong way.