I skipped a volleyball opponent's handshake after a personal taunt. Should I apologise without dropping the complaint?

At the net, I walked past the player who had spent the last set telling me I was only on the team because I organised the rides. I shook hands with the others. Our captain noticed and said, "That made all of us look bad."

The opponent called it a joke when I objected during the game. I played on and didn't raise it with the official. By the handshake line I wanted to show that I wasn't happy to laugh it off.

Now I wonder whether refusing the handshake communicated anything beyond my own anger. A teammate offered to mention the remark to the organiser, but the captain says an apology should come first.

Would you acknowledge the skipped handshake while still reporting the specific taunt, or does that turn one sour exchange into too much process? I don't want the team to have to defend my reaction. I also don't want "sportsmanship" to mean I'm the only person asked to change.