Is it fair to stay at a party after my wife says she's ready to leave?

My wife caught my eye from the doorway and said, "Shall we go?" I put down my drink, said goodbye and followed her. On the way home I admitted I'd wanted to stay. She asked why I hadn't said so.

We usually arrive and leave together. She enjoys the first couple of hours; I often relax into a party just when she has had enough. We can each get home independently, and neither of us objects to the other doing that in principle. In the moment, though, staying feels like telling the room I'd rather be there than with her.

She says I'm making the choice silently and then making her feel guilty about it later. That's probably fair. How do we make separate departures feel ordinary rather than like a statement about our marriage? I want to stop turning her question into an obligation she didn't actually give me.