Do I have to give my partner a house key because he gave me one?

My boyfriend gave me a key to his flat and said, "So you don't have to wait outside when I pop to the shop." I thanked him. A week later, he asked when he was getting mine.

We've been dating for five months and stay at each other's places regularly. I like arriving to find him there. I haven't decided that I want either of us coming into my flat when I'm out, and I had understood his key as something he wanted to offer, not the start of an exchange.

He says it feels unequal that I can use his space in a way he can't use mine. I can see the difference. I would rather give his key back for now than give him mine just to make it even, but that might look like taking a step backwards.

How do I talk about the pace of access without making it sound like a measure of which one of us trusts the other more?