Shame comedy
You made a joke at someone else's expense, didn't you?
That's probably why you're here.
I know you didn't mean any harm, but harm was done. When you joke at someone else's expense, you make them feel bad. They probably didn't want to show it, because the recipient is expected to be a "good sport" about it, but it probably bothered them.
"Shame comedy" is when you make a joke with someone else at the butt of it, especially when it exploits something they might be insecure about. It's "punching down", and it's not cool. People won't think you're funny, it will just come across as conceited.
Saying "it's just a joke" isn't an exuse, it's not up to the person doing the hurting to decide whether it's OK or not.
Don't do shame comedy. Be better.