I think the UK retailer is wrong. If they're selling to a non-UK resident and delivering the goods to an address outside the UK, the transaction can be zero-rated. For compliance purposes the UK retailer needs to retain evidence of export — invoice addressed to non-resident purchaser, shipping documents evidencing delivery of the goods outside the UK.
But it may be that the UK retailer doesn't know this, or does know it but has so few export customers that they can't be arsed to set up a system for managing, implementing and documenting the zero-rating of export sales. You can't force them to if they don't want to.
Look for the same ornament from a better-resourced retailer. Or look for a different ornament from a different retailer. Or grit your teeth and suffer the double VAT. Sorry.