Do you not think it's a bit unfair one group of staff have flexi time while another doesn't or worse again only one person. I take it from your post, you and the other staff members who cover the desk at lunch have flexi.
As purple said read your contract, if your contract states your starting times and finishing time, any changes will need to be agreed.
If you agree to changes (which I think you should, but I work in HR so I would say that), there should be some rules that ensure there is a fair system in place so that no one can 'bugger' off when they feel like it.