We have similiar clauses in our contracts at work, and despite the fact that site work is quite frequent (and an essential part of the business in my line of work) we are consulted on each proposal and I've never heard of anyone being forced into a site position (for any period of time) that they didn't accept.
Personally, as I am just out of college a short time, I have no problem with this and spend well over 90% of my time on site. Many of the staff, with young children etc. are not so willing, similiar to yourself, but have never been pressured into it or treated poorly because they declined requests to go on site (it does mean missing out on additional pay etc. so anyone who can do it usually does so the company aren't exactly stuck for site staff).
To be honest, I'm not sure what comeback you'll have seeing as it's in your contract. I only mentioned my own experience of it to suggest that even though the company can/could/may enforce this, it's quite poor of them to do it in this manner.
Do check it out with any sources you can find, such as DETE above, but if you don't have much luck I'd certainly consider it as a chance to review your position and consider if your employer is treating you as you expect to be treated.