You need to look at the wording of the right of way.
Is the right of way across his land? If yes, you are in a bind.
As an urban solicitor, I would downright put people off buying a property with any form of right of way. Its almost inevitable that a row will develop at some stage.
For the moment, you should try and talk nicely to your neighbour - if, as you say, this is a holiday home for you, then you need to be careful not to sound like one of those annoying townies who don't really get rural living!
mf