Have you the right of way officially documented and registered by a solicitor?
That should satisfy the bank that you have iron-clad access to the house and can't be denied that access by any third party in the future.
However, what the bank may be being cautious on is not your right of way but their right of way should they need to repossess - this might be what's really on their mind - I recall something on this before - talk to your solicitor, what suggestion does he have