If you have an agreement and broke it you will be responsible. If you clearly communicated the agreement to your builder and issued him /her with a clear set of construction designs clearly setting out the lines and dimensions of the project then responsibility for the mistake will lie at their feet.
What happened along the way? Surely as the footings wall rose out of the ground the mistake should have been spotted in the misalignment of the blockwork? Questions should also be asked of your site supervisor / engineer / architect who would normally pick up on items like this.
A lot will depend on the information provided to the builder and the accuracy of this information. Naturally everybody is going to say they aren't responsible and the can will be left with you. If your information chain is correct you should be able to establish clearly who is responsible and their respective insurance policies should cover the reinstatement costs.
To answer your direct question. The thing is to identify and agree a mistake has happened, inform your neighbour that you are going to deal with the relevant individuals, you could possibly ask them (if it is you and them) what they would like to happen, be prepared for "give us back our 4" of land".
I don't think you can offer anything other than to consider buying the 4" of land, trespass has occurred.
Although you could ask Pat . . .