MY dad had the very same situation. Neighbours built on party wall and said he could use the new wall for any extension in the future.
Subsequently the guy sold the house, my parents decided to extend and the new neighbour told them to get lost and not go near the wall.
Now they have an extension that is not as big as it should have been and it looks awful and uneven at the side which approaches the neighbours.
It has definitely effected the house value.
On another note, My wife really like one of the houses we were viewing about 6 months ago, but i saw that the neighbours had built on the party wall, so would not agree to buy the house. You can guess why.