If the right of way is used only by this neighbour, I would sell the strip of ground to him and get him (as part of this deal) to construct a good, permanent boundary between your property and his. This will bring closure of a sort, as there will be no right of way to argue over in the future. Strong fences make good neighbours and all that........
If the right of way is used by both you and the neighbour, then obviously you are stuck with a permanent relationship. You were right to speak to him, but I would not be inclined to take it further. I think you have done enough to assert your rights. If he abuses the situation further, you might consider going down the legal route, but these things can fester for years, so if you can tolerate the situation, leave well alone.