The guy is definitely chancing his arm - call him up (or call through to his voicemail if you aren't good with confrontations) and tell him in no uncertain circumstances that he's not getting another penny.
It's the same in any business - you asked for and agreed on X amount of work for Y amount of money. Just because he brought Z into the equation, that's his problem, not yours.
if you are drawing down on a loan you must have a solicitor, tell him you are not paying any more, if he still annoys you get your solicitor to write him a letter.
I agree... you shouldn't have to pay the extra amount. But also think you should deffo change the locks. We had plasterers / electricans etc in the house and when they were all done, we didnt bother asking for keys back, just changed all the locks.