In my honest opinion, once the contracts had been signed, snag list doneand you had moved in he should not have a copy of the key. I would have changed the locks immediately upon moving in. I can see how you might think it was ok for him to keep a copy initially.
however, he should not have access to your house whenever he likes. it should only be on your terms while you are there and when you let him in
I agree with all above. Change the locks back and front, whatever the cost