Hey Eve, been through something similar myself. Two things to say. (1) I think you most certainly should go to your solicitor about this immediately, if you have not already done so. (2) And definitely don't pay that contractor one more cent unless your solicitor advises it.
It is so much harder to get justice / fair play for yourself if you have already handed over money compared to keeping the money until the work is satisfactorily completed.
These disputes can be quite slow to resolve, but with good legal advice things should work out for you. Best of luck.