Have you told the builder that you're willing to forego the deposit? His solicitor's might be making noises in order to eventually 'let you off the hook' if you're willing to forego this. Personally, I wouldn't be willing to lose this money lying down! I'd try to negotiate a lesser amount.
At the end of the day, the builder is in the right, you signed a contract and put your money down. Presumably, you initially paid a booking deposit and had X amount of weeks to sort out finances/pull out etc. Technically he can force you into completion AFAIK and he can afford to pay the legal bills more than, I imagine, you can.
Are you sure the property has gone down by 10K? Flippers can afford to undercut the builder by a huge margin, as they originally bought in the first phase, but this does not mean the properties, themselves, have devalued. Unless you're thinking of buying in a different location altogether?