It would appear simple. The builder owns the house and is selling it to you.
He did engage an EA to sell the house, but unfortunately he forgot to tell the EA the house was no longer for sale once you put down your deposit.
So the other buyer issue is between him the builder and the EA.
It would be the same if someone forgot to tell their EA they have taken their house off the market.
The other party can't threaten you with legal action, nor can they threaten the builder with legal action.
Not sure - myabe they can accuse the EA of advertising a property for sale that is not for sale.
Now unless the situation is that the other party are further along in the sale process than the builder is letting on. But because you have offered higher he's wanting you instead.
The builder shouldn't have their deposit himself, the EA should have their deposit.
Ask your Solicitor to look into it for you with the Builder's solicitor.
Bear in mind the builder may not be being upfront with you...