looks like a messy probate

There is a will. That sets out your late father-in-law's decision on what should happen to his property, and the executor is bound by that. Your wife and her sister cannot run it as a quasi-democracy where two beneficiaries can outvote the third one.

It looks as if you do not have the resources to pay her brother his share without the house being sold.