Do you have an exit plan for the farm down the line? You mention your spouse has an attachment to their rental property, but you are holding a very valuable farm as well....I do get the attachment to land, especially if it was family owned for a long time. And if it is increasing in value or has future potential use (eg sites for houses for your children) then I get holding onto it. But try to apply the same rigour in thinking to the farm as you are to the investment property.
Do you really need to farm it to generate that income, or could you get it from lease income? Is is required that you maintain status as a farmer for other reasons? When you say part time, are you missing out on a lot of leisure or family time by keeping up your farm work and a full time job? Is this sustainable as you get older?